Guild Wars 2

Thank you for your work on fixing the lag. I’m actually optimistic that this will make difference in WvW.

On the other hand, what about FPS issues? Apparently a lot of players, despite having top-of-the-line PC, are still having experiencing below 20fps on very crowded events.

Skill lag can get pretty bad when the Wv3 areas are full, hope you can fix that sooner than later.
But don’t you feel like altering skills to optimize them might not cause more trouble than it may solve? I hope it works out, but sounds sort of risky.

Btw, something is broken with the Q: at least I didn’t intend to use this feature…

Hello Bill,thank you for taking time to speak with us and working on this very important and “pain in the kitten ” for WvW community problem.

1. My 1st question is : Can you be more specific of what “optimization” means and will it affect the skills as we know it by any negative way ?
2. My Second question is : Wasn’t all these problems caused because you have to broadcast every single condition/buff to everyone which is currently pushing your systems and servers/internet to its limits ?
3. My 3rd question is :If you ever have a solution to this are there any plans to lift the AoE Limit for both WvW/PvE/PvP because thats why this AoE limit exist in 1st place

• Isn’t skill delay in big battles (WvW or PvE like Taco) a direct result of those?

Just wanted to say thank you for the update. These are the kind of posts that I would like to see happen more often, because it can be frustrating when we feel left in the dark. Keep up the great work!

I have nothing constructive to offer, but wanted to congratulate you for your efforts and for being trasparent with us on the matter.

I just have an observation I have noticed. My memory usage for GW2 skyrockets during lag. As in 3 million k plus. When it is less than a million, my game is smooth as silk.

Thanks for the update. May I suggest that you start working through the skills in order of longest cast time first? Those seem to be the ones that are affected the most by lag.

Thanks for informing us about the process. I do have one question. You said that when that patch note comes through, expect a very different WvW experience. should we expect to see these optimization on 15th October? or was that just writer’s freedom of speech?

20131005 – Prior to the update on 03 September 2013 I had no lag issues what so ever. I was however having a lot of issues with a new sound card I had purchased (testing the third factory replacement) so when my FPS dropped from an average of 4(with all the eye candy turned on max) 5 in Lions Arch to 3 (in performance mode!) and lots of skill lag, I thought nothing the sound card driver was the cause of it. Since that time I have formatted the hard drive and did a full reinstall of Windows and reinstalled my old reliable sound card, my FPS went from 2 up to 12 (still in performance mode) with about the same number of people around. Since that time I have seen some improvement with recent optimizations, I can reach 18FPS (yes, still in performance mode) and most of the skill lag is gone. Almost everyone I know of that uses both an AMD processor on an x90 chip-set motherboard and an higher end AMD video card seems to have the same issues.
I want to make sure you understand, I don’t give a kitten about WvW or any other form of PvP, I am all about PvE. So here is my question; Will these upcoming optimizations improve the situation for us?

Edit: 20131006 – Today I had my sound card drivers crash when I logged into Guild Wars 2. My frame rates are almost back to normal and most of the skill lag is gone. I really dislike playing with that kind of disadvantage however.
After restarting my computer I expected the issue to return. It did not, which would indicate that the sound card driver was at fault, except for one thing. Before I had a single core always running about 50-75%, now I have five cores running between 30-75%. I think it may have been an issue with the processor driver. Either way, I am glad that it appears fixed.

My main issue with lag is that when I have powers with casting times, I get used to a certain pacing, but then the casting times change under lag, so I’ll click on one ability, wait an appropriate time, click on another, and then the first will get skipped and never go off. Whatever you do, just ensure that every ability I click on will activate.

I’m sorry I don’t have anything more concrete or helpful to say, but I really, really love it when you keep us in the loop like this.

Thanks for letting those of us who’ve been seeing this, checking our ping times, monitoring our client performance and occasionally shedding some hair that we’re not crazy (well, okay, not because of this, at least).

Fair winds and following seas, dear friend.

Thank you but what about client side optimizations? We were promised some major engine/client side optimizations to improve our fps in WvW 6~ months ago, has this project been dropped because I haven’t seen anything regarding client performance since then

I wish you kept us this in the loop for bleed cap: https://forum-en.gw2archive.eu/forum/game/gw2/Condition-Catastrophe/

In a vain attempt, I’ll try to keep this post away from corporate politics. Regardless, I believe that certain corporate-wide policies need to be re-examined in terms of game development & maintenance, patch/update presentation and community interaction for the future.

About transparency: not limited to recent WvW scandals and community outrage, I believe we could use some form of issue-tracker and update-scheduler, present on forums or a "sister" website to main guild wars 2 website (and definitely visible as a tab on main website, as well as a link from the forums).

Issue-tracker:

- a list of all current problems with the game, separated by tabs (graphics, sound, gameplay etc.)
- every problem defined: what, how, why, temporary workarounds, ETA (be it a week, month or unscheduled)
- every problem discussed: separate thread purposefully moved away from general technical/support forums

The purpose:

- community is informed about bugs, knows they’re being worked on: less duplicates and false-positives, feedback about singular issue concentrated in a single thread
- technical forum is still accessible to users requesting personal help, or bringing up new/unconfirmed issues (less forum clutter - identified issues are moved to a separate tab or place, to reduce the chaos)

Update-scheduler:

- less technical and more general in nature, outlines game going forward in new directions (be it fixing a core gameplay system/feature, or adding new systems/features, not limited to: maps, professions, skills, events, game modes, encounter types, conditions/boons/states)
- fosters community dialogue & feedback before features go live (including: philosophical, mechanical and social consequences for the game and it’s players)

Last Words

In varying intensity and form, we’ve already seen some of these steps taken by Anet: "creative design" blogs, feature blogs and posts, progress/update posts, feature videos. What is lacking, however, are two things: consistency of approach and organisation.

Organisation: central go-to place to study and discuss existing game problems in depth, as well as game-going-forward features which are in the works, but with more flexible schedule and final content/design.

Consistency: all current issues and future features are documented, catalogued and freely discussed with the community as a prime community relations policy, NOT "as exception" rule: when under duress of public outrage, trying to limit damage after scandal/leak or "almost everything is ready so now we talk about it" mentality.

Summarising, the irrational fear of: exploits becoming public knowledge, developers being publically lynched for missing a deadline/feature, needs to go.

Exploits: make a public list of bannable offenses, gather public feedback about them and fix them faster.

Missing a deadline/feature: players have to understand that not every feature will get implemented on time, at all or in the form outlined in early design sketch or philosophy - simultaneously, playerbase as a whole shouldn’t be punished for simple curiosity and interest in the ongoing health of game and it’s continuing success.

Purposefully brick-walling and withdrawing critical information about game issues and your policy towards fixing them until the very last moment causes permanent community damage - loss of players and declining morale. I believe we deserve better.

In short: please be more upfront about your game design, talk with us openly and accept constructive criticism where it’s due. Failing that, we will see more hostility towards developers and the company, more conspiracy theories and generally poor player feedback. Worst communication is silence.

Trivia:

- "Roadmap" http://www.planetside2.com/roadmap
- "the problem, what we’ll do to fix it and the philosophy behind it" post https://forums.station.sony.com/ps2/index.php?threads/operation-make-faster-game.150281/
- "update/issue" discussion thread https://forums.station.sony.com/ps2/index.php?threads/october-on-going-performance-improvements.152361/
- "when forum or blog post is not enough" https://www.youtube.com/watch?v=EaHgTR4yyuU

Start with the super-lag skills please. Things like Ride the Lagning and Burning Retreat.

I thank you and my poor keyboard (which I smash like mad when skills don’t fire on time) thanks you.

When you say “optimization”, do you mean making your engine more compatible for multi-core systems? Right now your game’s engine runs primarily on single-thread, which Intel CPU’s seem to thrive on; while at the same time your game’s engine is poorly designed for multi-thread CPU’s, which AMD is best at.

Now, the problem is that a multi-core AMD doesn’t have good single-thread speeds, which on GW2 drives the performance down. Why is this? Because the video game -a graphical projection on a computer screen- is for some reason more prioritized for CPU usage instead of GPU usage. Why? I haven’t a clue. Maybe it’s because the high use of physics within the game, don’t really know. Either way, it’s a problem.

Demosthene.2195:

…

Trivia:

- “Roadmap” http://www.planetside2.com/roadmap
- “the problem, what we’ll do to fix it and the philosophy behind it” post https://forums.station.sony.com/ps2/index.php?threads/operation-make-faster-game.150281/
- “update/issue” discussion thread https://forums.station.sony.com/ps2/index.php?threads/october-on-going-performance-improvements.152361/
- “when forum or blog post is not enough” https://www.youtube.com/watch?v=EaHgTR4yyuU

This is off-topic, but this is the level of transparency I would like to have. I think it would be beneficial for everyone if you give us insights on what are you doing with current game issues. We could understand what issues you’re dealing wih instead of going :

“Omgz skill lag, why not fix already, so easy, my grandma can do it, QQQQQQ, buy better server QQQQQ,”-Random GW2 player

I imagine is not as easy as buying/renting better server hardware? or internet with more bandwidth, less latency? if that is not the solution, I wonder where the bottleneck causing this lag is?

Great post Bill, I’m so pleased to see this! I was hoping the focus would shift a little to performance at some point because there are definitely issues that let the game-play down a bit here and there, mostly down to skill lag and FPS (on a NVidia GTX 670).

Besides these issues, I find the GW2 game-play some of the best I’ve ever come across, I’ve never felt so in control of a character (d/d ele in my case).

There is a graphical issue I often come across, and that is the character being half embedded in a rock or something else that is meant to be solid, such as standing on a wall in WvW to find my character’s feet are part of that wall, though that’s just a minor gripe. It just makes it feel a little unfinished..

I’m also finding a lot of “sound lag” when I first go into Lions Arch. I usually use RTL for instance to go straight to the trading post, and burning speed and lightening flash. The skills work almost immediately, but I’ll get all the sound FX bunched up together a good few seconds later (after that initial lag it is fine).

As a professional developer myself it would be great to have posts of a more technical nature such as shown in the link2 Moka.9641 provided, but I suppose it is understandable if you don’t want to go that far.

Anyway thanks again for the update!

I don’t know, but ever since the teq patch I’m getting lag and DCs regularly now, specially during prime time. I guess DCing players during prime time is a way to relieve stress on the server.

Take a look at tech support and bug report forum, there are plenty of players who are having the same issues all over the world. Please bring back my old gaming experience where by I do not get DCs every 5 mins before you further “optimise” the game server, cheers.

Thanks for the optimization and communication.

I have one question regarding this section if you don’t mind:

BillFreist.1527:

Q: My Second question is : Wasn’t all these problems caused because you have to broadcast every single condition/buff to everyone which is currently pushing your systems and servers/internet to its limits ?
A: This is a good question. Buffs are a large part of Guild Wars 2 and at first glance you’d definitely think that telling everyone about them would be really expensive. In reality, this is only a hit with server->client traffic, which has been significantly improved with the handful of bandwidth optimizations that have already gone out. As far as server processing, its pretty minuscule (thankfully).

With the optimizations that have already “significantly improved” bandwidth, and with those that are to come, do you think it would one day be possible to increase the condition cap in PvE? So far the answer given as to to why they can’t increase it is because of bandwidth concerns, so it stands to reason that if the game is better optimized, they shouldn’t be held back any longer.

Thank you for sharing your work progress with us Bill.

I hope you keep us posted once in a while when you reach a milestone!

P.S. I just couldn’t resist to not upload this !

Thanks for keeping us in the loop Bill

Thanks for the update bill, I only have 1 question.

What about the Mac client?

Hi Bill, thanks a lot for working on this, keeping us updated, and especially for responding to our questions! I think this is definitely what a lot of us want to see following Chris Whiteside’s “Collaborative Development” thread.

I’ve noticed that fights in Garrison and Stonemist tend to have more skill lag than comparable fights in other keeps or in the open field. My guess is that this is because the collision detection is more complex (multiple floors etc.) – if that’s correct, is this something you’re also looking at optimising?

More importantly, I was wondering since it seems you agree that WvW suffers the most from skill lag and that you said you’re going to optimise every skill one by one, would it make sense to prioritise the skills that are most widely used in large-scale WvW builds?

While there are obviously variations, the basic templates are by far the most common and you could probably get 90%+ of the effect by optimising only 25% of the skills.

This might change slightly with the patch (although I expect not much based on Jonathan Sharp’s teaser) but here’s a list that I hope will help you if you do decide to prioritise common WvW skills and don’t already have that data with your internal tools. I’ve listed the classes very roughly by their frequency in the meta.

Warrior
Weapons: Hammer, Sword/Warhorn (rarely Axe/Axe or Greatsword).
Skills: Healing Signet, Healing Surge, For Great Justice, Shake it Off, Balanced Stance, Doyak Signet, Endure Pain, Signet of Rage, Warbanner. (6xRune of the Soldier with Vigorous Shouts)

Guardian
Weapons: Staff, Hammer, Greatsword (rarely Mace/Focus).
Skills: Shelter, Signet of Resolve, Stand Your Ground, Hold the Line, Save Yourselves, Wall of Reflection, Sanctuary, Signet of Judgment, Renewed Focus, Tome of Courage.

Elementalist
Weapons: Staff by far most frequent (some Dagger/Dagger).
Skills: Ether Renewal, Lightning Flash, Mist Form, Armor of Earth, Arcane Wave, Tornado, Glyph of Elementals.

Necromancer
Weapons: Staff, heavy use of marks. Second set more open but often axe/dagger.
Skills: Consume Conditions, Well of Suffering, Well of Corruption, Well of Darkness, Spectral Wall, Plague.

Mesmer
Weapons: Staff, Scepter/Focus, Sword/Focus (rarely Greatsword).
Skills: Ether Feast, Veil, Null Field, Feedback, Blink, Timewarp.

Engineer: Pistol/Shield, Healing Turret, Bomb Kit, Throw Mine, Elixir Gun, Flame Thrower, Rocket Boots, Supply Crate (note: supply crate is one of the most lag-prone skills I know sadly).

Thief: Shortbow, Hide in Shadows, Smoke Screen, Shadow Step, Shadow Refuge, Blinding Powder, Caltrops, Basilisk Venom, Dagger Storm.

Ranger: Sorry, no clue.

I hope this helps and that you continue to keep us updated on your progress!

One thing I don’t get. In wvw PASSIVE effects NEVER LAG. They are always there, and always work. (Healing signet warrior for instance). Why is that so? Why do active skill lag while passive ones don’t. Honeslty at the moment this makes stonemist fights a bit unfair (passive heal so much strong then active heal, cause they won’t activate).

Thank for anwser some question, and Thank you for the effort.

Phoebe Ascension.8437:

One thing I don’t get. In wvw PASSIVE effects NEVER LAG. They are always there, and always work. (Healing signet warrior for instance). Why is that so? Why do active skill lag while passive ones don’t. Honeslty at the moment this makes stonemist fights a bit unfair (passive heal so much strong then active heal, cause they won’t activate).

I’ll take a shot at this.

In very large battles when the server is lagging – time takes longer. The server only lets clients use skills after they are off cooldown. If the server is taking 2 seconds to process 1 second worth of data that means the real cooldown of your ability has doubled. The problem is the client itself has no way of knowing how slow the server is going so the timer on your skillbar only uses real time. So if your heal has a 20 second cooldown and you are in a laggy fight, the server might only think 10 seconds has passed when you try to hit the skill again, thus the skill wont activate because it is still on cooldown according to the server.

Passive effects i’m not absolutely sure about, but they might only be half effective in laggy situations as well, for example you only get 10 seconds of healing in a 20 second period. The major advantage of passive abilities is that you don’t need to know when its off cooldown and useable.

BillFreist.1527:

Q: My 3rd question is :If you ever have a solution to this are there any plans to lift the AoE Limit for both WvW/PvE/PvP because thats why this AoE limit exist in 1st place Isn’t skill delay in big battles (WvW or PvE like Taco) a direct result of those?
A: <<< The AoE hit limits exist for more than just performance reasons.

But do you not find it problematic that one of the most broken gameplay tactics in the game is for a zerg to just ball into a little clump so that only ever 5 players take damage from AoEs?

Can you look at the suggestion I made here? https://forum-en.gw2archive.eu/forum/wuv/wuv/Don-t-make-a-new-map-Solve-the-skill-lags/page/2#post2783543
Essentially, I’m asking whether it’s possible to ration object interactions so as to cut down on the runaway exponential growth of computing load.

BillFreist.1527:

Part 2:

…snip…
Q: When you say “optimization”, do you mean making your engine more compatible for multi-core systems? Right now your game’s engine runs primarily on single-thread, which Intel CPU’s seem to thrive on; while at the same time your game’s engine is poorly designed for multi-thread CPU’s, which AMD is best at.
A: As I outlined above, this is mostly server changes that I’m involved with, but there are client optimizations in the works, but we don’t know when those will go out. Optimizing is extremely dangerous and requires a lot of testing. The server side changes are more controlled in the sense that we don’t have to worry about testing on different hardware configurations like with the client. Its unfortunate, but necessary to ensure that the changes don’t end up doing more harm than good.

The engine isn’t as single-threaded as you might think. It will only use as many threads as your machine can support. if you have a 4 core processor, you may only get 2 threads plus an additional 2 worker threads for processing things like animation. Though I we are definitely looking at other ways to improve other game systems to take better advantage of using multiple threads while they’re not in use.
…snip…

Thank you very much for your detailed answers to the many questions posed you and the rest of the development team.

As for the section quoted above and if you refer to my above post you will not that the lack of threads may not be an ArenaNet issue. Since then I have checked several of my friends computers, almost all of which have AMD processors. Of the six that are using FX class processors I found that only twice did the chipset/processor driver install properly. I can not speak to Intel processors, but I would as that you advise people experiencing low frame rates to uninstall their chipset and processor drivers, and then reinstall them starting with the oldest version first. This was the only way I found that consistently corrected the multi-threading issues in Guild Wars 2 and a few other games.

Thank you again for everything you do for us that we never get to hear about!

Phoebe Ascension.8437:

One thing I don’t get. In wvw PASSIVE effects NEVER LAG. They are always there, and always work. (Healing signet warrior for instance). Why is that so? Why do active skill lag while passive ones don’t. Honeslty at the moment this makes stonemist fights a bit unfair (passive heal so much strong then active heal, cause they won’t activate).

Draygo is sorta right, but the explaination doesn’t really explain what you’re describing.

The closest explanation I have is that the thread, or daemon or code, that handles passives is not getting bogged down while the thread that handles cooldowns or skill activations is getting bogged down. Since the code for passives isn’t getting delayed or slowed, they always work.

Imagine it like a couple of people in an office. The person that handles active skills has a lot of work to do. The person that handles passives has little work to do. The passives person would naturally get done faster.

Have you experimented/considered auto adjustable settings for the client? Say if FPS drops below certain levels it turns off selected features or scales them down and if it goes back above it turns them on/higher again (like downsampling).

If possible it would be nice to have them for each setting (that can be adjusted on the fly and dont cause too much of a spike) separately with a global desired FPS option at 30/45/60.

In particular this would be useful for shadows and reflections as they are the most demanding on the CPU.

It should help most people due to the variance in client performance due to forcing people into zerging certain events and it should be a relatively quick thing to implement.

Apart from that if you could maybe extrapolate on the things you are doing for client performance it would let people know that you are actually working on something significant. SOE has done some good reassuring with this:
https://www.youtube.com/watch?v=EaHgTR4yyuU
https://www.youtube.com/watch?v=bgOsrJaibjM

Finally IMHO you need a bigger leash on content creators. Most people who wanted a redesigned Tequatl fight didnt really ask for 100 people to run to single spot and fire off some 1000 skills in a matter of seconds. Not to mention all the other forced zergfest updates in the last few months.

Awesome posts, very insightful. As a WvWvW player one thing that always surprises me in large skill lag battles is that my thief suffers a great deal less than my necro. Of course I can’t play both in the exact same battle but massive three ways in Stonemist I always wish I was on my thief who seems to suffer much less from skill lag on weapon skills (but probably same skill lag with primary heal).

Possibly something to do with animation or complexity of skills (less conditions?) or are thiefs just handled/prioritized first?

I wish you luck with this project, if you can get anywhere near the improvements made to culling you may become as legendary as Patrick.

Thanks for keeping us updated on this.

Concerning skill lag, in my own experiences in PvE with ~50 players trying to down a dragon, I’ve noticed that the utility skills (7, 8 & 9) are the ones that are least likely to fire off in a timely manner under these conditions, but conversely, the heal skill will almost always work as desired when commanded.

Say on an ele, meteor shower is a bit 50/50, sometimes it will fire off on command, other times it just won’t, and my guess is the aforementioned AoE limit. Having said this, a utility skill such as Arcane Blast which is a simple single-projectile skill with a guaranteed crit, will almost never fire off until about 10 seconds after I’ve started mashing the key, and the number is worn off the keyboard.

Again this is just my observation and only really becomes an issue over weekends when the playerbase periodically peaks; normal times during the week there’s ~30 players at any given event and this isn’t as much of an issue, or if it is then it’s not as noticeable.

Kudos to you for keeping us in the loop.

BillFreist.1527:

I’d like to create a central place for you all to get the current state of where we stand on optimizing the servers and relieving what most of you have come to know as “skill lag”.

Does Arena-Net have any intention of investing in upgraded (Newer/Better) Server hardware and related solutions?

One of the main reasons World of Warcraft is losing people is because they have in so many words refused to put any money at all into better infrastructure. They have instead opted to “phase” things or “optimize” pre-existing bargain bin hardware.

Any input you could provide would be much appreciated.

Sounds awesome!

One thing I noticed is that when the Scarlet events released I didn’t have any skill lag in the Scarlet events (WvW remained much the same). It looks like you’ve already finished with this thread; still I am interested to know whether WvW and PvE share the same server architecture and whether you have thought about capitalising on any of the improvements they have clearly been made in PvE.

Voland.5862:

Does Arena-Net have any intention of investing in upgraded (Newer/Better) Server hardware and related solutions?

Not all problems can be scaled vertically – quadratic problems like game servers are particularly poor candidates for vertical scaling. This is because, e.g. if you double your CPU speed you will only get a 50% speed improvement instead of the 100% you would expect. To compound the issue it may not even possible to get a system that is twice as fast as the current ones (because they might be close enough to the bleeding edge that only, say, a 0.3ghz jump is possible).

Not everything in WvW is quadratic, but “skill lag” is the symptom of one area that it is (everyone dropping AOEs) – so pulling out the wallet and improving the hardware would probably only make a marginal difference that us players wouldn’t really notice.

The best thing to do always is to improve the efficiency of your algorithms (or I would rather say, decrease the inefficiency). At the time you would hypothetically get more hardware you would have more bang-for-buck because of previous optimizations.

zamalek.2154:

Sounds awesome!

One thing I noticed is that when the Scarlet events released I didn’t have any skill lag in the Scarlet events (WvW remained much the same). It looks like you’ve already finished with this thread; still I am interested to know whether WvW and PvE share the same server architecture and whether you have thought about capitalising on any of the improvements they have clearly been made in PvE.

Voland.5862:

Does Arena-Net have any intention of investing in upgraded (Newer/Better) Server hardware and related solutions?

Not all problems can be scaled vertically – quadratic problems like game servers are particularly poor candidates for vertical scaling. This is because, e.g. if you double your CPU speed you will only get a 50% speed improvement instead of the 100% you would expect. To compound the issue it may not even possible to get a system that is twice as fast as the current ones (because they might be close enough to the bleeding edge that only, say, a 0.3ghz jump is possible).

Not everything in WvW is quadratic, but “skill lag” is the symptom of one area that it is (everyone dropping AOEs) – so pulling out the wallet and improving the hardware would probably only make a marginal difference that us players wouldn’t really notice.

The best thing to do always is to improve the efficiency of your algorithms (or I would rather say, decrease the inefficiency). At the time you would hypothetically get more hardware you would have more bang-for-buck because of previous optimizations.

FYI, Arena-net hasen’t exactly been very transparent in terms of their infrastructure.

Worst Case Scenario: They are doing what blizzard did/still are doing. They snatched up a good deal of outdated hardware and put an MMO on it.
Best Case Scenario: The hardware is up to date enough that all they need is some serious effort into optimization and restructuring to resolve the issues full stop.

The most likely scenario
They have decent hardware but the poor choices made in terms of how the game as a whole was implemented both client and server side will be a time sink that will eat optimization efforts while only yielding minimal improvement.

Awesome to see a thread like this exists. Much appreciated Bill.

This thread is more relevant than ever given the sharp increase in lag in the last 2 days. You guys have to come up with something fast because WvW has been verging on unplayable the last 2 days with the lag.

^

WvW is unplayable. We queue for 1-2 hours (mid tier server) and jump into 10 seconds of massive unplayable skill lag. All you can do is run forward and die, to the point even a sentry nearly killed me.

WvW imo is very broken right now, I don’t know if doing the seasons/archievements (which in my opinion are awesome ideas) are executed at the right time. There should have been some sort of fix in place first. Maybe adding the archievements shouldn’t have taken place at the start of the first WvW season either. Now you have “unfair” and “biased” results because a large amount of the PvE community is playing WvW at this point.

Just my two cents here.. I hope WvW becomes playable again soon because it’s what I love the most in GW2.. consumes all my time playing GW2 and now I can basically not play the game I like to be playing.

Voland.5862:

FYI, Arena-net hasen’t exactly been very transparent in terms of their infrastructure.

Worst Case Scenario: They are doing what blizzard did/still are doing. They snatched up a good deal of outdated hardware and put an MMO on it.
Best Case Scenario: The hardware is up to date enough that all they need is some serious effort into optimization and restructuring to resolve the issues full stop.

The most likely scenario
They have decent hardware but the poor choices made in terms of how the game as a whole was implemented both client and server side will be a time sink that will eat optimization efforts while only yielding minimal improvement.

Assuming your most likely scenario, it could actually be an issue with design decisions not taking technical limitations into account. There is a good reason for why the combat in traditional MMORPGs were designed the way they were. Skill cooldowns, global cooldowns, client throttling were all systems or constraints put in place to alleviate bandwidth and server processing. The action heavy nature of Guild Wars 2’s combat translates to a heavier burden on the server side that wasn’t experienced in traditional MMORPGs.

There’s also a lot more overhead on server hardware for MMORPGs in general than other online games. You have more than just a single server for each world/realm. You have what are known as server clusters, which are a group of servers that pool their resources together to function as a single unit. With this setup, you don’t really get 100% of the resources because they are separate systems. They need to communicate with each other and there is always going to be latency. We’re still only looking at world servers.

All of the data, which includes skill inputs, needs to go through the world servers and into a database hosted on a database server which is running a database management system like Oracle, MySQL, etc. Every time you input a skill, that skill is sent by your client to the application (world) server, and the application server generates a query to the database residing on the database server. I think it would be safe to assume that the database and application servers all reside on the same network. Even on a 10GB Ethernet network there’s still going to be slight delay. That’s not to mention the I/O lag you get from moving data between memory and storage on the database server itself.

As for optimizations there are a few things that can be done if they aren’t already. I do think they have been done, are being considered already, or are technically infeasible. Good database design requires normalization of data, to eliminate insert, update, or delete anomalies, which means you’ll end up with a lot of heap (data) tables. Joining these tables in queries will produce a lot of overhead unfortunately so database designers will often try to do the minimum amount of normalization needed. They will also use indexes to make searches faster by narrowing down the search area of a given table. Yes, they could also improve their algorithms on the application server but these things could end up making things worse in other areas so thorough testing will be needed.

For the hardware side they could invest in NAND flash. I’ve never actually used it myself as it’s still fairly expensive right now. It’s really a question of whether it’s actually worth the investment. How much of a performance gain in percent is achieved relative to the dollar amount invested?

Another solution that ArenaNet could consider is to have both the application server and database server running on the same host as virtual machines. It’s what is referred to as a virtual application. This could significantly reduce the latency on server side processing but it would also require a significant investment of both time and resources into rebuilding the infrastructure to make it all happen. It all comes down to performance gain vs money invested.

Voland.5862:

zamalek.2154:

Sounds awesome!

One thing I noticed is that when the Scarlet events released I didn’t have any skill lag in the Scarlet events (WvW remained much the same). It looks like you’ve already finished with this thread; still I am interested to know whether WvW and PvE share the same server architecture and whether you have thought about capitalising on any of the improvements they have clearly been made in PvE.

Voland.5862:

Does Arena-Net have any intention of investing in upgraded (Newer/Better) Server hardware and related solutions?

Not all problems can be scaled vertically – quadratic problems like game servers are particularly poor candidates for vertical scaling. This is because, e.g. if you double your CPU speed you will only get a 50% speed improvement instead of the 100% you would expect. To compound the issue it may not even possible to get a system that is twice as fast as the current ones (because they might be close enough to the bleeding edge that only, say, a 0.3ghz jump is possible).

Not everything in WvW is quadratic, but “skill lag” is the symptom of one area that it is (everyone dropping AOEs) – so pulling out the wallet and improving the hardware would probably only make a marginal difference that us players wouldn’t really notice.

The best thing to do always is to improve the efficiency of your algorithms (or I would rather say, decrease the inefficiency). At the time you would hypothetically get more hardware you would have more bang-for-buck because of previous optimizations.

FYI, Arena-net hasen’t exactly been very transparent in terms of their infrastructure.

Worst Case Scenario: They are doing what blizzard did/still are doing. They snatched up a good deal of outdated hardware and put an MMO on it.
Best Case Scenario: The hardware is up to date enough that all they need is some serious effort into optimization and restructuring to resolve the issues full stop.

The most likely scenario
They have decent hardware but the poor choices made in terms of how the game as a whole was implemented both client and server side will be a time sink that will eat optimization efforts while only yielding minimal improvement.

I think most of the issue is poor optimization and a lack of virtualization. The combat system alone needs an absurd amount of number crunching per player. Take conditions for example. The condition system currently calculates and recalculates every single condition you apply in real time in order to account for stat changes from boons/food, additionally, it calculates condition damage on the server side, rather than the client side, and has to track the damage across stacks. It does that per active character, including for mobs that cause conditions. It also retroactively buffs already applied conditions to account for things like Might. This is one of the reasons the condition cap exists (that and the whole wvw/pvp balancing with pve issues). It also actively knocks the lowest damage condition stack from the total stack when in a group.

This means per 5-man party, the game could be accounting for a total of 5-15 conditions per attack (including sigil and skill condition adds) plus the already applied conditions, calculates and recalculates the whole group in case of stat change, re-applies and has to repeat that process every single tick of every single condition, and every single attempt to apply the conditions. Now think of the zergs we get occasionally get on champions, in wvw, or on world bosses. That probably accounts for 10-15% of the server side skill lag we get. Guild Wars 2 is the only game I’ve played that does all condition calculations server side, or updates already applied conditions based off new buffs, or makes players conditions over-write each other, most games use a player individualized stack to make balance of conditions easier.

Syndrilious.9632:

Hi Syndrilious,

I am completely ignorant as far as the background and what needs to be done as far as server or game client. I can only provide feedback from just an average user.

If I understood correctly GW2 has on the WvW feature one of its unique features (and quite frankly it is awesome). I do understand that when massive zergs join together and clash things tend to be a bit “slower”. As a player it is kind of a bummer… True, but manageable.

In the beggining there was some lag but I noticed that it went on getting better with the work invovled over time. However, since the WvW leagues and Bloodlust feature I noticed the lag and (something I had never experienced thus far) skill lag and NPC’s lag. One can say it has more people palying it (which is good – the more the merrier).

I understand that new features bring new issues and complication. I relate to that. However, it is hard for me to understand how can Arena Net have one of its core game features have more features which will obviously bring more people into it and not be fully ready for them (especially when they claim that they bring it out only when it is ready to launch).

Quite frankly I would rather not have WvW League (which in my server brought more conflict than benefits – but that is a totally different story not relevant for this thread) and have maps that do not lag with big zergs than have a league that you cannot play well enough because you simply cannot play.

I do not want to sound bitter or picky. I just completely love this feature (WvW) and obvisouly I may be sensitive when things do not go as they are planned. This needs to be addressed and I am happy to know that staff of Arena Net (BillFreist) officially made a statement on this. At least we have a confirmation that things (even though slowly) are being addressed. Let’s see how much time we will have to wait to see them implemented and detect them!

Bill, thank you very much for taking the time to create this thread, giving us such detailed information on what you are doing to save us from this horrible skill lag, and replying to our questions. Much appreciated, and I just have to add that you’re the nicest guy I personally have ever seen to interact with us on these forums. A very special thanks for that alone.

Some observations from my own gameplay experience in wvw which might perhaps be helpful to you and your team:

- Skill lag and weird fps drop issues significantly intensified for me with the Teq patch.
- I’m saying “weird” fps drops because normally I don’t experience it to that degree, and I never experience it in any other game I play.
- I get horrible skill lag and weird fps drops ONLY when our group (small or large) is outnumbered by the enemy. Never the other way around. So, it doesn’t seem to matter how many players are currently fighting on a given area, if we are, say, 25-30 and the enemy numbers are higher than that, I experience these two issues to such an extent that hardly any skill goes off.
- The skills that almost never go off are the utility skills in my bar (heal skill in particular), hardly ever my weapon skills.

I don’t know if other players have already mentioned similar observations or not, but these are my experiences in wvw fights. With the introduction of the Leagues, all of the above have gone from terrible to unplayable for me. Finally I had to exit the game last night because I simply wasn’t able to function in my role.

Wish you guys best of luck in identifying and “slaying” the culprits as soon as possible