Guild Wars 2

I’m on Gandara EU (silver) and skill lag all over the place. . It’s near to unplayable and it’s such a shame for this awesome game mode. Can you please update us on your progress on the skill lag issue

It’s a shame for the whole game, as this happens in WvW in Tquatl etc …
Ami honestly I can not believe that after more than a year, we continue with these problems, sound problems, and only penseis on how to make money from the auction. Start by having people happy in the game, with a good performance and fun and your money will come.

Hey bill!
Great you explain us what’s going on with the skill lag and great to hear that you guys are working on it!

Though I’m still confused by what happened the last week.
Basically what has been going on is a few months ago there was a post about a team of arenanet saying they were working on the skill lag step by step, which you mentioned in your first post. Of course, like you also mentioned, the first steps are barely noticeable, but what actually happened after the patch?

You probably already know what’s going on, the skill lag got worse, and not just a bit. Pretty much at every fight there’s skill lag now, which wasn’t before the patch. So what happened? Is it a bug which came along with the leagues or is it the leagues itself causing the problems? Or maybe totally something different?

Hope to hear an answer soon from you,

Doiid

BlackDevil.9268:

Hey bill!
Great you explain us what’s going on with the skill lag and great to hear that you guys are working on it!

Though I’m still confused by what happened the last week.
Basically what has been going on is a few months ago there was a post about a team of arenanet saying they were working on the skill lag step by step, which you mentioned in your first post. Of course, like you also mentioned, the first steps are barely noticeable, but what actually happened after the patch?

You probably already know what’s going on, the skill lag got worse, and not just a bit. Pretty much at every fight there’s skill lag now, which wasn’t before the patch. So what happened? Is it a bug which came along with the leagues or is it the leagues itself causing the problems? Or maybe totally something different?

Hope to hear an answer soon from you,

Doiid

Well said! I would like to hear some tech details behind what’s happening to create these conditions.

Dr. Clarke-Willson had a great talk here (slides only, unfortunately), maybe an explanation in the context of that talk?

Hi,

Devan asked us to post in this via another thread. The lag issue is absolutely serious for many of us in the WvW league. Match 1 is our most critical match-up and lag is affecting people ability to play this game, We have half our 50 man guild not logging on till this is fixed, we have another large guild stop playing all together until this has stopped and yet another 100 man guild transfer to another server during the wvw league because of how bad this is, again seriously affecting us. All attributed to server side lag.

It is killing our guild and the game for many. Please see if there is anything you can do in the interim to address this??

BillFreist.1527:

Over the past several months we’ve had a small team working on ways to relieve some of the load from the game server while there are large concentrations of players. Most of the work thus far has gone pretty much unnoticed from the player’s perspective, but there have been pretty large steps made towards the issue. This continues to be a large and high priority issue for us so we can continue to make improvements to make this a thing of the past.

You need a “large” team. There needs to be noticeable improvements every patch.

This is not a troll post and I appreciate that it’s being looked into, but I’m going to tell you this one simple fact: Skill lag makes the game unplayable, players abandon unplayable games. It is a testament to how much player interest there is in WvW and the large scale PvP concept that players have stuck with Guild Wars 2 in spite of this skill lag for as long as they have. What I hear all the time in Teamspeak from the WvWers is that this game is “it” right now, there are no other serious choices for WvW. All anyone has to do really is introduce a game featuring the WvW concept without the skill lag and poof, there goes your entire WvW population which incidentally is the group who spends the most money on this game.

Again I’m not trolling, I’m just underlining the fact that first and foremost you need to fix the skill lag problem which has been around since launch a year ago or you will lose all of your WvW players to other games. It needs to be noticeable improvement every patch, it needs to be a “large” team working on this. You need this more than you need new PvE content.

Jubelum.3472:

BillFreist.1527:

Over the past several months we’ve had a small team working on ways to relieve some of the load from the game server while there are large concentrations of players. Most of the work thus far has gone pretty much unnoticed from the player’s perspective, but there have been pretty large steps made towards the issue. This continues to be a large and high priority issue for us so we can continue to make improvements to make this a thing of the past.

You need a “large” team. There needs to be noticeable improvements every patch.

This is not a troll post and I appreciate that it’s being looked into, but I’m going to tell you this one simple fact: Skill lag makes the game unplayable, players abandon unplayable games. It is a testament to how much player interest there is in WvW and the large scale PvP concept that players have stuck with Guild Wars 2 in spite of this skill lag for as long as they have. What I hear all the time in Teamspeak from the WvWers is that this game is “it” right now, there are no other serious choices for WvW. All anyone has to do really is introduce a game featuring the WvW concept without the skill lag and poof, there goes your entire WvW population which incidentally is the group who spends the most money on this game.

Again I’m not trolling, I’m just underlining the fact that first and foremost you need to fix the skill lag problem which has been around since launch a year ago or you will lose all of your WvW players to other games. It needs to be noticeable improvement every patch, it needs to be a “large” team working on this. You need this more than you need new PvE content.

Whether or not you meant it to be this is really condescending and kind of annoying.

I’m quite sure these guys know that it’s a serious problem, and that it is going to negatively impact their player base. They’re not idiots, you know? I’m quite sure there are project managers, CEO types, customer service people, etc., etc., all aware it’s an issue and breathing down these guys’ necks. Do you ever stop and think how tiresome it must be to read stuff like this?

I think there’s plenty of causes for legit frustration around this issue (my guild is WvWing right now but I’m sitting in queue reading the forums, and I’m not sure I even want to play WvW because the lag is so bad), but it’s the most annoying thing in the world to have someone preaching at you about what you need to do like you don’t already know. Especially in this particularly condescending, “Now, I’m not trolling bud, but I’m just letting you know…” tone. Do you think the QA devs of all people don’t know it’s a serious issue?

Source: I’m a dev who absolutely HATES stuff like this, either from users or co-workers.

edit: I guess there’s a chance I’ll get infracted in some way for this post myself, since it’s kind of combative, but sheesh.

edit2: And this whole “you need a large team” thing is garbage. Throwing more people at a problem is counter-productive. Mythical man-month, etc.

Good luck to Bill and his team. I hope his employer supports his work by providing upgraded hardware to complement the coding changes he is making.

anything happened yesterday? the lag was less

To Bill, do you think that toning down the gfx of all the different skills/aoe effects will be better for the server? Or maybe even server and client performance.

Hi
With all the skill lag commotion since the last update I’d like to know if there were already any changes on the last patch.

Before on zerg v zerg fights i tried to activate skills but the icon kept on flashing the queue animation until half the people were dead (basically no skill activation) and it was mostly limited to the fight zone and close by areas.

Now it’s possible to tell when a zerg fight starts on the other side of the map because of the same issue (no skill activation).
Also lately i’ve been having trouble with recharge times too. I noticed that on my warrior shouts would recharge but i couldn’t use them (so i thought zerg fight → no skill activation) but then i noticed that my signets recharge time was over but the passive effect didn’t start for quite some time.

Anyone else noticing recharge time discrepancies as well?

I’ll throw my $.02 in. Lag has been pretty rare for me, pretty much since release except for a few specific instances. I feel I’ve been pretty fortunate. I’m not sure what happened the past 2 days but something seems to have changed. Skill lag in LA for costume brawl and in WvW was very bad. Bad to the point I had every skill flashing at one point with no effects. The odd thing was, I could see everything happening around me in real time, the only thing I could not do is use my skills. Damage and heals were also very slow to register on only me. While talking to my guildies in Vent they said they could see me running into a wall. So somewhere there is a disconnect for certain server-client info packets but not all? Hope this can get straightened out soon. Let me know if you want further info on anything.

Edit: Friday evening, several guildies and I all have been experiencing massive skill lag. Doesn’t matter where we are, how many are around us. I’m not sure what changed in the past few days but this is becoming unplayable at times. Sad face.

Let’s cut through the kitten on this. A-net has received hundreds of dollars of my money. The game was great at first, but they have oversold their capacity. I feel like I paid for a first class airline ticket, got bumped and had to sleep in the airport, only to get up the next morning to be put on a broken down bus with standing room only.

A-net knew this was a problem, so what did they do? They put the game on sale for 40% off and put out a whole table full of carrots (a WvW ‘tournament’ full of AP achievements) to lure even more people in. Is this a sign of desperation, or what?

after this horrible season start wvw is one big joke (w8 many hours in queue/lags/skill lags or complete stop work/random DC) you got lags when fight in zerg or small group or 1v1 and i really love when in fight 1v1 my skills stops work and enemy player easy kill me and anet really good idea put AP to wvw !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!

In many other games, including GW1, the server sometimes slowed the game pace down to save the server from overloading. Its effective, and in PvP effects everyone equally.

Why do not GW2 do this? Would 75% or even 50% WvW gameplay kill the game experience more than skill lag currently do?

I’ve faced skill lag in Lunatic Inquisition map.
I also faced a point whereby I couldn’t even activate any of my lunatic skills while being a ghost to chase and hunt down the villagers.
The villagers were wondering what the hell is wrong with this ghost who just move around and not using any skills.

Pressing the keys (1 to 0) and using mouse clicks on skill do not work.
I ended up have to re-login to the same character to get the keys working.
Is this really a skill lag??

“Lag” can be:

1. the server is running a bit behind

2. packet loss or delay from client to server

3. packet loss or delay from server to client

4. a long work queue in the client, in which messages from the server wait so long they’re useless (and then execute and render on one’s screen)

5. N^2 work going on on the server, due to too many people in proximity

6. N^2 work going on on the client, due to too many people in proximity

7. Graphics texture cache thrashing

8. waiting for the hard disk (but anyone who cares about performance in this game uses a new generation fast SSD anyhow, the zone loads are absurdly slow with a rotating disk)

it would be really useful to have some time stamped logs, or even some performance indicators from the client (the server is too much to ask) so that the source of lag can be identified. I’ll even agree not to hold you responsible if the indicators tell me to buy the wrong thing to upgrade my client.

or if the lack of data causes ArenaNet to invest in the wrong place to fix it

edit: there are some techniques used in networking to mark packets as candidates for discard if something gets overloaded downstream (sometimes called “coloring” or “marking”, ignoring the nuances of both terms). My guess is the fix for clients falling on the floor during big PvE events or keep/stonemist sieges is that the work the client is doing needs to be ruthlessly prioritized.

if there were a single statistic I’d like it’s a graph of queue length for work in the client, measured once per graphics frame (or at 20 millisecond intervals). Still remember DAoC where the client would declare an LD and drop the player…and didn’t bother to check for the keepalive packet which had been cooling its heels in the client’s work queue for several seconds at that point.

But then DAoC instantiated each spell effect in a heavy weight call to the (purchased) client engine…the same heavy weight call used to instantiate a player. No one would be foolish enough to do that in 2013, would they?

Thanks for this really interesting post and the links, i enjoyed the presentation of cameron. I really hope that the fixes you will introduce can ease the situation we are facing in wvw. To be honest the workaround or temporary things you spoke about did something noticable. In the last few days i encountered a lot of less skill lag than before. Can´t wait until the permanent fix comes, thanks again.

Great update Bill and really reassuring.
And thanks for using non-tech/corporate speak so people like me can understand

Thank you for the update Bill. It was really informative and put a lot of things into perspective. Great post. I hope that the improvements come without very many hitches.

Thanks for the explanation, Bill, it was great to read, and I look forward to the upcoming fixes. I book marked the GDC video so I can watch it.

Spork.2375:

Ugh, I really want to come up to Bellevue and see what you guys are doing. It’s frustrating as a developer to sit by and hope someone else figures it out. I’m sure I’m not alone here on that feeling. So get your NDAs printed up and ready to be signed.

As a developer or creator of original content, you must be aware that there are many ways to approach or code a feature or fix a problem. Working in a similar field, I know how easy it is to [peer] review a product and suggest an improvement that the original content developer did not think of. If Anet brought in 10, 20, 50, or 100 more developers you’d have that much more creativity and knowledge to apply to your product and improve or speed up the product (i.e. SETI@home or graphics render farms).

But there is a limit, based on funding, space, simultaneous development, etc, to how many coders or people can be applied to development. Like most products that are in constant development, there are production phases or iterations, where you spiral the development and come and revisit your initial or follow up implementation of a feature to re-evaluate or improve/fix what you are looking at.

The skill priority implementation is one aspect that is being reevaluated, as quoted by Bill above (i.e. seemed good at the time, after a second look, maybe not the best way). At first blush, it seems easy to say that all you need is an afternoon to see how things are done, and (possibly) assume you can gen up a better way. But 99 times out of 100, it’s not that easy.

Just continue giving feedback as the devs are obviously engaged with the community and listening (even if it doesn’t always seem that way).

Bill, thanks a lot for the recent improvement!
I haven’t had the chance or misfortune of finding myself in an intense 3-way battle since I wrote my previous post, but generally speaking it does feel better now

As far as skills just not executing (noticed some people claiming utilities are more susceptible to this), its mostly just a race-condition as far as processing on the server.

I am one of those peeps, and it was no “claim”, neither in my own experience nor in my friends’. Our entire TS had the same issue, and many other people in other threads have already mentioned it. My autoattack always works, the remaining weapon skills get delayed but do go off after a (very annoying) while, which makes me think they have some sort of priority? To further clarify, my weapon skills keep flashing when pressed and then do go off eventually; but the utility skills plus the elite do not even flash, no matter how brutally I attack the keys. They just do not get triggered. So, yes, I guess it must be a bug as you explained, and I’m happy you’ll be looking into it

So, are any of these changes going to effect all the “black screens”, “error messages” and Game Crashes? Skill lag is one thing.. but whole groups of people getting kicked out of WvW, during prime time.. where there is a 3hr queue to get into the map is getting old. really fast.

My connection tonight (ping):

Home servers: Loss Rate 0%
Google: Loss Rate 0-2%
Yahoo: Loss Rate 5-7%
guildwars2.com: Loss Rate 66%
206.127.159.84 (my game IP gw2): 100% timed out!!

Wonderful work! I played 10 minutes and I was disconnected.

Avanax.8492:

My connection tonight (ping):

Home servers: Loss Rate 0%
Google: Loss Rate 0-2%
Yahoo: Loss Rate 5-7%
guildwars2.com: Loss Rate 66%
206.127.159.84 (my game IP gw2): 100% timed out!!

Wonderful work! I played 10 minutes and I was disconnected.

Don’t forget pings are handled on a best effort basis by the destination, and under load need to be discarded as part of protection against DDOS attacks.

That having been said, was Windows downloading a patch in the background? That sounds like an internet connection problem at your ISP. Oh, one other thing: I had a DSL modem go bad once in a way that it ran really hot but the bit rate decayed until I couldn’t even load my ISP’s support page to file a support ticket.

Yes, every dropped packet is roughly a 2 second TCP timeout and retry. Multiple dropped packets back to back will probably trip the client’s DC trigger. Ouch.

is nov.12. valid?

206.127.159.84 blocks ICMP so you cant just ping it.

If you have a packet loss of 5-7% to yahoo, you have a serious problem and need to get your ISP to look at the signal quality to your modem, or you may need to have your modem replaced.

I would be interested to know what you did technically; if you are able to share those details (if not here then on gamedev or something – it’s been a while since us indie devs have had anything more than the Age of Empires and Halo network coding postmortems to mull over).

So skill lag last few weeks for me has been getting much worse, to the point that all skills are flashing but massive delay on activation or not-activating as the case maybe, this is also not just now in WvW but in PvE as well, plus seems more troubled around certain classes (especially human thief ).

I see there was a mention of increased of WvW players creating lag issues and large queue times for WvW (nearly 2hrs for me last night), well what did you expect when you put PVE Achievement content into a WvW environment, now the WvW slots are being swallowed up by people solely intent on PvE achievements , leaving long queue times putting more strain on the server and generally disrupting the flow WvW (+ Im fed of the whining as well from the WvW players about it)

Unhappy? Yes!!!

BillFreist.1527:

Some other assumptions I’m seeing is about our server hardware and inefficient use of communication between them and the game’s database. I’ll go ahead and put these assumptions to rest. Our server hardware was actually purchased new right around the launch of the game, rest assured its not out-dated.

The hardware can be as new and as powerful as you like. The problem for EU players is that you chose to locate your ‘EU’ servers in Texas, and no matter how powerful the server hardware you’re going to get more lag if the data has to go from (say) France to Texas and back than if it had to go from France to the Netherlands and back.

I was in a semi 3-way SMC fight on reset night (the 3rd server came in near the end of the fight (Sanctum of Rall, Tarnished Coast, Magumma match-up)) and was pleasantly surprised to find that the skill delay on my skills was significantly shorter, down from the expected 10+ seconds to around 3 seconds. PvE wise, I rarely notice skill lag so I can’t say much about it there. Good job on the changes so far.

BillFreist.1527:

Carzor Stelatis.9435:

BillFreist.1527:

Some other assumptions I’m seeing is about our server hardware and inefficient use of communication between them and the game’s database. I’ll go ahead and put these assumptions to rest. Our server hardware was actually purchased new right around the launch of the game, rest assured its not out-dated.

The hardware can be as new and as powerful as you like. The problem for EU players is that you chose to locate your ‘EU’ servers in Texas, and no matter how powerful the server hardware you’re going to get more lag if the data has to go from (say) France to Texas and back than if it had to go from France to the Netherlands and back.

False. They are located in Germany.

How did you come to this conclusion?

You can thank the various geoip databases and the tools that rely on those databases. Many will report the EU servers as being in Texas because the IP is owned by NC Interactive/NCSoft West which is based in Austin, Texas(but you probably know that part already).

Carzor Stelatis.9435:

BillFreist.1527:

Some other assumptions I’m seeing is about our server hardware and inefficient use of communication between them and the game’s database. I’ll go ahead and put these assumptions to rest. Our server hardware was actually purchased new right around the launch of the game, rest assured its not out-dated.

The hardware can be as new and as powerful as you like. The problem for EU players is that you chose to locate your ‘EU’ servers in Texas, and no matter how powerful the server hardware you’re going to get more lag if the data has to go from (say) France to Texas and back than if it had to go from France to the Netherlands and back.

Do a tracert on one of the game sever IP’s (the ones you get when you type /ip) or look at the ping values in resmon for gw2.exe, I get values in the range of 20-80 from the UK, that’s normal for a connection to a server in germany, the US would be 130+ (I used to be on a US server in GW2, and I can tell you I felt the ping difference at once when I moved EU)

As far as skill lag, the only place I have had any problems popping skills has been in large 3 way fights (A nice big garri fight, almost 3 whole zone blobs), and even then it as around 0.5-2s at its worst rather than upto 4s, a massive improvement, it has made WvW a lot more enjoyable for me thats for sure!

In normal WvW fights I haven’t had any problems, and it seems the whole map doesn’t start lagging anymore when a large fight breaks out somewhere.

BillFreist.1527:

So during the November 12th release, we rolled out a couple of changes that significantly reduced skill lag. One mostly targetted PvP and WvW, whereas the other was more high creature concentrated PvE areas.

Curious to see if you, as players, have noticed the impact (or haven’t noticed it because the game feels as it should). Of course WvW will still see occasional spikes during large 3-way encounters, but those a significantly shorter and the server can now recover from them almost immediately.

Needless to say, more improvements are well underway.

-Bill

Yes, we were discussing at guild how skill lag was lower than before in mass fights at lord’s room. For me, skill lag is almost unnoticeable now and in mass fights you can actually blast stuff in 100 man clash after some delay and not only press 1 and hope it hits.. Altho there were reports by some players that their frequency of crashes increased presumably after you rolled those changes.

All in all very good job, it is very noticeable.

As a hardcore WvWer, I just am amazed by the lagless encounters after patch. You guys achieved an amazing success.

My feedback will be as following;

I and my friends have noticed more rubberbanding as before. Thus let me think you guys lowered the frequency on location repoting between server and client. This is something i couldnt not careless. Just leaving a little feedback here.

Really thanks to Dev Team. Now time to improve FPS during heavy fights

BillFreist.1527:

So during the November 12th release, we rolled out a couple of changes that significantly reduced skill lag. One mostly targetted PvP and WvW, whereas the other was more high creature concentrated PvE areas.

Curious to see if you, as players, have noticed the impact (or haven’t noticed it because the game feels as it should).

During some PvE meta events this weekend I saw the typical <10 FPS due to too many entities in the one area, but the skills were ok for the most part. No obvious skill activation lag from what I saw.

I’m not sure about the FPS though, my machine isn’t exactly old but it is very capable, but not Intel powered. I’m thinking that’s where the low FPS is coming from, too many entities (toons, skills being activates, projectiles, etc.) and not enough processing power to smooth it out.

For me it made a great difference. Even in EB i get nearly no lag now. So far looks pretty good for me.

Any news about client side optimizations?

I used to get skill lag in keep fights between 2 sides and I haven’t since the patch. Haven’t been in any 3 way SM fights so can’t comment on those.

I’m disinclined to believe the server side changes have much to do with the dramatic (3x-4x) increase to my frame rate seen 3 days ago though I could see it possibly playing a minor role. Has there been any undocumented under the hood changes to the client in the past few days that may explain this sudden change? I’m using a Phenom II that is commonly accepted as insufficient for playing this game on the forums and previously lived up to that reputation. I’m now miraculously experiencing a totally different GW2 that’s encouraged me to stay with the game I was very near leaving. What gives? I haven’t noticed any one else experience this and in fact am still hearing people with superior systems experiencing what I was previously. I’m excited as all hell but feel like its too good to be true and i’m going to wake up from a dream any second. I hope this doesn’t come off as a complaint i’m just curious as to what it could mean for others if its a sign of whats possible.

Really <b>Thanks</b> for you work, i havent already had any skill lags for 1 month, on reset aswell. You stacked all condition demage into 1 whole, I think it helped with lags and seems better xD

personally I haven’t had any skill lag in WvW since the patch, until last night. there was a 3 way battle in Hills on SoS borderland with about 40-50 each of SoS, BG, and JQ. for about 5 minutes we started getting 3-5 sec skill lag, but it seemed to recover quickly.

thanks for the improvements!

I must admit that skill lag for me has been little to non existent which is a bonus. I have seen a huge improvement from what it used to be. Only problem now is that when I die I can only blame my lack of skill and not the skill lag.

Thanks for the improvements as its made WvW alot more enjoyable.

BillFreist.1527:

Curious to see if you, as players, have noticed the impact

• Skill lag has completely disappeared.
• It seemed that last night there were scenarios of server overload on Gandara Borderlands (nothing happening, and then the fast-forward scenario while the server catches up with the load, all the skills that I spammed in panic would still pop).

From a playability standpoint I feel that server overload isn’t the worst thing in a game like GW2. As long as my skills go off I’m a happy camper (even if the blobbadons skills all go off under me at the same time and I am suddenly on top of 10287213 AOE rings).

Good work guys. Keep it up, I’m looking forward to seeing further improvements. The only criticism is that I find that battleping.com still improves the reliability of my in-game experience quite a bit (I think that might always be a problem as I have 300ms ping).

Still getting a lot of skill lagg tbh. Some days it’s barely playable and I just stop playing then.

I’m in the UK on Piken Square just doing low level PvE, and wow, lag is as bad as it ever was.
Started for me about SSCove content and has never gotten better.
Yes, i’ve done all the testing and moaning, coming and going between ANet, ISP and hardware, kept up to date with latest patches and forum chatter, but been lagged out of game for months now. So i went back to other games.
I return every few weeks to patch, repair and play. Within 2 mins i’m attacking fireflys or some other low level enemy and bam!

skill lag, skill fail, rubber-banding, fps drop <10-20, DC and eventually have to quit.
I don’t mind a bit of lag etc, i’m not daft, but i’ve seriously been unable to play guild wars 2 in 6 months.

I hope more fixes are in the works, as sadly the Nov update didn’t fix anything for this PvE thief. But i’m glad others are getting a boost.

Peace

I do hope this is addressed soon. The lag usually makes the game unplayable during the afternoon, no matter if I’m in WvW, LA, some abandoned map like Diessa or an instance/dungeon.