Guild Wars 2

Stephen, before anything, I’d just like to say that I greatly appreciate what you’re currently doing. You’re communicating with us, on a Saturday, about something that has largely been ignored previously. Great job!

Stephen ClarkeWillson.1802:

The silver lining as they say is that the perf counters are more accurate than they were and I think I see the problem now; it’s going take some engineering to fix properly but I have a band-aid in place now that reduces the occurrences (at least as I perceive them via my graphs).

What do you mean with “some” engineering? Is this something you believe can be fixed in a short time frame, or is it something that will likely take months?

Magnosian.8750:

Stephen, before anything, I’d just like to say that I greatly appreciate what you’re currently doing. You’re communicating with us, on a Saturday, about something that has largely been ignored previously. Great job!

Stephen ClarkeWillson.1802:

The silver lining as they say is that the perf counters are more accurate than they were and I think I see the problem now; it’s going take some engineering to fix properly but I have a band-aid in place now that reduces the occurrences (at least as I perceive them via my graphs).

What do you mean with “some” engineering? Is this something you believe can be fixed in a short time frame, or is it something that will likely take months?

One of the first thing software developers learn…is never put a timeframe on ANYTHING.

Hey there, thought I’d just share my experiences with the lag so far. I’m playing on Piken Square (EU) at the moment, and for the past 7 or so months I’ve gotten increasingly involved in WvW. Ever since the stability patch, I’ve noticed, as well as almost the entire server, horrible lag spikes throughout the game. While its certainly the worst while in WvW, I have felt lag while running around in regular maps, while doing dungeons, while in the PvP lobby, and while sitting in LA trying to use the trading post.

I first noticed this lag the Saturday first following the stability patch. I commanded our Friday reset for a few hours, and actually DID NOT notice bad lag throughout the 3 or 4 hours I was tagged up. However, the next day, Saturday, I was tagged up again for around 3 hours, and it was the worst lag I have ever experienced in any sort of video game I have ever played. Even when we weren’t fighting, our skills were taking a few seconds to go off, switching weapons took a few seconds, and going through portals took a few seconds. However, during the times we did fight it was impossible to play. I literally ran through the enemy group of at least 60 people, taking almost no damage because, just like myself and the 50 people following me, no one could use any skills.

The weeks following I’ve experienced bad lag spikes, none as bad as that first time, while following a commanded, leading open/guild raids, or just running around in small groups or alone. Recently, I’ve noticed less lag, whether that was because of something you guys did, or just coincidence I don’t know. Yesterday, I was tagged up for about 4 hours for our reset, and had around 60 people on me. There was slight lag running around the map, and when we fought an enemy group there was more lag, however it was playable. But when we would encounter both of the enemy servers at once, the game was again unplayable. All of us would rubber band, warping around, skills were not usable, we couldn’t switch weapons, and it got to the point where I actually just brought all of our players to the other side of the map avoiding the other two groups, and the fights our server loves :P, to be able to play the game.

Obviously I am like the many other people in this thread, glad and appreciative to know we have someone from arena net seeing the problem, and many thanks to all the work and updates you’re giving us, especially on a Saturday. Good luck fixing this, I would love to be able to play WvW without problems again, as it is my favorite part of the game at this point. Cheers

Noobix.3958:

Magnosian.8750:

Stephen, before anything, I’d just like to say that I greatly appreciate what you’re currently doing. You’re communicating with us, on a Saturday, about something that has largely been ignored previously. Great job!

Stephen ClarkeWillson.1802:

The silver lining as they say is that the perf counters are more accurate than they were and I think I see the problem now; it’s going take some engineering to fix properly but I have a band-aid in place now that reduces the occurrences (at least as I perceive them via my graphs).

What do you mean with “some” engineering? Is this something you believe can be fixed in a short time frame, or is it something that will likely take months?

One of the first thing software developers learn…is never put a timeframe on ANYTHING.

“For an initial, high-level estimate for a business case then the key things are:

Speed. Whatever method you use it needs to be quick. The whole point is the stakeholders aren’t sure if it is even worth doing the project – which is why they need the numbers for the business case. If the business case was solid they wouldn’t need your estimates. The bulk of these projects won’t go ahead so it is important that too much effort isn’t expended providing the estimate.
Give a range. Make it broad. You have had no time to analyse requirements, workshop with stakeholders, validate assumptions. A wide range tells the recipient of the estimate “Software projects are naturally complex and risky – if you want a proper estimate you need to give me more details and more time”. The problem with giving a single number or a narrow range is that it paints you into a corner by setting expectations before any real analysis is done.

I find the best technique to pick a comparable project that “feels” the same. “Feel” is completely subjective – but with this kind of estimate my experience tells me you won’t find objective measurements. Then provide a wide range. I’ve read some books that say a range of -50% to +100% is good but it depends on many factors.

For a detailed, low-level estimate:

You need a baseline. If the baseline isn’t stable the estimate is meaningless.
Bottom up is best. Get a detailed work breakdown, estimate each component then roll it up into a larger number. I find planning poker to be a great technique here.
Document contingency. Make it clear where any contingency (if any) is added. Is it added to each line item? Or to the whole estimate? Or to specific risks? Or is there none?
State your assumptions. Validate as many as possible given the time frame.
State explicitly what is included and excluded in the estimate. For example, is review included? Are technical delays included?"

DarkSyze.8627:

copypasta

I see DarkSyze is harbouring dreams of becoming yet another Dilbert-esque middle manager

Stephen ClarkeWillson.1802:

Yo – turns out I can temp move myself into Blackgate and experience the lag level for myself! Don’t worry – I won’t actually play for points. I’m “Asuradly So”, a little Asura engi in-game hanging at Blackgate Hill, if you want to say “hi”.

Of course, now I have forums, graphs, and actual game to watch at the same time. Should be interesting! I think I need more monitors.

Sweet. Everybody, time to all crash BGBL. Let’s give this cool tech guy a heck of a lag experience.

By the way, thanks for spending the time to dig into this problem. I’m glad for one, we no longer have to argue about whether there is skill lag issues or not, the tech guy admit to it. Forum posters sure wasted a lot of time arguing about it. Secondly, finally someone explaining what happened, admit to a patch that fixed something but broke it for WvW.

Acknowledgement and communication is the first step to getting something resolved. Now that Anet admit to it, explained a bit about what happened, I think we can all breath better and let them work on it. Hopefully you can find out the root of problems soon! Thanks again for spending the time on the weekend to check this out.

bigblah.3285:

DarkSyze.8627:

copypasta

I see DarkSyze is harbouring dreams of becoming yet another Dilbert-esque middle manager

Either I am having a heart attack, or the WvW forums just got 18 red posts in 6 hours! O.o

K read through all the new posts, and it sounds good, Stephen thinks he’s found the problem, and with NA primetime coming in, they can collect more data to make sure. Let’s hope the fix is around the corner!

Thanks again Stephen for looking into this for us!

Keep up your good work!

Its nice to ‘see’ a dev working on the actual wvw issues!

So excited to see a mod again!

I am on BP the 24th ranked server in NA and we are having severe lag with every fight. Our massive BL rolling zerg is 15 strong. Look out T1. The Lag is with us.

Thank you for taking the time to work on this issue on a Saturday before a holiday! It’s appreciated that you are taking the time to engage with us and look at our feedback and give us information on what you’ve found so far. You’re welcome to follow my group in EB on the Jade Quarry server!

Stephen ClarkeWillson.1802:

I don’t post a lot but this seemed like an opportunity in real-time to correlate the player experience with a big pile o’ graphs.

But if this leads to a fix then it’s be best red posts WvW has had since launch most likely!

This thread is amazing, an Anet dev actually playing/testing/fixing WvW.

Was on Mag BL around 30 minutes ago. BL was quiet except for some 6-8 man skirmishes and I was getting lag. 3 or 4 bouts where the world stopped and a few seconds later, catch-up time.
Not sure what that tells you, hopefully something.

Stephen ClarkeWillson.1802:

Yo – turns out I can temp move myself into Blackgate and experience the lag level for myself! Don’t worry – I won’t actually play for points. I’m “Asuradly So”, a little Asura engi in-game hanging at Blackgate Hill, if you want to say “hi”.

Of course, now I have forums, graphs, and actual game to watch at the same time. Should be interesting! I think I need more monitors.

Best dev 2015 for use of puns.

I’m in T4 NA and my group of 7 is periodically lagging out to disconnect every 5 minutes or so. We are all feeling the lag but only 1 or 2 will dc fully each time.

Stephen ClarkeWillson.1802:

This is great information. The DC’s are a different problem (still important of course).

Thanks for staying on top of this.

FYI, in Tier 1 BGBL, entire map is lagging/teleporting. Guildies reporting a few dc’ed. Map is queued.

Come to BG BL right now. It’s bad n_n Lagging out/teleporting and a few people just poofed totally. We don’t have a queue of people here, but I’d bet TC and BG do.

Stephen ClarkeWillson.1802:

Yo – turns out I can temp move myself into Blackgate and experience the lag level for myself! Don’t worry – I won’t actually play for points. I’m “Asuradly So”, a little Asura engi in-game hanging at Blackgate Hill, if you want to say “hi”.

Of course, now I have forums, graphs, and actual game to watch at the same time. Should be interesting! I think I need more monitors.

And, I just read this now and missed the fun of hunting an ATag??? If you ever want to test lag, just ask anyone on JQ, BG, or TC. We, as a Tier, would gladly run some 3 way Omni blob fights and push your server’s to the limits. You don’t have to wait for it to happen, just ask.

Eu player, On my NA Blackgate account also getting lag, ping is 163, and otherplayers are rubber banding a bit

Update, just got onto EB, lag is worse there,includes skill lag now as well

Bad lag in the SM fight just now. :\

Stephen ClarkeWillson.1802:

Yup, saw that one! I have to go now. Feel free to keep posting your experiences and I’ll catch up on them later. G’night!

Goodnight, thanks for looking at the graphs for hours! I’d get crazy after 15min. Hopefully it’ll all get sorted soon.

Yo guys, I gave him as much info and showed him as much as I possibly could in game. I Dced a few times, idk how many other people DC during these situations, but if you have and you dont think it was on your end post here. I saw our commander dc also so I dont think it was just me. A thanks goes out to JQ for making it lagtastic for him to see and measure it on his graphs.

I think its the cpu doing all that extra work trying to figure out how many stacks of stability players have left.

slamfunction.7462:

Stephen as a result of the server issues pertaining to WvW this evening and in completing a WvW Report for my server on our private forums, we found that the queuing system for the T1 servers are problematic.

More specifically, Jade Quarry appeared to be able to have more players on a map than either Tarnished Coast or Blackgate.

Typically, Tarnished Coast only experiences 1-2 maps being queued on any given Saturday night at 9PM EST. This evening, we experienced ALL maps being queued with out the on-map numbers to support why we were queued, when JQ had twice as many numbers.

I am not going to argue the point that this proliferated an unbalanced competition, but please provide information to discredit what my server and I were experiencing. We do use superior map coordination and do member/head counts to confirm these numbers daily and nightly.

Hi Dev,

Would you please look into green BLs across every match up? That particular BL has been lagging for the past 2 weeks and it’s close to unplayable due to the lag.

There are different kind of lags and they need to be treated differently:

1. Server side lag, that shows up as – skill lag – for everyone on the map, at the same time, is the main problem, that started since the stability/camera change patch

2. temporary high ping, delays, rubber-banding and other anomalies that are not happening to everyone at once, or affects only a few – these likely are related to local ISP & traffic congestion

3. FPS drop – these can be bugs in engine, and most likely will affect those with low end hardware or specific hardware configurations

While all these are problems, they are different problems and require different solutions.

slamfunction.7462:

slamfunction.7462:

Stephen as a result of the server issues pertaining to WvW this evening and in completing a WvW Report for my server on our private forums, we found that the queuing system for the T1 servers are problematic.

More specifically, Jade Quarry appeared to be able to have more players on a map than either Tarnished Coast or Blackgate.

Typically, Tarnished Coast only experiences 1-2 maps being queued on any given Saturday night at 9PM EST. This evening, we experienced ALL maps being queued with out the on-map numbers to support why we were queued, when JQ had twice as many numbers.

I am not going to argue the point that this proliferated an unbalanced competition, but please provide information to discredit what my server and I were experiencing. We do use superior map coordination and do member/head counts to confirm these numbers daily and nightly.

We only had 2 map queues tonight as well. Those were EB and JQBL. Besides, stop talking matchup craziness when this thread is about lag.

Terrible lag all night (4/4) on HoD BL and in EB. No queue on either map while I was there though EB was queued before we switched over there.

Last night (4/3) had lag all night on NSP BL. Everyone was complaining about it in TS so not an individual issue.

I don’t know about for anyone else but the lag started getting really bad after the latest patch was released and seemed to get worse as the hotfixes were released on

Can’t say I know much about game server programming but cpu load may not tell all since threads/processes could be locked up waiting for io or just mutex locks. like a a message queue that gets locked up, so maybe try to write some strafe scripts or similar would be better?

Holy Stephen xD, you have been busy till around 4 o clock i see. Take a good nights rest and happy easter to everyone.

And about this, guys see it like this. Its like an sickness it always have a root, even if you have dozens of bacteria flying around, it always had a core problem(like how come there is no more immunity for those bacteria?). And that is the thing Anet is searching for at the moment.

The main root of the problem hmmm, maybe processing the data is the place where its going wrong. I mean that it just becomes to much at the same time. Everyone spamming their skills, the amount of people loaded, even if your mind is big, you can only take 4 things at the same time in your short term memory.
Maybe its the same with this.

I dont know what to do though, i aint a guy that has been in to much dept about it. But maybe this can help a bit.

Great to see someone actually looking at the problem and confirming it. On a weekend!

Thank you.

Thanks for working on it. Good to hear that map population reduction is a temporary fix. Queues exceed 100 on EBG for NA prime on reset already. We want to keep the game “massively multiplayer”.

Most players noticed a correlation with all the new calculations being done as a result of condition and stability changes with the latest wave of server lag. Has that been looked into?

Even though it’s awesome you guys in anet are working to fix the lag, but I wish the lower population cap doesn’t last for long. I bet most of the t1 – t2 servers in NA have big queues during their primetime, and so do we in EU t1 – t2 (atleast in FSP) since we could already queue up every border with just guild raids. That’s about 3 guilds/border (maybe fourth if it’s a small guild/or if they want to roll with smaller numbers and queue for half the evening) and with the new cap you could barely squeeze three guilds in the border.

So i had a short chat with you yesterday when playing on my us-account about lag verses US/EU. I play from Sweden and my ping are roughly around 190-220 but still playable, it is actually the way EU was before we got this horrible lags that we have now, so i am used to play in this sort of environment, my skills are working, just feeling a tad of lag. On EU-server my ping mostly are around 60-90 in wvw, but the lag problems are far worse.

A few things i want to add. I understand what you try to say, that your server is not the problem, but i would say the traffic coming in to it is. Of all the tracerts i done my self and about hundreds i seen here and from other sources (forums, friends and the like) on the EU-server there is always the last link, the jump over to NCSoft that seem to be a big problem. Package loss is huge there and keep on going inside of NCSoft.
I do also believe that NCSoft have the biggest possibility to do something about this, since it is the last jump to them that causing the issues.

I try to understand the problem and not going mad over it, but here is my thought. You are right about that it is not your players problem, our connection in most cases are working as intended, just as your server most probably work as intended, and the problem is some where inbetween. BUT if nearly all your players in EU/US experience lags in some way, then it really must be you guys that fix it and adjust to it, because your game is not working for the players.
I realize the reasons and i understand the problem and how hard it can be to fix it, but not fix it is not a option, because we as players can’t at all, so it really is up to you.
As it is now people are quitting over it, guilds are breaking you are leaking players and it is so close to expansion and if the expansion meet this kind of lag, the new players we could get in to wvw will not stay for long.

Ok as for DC problems. I have two of them. One is a bug, were the game simply freezes and i get a message saying that a very serious problem happen and game is shutting down please send report to Arena. What that problem is i have not clue because it does not say, and i have sent the report many times now. I notice this happens when i make a fast turn with the camera. Might be a coincidence but that is when it have mostly happen.

The other DC problem seem to be due to lag. It starts with skillag and this feeling that you are walking in glue, like there is a 50 man blob around the corner. Then people start to rubberband around me and a few min later they run in to walls all whilst i walk around in my own world not being able to use anything just jump around and move over the map. Sometimes i die when this happen, but i don’t see that, i just move around until BAM DC and 50 man queue to wait through. The other day a commander told us to get our butts down to the bridge at Hills and stack on him. Problem was that we all were stacking at him at the gate of Hills.
This is pretty much what all people say when they dc, and pretty much all experience the same thing. Sometimes whole groups dc or enemie groups vanish in front of us, because they dc.

Nieny.9418:

Ok i have only one more question , why only GW2 have problems like this, any ESO or WoW or other mmo… only we have skill lags, lags and others things..

wut? What are you talking dude? ESO is the worst lag fest in all of MMO history. Have you ever been to their type of WvW? It’s unplayable almost…since release btw. Just check their forum before you post anything lol

I only had very small lag spikes in big battles whereas some of my guild had a total d/c…still supporting the ISP issue hypothesis.

https://youtu.be/8ZCysBT5Kec
(^_^)

Stephen,

Thank you very much for talking to us about this issue and keeping us up to date. It is honestly a breath of fresh air and I can’t tell you how much we appreciate it.

With that said, if there is any kind of coordinated testing that you want to perform during NA prime time, the player community in T1 has the relationships across server and communicates enough with each other to be able to coordinate anything you may possibly need. Believe me when I say that we will be more than willing to assist you in anything and everything you may need to get this issue resolved. Our goal is one and the same, a fun and playable WvW experience.

Thanks

Something I’ve noticed: Even with small 10v10 fights in a keep, the lag occurs.

It doesn’t even require a full map. We’re getting massive spikes and “discos” while trying to hit EB red keep this morning.

Stephen ClarkeWillson.1802:

Thanks. I’ve been whispering with BATMAN in-game to compare his experience to the various graphs I can see. So far this is going way better than EU peak time, but he says there are a lot of disconnects. That would be a different problem – is anyone else seeing a lot of that in T1 NA?

When the FPS lag starts up and start rubber banding a bit is when I get DCs during NA play. Typically I will see the commander start to rubber band and than he will band into a wall and stay there. About 4-5 seconds later I get disconnected. I don’t think the issue is ISP related as TS3 and my video streaming neither lagged nor hiccuped only GW2.

Nieny.9418:

Ok i have only one more question , why only GW2 have problems like this, any ESO or WoW or other mmo… only we have skill lags, lags and others things..

I have been playing ESO since launch. PVP lag has been a serious issue since update 6 in particular. I love ESO but severe server lag is something they are still dealing with in PVP, and the pop cap in PVP has been reduced at least once since launch.

I played a full match in eotm last night on DB. I really didn’t notice much lag like what people have been describing. It was pretty late at night but we did have three big zergs for a long time from each side and a few large battles.

But then this morning there was a new round with a probably full map and there were a few times where things seemed to “catch up” much moreso than the earlier match.

Never quite 10+ second skill lag but maybe 3-4 seconds at some times. I did still see some skill lag in world bosses that was worse than eotm.

Inferno.2315:

I played a full match in eotm last night on DB. I really didn’t notice much lag like what people have been describing. It was pretty late at night but we did have three big zergs for a long time from each side and a few large battles.

But then this morning there was a new round with a probably full map and there were a few times where things seemed to “catch up” much moreso than the earlier match.

Never quite 10+ second skill lag but maybe 3-4 seconds at some times. I did still see some skill lag in world bosses that was worse than eotm.

Normally I would say “then why post here if you dont have the problem” but this could be useful infomation if you really wasn’t seeing any problems.

Maybe you could post some info about your setup that could relate to the problem. Maybe its hardware related.

Nieny.9418:

Ok i have only one more question , why only GW2 have problems like this, any ESO or WoW or other mmo… only we have skill lags, lags and others things..

LOL what? WoW had the worst lag ever..

Looky here https://youtu.be/JPdpMUoxxEQ

Hylgeriak.8250:

Nieny.9418:

Ok i have only one more question , why only GW2 have problems like this, any ESO or WoW or other mmo… only we have skill lags, lags and others things..

wut? What are you talking dude? ESO is the worst lag fest in all of MMO history. Have you ever been to their type of WvW? It’s unplayable almost…since release btw. Just check their forum before you post anything lol

I only had very small lag spikes in big battles whereas some of my guild had a total d/c…still supporting the ISP issue hypothesis.

That my be the case for some people but not for all of us.

I have uploaded an screenshot while I was having the massive skill lockup and as you can clearly see the server return time was no more than 30ms. Yes one or two hops along the route was pinging out high but thats because the router are under load and give lower priority to a ping packet but as you can clearly see it doesn’t affect the overall rout time and thats what your looking for in a trace router test.

<a href=“http://imgur.com/EHO0K80”><img src=“http://i.imgur.com/EHO0K80l.jpg” title=“source: imgur.com” /></a>

if that still isn’t working try this link

http://imgur.com/EHO0K80