Guild Wars 2

I don’t post a lot but this seemed like an opportunity in real-time to correlate the player experience with a big pile o’ graphs.

Also … it is sometimes an ISP. Things are rough at peak time in some parts of Europe right now. Amazon and Netflix have expanded their streaming offerings and the intertubes in EU are sometimes a bit clogged up at peak time. This is gross generalization because the EU is a big place and YMMV. The thing with connecting to our game is that there is your ISP, one or more other ISPs in between, and then one of our ISPs (we have three I think). Usually the trouble is with one of the ISPs in the middle. That’s why it is hard to fix; you call your ISP, we call our ISP, but it’s hard to get someone to call the ISP in the middle and then get the ISP in the middle to do something about it when they are just trying to make it through a night of lots of video streaming.

I don’t think it is the skill change. I do think the thing we did to reduce PvP (and also some PvE) lag made WvW lag worse; or, put the opposite way, WvW lag was bleeding over into the other map types, and we fixed that. The silver lining as they say is that the perf counters are more accurate than they were and I think I see the problem now; it’s going take some engineering to fix properly but I have a band-aid in place now that reduces the occurrences (at least as I perceive them via my graphs).

T1 EU EB must have just had an epic half hour fight. Holy moly. Seems to be recovering now.

I’m not forgetting about NA. It’s just less busy at this time of day. I can see a big spike 17 hours ago in NA which would be … kitten … 7:00 p.m. Pacific time?

Whatever I do to help EU automatically helps NA. T1 definitely lags more often than other tiers – at least on my graphs. But your experience may not match my graphs, which is the tricky problem I face – which graph is the most informative and will help find the problem? We collect hundreds of thousands (maybe a million?) perf samples every minute so figuring out the ones that matter is key. And we, as a team at ArenaNet, have already looked at the obvious ones, and it’s not correlating. One thing with the recent server fix is that some perfs are more accurate now, which is a huge help, as you can imagine.

It’s correct that the server lag (skill lag) will be map-wide. You don’t have to be in the big battle but the big battle is the source of the lag if it’s in the same map as you.

Yup, it’s bad for EU tier 1 Eternal Battlegrounds right now. The cool thing I just discovered is another perf counter that correlates better with the lag than total CPU. Y’all in a big fight right now? I guess if you are you’re not posting to the forums.

The best thing you can do to help is keep playing while I try to correlate lag with the root problem. I’m now watching the t1 EB CPU specifically and while I don’t see 100% I see CPU in excess of 90% still and that seems to correlate with lag spikes. Hmm.

What changed with this patch, and why I’m watching on a Saturday, is that we reduced PvP lag (a lot) with a scheduler change. In testing we saw that WvW got worse, so we reserved extra server capacity for WvW – we dedicated hardware to WvW battles that isn’t shared with the rest of the game. But that seems not to have fixed it for really big battles. I will say I think the lag spikes resolve themselves faster than they used to, but they might also spike up faster, which is when they are most noticeable to a player.

Things are stable again. My working hypothesis is that there is a bug in the “catch up” code. It’s not really necessarily bad for a map to exceed 100% CPU for a short period of time – the game is very tolerant of small lag even in WvW battles. But I’m wondering if sometimes a map starts working hard to fix a tiny blip which actually makes things worse for several minutes. If that’s the case it would certainly show up in big fights. I’ve lowered the map cap to keep CPU below 85%-ish to see if there are fewer lag spikes.

… and it’s resolving itself. That one lasted six minutes.

Oops. There goes another one in Eternal Battlegrounds. Dang.

At this exact moment all looks good. For me, it’s a bit like looking at all of the green characters dropping down the screen in The Matrix movies and then inferring your experience. Let me know on this thread how it goes the rest of the night (Europe time). I’ll keep watching graphs.

I see another spike on a borderlands map now. Hmm.

In the past CPU usage would cause server lag – 100% CPU usage is not good. However we’re not seeing that now so I’m not sure lowering the map capacity will fix it (but I might try it). Instead now we’re seeing “lag curves” (server side) that grow over about a five to ten minute period and then resolve themselves. For instance I see an Eternal Battlegrounds map right now in Europe just had one of these spikes. Over a two minute period the server fell behind by 40 seconds (yikes!) and then caught up in the next 2 minutes; the server CPU did not exceed 80%.

Hi all – we’re aware of the problem and working on a solution.

I verified I could not zone into DR in the EU. I modified the server routing to avoid this hardware server and now I can zone in.