Guild Wars 2

Arenanet, I have had trouble most of the day with Guild Wars 2, and usually do not.

I used my router to tracert.

There is a failure between Level 3 and your servers.

The trace shows my connection failure starting at the Level 3 node – 4.68.71.217 this is a Level 3 node per Arin WHOIS. Level 3 seems to have a very high packet loss rate (over 25%) going to your data center. Note Level 3 is a company name of a large Internet backbone provider not a code word. Several Level 3 nodes seems to be failing in a cascade. 25% from one, 25% from the next. This eventually causes your servers to drop my connection.

Someone in your network department may want to contact Level 3 or the data center network department where your servers are located and inform them access to your servers in the U.S. is failing due to Level 3 nodes not working properly.

It does not appear to be an Arenanet problem, but instead a problem with Level 3 delivering packets to Arena Net.

The problem I’m reporting is as of 8/17/2015.

https://forum-en.gw2archive.eu/forum/support/support/Ping-Issues-merged/first

Gaile Gray.6029:

We are tracking these issues. We’d like to know a bit more:

• What are you experiencing?
• Are you getting any error codes?
• What is your ISP?
• What is your location?
• Could you run a traceroute and post it here? (Feel free to remove any personally identifiable information, or feel free to send the traceroute in a ticket to support.) It would really help!

Thank you so much for helping as we work to solve these issues.

(I may amend this list to ask more.)

I can tell you packets aren’t being delivered to Arenanet from Level 3. My ISP doesn’t matter, as the ISP traces cleanly to Level 3. At level 3, first node listed in my OP fails with 25% packet loss.

Location doesn’t matter, YOUR location isn’t getting the packets handed off by Level 3.

The error code of Guild Wars 2 doesn’t matter, as it’s a high network packet drop rate issue.

Someone should get on the phone to your data center network department, or if you control the network for your servers, someone should contact level 3.

The traced failure looks like this from my windows command line:

4 13 ms 22 ms 12 ms be-22-ar01.chartford.ct.hartford.comcast.net [68.87.182.197]
5 19 ms 19 ms 17 ms he-1-12-0-0-ar01.needham.ma.boston.comcast.net [162.151.53.145]
6 16 ms 15 ms 17 ms 4.68.71.217
7 * * * Request timed out.
8 * * * Request timed out.
9 67 ms 65 ms 65 ms 4.59.197.34
10 56 ms 55 ms 55 ms 64.25.32-9.ncsoft.com [64.25.32.9]
11 56 ms 54 ms 55 ms 64.25.32-26.ncsoft.com [64.25.32.26]
12 58 ms 56 ms 57 ms 64.25.32-82.ncsoft.com [64.25.32.82]
13 55 ms 55 ms 55 ms 64.25.33-80.ncsoft.com [64.25.33.80]

It’s 17 milliseconds to Boston from Hartford, after which Level 3 gets my packets. Level 3 fails to deliver them to your facility.

I assume your servers are in Texas. Level 3 eventually drops almost all packets and my client is disconnected. Ping gets progressively longer over a few seconds.

This isn’t an Arenanet problem, it’s a Level 3 to your Datacenter problem. There shouldn’t be timeouts in the hand off at the exchange in Boston. Level 3 isn’t working properly. Comcast is my provider, but they are delivering to Level 3 in about 17 milliseconds. It’s not a Comcast problem, a Arenanet problem, it is a Level 3 problem.

I still wonder if the problems of the last days were also related to Level 3, since
the german Telekom routed over Telia in Sweden to Frankfurt instead of Level 3
like normally, and now we go over Verizon.

Level 3 owns the class A address space that starts with IP address 4. If you see 4.xx.xx.xx it’s Level 3. They own all IP addresses that start with 4. If you run a trace route via your router or command line, see if traffic is moving through a node with a 4 at the start, if so, Level 3 is involved. You’ll need to determine if Level 3 is having issues by examining your trace route. My router tells me more, but doesn’t offer an easy way to cut and paste. It indicates high packet loss rates through the level 3 nodes.

Since you seem knowledgeable about this I have to point out that routers can be configured to not reply to a ping so you are guaranteed 100% packet loss between the hops from when the packets get on and off of Level 3’s backbone.

Target Name: 64.25.33-80.ncsoft.com
IP: 64.25.33.80
Date/Time: 8/17/2015 7:32:37 PM to 8/17/2015 7:36:52 PM

Hop Sent Err PL% Min Max Avg Host Name / [IP]

4 18 0 0.0 20 51 25 [32.223.104.185]
5 18 0 0.0 19 65 25 ae4—-0.car02.wlfr.ct.frontiernet.net [74.40.71.101]
6 18 0 0.0 30 187 44 ae11—-0.cor01.asbn.va.frontiernet.net [74.40.5.217]
7 18 0 0.0 30 157 42 ae0—-0.cbr01.asbn.va.frontiernet.net [74.40.2.174]
8 18 0 0.0 30 132 39 ae75.edge3.Washington4.Level3.net [4.53.116.45]
9 18 18 100.0 0 0 0 [-]
10 18 18 100.0 0 0 0 [-]
11 18 0 0.0 63 126 79 [4.59.197.34]
12 18 0 0.0 63 98 70 64.25.32-9.ncsoft.com [64.25.32.9]
13 18 0 0.0 63 165 77 64.25.32-26.ncsoft.com [64.25.32.26]
14 18 0 0.0 66 228 92 64.25.32-82.ncsoft.com [64.25.32.82]
15 18 0 0.0 63 204 73 64.25.33-80.ncsoft.com [64.25.33.80]

I don’t see an issue here.

I use and old version of visual route for that .. much better readable than tracerts
especially in forums.

And i know that Level 3 is NOT in the routes here, but i remember whenever i had
problems it was at Level 3 in Frankfurt. And now they are out suddenly and we had
lags like hell the last days when it went over Telia. Looks for me simply as if
Level 3 in Frankfurt was bombed down or whatever.

Yes, but my router can ping via tcp or udp and I can select a non-ping port. I was able to test connectivity with udp port 80 and get traffic results. If you are using Windows or Linux there are options to specify a different port and to use tcp or udp via the command line. I do not know the command or syntax for the Apple OS. However if you are having similar issues, then it may be worth noting you are also connecting to Level 3. They own all the address space that starts with 4 (they own what is called a class A address space).

If it starts with 4, it’s an IP address owned by Level 3.

My router doesn’t permit me to easily copy and paste results. However it provides a lot more information than the tracert command. The failing nodes are at Level 3, not at my ISP (Comcast) or Arenanet.

I don’t know if this helps, but here is my tracert against 64.25.33.54

17 ms 57 ms 16 ms 4.68.71.217
* * * Request timed out.
* * * Request timed out.
56 ms 57 ms 56 ms 4.59.197.34
* * * Request timed out.
58 ms 56 ms 57 ms 64.25.32-26.ncsoft.com [64.25.32.26]
58 ms 57 ms 60 ms 64.25.32-82.ncsoft.com [64.25.32.82]
57 ms 57 ms 57 ms 64.25.33-54.ncsoft.com [64.25.33.54]

Behellagh.1468:

Since you seem knowledgeable about this I have to point out that routers can be configured to not reply to a ping so you are guaranteed 100% packet loss between the hops from when the packets get on and off of Level 3’s backbone.

Target Name: 64.25.33-80.ncsoft.com
IP: 64.25.33.80
Date/Time: 8/17/2015 7:32:37 PM to 8/17/2015 7:36:52 PM

Hop Sent Err PL% Min Max Avg Host Name / [IP]

4 18 0 0.0 20 51 25 [32.223.104.185]
5 18 0 0.0 19 65 25 ae4—-0.car02.wlfr.ct.frontiernet.net [74.40.71.101]
6 18 0 0.0 30 187 44 ae11—-0.cor01.asbn.va.frontiernet.net [74.40.5.217]
7 18 0 0.0 30 157 42 ae0—-0.cbr01.asbn.va.frontiernet.net [74.40.2.174]
8 18 0 0.0 30 132 39 ae75.edge3.Washington4.Level3.net [4.53.116.45]
9 18 18 100.0 0 0 0 [-]
10 18 18 100.0 0 0 0 [-]
11 18 0 0.0 63 126 79 [4.59.197.34]
12 18 0 0.0 63 98 70 64.25.32-9.ncsoft.com [64.25.32.9]
13 18 0 0.0 63 165 77 64.25.32-26.ncsoft.com [64.25.32.26]
14 18 0 0.0 66 228 92 64.25.32-82.ncsoft.com [64.25.32.82]
15 18 0 0.0 63 204 73 64.25.33-80.ncsoft.com [64.25.33.80]

I don’t see an issue here.

If your connection isn’t dropping, that is good. From Boston, there is no working connection to Arenanet via Level 3. Comcast routes to Level 3 at Boston, once the interchange node accepts the traffic, it’s liable for delivery.

Level 3 isn’t delivering in my case, and it seems to be affecting many users. If it isn’t affecting you, that’s great. It may mean a problem with their Boston node, or with the fiber cables used to connect Boston to Arenanet.

It’s beyond the scope of Comcast or Arenanet to resolve, Level 3 is responsible for delivery of the traffic it accepts.

There is a downdetector site, for Level 3 the webpage is here – https://downdetector.com/status/level3 currently it reports problems in Connecticut and Texas. Assuming Arenanet is in Texas, it could be experiencing problems.

Arenanet are you in or near San Antonio? A second map is here – https://downdetector.com/status/level3/map it appears today (8/17/2015) Level 3 is having some network issues.

pandaexpress.1458:

There is a downdetector site, for Level 3 the webpage is here – https://downdetector.com/status/level3 currently it reports problems in Connecticut and Texas. Assuming Arenanet is in Texas, it could be experiencing problems.

Arenanet are you in or near San Antonio? A second map is here – https://downdetector.com/status/level3/map it appears today (8/17/2015) Level 3 is having some network issues.

So does Guild Wars 2.

https://downdetector.com/status/guild-wars-2

Just to chime in here, I am from CT and getting level 3 packet loss. Kinda glad that the ping issues I am experiencing isn’t from my end, kinda of a hassle to learn how to fix this stuff myself lol.

Ytlayol.5864:

Just to chime in here, I am from CT and getting level 3 packet loss. Kinda glad that the ping issues I am experiencing isn’t from my end, kinda of a hassle to learn how to fix this stuff myself lol.

yeah you’re not kidding.

Just to add to the conversation. I am also in CT and getting about a 50% packet loss when I ping Anet. I haven’t been able to play all day. Is there something we can do on our end to facilitate a fix?

Here’s my trace. It also looks like a Level(3) issue.

Tracing route to 64.25.38-247.ncsoft.com [64.25.38.247]
over a maximum of 30 hops:

4 12 ms 11 ms 11 ms be-23-sur01.chartford.ct.hartford.comcast.net [6
8.85.88.241]
5 12 ms 11 ms 11 ms be-82-ar01.chartford.ct.hartford.comcast.net [68
.85.106.197]
6 15 ms 27 ms 15 ms he-1-7-0-0-ar01.needham.ma.boston.comcast.net [1
62.151.53.225]
7 13 ms 21 ms 14 ms 4.68.71.217
8 * * * Request timed out.
9 * * * Request timed out.
10 53 ms 53 ms 53 ms 4.59.197.34
11 53 ms 53 ms 53 ms 64.25.32-9.ncsoft.com [64.25.32.9]
12 53 ms 51 ms 54 ms 64.25.32-26.ncsoft.com [64.25.32.26]
13 53 ms 53 ms 57 ms 64.25.32-82.ncsoft.com [64.25.32.82]
14 54 ms 53 ms 55 ms 64.25.38-247.ncsoft.com [64.25.38.247]

Guild Wars 2 has been literally unplayable for me for 7 hours due to this. Anyone have any idea how we can push to get it fixed? Can we bug Level 3 or something?

Sounds more like an issue with Comcast in CT or maybe the L3 gateway in Boston. I’m on DSL in CT with Frontier (don’t judge) and it seems fine. As my PingPlotter shows I’m hopping through Virginia, DC and then onto the L3 backbone.

Here I am hanging in DR.

Target Name: 64.25.38-247.ncsoft.com
IP: 64.25.38.247
Date/Time: 8/17/2015 10:45:39 PM to 8/17/2015 10:50:24 PM

Hop Sent Err PL% Min Max Avg Host Name / [IP]

4 20 0 0.0 20 55 24 [32.223.104.181]
5 20 0 0.0 19 76 25 ae5—-0.car02.wlfr.ct.frontiernet.net [74.40.71.105]
6 20 0 0.0 30 62 33 ae11—-0.cor01.asbn.va.frontiernet.net [74.40.5.217]
7 20 0 0.0 29 60 33 ae0—-0.cbr01.asbn.va.frontiernet.net [74.40.2.174]
8 20 0 0.0 30 88 36 ae75.edge3.Washington4.Level3.net [4.53.116.45]
9 20 20 100.0 0 0 0 [-]
10 20 20 100.0 0 0 0 [-]
11 20 0 0.0 76 91 82 [4.59.197.34]
12 20 0 0.0 76 146 85 64.25.32-9.ncsoft.com [64.25.32.9]
13 20 0 0.0 79 113 84 64.25.32-26.ncsoft.com [64.25.32.26]
14 20 0 0.0 78 120 86 64.25.32-82.ncsoft.com [64.25.32.82]
15 20 0 0.0 77 122 84 64.25.38-247.ncsoft.com [64.25.38.247]

Edit: Also check the Windows Resource Monitor if you want to check actual packet loss in GW2.

Behellagh.1468:

pandaexpress.1458:

There is a downdetector site, for Level 3 the webpage is here – https://downdetector.com/status/level3 currently it reports problems in Connecticut and Texas. Assuming Arenanet is in Texas, it could be experiencing problems.

Arenanet are you in or near San Antonio? A second map is here – https://downdetector.com/status/level3/map it appears today (8/17/2015) Level 3 is having some network issues.

So does Guild Wars 2.

https://downdetector.com/status/guild-wars-2

I’m uncertain why you are introducing this into the thread?

Our problem isn’t with Arenanet, but with Level 3.

The downdetector won’t be meaningful for Guildwars 2 servers, when Level 3 is the problem.

Ytlayol.5864:

Just to chime in here, I am from CT and getting level 3 packet loss. Kinda glad that the ping issues I am experiencing isn’t from my end, kinda of a hassle to learn how to fix this stuff myself lol.

If you can VPN to NYC, and bypass whatever mess Boston is going through, it is possible to access Arenanet.

I have done this. Forcing the route to NYC bypasses whatever is wrong.

Behellagh.1468:

Sounds more like an issue with Comcast in CT or maybe the L3 gateway in Boston. I’m on DSL in CT with Frontier (don’t judge) and it seems fine. As my PingPlotter shows I’m hopping through Virginia, DC and then onto the L3 backbone.

It is not Comcast. A Comcast VPN from Hartford is routed to NYC without going to Boston. Going from NYC to Arenanet is no problem using Comcast.

Bypassing Boston solves the problem, which is clearly with Level 3, and not Comcast.

If you need additional explanation about how this works, let me know.

To Arenanet, someone get to Level 3, and have them fix the issues which seem related to Boston.

pandaexpress.1458:

To Arenanet, someone get to Level 3, and have them fix the issues which seem related to Boston.

Please, Anet! Help us play again!

pandaexpress.1458:

Behellagh.1468:

Sounds more like an issue with Comcast in CT or maybe the L3 gateway in Boston. I’m on DSL in CT with Frontier (don’t judge) and it seems fine. As my PingPlotter shows I’m hopping through Virginia, DC and then onto the L3 backbone.

It is not Comcast. A Comcast VPN from Hartford is routed to NYC without going to Boston. Going from NYC to Arenanet is no problem using Comcast.

Bypassing Boston solves the problem, which is clearly with Level 3, and not Comcast.

If you need additional explanation about how this works, let me know.

To Arenanet, someone get to Level 3, and have them fix the issues which seem related to Boston.

Can you send me a basic guide how to do this? Thanks in advance

Same issue. I’ve traced the issue to Level3 (before coming here and seeing this topic).

Tracert is indicating packet loss at Level3 and pinging results in TTL expiration every few seconds (IP 4.59.197.34).

If it helps, I’m on Comcast in NY, mid Hudson Valley (the issue doesn’t seem to be Comcast, only connections through Level3)

ANet-anything you can do?
Edit: for what it’s worth, I’m very close to the CT border. Probably going through Boston too (I haven’t checked yet).

I live in CT. near the RI border and I’m having the exact same issue. Comcast as well.

Just a brief follow up. I guess it is a Comcast issue. They’re having issues in both CT and Boston.

https://downdetector.com/status/comcast-xfinity/map/

Interesting, last night results were slightly different no loss through comcast, just when reached level 3, this morning comcast very lossy, and level 3. I called level 3 and they said they don’t have any issues in ct at the moment and said to contact provider and let them know and if they have an issue they’ll contact level 3.

Bannaar.1506:

Interesting, last night results were slightly different no loss through comcast, just when reached level 3, this morning comcast very lossy, and level 3. I called level 3 and they said they don’t have any issues in ct at the moment and said to contact provider and let them know and if they have an issue they’ll contact level 3.

What number did you use to contact level 3? I would like to call them also, but could not find a number that seemed appropriate.

From what I’m gathering, the failing Level 3 node(s) aren’t necessarily in CT. Most of the trace routes here are going to Comcast in Boston and then being shipped off to
4.68.71.217 and then fails a few times before reaching 4.59.197.34. According to iptrace.in, these are in California and Texas, respectively.

Nilanil.4721:

Just a brief follow up. I guess it is a Comcast issue. They’re having issues in both CT and Boston.

https://downdetector.com/status/comcast-xfinity/map/

I’m not sure that Comcast also having issues proves this is a Comcast issue. The trace routes seem to indicate the Comcast points are being hit successfully and that the failures begin when reaching the Level 3 points. From what I gather the problem is that requests from Comcast users in the Boston and CT areas are put on a path to those points. You can see this in the traceroutes and pandaexpress who used a VPN to change his path to arenanet.

sanadin.8317:

Nilanil.4721:

Just a brief follow up. I guess it is a Comcast issue. They’re having issues in both CT and Boston.

https://downdetector.com/status/comcast-xfinity/map/

I’m not sure that Comcast also having issues proves this is a Comcast issue. The trace routes seem to indicate the Comcast points are being hit successfully and that the failures begin when reaching the Level 3 points. From what I gather the problem is that requests from Comcast users in the Boston and CT areas are put on a path to those points. You can see this in the traceroutes and pandaexpress who used a VPN to change his path to arenanet.

Sorry, I meant to say that it could still be a Comcast issue, not that it IS a Comcast issue. Either way, it seems to be resolved now (at least for me).

Interesting even though I don’t have a problem with GW2, different ISP in CT, the TuneIn stream from the local radio station on I listen to wasn’t terribly stable yesterday. They might use Comcast or also hopped to Boston’s L3 node. Still had problems this morning but all is fine now.

Are CT Comcast people still having issues?

I still cant connect

finally can connect

I have the exact same problems, pandaexpress.

I’ve contacted my ISP about it many times to see if there’s anything that can be done from their end and they’ve told me no every single time.

:(

pandaexpress.1458:

I can tell you packets aren’t being delivered to Arenanet from Level 3. My ISP doesn’t matter, as the ISP traces cleanly to Level 3. At level 3, first node listed in my OP fails with 25% packet loss.

Location doesn’t matter, YOUR location isn’t getting the packets handed off by Level 3.

The error code of Guild Wars 2 doesn’t matter, as it’s a high network packet drop rate issue.

Someone should get on the phone to your data center network department, or if you control the network for your servers, someone should contact level 3.

The traced failure looks like this from my windows command line:

4 13 ms 22 ms 12 ms be-22-ar01.chartford.ct.hartford.comcast.net [68.87.182.197]
5 19 ms 19 ms 17 ms he-1-12-0-0-ar01.needham.ma.boston.comcast.net [162.151.53.145]
6 16 ms 15 ms 17 ms 4.68.71.217
7 * * * Request timed out.
8 * * * Request timed out.
9 67 ms 65 ms 65 ms 4.59.197.34
10 56 ms 55 ms 55 ms 64.25.32-9.ncsoft.com [64.25.32.9]
11 56 ms 54 ms 55 ms 64.25.32-26.ncsoft.com [64.25.32.26]
12 58 ms 56 ms 57 ms 64.25.32-82.ncsoft.com [64.25.32.82]
13 55 ms 55 ms 55 ms 64.25.33-80.ncsoft.com [64.25.33.80]

It’s 17 milliseconds to Boston from Hartford, after which Level 3 gets my packets. Level 3 fails to deliver them to your facility.

I assume your servers are in Texas. Level 3 eventually drops almost all packets and my client is disconnected. Ping gets progressively longer over a few seconds.

This isn’t an Arenanet problem, it’s a Level 3 to your Datacenter problem. There shouldn’t be timeouts in the hand off at the exchange in Boston. Level 3 isn’t working properly. Comcast is my provider, but they are delivering to Level 3 in about 17 milliseconds. It’s not a Comcast problem, a Arenanet problem, it is a Level 3 problem.

All this trace shows is that you are getting a latency spike from Hop 7 on to your destination. You are not getting any Timeouts, its simply that Level3 disabled ICMP replies from Hops 7 and 8. You can change port in your router all you like but Ping is always ICMP (protocol not port type) and if a router does not respond to ICMP you cannot get a respond from it.

Either Anet or your ISP can open a head to head ticket with level3 to End to end test this issue and get it fixed since BOTH of those ISPs are transporting through Level3.