Increased network latency to Europe servers — part 2

The problem I spoke about in my previous post is still occurring. Blizzard’s official thread for the problem is:

This morning I woke up to read a thread on the forums which indicated most Belgian players can’t connect at all. Blizzard, in their infinite wisdom, recommended the users use IPCONFIG /FLUSHDNS and reset their DSL/cable modems. A few of the users pointed out the same thing I have: the problem is with Telia, particularly their POP in Paris.

While I was asleep, apparently some members in our guild who live in Sweden were also seeing crazy latency and bizarre “bursty” lag.

Here are some active forum threads indicating the problem is getting worse:

I tried to log on, but got stuck at the “Success” screen. I cancelled, retried, and logged in moments later — only to find my latency was between 4900ms-7040ms, consistently.

The mtr I had running continues to show the problem is with Telia’s Paris POP. Again, this is NOT ICMP de-prioritisation which is being seen. My guess at this point is that the Paris routers are being overloaded, either from a DoS or simply too much traffic in general.

                                        Packets               Pings
 Host                                 Loss%   Snt   Rcv  Last   Avg  Best  Wrst
 1. gw.home.lan                        0.0%   486   486   0.4   0.3   0.3   0.6
 2. ???
 3. ge-2-19-ur01.santaclara.ca.sfba.c  0.0%   486   486   8.5  10.6   7.8  68.0
 4. te-0-3-0-0-ar01.sfsutro.ca.sfba.c  0.0%   486   486  10.8  12.1   9.0  39.3
 5. pos-0-5-0-0-cr01.sanjose.ca.ibone  0.0%   486   486  19.1  13.4  10.5  37.4
 6. xe-10-0-0.edge1.SanJose1.Level3.n  0.0%   486   486  43.0  17.5  10.3 236.4
 7. vlan89.csw3.SanJose1.Level3.net    0.0%   486   486  27.2  17.9  10.5  38.5
 8. ae-83-83.ebr3.SanJose1.Level3.net  0.4%   485   483  19.8  18.3  10.5  37.1
 9. ae-2.ebr3.LosAngeles1.Level3.net   0.6%   485   482  24.4  27.7  19.9  49.4
10. ae-73-73.csw2.LosAngeles1.Level3.  0.0%   485   485  22.5  27.3  20.0  49.6
11. ae-2-79.edge1.LosAngeles9.Level3.  0.0%   485   485  47.5  28.5  20.2 172.2
12. telia-level3-ge.LosAngeles9.level  0.0%   485   485  22.3  25.8  20.0 147.0
13. ash-bb1-link.telia.net             0.0%   485   485  87.3  87.5  83.4 128.9
14. prs-bb1-link.telia.net            10.3%   485   435 185.5 187.0 182.4 213.4
    prs-bb1-link.telia.net
15. ffm-bb1-link.telia.net            11.1%   485   431 194.7 200.5 191.2 359.5
    ffm-bb1-pos6-0-0.telia.net
16. ffm-b4-link.telia.net              8.5%   485   444 195.2 204.5 192.9 391.2
17. ???

And a couple hours later, things are worse:

                                        Packets               Pings
 Host                                 Loss%   Snt   Rcv  Last   Avg  Best  Wrst
 1. gw.home.lan                        0.0%  1288  1288   0.4   0.3   0.3   1.3
 2. ???
 3. ge-2-19-ur01.santaclara.ca.sfba.c  0.1%  1288  1287  10.1  10.7   7.7 204.9
 4. te-0-3-0-0-ar01.sfsutro.ca.sfba.c  0.1%  1288  1287  11.4  11.9   9.1  32.8
 5. pos-0-4-0-0-cr01.sanjose.ca.ibone  0.0%  1287  1287  11.7  13.3  10.6  37.9
 6. xe-10-0-0.edge1.SanJose1.Level3.n  0.0%  1287  1287  12.3  17.7  10.3 198.7
 7. vlan99.csw4.SanJose1.Level3.net    0.0%  1287  1287  11.1  17.9  10.3  45.2
 8. ae-93-93.ebr3.SanJose1.Level3.net  0.0%  1287  1287  17.0  18.3  10.9  56.4
 9. ae-2.ebr3.LosAngeles1.Level3.net   0.1%  1287  1286  26.6  27.2  20.0  61.8
10. ae-73-73.csw2.LosAngeles1.Level3.  0.1%  1287  1286  30.5  27.8  20.1  53.6
11. ae-2-79.edge1.LosAngeles9.Level3.  0.2%  1287  1285  24.0  28.2  20.1 187.1
12. telia-level3-ge.LosAngeles9.level  0.0%  1287  1287  23.6  25.8  20.0 170.9
13. ash-bb1-link.telia.net             0.2%  1287  1284  94.6  87.2  83.5 149.3
14. prs-bb2-link.telia.net            12.5%  1287  1126 200.5 191.7 184.2 237.5
    prs-bb2-link.telia.net
15. ffm-bb2-link.telia.net            16.7%  1287  1072 216.8 204.5 193.6 360.3
    ffm-bb2-link.telia.net
16. ffm-b5-link.telia.net             13.8%  1287  1109 215.5 202.9 194.1 401.3
17. ???

People are also having problems with instances not working, but that’s a completely unrelated issue (one that Blizzard should be able to fix).

Anyway, players need to understand two things:

  1. It doesn’t matter what Realm you play on, the problem is affecting most people, on all realms. A single traceroute won’t help; people need to be using tools like mtr, WinMTR, or PingPlotter to show the problem. The issue is really affecting all people because the EU realms (servers) are in France — do the math
  2. The problem is not with Blizzard, their hardware, or their servers. The problem is with Telia (a.k.a. TeliaSonera, a transit provider for Internet traffic (think Internet backbone); they’re Swedish, but their network is not limited to just Sweden) — specifically their routers at their Paris POP. This isn’t Blizzard’s fault… however, it’s obvious Blizzard has some relationship with Telia (they probably peer with them; I haven’t looked at AS paths), so Blizzard should be able to put pressure on them (I do this at my job quite often, specifically with AT&T, Verizon, and Abovenet)

I’m almost to the point where I’m going to bust out Wireshark and analyse packet captures, as I have a feeling there are a ton of TCP retransmissions going on.

Also, as I stated before, the packet loss problem is easily confirmed using Telia’s own looking glass — you might need to run a couple traceroutes, but the packet loss is apparent.

Finally: if there are any European WoW players out there who frequent the forums, can you do me a favour and post in one of the threads, or post a new topic, pointing Blizzard (and players) to these entries in my blog? I’d do so myself, but… well, I can’t really go into why I can’t do that. I think players would appreciate knowing there is someone out there who understands the technical nature of the problem (they can also comment in my blog without an account). It might also give Blizzard the necessary information they need to slap Telia around.