Please note, updates to this post can be found further down the page.
It seems that Telia, one of the underwater transatlantic cables which connects the Internet between Europe and the US, has been reportedly cut or damaged. The specific cable believed to have been cut was TAT-14, the route of which can be seen below as it travels from the US through to the UK, France, the Netherlands, Germany and Denmark.
For a large number of Internet users in Europe, this resulted in outages of many popular sites including Dropbox, and sites using CloudFlare, until hosts were able to re-route web traffic via other unaffected routes.
Luckily, there are many other transatlantic cables, so this didn’t take too long, and it was at most an hour or so before sites could be accessed again – although there was considerable congestion afterwards, as can be seen by the below screenshot of Akamai‘s website late on 19th May.
News quickly spread on a number of online forums including:
TheRegister also published an article on this – although fairly late after it happened.
Digital Ocean and CloudFlare seemed to be the two companies mostly affected (who it is assumed have transit links with Telia), but many other websites and web hosts were indirectly affected through customers using an ISP which uses Telia as one of their connections.
Digital Ocean:
CloudFlare:
A number of users also took to Twitter to find out if they were the only ones experiencing problems accessing certain parts of the Internet:
UPDATE (11:45pm GMT – 19th May):
Telia has released a statement via Twitter which explains the cause of the outage as a ‘planned update gone wrong’
Which roughly translates from Swedish to:
“Our previous disruption on our internet traffic against the United States and Asia is eliminated since 23 o’clock, it was a planned update went wrong”
UPDATE 2 (12 midnight GMT – 19th May:
Telia have now published an article in Swedish, in which Bo Djurberg (Manager at Telia) summarises the issues they had.
Again, roughly translated this is:
Problems with Telia Internet
Telia had tonight big problems with their Internet services.
Users could not get into websites registered in the United States.
– There was a scheduled job that went a bit wrong, says Bo Djurberg, manager at Telia.
The problems started at 22 o’clock on Monday evening.
Lots of customers had problems with the internet connection. According to Telia these involved a planned update that went wrong.
– They had problems reaching websites in the U.S. and vice versa, says Bo Djurberg.
At 23 o’clock the problems were remedied, according to Telia.
– Now everything will work out. All traffic was down for about an hour, says Djurberg.
Exactly how many users are affected is not clear.
UPDATE 3 (8am GMT – 20th May):
A further statement by Telia says that the issue was caused by a software ‘route change’ rather than the physical cable being cut. This explains how Telia were able to fix the problem so soon – something which likely wouldn’t have been so soon had it been cut.
UPDATE 4 (10am GMT – 20th May):
Further confirmation of this being a ‘planned update’
This post will be updated as and when more information is released.
UPDATE 5 (4pm GMT – 20th May):
This will likely be the last update, so I just wanted to summarise all of the updates and new developments since this post was first published.
This morning, Telia announced that the disruption was due to a ‘planned update went wrong’. There was no physical damage or cut to any of the cables, so it seems they were doing some form of software update for the cable (which we believe was TAT-14), which didn’t go as planned and resulted in the cable failing to carry data and going out of service.
Thankfully the downtime last night didn’t last too long as ISPs and web hosts were quick to re-route traffic via alternate transit carriers. There are many other transatlantic fibre cables, so this wouldn’t have been to hard – although congestion was caused as traffic had to be funneled to these other routes.
Were you affected by the Telia issues? Please get in touch (adam[at]adamowen.co.uk), or add a comment and I’ll feature your quote.
I also wrote another article for my company here – http://www.spiralmedia.co.uk/slow-internet-last-night.
