Server farms under strain from increased streaming & videoconferencing during Covid-19 outbreak

Across the globe, hundreds of millions of people are using their Internet connection more intensely than they did just a few weeks ago. Usage ramped up so quickly that the infrastructure is being stretched past its limits.

I don’t have any tips. You don’t have to do anything differently. It’s just another interesting way that Covid-19 is turning our economy and culture upside down.

In the last two or three weeks, when we were confined to our homes, we immediately began to expect our home Internet connections to do some new things. We’re streaming far more video over Netflix and other services. We’re working remotely over VPNs or through connections like Splashtop to our office computers. We’re participating in online video meetings using Zoom, WebEx, or Teams. We’re using Zoom for online workouts or to stay in touch with family and friends. The kids are sitting down to online classes. We’re playing more online games with Playstation and Xbox.

ISPs

The connections to our houses – the “last mile” – are handled by thousands of ISPs in hundreds of countries around the world. Those companies – from behemoths like Comcast and Spectre to local companies like Sonic – have excess capacity and sophisticated tools to manage bandwidth. None of them were ready for an explosion of demand in a period of two or three weeks. They all claim to be entirely on top of things, of course, but there are signs of stress here and there. The ISPs will have more trouble keeping up in densely populated urban areas, and some ISPs have less extra capacity ready to throw at their customers.

You’re not likely to notice reductions in speed in the US. Even slowdowns deliver enough bandwidth to stream videos without stuttering – perhaps lower resolution video, but with luck, nothing will stop working.

But the increased demand, and slowdowns from overwhelmed ISPs, are quite real. Broadband Now released a report a few days ago comparing speeds in 200 cities for the week of March 15-21 to speeds a few weeks earlier. In 88 of the cities, there was at least some degree of network degradation. Twenty-seven cities saw speed reductions of more than 20 percent. “Three cities—Austin, Texas; Winston-Salem, North Carolina; and Oxnard, California—have experienced significant degradations, falling out of their ten-week range by more than 40 percent,” the report said.

Streaming services

Netflix accounts for almost 15% of the world’s bandwidth usage. It’s so significant that last week EU Commissioner Thierry Breton called for Netflix to reduce streaming quality in Europe to minimize the effect of its traffic. Netflix agreed to lower the resolution of its video streams across the board in Europe. So far, Netflix has not made a similar change in the US.

YouTube changed its default to standard definition around the world after EU officials expressed concerns. (You can manually change the quality of individual videos to 1080p or 4K.)

The other video streaming services are experiencing huge, unexpected growth. HBO Now reports that movie watching is up 70 percent. Disney Plus saw surges in subscription signups between March 14 and March 16, when social isolation began in earnest in the US. The video streaming services have so far avoided major outages, even in the face of the increased demand.

Video conferencing & cloud services

There have been massive increases in demand for video conferencing software like Zoom and WebEx, and remote access solutions like Splashtop. Zoom saw an increase of more than 40 percent in logins between March 13 and March 20, along with an increase of nearly 50 percent in its stock price. WebEx usage increased by 20 percent overall, and far more in February in China, Singapore, Japan, and South Korea. Splashtop CEO Mark Lee estimates that the company has seen “over 10x the usual demand.”

None of these companies have reported extensive outages, but behind the scenes engineers are scrambling to increase capacity and avoid slowdowns. Zoom had at least one widespread incident of “degraded performance” and “partial outage” on March 16.

Meanwhile, Microsoft is struggling with the success of its enterprise cloud services, which are run from huge server farms located all over the world. For years Microsoft has been steadily increasing capacity to stay ahead of demand, but the recent surge has been overwhelming and Microsoft has been forced to impose temporary restrictions. A few days ago, Microsoft reported a 775 percent increase in overall cloud services usage in regions that are enforcing social distancing or shelter in place orders.

Microsoft Teams has been widely adopted by enterprises for collaboration and video conferences, making it the perfect tool for employees who are suddenly working from home. Usage has surged, making it worse that Teams recently had its second outage in Europe in as many months. Mary Jo Foley reports that some Microsoft Azure cloud customers in Europe say they are hitting capacity limits or are unable to spin up virtual machines in their regions. Microsoft promises that it is ” expediting the addition of significant new capacity that will be available in the weeks ahead.” In the meantime, there are “a few temporary restrictions designed to balance the best possible experience for all of our customers.”

Games

Security company Cloudflare reported 400 percent growth in online gaming in the last two weeks. So it’s no surprise that Sony is slowing down Playstation game downloads in Europe and the US to preserve Internet capacity overall. Xbox Live has had a string of outages, leading MS exec Phil Spencer to acknowledge the “unprecedented demand” and promise that the Xbox teams are “working diligently to ensure we can be there for our players.”

Steam broke its all-time record for concurrent users over the weekend of March 15, with more than twenty million gamers online at once. No outages so far.

Oh, one more familiar name: Google claims to be unruffled by the shifts in demand. Google’s Senior VP for technical infrastructure says that Google’s systems were built for peaks like Cyber Monday online shopping and the World Cup finals. “It’s natural to wonder whether the Google network can handle the load. The short answer is yes. Peak traffic levels are well within our ability to handle the load.”

Every company is scrambling to increase capacity. Although it’s only been two or three weeks in the US, working from home feels like the new normal. Perhaps we’ve already seen the biggest spikes.

The Internet is under stress. Do your part to relieve things – take a break and go wash your hands.

Share This