How Verizon, AT&T, and T-Mobile responded to Hurricane Ian

On September 28th, Hurricane Ian hit southwest Florida, lashing the area with category four winds and proceeding to cross through the state and then into the Carolinas and Virginia. As someone with friends, family, and co-workers dotted throughout the sunshine state, I’m used to the fear that comes with seeing photos of familiar places inundated with water and the uncertainty of not knowing when you can expect to hear from the people you care about. Even relatively minor storms can temporarily knock out power and cell service; hurricanes can do so for days or weeks, extending the time spent waiting for the “I’m okay” texts or phone calls.

Ian took out its fair share of towers in Florida; according to the Federal Communications Commission, nearly a fifth of the cell sites in its path were put out of service at one point. The Associated Press and NPR relayed stories of citizens unable to call for help or to reassure their loved ones that they’d survived the storm, reporting that issues were especially bad in Lee County, home of Fort Myers and one of the hardest-hit areas — over 65 percent of the cell sites in the county were offline the day after the storm, according to the commission.

But just over a week later, that number was down to nearly 6 percent. That’s thanks to years of cell carrier experience planning for and responding to extreme weather events. Here’s a look at how cell companies plan for and respond to hurricanes and the efforts they take to keep people online during disasters.

Before the storm

In the days before the storm hit, the carriers began getting ready to address damage and outages and preparing their infrastructure to avoid those things when possible — this involves getting crews and equipment into the state, figuring out what parts of their networks they need to reinforce, and taking steps to make sure that as much of their infrastructure as possible will be able to keep on working even when power goes out.

In a blog post, Verizon says it had engineers checking on the battery and generator backups at its cell sites, making sure that they’d kick in correctly if they lost power. The carrier also says it hurricane-proofs its switching facilities and has redundant fiber connections to keep them online. T-Mobile has a similar post, which talks about how it staged equipment such as portable cell sites mounted to trucks and generators.

Drone shot of a field filled with equipment and trucks.

T-Mobile’s staging area, filled with portable cell cites ready to roll out to affected areas.
Image: T-Mobile

For AT&T, preparations involved similar steps. Chris Sambar, an executive at AT&T who oversees the carrier’s network, says it put up sandbag barriers and caulked doors at its operations centers, made sure there was staff on-site to deal with any issues that cropped up (and that those workers would have food and water for a few days), and staged its repair equipment and crews just outside the forecasted route or at hardened locations in the path. He also said that portable cell sites are useful because they can connect to satellites for data and then act as temporary cell towers that phones connect to.

Some things call for triple redundancy — the networks used by first responders are one of them

Sambar says that AT&T has three routes into mission-critical offices in case one of its providers has a line get cut. That wasn’t always the case: “We had two routes into the main switching center during Hurricane Ida, and we learned that we need three,” he said.

After making that decision, AT&T “also looked at the routes around the nation at the major hub offices” to see if the conduits it relies on were overground, underground, or going over potentially vulnerable infrastructure, such as bridges. “We kind of did a sweep over the last year and a half to really look at how we can harden the network even more than we had already hardened it years ago when it was built.”

During the storm

As the storm hits, it’s mostly a waiting and monitoring game. Sambar says that each of AT&T’s cell sites has an automated alarm system that’ll alert the carrier if it loses power or data, letting the company track where issues are happening. “In storms like this, a vast majority of the issues are power,” he said, noting that connectivity issues actually crop up more often during recovery efforts, as power companies are clearing debris and downed lines — sometimes, he said, AT&T’s fiber accidentally gets cut in the process.

That’s not to say there weren’t problems with connectivity; residents on Sanibel Island were stranded when the bridge connecting it to the mainland was severely damaged. According to Sambar, the fiber that connected that island also ran along the bridge and didn’t survive the storm. On the afternoon of the 28th, Verizon’s updates page noted that the carrier saw several sites go out of service along the west coast of Florida from Naples to Tampa. T-Mobile noted that commercial power outages knocked some of its towers offline as well.

T-Mobile and Verizon told their customers in affected areas they’d be getting unlimited calling, texting, and data, and AT&T promised to waive any overage fees.

After the storm

Once it’s safe, crews can start working to deal with outages that occurred during the storm. Throughout the week following the hurricane’s landfall, the carriers posted updates about how their networks were holding up and explained the steps they were taking to repair their infrastructure; Verizon noted deploying mobile cell sites, along with repairing and refueling the generators that were keeping some equipment online. According to Governor Ron DeSantis, 100 of the portable cell towers had been deployed in southwest Florida by September 30th.

Image of a trailer with two satellite dishes, as well as a large mast sticking out of the top.Image of a trailer with two satellite dishes, as well as a large mast sticking out of the top.

One of Verizon’s mobile cell sites, which communicates with satellites and then acts as a temporary cell tower.
Image: Verizon

The carriers had a lot of issues to contend with. According to a report from the FCC, cell site outages in Florida peaked around 17 or 18 percent on the 29th. By the 4th, around three percent were offline, just about 1 percent more than had been before the storm. In some counties, however, the percentage of outages was much higher — DeSoto County had about 20 percent of its 26 cell sites out (which was actually worse than the day before, when only 15 percent of its sites had issues), and Lee county had around 14.5 percent outages.

Chart showing the percent of cell sites out of service in the disaster area by date. It shows a sharp increase from September 28th to the 29th, then a gradual decrease from September 30th to October 4th.Chart showing the percent of cell sites out of service in the disaster area by date. It shows a sharp increase from September 28th to the 29th, then a gradual decrease from September 30th to October 4th.

It’s taken some time, but the carriers have gotten a good chunk of the cell sites back online, whether that means restoring power or connectivity.
Chart: FCC

The report does note that “the number of cell site outages in a specific area does not necessarily correspond to the availability of wireless service to consumers in that area.” Basically, because cell towers often provide overlapping coverage, 20 percent of them being out doesn’t necessarily mean that 20 percent of the area doesn’t have service.

The infrastructure damage caused by hurricanes can make it difficult for even the heavy-duty and high-water trucks to get equipment to where it needs to be, requiring extra measures. Sambar mentioned that AT&T used an amphibious vehicle to get a satellite to Sanibel Island, and the company posted on its site about how it used a helicopter to deploy a portable cell site to Pine Island. He also said that the company uses drones for reconnaissance to remotely check out potentially inaccessible cell towers and assess them for damage. Verizon posted about doing this as well and said that it had a drone “providing cellular coverage from the air” for search and rescue teams on Sanibel island.

1/2

Verizon’s cell service drone.
Image: Verizon

In areas where the damage isn’t as extreme and there are cell towers left standing, the companies have been working to keep each other’s customers connected. During a press conference, DeSantis said: “All the companies, I think, now are allowing the other customers to roam on their network,” though he did warn citizens to use regular calling instead of Wi-Fi calling. On its site, AT&T said its network has handled “more than 47 TBs of traffic from other carriers.”

The carrier prepares for and handles that much additional load by having its network engineers do traffic assessments before they open the network up, estimating how much extra traffic they’ll have to deal with, Sambar says. After other customers come online, they’ll monitor the network to make sure everyone is able to use it properly; “if we start to see serious degradation, then we’re going to have to push one or both of them back off.”

He also said that the sharing shouldn’t affect customers on FirstNet — the First Responder Network Authority — as they have dedicated infrastructure and spectrum. FirstNet is a government authority that’s partnered with AT&T, and it’s tasked with keeping first responders online, even during disasters. According to Sambar, the carrier has recovery equipment dedicated to FirstNet, and its duty to keep the network online was part of the reason it went for triply redundant connections. Verizon also has a service with the same goal called Frontline. Both carriers mention the extra work and support they’ve done to make sure that rescuers can communicate with each other and with the people coordinating their efforts.

The future

While the need for wired and cellular infrastructure won’t be going away anytime soon, we’re heading toward a future where it’ll be possible to communicate with loved ones and rescue personnel in other ways, which could potentially make service outages less scary and dangerous.

In about a month, the iPhone 14 will be getting a feature that allows people to communicate with emergency responders via satellite. While it won’t necessarily help people who are just looking to let their family know they’re okay, it could give people who are in immediate danger a chance to contact rescuers in a way that may not have been accessible before.

For people without the latest iPhone, there are other potential solutions coming in the near-to-mid term. T-Mobile has partnered with SpaceX to provide service to 5G-capable phones via satellite, and companies like Lynk and AST are working on similar tech. T-Mobile’s service is probably a year or two away at this point, and while Lynk has already proven its ability to send messages to phones from space, it still has to team up with a carrier and get approval before it can start providing commercial service.

Still, it’s foreseeable that within a few hurricane seasons, people could have some modicum of connection before the carriers roll in with their mobile cell stations. That could especially be a boon for those in rural areas. There were hints of how this could work this year; according to CNN, the state is using SpaceX’s Starlink satellite internet to help restore connectivity in some areas.

Of course, there are still limitations. Communicating with satellites can come with a lot of latency, and if you’re doing so with your phone, you won’t necessarily want to be burning a very limited battery when it’s unclear how long it’ll take to get the power back on. But for those of us waiting to hear from loved ones in disaster areas, one or two messages could mean the world — and for those in disaster areas, the ability to get word out could genuinely wind up being the difference between life and death.


Source link

Leave a Reply