How a computer problem can shut down an airline like Delta

The power outage that left thousands of Delta passengers facing worldwide flight cancellations and delays Monday shows how computer-dependent we've become — and airlines must decide if their backups are good enough, an IT expert says.

'We've kind of painted ourselves into a corner where we must rely on computer systems,' prof says

Delta Air Lines grounded by power outage

5 years ago
At least 650 flights were cancelled and thousands more were delayed after a worldwide computer system outage 2:04

The system outage that left thousands of Delta Air Lines passengers around the world facing flight cancellations and delays on Monday shows how computer-dependent society has become — and airlines have to decide if their backup technologies are good enough to deal with that reality, a Canadian computer networking expert says.   

"We've kind of painted ourselves into a corner where we must rely on computer systems," said Srinivasan Keshav, a professor of computer science at the University of Waterloo.

"[But] we have now been able to build systems which are very tolerant of losses, of parts of the system being taken down."

The key, Keshav said, is to adopt the model that technology leaders like Google have — known as "system fault tolerance," which assumes any single component in a computer network can fail at any time, but it doesn't matter because there are multiple backup measures in place at every level of the system.   

"Failures are not exceptions. Failures are kind of normal," Keshav said, noting that companies like Google or Amazon have dozens of servers "dying every day," but with upward of 100,000 servers on hand, the systems don't crash.

Power outage a 'surprising' cause

Delta Air Lines said the cause of Monday's mess was a power outage at its base in Atlanta, Ga., at around 2:30 a.m. ET. In a statement posted online Monday afternoon, the airline said systems were once again "fully operational" and flights had "resumed hours ago but delays and cancellations remain as recovery efforts continue."

The fact that a power outage was to blame is "surprising," Keshav said, because "it's the one thing you wouldn't expect to have happen because that's easy to get right."  

Airline data centres usually have two layers of backup — diesel generators and batteries — to protect "critical systems," he added. 

"When you look at a complex computer system such as the one that Delta runs, there's many layers of the cake, so to speak. At the bottom is power," Keshav said. 

Mark Duell, vice-president of operations for the global aviation tracking website FlightAware, said airlines "go to great lengths" to make sure backup systems, including several power sources, are in place in their data centres. 

"Everything from bringing in power from the utility on opposite literal sides of the building, just so [a] single backhoe can't take them both out at the same time; having more generators than they need so that they don't need all the generators to be operable; having... multiple battery backup systems internally to cover everything until the generators come online," Duell said.

Delta passengers, including four-year-old Lisette Hamann, lower left, and older sister Harper, wait at a ticket counter at Newark airport in New Jersey on Monday, among thousands of stranded Delta customers around the world. (Seth Wenig/Associated Press)

"And then down to the point of literally each computer, each server in the data centre is plugged into two different power strips and has two power supplies that are redundant."

Although he doesn't know specifically what happened in Delta's case, Duell said it was likely that the problem extended beyond a basic utility failure, since the batteries and generator backups should have kicked in.

"It was probably more than one failure," he said.  

Safety not at risk

Both Duell and Keshav emphasized that the computer system outage would not have posed a risk to passengers in flight. 

"The airplane is entirely independent of the ground in terms of continuing to fly," Duell said. 

That's because airlines use "decoupling" in computer system design, Keshav said, meaning systems involved in actually operating the aircraft are independent from other systems like reservations or flight schedules. 

The reason a system outage like this one has such an impact, Duell said, is because airlines stop and cancel flights for safety reasons when they can't get access to important computerized information like passenger counts, how much baggage has been checked or fuelling records. 

"You run into those sorts of dependencies where they can't move things, but anything already moving is not in any real danger," he said. 

'Critically examine' infrastructure

Delta isn't the only airline to have experienced a recent system failure. 

Last month, Southwest Airlines cancelled more than 2,000 flights over several days after an outage that it blamed on a faulty network router.

United Airlines has suffered a series of delays since it merged with Continental as the technological systems of the two airlines clashed. 

Perry Higgins, left, and Alaina Whittaker check for updates from Delta Air Lines at Toronto's Pearson airport on Monday. Computer problems at the U.S. airline forced the couple to cancel their plans to get married that afternoon in San Francisco. (Nick Boisvert/CBC)

"It's something that happens from time to time," Duell said. "There's no particular airline that is immune to these [problems], and from what we've seen, there's none that are particularly prone to these."

Although Keshav doesn't know what measures specific airlines have already taken, such large-scale failures could be prevented if they invest in rigorous systems "that tolerate fault and assume faults are going to happen."

But that would entail expensive and complex engineering, requiring the replacement of legacy systems built years ago, he said. 

"Banks, airlines, things like that which have been around for a while... need to at some point critically examine their infrastructure." 


Nicole Ireland is a CBC News journalist with a special interest in health and social justice stories. Based in Toronto, she has lived and worked in Thunder Bay, Ont.; Iqaluit, Nunavut; and Beirut, Lebanon.

With files from The Associated Press


To encourage thoughtful and respectful conversations, first and last names will appear with each submission to CBC/Radio-Canada's online communities (except in children and youth-oriented communities). Pseudonyms will no longer be permitted.

By submitting a comment, you accept that CBC has the right to reproduce and publish that comment in whole or in part, in any manner CBC chooses. Please note that CBC does not endorse the opinions expressed in comments. Comments on this story are moderated according to our Submission Guidelines. Comments are welcome while open. We reserve the right to close comments at any time.

Become a CBC Member

Join the conversation  Create account

Already have an account?