The network was crippled. Days before Thanksgiving, Sony Pictures employees had logged onto computers that flashed a grim message from a hacker group calling itself Guardians of Peace.
- FBI head calls North Korea a cyber threat
- North Korea blasts U.S. for sanctions over Sony attack
- Sony hacking leads to U.S. sanctions against North Korea
Soon personal information for tens of thousands of current and former workers was dumped online, including Social Security numbers and the purported salaries of top executives.
Five Sony-produced movies, including the unreleased Annie, appeared on file-sharing websites. Thousands of private, and sometimes embarrassing, emails hit the Internet.
"They came in the house, stole everything, then burned down the house," Michael Lynton, the movie studio's chief executive, said in an interview with The Associated Press on Thursday. "They destroyed servers, computers, wiped them clean of all the data and took all the data."
More than six weeks later, the studio's network is still down — and is expected to remain so for a few weeks, as techs work to rebuild and get it fully back online.
In that time, Sony has been thrust into the geopolitical spotlight as the target of an unprecedented corporate cyberattack that the United States has attributed to North Korea.
In a wide-ranging interview Lynton talked about the isolation and uncertainty created by the attack and the unique position the company found itself in, in a case that's undoubtedly being closely watched in boardrooms around the world.
"We are the canary in the coal mine, that's for sure," Lynton said. "There's no playbook for this, so you are in essence trying to look at the situation as it unfolds and make decisions without being able to refer to a lot of experiences you've had in the past or other peoples' experiences. You're on completely new ground."
In the early hours of the hack, workers scrambled to find ways to communicate with the studio's 7,000 employees and keep the business running.
Some dug through basement boxes for old BlackBerrys so they could email securely and others turned up long-unused check cutters so workers and vendors could get paid by paper check.
A close-knit senior management team of 10 to 15 people relied initially on word of mouth, an emergency notification system and town hall meetings to disseminate information and calm fears.
Managers were told to be visible during commissary mealtimes and gathered with 80 to 90 employees at a time at buildings across the lot to offer updates.
"People relied on each other and it's a good thing they relied on each other, because there wasn't a lot of assistance coming out of the community, except for the FBI," Lynton said.
While most Sony employees already were on the Everbridge emergency notification system, workers recruited the rest to sign up. If he had to do it again, Lynton said he would have made it mandatory to already be on it.
Senior managers created text and phone trees to communicate and held twice-daily meetings. Thirty to 40 people worked day and night through the Thanksgiving holiday.
When employees arrived to work on Monday, one week after the Nov. 24 hack, a concierge-like desk greeted them to help get them signed onto a temporary email system set up by the technology team.
The focus, Lynton said, was on answering questions and curbing fears as well as maintaining operations. People were upset and scared and managers were tasked with trying to assure them and providing information updates two to three times a week. Focusing on operations and making employees feel safe helped keep Lynton and his senior managers afloat.
"As long as you could stay true to that, it felt OK, it actually felt OK," he said. "They weren't ideal circumstances, and of course, when you went home your kids or your spouse would say to you, `Geez, how is it,' and it's a very difficult thing to recount because every day you go into the office thinking one thing and go home with a completely different set of events than you'd imagine. All my colleagues felt the same way."
The Federal Bureau of Investigation and investigative firm Mandiant were brought in within the first week.
'We probably in retrospect should have said we're exploring other options.' - Sony CEO Michael Lynton
Lynton is effusive in his praise of the FBI, which camped out in a special set of rooms in the centre of Sony's lot and conducted multiple hour-long "clinics" on a sound stage for 400 to 500 Sony employees at a time. The meetings covered identity theft and also some computer security tips.
But with constant data leaks and rolling threats coming in from the hackers, Lynton said his team had to work hard to not be too reactive and to make measured decisions.
"The whole series of events, not just for myself, but for everybody in the company, had so many twists and turns to it that every time you thought you were going down a path, every time people thought we got this in hand, the next thing you knew we'd have another threatening email come through two days later or another series of events," Lynton said.
And the story was about to take another turn. As the studio grew closer to a planned Christmas Day release of the Seth Rogen and James Franco comedy The Interview, which spoofs an assassination of North Korean leader Kim Jong-un, the hackers shifted from providing fodder for gossip columnists to instead fanning fears of terrorist attacks.
Threats of violence reminiscent of Sept. 11, 2001, against theatres planning to show The Interview prompted major theatre chains to pull the movie, forcing Sony to say it would cancel the film's Christmas Day release.
The company immediately faced a litany of criticism over free speech and censorship from all sides, from Hollywood actors like George Clooney to the president of the United States, who said during a press conference that Sony "made a mistake." In the midst of the firestorm, the FBI formally announced it had linked the attack to North Korea.
"We were so taken by surprise by the events … that we didn't have a plan at that moment to go forward," Lynton said.
But Sony always planned to release The Interview, Lynton said, it just initially didn't know how to.
"We probably in retrospect should have said we're exploring other options, because that's exactly what we were doing," he said. Almost immediately after the theatre chains pulled out Lynton said he was on the phone trying to find a way to get the film out, especially after suffering through the prior three weeks of data dumps and "what could only be described as extortion."
"We'd already spent a lot of money, millions and millions of dollars, to get a national audience to release a picture; the last thing you want to do is then haltingly bring the movie out," he said.
Called Google CEO
But cable, satellite and digital companies told Sony they were wary of running the film during the holidays, a traditionally high-selling period, out of fear of becoming targets for hacker attacks too.
Lynton then called Google CEO Eric Schmidt, who he recalled told him: "This is what we've been waiting for."
Schmidt agreed to help get the film out on Google Play and YouTube. Sony built its own website and Microsoft's X-box and Apple's iTunes also ultimately agreed to release the film, Lynton said. Sony purposely priced the online version of The Interview at $5.99 US rather than a typical $9.99 or higher to avoid accusations of price gouging and to ensure more people could see it after the free speech criticisms it had weathered.
The movie launched online on Christmas Eve and independent theatres also stepped up to screen the film on Christmas Day. Sony became an unintended test piece in a new film release strategy of putting out streaming video at the same time as a theatrical release.
The film, which had a $40 million production budget, has so far made more than $31 million from its online and on-demand release, Sony said earlier this week. That is the most lucrative digital release for a Hollywood film so far.
The Interview is currently playing in 558 theatres and has been rented, streamed or purchased 4.3 million times. It had originally been forecast to earn about $30 million in its opening weekend alone in a few thousand theatres, however.
Lynton said the studio views the release of a film on on-demand video and in independent theatres as "still experimental."
"You would never take a movie of this size and do what we did with it in the end," Lynton said.
"It's true, it proved to be that kind of experiment, but it certainly wasn't planned." The theatrical experience is important, Lynton said, especially for comedy "because people love to laugh with each other."
"Had this not come along the way it had, we would've proceeded exactly the way we planned to do it, which is to put it out on 3,500 plus screens," he added.
Throughout, Lynton said senior management focused on keeping the business going. One manager ensured Annie would still release over Christmas, another made sure Black Friday went off without a hitch for DVD sales and replenishment was made available for all DVDs, another focused on all television shows shipping on time.
As a result, the company didn't lose a single day of television or film production and Lynton said losses are expected to be "very manageable" and "not disruptive to the wellbeing of the company."
He declined to provide a cost estimate but said prior numbers that have circulated aren't accurate.
Meanwhile, most forensic on-site work at Sony is complete and remaining techs are focused on getting the system back online.
Lynton said it's likely another week before email is back up and running and two to three weeks before the network as a whole is back online. The company is now working on identifying lessons learned and trying to determine what should or shouldn't be changed going forward.
"I know that we were adequately prepared," Lynton said about the company's technology and security. "Just not for an attack of this nature. Nobody could have withstood an attack of this nature."