The day the hackers invaded…
and other tales from the techs
In December 1993, CBC Radio conducted a science experiment. The ingredients: a sprinkling of digitized audio (an episode of Quirks & Quarks, a Basic Black show, plus a few other samples), a dash of text, and one 56 Kbps modem. The result – the CBC Radio on the internet trial. It was judged a success, although it was enjoyed by only a few hundred people.
Over a decade later, the CBC.ca experiment continues under a new generation of scientists – or technicians, or IT guys (or geeks, if you must.) The scale is a little different: hard drives in the terabytes, bandwidth in the gigabits per second, half a million pages and an audience nearing four million unique visitors each month.
But the progression hasn’t always been smooth. Like any other media operation, CBC.ca has endured countless peaks and crashes, upgrades and hacks, the successes and scares that come when the whole world knocks on your door at once.
Somewhat amazingly, many of the people who keep this enormous operation ticking along in 2006 are the same ones who held its hands during its first baby steps. They experienced the dot-com bubble, Y2K and 9-11, and lived to tell the tale. We asked them to pick three defining moments in CBC.ca’s technical evolution.
1997: Ghosts in the machine
Paul Livi will never forget the story of the haunted house in Emeryville, Ont. Not because the CBC network analyst believes in ghosts, but because of the scary thing that happened to the CBC.ca website.
The story on CBC Radio was terrifying. For several months in early 1997, Debbie and Dwayne Tamai of the small town of Emeryville had been terrorized by an electronic intruder.
“Sommy,” as the invisible assailant called himself, took over the Tamai’s entire house. He turned appliances on and off at will, eavesdropped on phone calls and voicemail, and caused the lights to flicker and the TV to change channels.
The media dubbed it “The Emeryville Horror,” speculating that a hacker or cyber-stalker was to blame for a haunting worthy of Poltergeist.
Actually, it was the Tamai’s 15-year-old son, Billy.
The unhappy teen eventually confessed that he was simply flicking the basement circuit breaker and picking up extension phones to freak out his parents.
CBC Radio wasn’t the only media outlet suckered by “Sommy.” NBC and The Discovery Channel sent in teams of security experts to sweep the Tamai’s house, and the Toronto Sun included the tale in a story headlined, “Who knows what evil lurks deep in the heart of… CYBERSPACE.”
But the day Sommy’s identity was revealed, a real cyber-intruder made CBC pay for being so gullible.
On April 20, 1997, a notorious hacker group called The Brotherhood of Warez broke into the CBC website, an apparent retaliation for the negative portrayal of hackers in the Sommy story.
The CBC.ca homepage was replaced with a message headlined, "The Media Are Liars."
“Hackers and phone-phreaks are not what the media portrays,'' their message read. “They are not the huge threat that certain people would have us believe.'' They said the media (and CBC) “sensationalizes issues like this one to sell papers, make headlines, and get ratings.”
CBC’s technical team went into overdrive, and the site was taken down within an hour. Paul Livi, the CBC network analyst assigned to investigate the hack, recalls the event clearly.
“It was bad timing,” Livi says. "It happened at the exact time that the Bre-X stuff was going on, which was a major news item at the time."
But Livi discovered that things could have been much, much worse. Though they had gained extensive access to the CBC.ca servers, the hackers left everything but the homepage untouched.
"I'll give these guys credit – they could have killed us, really put us off the air,” Livi says. "They could have crippled us completely, but they didn’t."
CBC Technology manager Sergio Bettio believes the group was not only voicing displeasure, but sending a message about online security.
"The Brotherhood of Warez gave us a warning. It was an education on our part that we needed to batten down the hatches."
As a result of the hack, a wide array of security measures was implemented, from firewalls to closed ports and restricted user access – measures that continue to this day.
"The rule on the internet is that things only happen once, and then the doors shut," says Bettio. "It alerted us that we'll have to take security seriously. There are a lot of unkind people on the internet."
"It was a blessing in disguise,” Bettio says. “But it wasn't funny at the time."
1999: OK for Y2K?
“In a world that seems increasingly to be run by computers it's easy to forget how stupid they are,” wrote CBC.ca’s Gary Katz in 1999.
He was describing a looming doom known by the handy acronym Y2K, the “millennium bug” associated with the year 2000. It referred to a fear that outdated computer software programmed with two-digit dates wouldn’t be smart enough to figure out that the next number after 99 was 00 (and that this particular 00 wasn’t the same as the one for 1900.)
“Anything that depends on a computer might not work correctly on January 1, 2000,” Katz wrote in a CBC.ca Indepth. “The list of things that don't include a computer is shorter these days than the list of things that do.”
It’s easy to be smug when looking back at Y2K with 20-20 hindsight. But at the time, there were serious concerns that taxes and bank account interest would be miscalculated, power plants would shut themselves down, planes would fall out of the sky.
CBC.ca doesn’t run power plants or fly planes, so its potential problems were considerably less dire. But, like most major corporations, CBC spent considerable time and money upgrading and replacing systems that might not be “Y2K compliant”.
“1999 was a very busy year,” recalls CBC Radio webmaster Joe Lawlor. Lawlor was asked to work with a Y2K consulting team to audit and identify systems that might need to be upgraded.
The team foresaw few problems with software. The only program that failed compliance tests was an internal system called Prolog Insite, used to publish radio program playlists on the internet.
“The programmer was contacted and asked to adjust his code,” Lawlor says. “Because of the modular nature of his code, he had to change two lines.”
Hardware was another story. CBC.ca’s website servers were Y2K-compliant Sun systems, and passed muster. But the consultants decided all of CBC Radio’s RealAudio encoders and servers, which ran Linux, needed to be replaced.
The systems had been purchased just three years prior, but the consultants were wary of the operating system. “Linux was largely a new phenomena in 1999,” Lawlor explains.
New AST computers running Windows were purchased. But Lawlor, who had always trusted Linux, switched their operating systems after the consultants had left. They worked flawlessly when the clock ticked over to the new year.
Lawlor also went a step further.
“As an experiment, one of the old encoders was left running from 1999 to 2005,” he now admits. “It worked flawlessly as well.”
2001: September 11
Of all the trauma, all the ways the world changed following Sept. 11, 2001, it’s almost trivial to talk about the way it affected one media website.
But, if your job is to make sure Canadians can stay connected and informed in a time of crisis, it was anything but trivial – it was a watershed.
As the Sept. 11 events progressed, millions of people flocked to their computers to get information about the unfolding tragedy. The internet slowed to a crawl.
Google reports that on that day, searches for news content increased by a factor of 60. The search engine even posted a homepage message warning that many media sites were offline “because of the extremely high demand,” and it even advised web users to turn back to radios and televisions for audiovisual content.
CBC.ca was one of the sites knocked offline by the huge demand for information. It’s a day site techs will never forget.
“I was in the online newsroom at CBC.ca,” says system administrator David Brock. “I remember hearing gasps, standing up and watching the second plane crash, and I came to the realization that this was going to get very busy technically.”
As the CBC.ca site became overwhelmed and crashed, webmaster Joe Lawlor was keeping a log of the technical crisis:
- 8:45 : First Airliner crashes into World Trade Center in New York.
- 9:03 : Second Airliner strikes second World Trade Center Tower. CBC Web Sites go down. Are very slow and cannot be updated. Firewall is claimed to have failed.
- 9:10 : Alternate connection to network for Newsroom is started. Live streams are up and running.
- 10:00 : CBC.ca sites inaccessible. First connection of templates successful.
- 11:00 : Systems are watched. Breakdown examined. Firewall tested and examined.
- 13:00 : Arrowpoint Load Balance (LB) switch rebooted. Things work for a while... Home page starts having links to media.
And so on.
The site was brought back online by introducing some new machinery and connections, and by paring down homepage content. Media streams reestablished through combining the resources of CBC.ca, Radio Three, and an unexpected saviour called Akamai.
The company was emerging as a world leader in distributing web content, with servers around the world to cache content and deliver it quickly to local users. However, on Sept. 11, chief technology officer and co-founder Daniel Lewin was killed aboard American Airlines Flight 11, the first of the two jets to slam into the World Trade Center.
Though the company was reeling from the news, it had the presence to reach out to media websites to make sure they could deliver the unfolding story to an anxious public. A deal was quickly reached with CBC.ca, resolving traffic issues for the remainder of the crisis. The relationship between CBC.ca and Akamai continues in 2006.
But distribution was only one part of the problem. The infrastructure failures at CBC.ca led to a sea change in site technology. A new philosophy for dealing with traffic spikes was needed.
“Traditional think was that applications that got busy needed beefy machines to run them,” says David Brock, “so we had beefy machines running the website.”
“That day identified the problem with traditional think. New think led us to concentrate on pushing pages, rather than just having powerful machines. We added multiple servers in a farm, caching servers for static objects, and bandwidth so that they had enough pipe to push it all through.”
In layman’s terms: nine computers costing $5,000 each work much better than three $25,000 machines.
But developing a system that can handle massive and unpredictable leaps in traffic is a long and ongoing process.
“Volume isn’t an instant fix,” says Brock. It's a familiar complaint. In all three cases presented here – hackers, programming errors and traffic surges – quick fixes are a myth. There is only the short, sharp shock when a technical problem becomes painfully clear, followed by a long road toward making it better.
Paul Gorbould is one of CBC.ca's first full-time employees. He came to CBC in 1995 after receiving a masters degree in journalism from the University of Western Ontario. In 1997 he produced the CBC4Kids website and created sites for CBC Radio programs. Since 2002 he has worked on the award-winning CBC Digital Archives website.
Experiments & Evolution
News – Past