Seagate technician Mai Nguyen pulls samples of hard drive disks as part of the quality control procedure in the clean room at a manufacturing facility in Milpitas, Calif. (Tony Avelar/Associated Press)

In the digital age, it turns out you can never have enough data storage: Bigger, apparently is better.

Once people talked about large storage capacities in terms of gigabytes and terabytes — now it's petabytes. And next year exabytes. And soon zettabytes. The massive demands of Web 2.0 — all those pictures, videos and jokes being shared — compliance and privacy legislation along with a general corporate trend towards obsessively archiving and data mining every piece of information has created a surge in the storage sector.

The technology research firm IDC predicts that by 2010, nearly 70 per cent of the digital universe will be created by users, but organizations, public and private, will bear the responsibility — that is, the risk — for security, privacy, reliability.

Finding the right words

"Data sprawl" is also creating a new vocabulary. One megabyte is one million bytes or the equivalent of the information found in a small book. A gigabyte is a billion bytes — equivalent to a complete symphony or an uncompressed collection of today's top 40 songs. A terabyte is one trillion bytes — or all the X-rays in a large hospital. A petabyte is 1,000 terabytes: half the contents of all U.S. academic research libraries. An exabyte is 1,000 petabytes; to put that kind of storage capacity in perspective, five exabytes would comprise all the words human beings have ever spoken. Finally, a zettabyte is 1,000 exabytes — as many bytes as there are grains of sand on all the world's beaches.

Globally in 2006 we created about 161 exabytes of stored information, equivalent to three million times the information in all the books ever written, notes IDC. By 2010 the number is expected to quadruple.

Globally in 2006 we created about 161 exabytes of stored information, equivalent to three million times the information in all the books ever written. By 2010, the number is expected to quadruple.—IDC

Far from being just a cost of doing business, on the contrary, much of that data has value both in terms of future business decisions and insurance for past choices. It can be mined, analyzed for nuggets of information to make future decisions about products, services and enterprises.

And by safely and securely storing it, companies remain compliant with corporate governance legislation like the Sarbanes-Oxley Act in the United States and its Canadian equivalent, and privacy codes such as the Canadian Personal Information Protection and Electronic Documents Act (PIPEDA).

The business of storage

But building storage facilities, also known as data centres, is a costly venture: you need lots of power, real estate and bulletproof hardware — meaning lots of backup. And you need access to a pipe big enough to upload and download to the web. And finally you need a way to organize that data, tag it for future retrieval, and encrypt it against hackers and loss.

And, as Joe Tucci — the chairman, president and CEO of the data management giant EMC says — all that data needs a place to live, creating new business opportunities and revenue streams.

Some business are already ahead of the curve. Amazon.com, for example, rents out space on its massive arrays of networked storage. That's all fine and dandy, but it's not enough just to house it, Tucci stresses. Data must be tagged for retrieval, encrypted to protect it from unauthorized access and, of course, physically protected from the moment it is created.

"There are also some questions around the applications," adds IDC Canada storage hardware research analyst Vasu Daggupaty. "Will those documents open with Word 2000 in the future or will we have issues like we have now with Word 98?"

An internal view of a Savio hard drive.(courtesy Seagate)

Storing files requires a more proactive approach than just dump and forget, he says. Data must be managed throughout its life cycle, from creation to archiving, even deletion. "Though no one really deletes much anymore," he said. "That's the thing. With compliance rules, some financial institution are holding onto this data forever." And that's giving rise to a system of tiered data: that which is active and in play and stored locally; that which is requires infrequent access and is housed both locally and at a mirrored backup site; and archival data, which is needed in case of disaster for restoration or other unforeseen circumstance (such as for legal or compliance reasons).

Tucci sees data storage as an outsourced service, but Sun Microsystems is betting much of it will stay in-house — as is Dell Computer, which has also launched a line of storage hardware.

"You're responsible for the data if anything happens so why add to the risk?" asks Philip Kaszuba, vice-president of Sun Microsystems Canada's storage group, who argues in favour of in-house expertise.

Daggupaty says it will likely be a combination of both.

Growth industry

Simply building more hard drive arrays won't satisfy the burgeoning demand for storage, because hard disks are an unstable technology. They're too prone to mechanical failure, taking the data down with them.

Meanwhile, CDs and DVDs are susceptible to corrosion if the surface lacquer is scratched and the oxide layer exposed to air. There are also questions about how long an optical disc can hold data. Plus, CD and DVD discs have relatively small capacities and aren't as easily searchable as hard drives when it comes to retrieving information.

The answer for long-term storage of that third tier of data is tied to the resurgence of tape — yes, that archaic strip of magnetically coated plastic — running through a cartridge that looks like an old-style cassette on steroids. While it's slow, it's also virtually foolproof, barring a nuclear war, fire or tsunami. Tape units today are large, self-contained storage closets in which the cassettes are indexed and racked. Like a giant jukebox, demands for access to specific data trigger a servo-driven robotic arm which goes to the location of the cassette, pulls it and loads it into a reader.

"Two years ago we would have said tape was starting to recede but that's certainly not the case now," says Rob Daleman, Dell Canada marketing manager. "Our studies suggest about 81 per cent of companies are looking to increase their level of investment in tape and over half are using tape for compliance issues."

As he says, tape is stable and easily transported. Plus, unlike hard drives, which have motors that are usually spinning constantly, tape storage only draws power when needed. That makes it a hot commodity in a storage-hungry, energy-conscious environment. And while prices for storage — like every area of technology — continue to fall, demand is growing fast enough to keep revenues growing.

"We're talking about massive growth in the archive side," observes Kaszuba. "In talking to our clients, they planned for 25 per cent storage growth at the beginning of the year, but then come back partway into the year and say, we've already gone through that, we need to grow more."

[an error occurred while processing this directive]