As discussed yesterday, the incredible growth of storage capacity led to an attitude that storage was too cheap to manage. Excess data capacity seemed always to absorb any new demand. But the unchecked growth of data led to the serious issues that storage managers face today: Difficulties in protecting massive data sets, concerns about compliance and litigation, and storage budgets that refuse to shrink.

Why isn't storage getting cheaper? This series of articles attempts to answer this question:

1. Too Cheap to Manage
2. Too Much to Manage
3. Tiered Storage
4. The Glass Floor
5. Storage as a Service

Too Much to Manage

There is a name for the digital swamp we've found ourselves in: Unstructured data. That first word is the critical one: Open systems store files in a user-created directory tree, with each file residing in one location in that tree. This storage paradigm creates a maze of paths leading to pockets of data here and there. Although many organizations try to tame this mess with directory and file naming standards, there is only so much they can do.

As most have already discovered, the filename/directory location concept inherently limits the organization and usability of file systems. This is the reason we call this type of data unstructured! IT has responded with "resource management" and search tools that try to index unstructured data and make it more usable, but these systems can only go so far. Another attempted solution is content management systems, which organize documents with version control, keywords, and much more flexible filing concepts than a simple tree.

Although the proponents of organization have been successful in some locations, the majority of us still drown in unstructured data. It fills up our personal computers and file servers, hampering productivity and limiting the benefits of ever-increasing capacity and ever-dropping cost. But it's not just a lack of organization that is turning "too cheap to manage" into "too much to manage".

Although office files have expanded, multimedia killed the disk quota. The new XML-based document formats are actually quite efficient, but the proliferation of embedded graphics, audio, and movies causes them to balloon from tens of kilobytes to tens of megabytes or more. With our multi-megapixel digital photos, iTunes libraries, and ripped movies, giant disk drives are like a drug for digital pack rats. A basic laptop now has a 160 GB hard disk, and terabyte laptop drives are on the way, enabling the problem rather than forcing a solution.

We don't have storage problems, we have usage issues. Quotas are passe and storage resource management (SRM) and hierarchical storage management (HSM) software never took off. Hard system limits are about all that stops the flood of data, and vendors are working to eliminate these, too.

This is one answer to the core question of this series: Storage isn't getting cheaper because even as capacity expands, our use of space keeps growing unchecked. But there are other reasons as well: The next entry in this series will explore the extent to which tiered storage can address the issue of data storage cost. Tune in Monday!

• Why Men’s Wearhouse is a business for the people