Data Warehouse Upgrade/Expansion Choices

Data warehouses are designed to store data. Normally lots of data. Sometimes lots and lots of data. You get the point. The rate of growth of a data warehouse often exceeds initial expectations, especially those that meet user expectations.

Some data warehouses consequently become victims of their own success. Technology constraints then become an issue. "Too slow" or "not enough space" are the common user gripes. Are they ever happy!?!?!?!

Once up a time when a data warehouse platform had 'run out road', the predictable response was twofold - software and hardware upgrades:

    • software - upgrade the OS, upgrade the DBMS, maybe create more indexes.

    • hardware - more RAM, faster CPUs, more CPUs, more storage, faster storage.

A mixture of software and hardware 'upgrades' was/is often the knee-jerk response to end user gripes. In essence, whatever it took to buy more time. Not a real solution, more a deferral so that addressing the underlying problem can wait until a later date.

Upgrading and/or adding more 'stuff' to the traditional SMP+SAN/NAS stack is the time-honoured 'scaling up' of the single SMP 'fat node' that underpins the majority of data warehouse platforms.

So, what's the point???

Well, it has become apparent in the last few years that there are now more choices available to those who have run out of data warehouse road.

    1. SMP scale-up is still a choice (see above), although diminishing returns will remain an issue.

  • Rip and replace with an MPP 'appliance' e.g. Teradata, Netezza...or even Microsoft's upcoming parallel SQL Server edition (project Madison, the Datallegro purchase), Oracle's Exadata or HP's NeoView.

 

  • Enable the existing SMP stack with a complimentary MPP appliance e.g. Dataupia.

 

  • Rip and replace with 'roll your own' MPP .e.g GreenPlum.

Decisions, decisions...the good news is that there are many more choices available now when your data warehouse needs more oomph, more space, or both (most likely).

The bad news is, there is more choice!

With any luck I'll find time to talk about our experiences to date with each of these approaches, out there where it matters, in the real world.