A forecast: as data warehouses move to the cloud, the integrator's entire project economics change
Less heavy implementation, more iteration on data and metrics. What this means for companies buying IT projects.
This is another postdiction. Suppose that in three years cloud data warehouses become the standard choice for new projects - not an exotic experiment, but the first option considered at the start. What changes in how IT projects are built and how companies buy them? The economic shift that makes this plausible - the new pricing and pilot psychology that Amazon Redshift introduced - I covered in Amazon Redshift and the new economics of data warehousing.
Right now a large share of the budget on a typical data warehouse project goes to hardware and its setup: servers, licenses, installation and configuration, initial optimization. This is expensive, time-consuming, and requires narrow specialists. And that is just to start working with the data.
How the cost structure changes
When the warehouse is in the cloud, the initial installation stops being a project. It becomes a procedure. Hours instead of months.
The money that used to go to hardware and initial setup remains in the budget. The question is where it goes. On a good project the answer should be: into working with the data - modelling, transformations, quality, business logic. That is where actual value for the business is created.
On a bad project the money simply disappears, because the budget was shaped around the old model and nobody revised it.
Iterations instead of long implementations
Classic data warehouse implementation is a large project with a fixed scope, long phases, and a multi-month cycle from start to first working version. That model did not arise by accident: when infrastructure is expensive and complex, it cannot be changed often, so requirements are gathered in advance and as completely as possible.
In the cloud, infrastructure is cheaper and faster. This allows for iteration. Launch a minimal version, see what works, add the next layer. Rethink the data model a month later instead of a year later.
This is a fundamentally different way of working. It requires different contracts, different teams, and a different culture at the client side - a willingness to work with an unfinished product and provide feedback in cycles.
What changes for the integrator
A company that previously sold large projects with expensive installation loses part of its traditional revenue. Installation is no longer expensive.
But a new space opens up: advising on data model design, setting up pipelines, working on data quality, helping interpret metrics. This is different work - less monumental, more continuous.
Integrators that adapt in time will move from "the people who installed the server" to "the people who help think about data as an asset." That is a more valuable position, provided there is genuine expertise behind it.
What this means for the buyer
If you are choosing a partner for a data project in 2013, it is worth looking beyond whether the integrator can install and configure software. Ask:
- does the team know how to iterate, or do they only know how to do large final releases?
- how do they work when requirements change in the middle of a project?
- do they have experience working with data in a business context, or only technical experience with specific products?
- what does their support model look like after launch?
The cloud lowers the infrastructure barrier to entry. That is a good thing. But analytical and architectural complexity does not go away - it just moves from the hardware level to the data and logic level. And that is precisely where real expertise is needed.
A simple test question
When someone proposes a project to "move to a cloud data warehouse," ask: what exactly are we going to do with the data after the warehouse is up? If there is no clear answer, you are being sold infrastructure, not a solution. Infrastructure without an answer to that question creates no value.