Real-time analytics: when it works and when it is expensive theatre
Streaming data and real-time dashboards have become a fashionable requirement. I look at when this actually solves a real problem.
In almost every conversation about a new analytics system, at some point someone asks: "can we do this in real time?" Sometimes this is a genuine business requirement. More often it is a signal that someone saw an impressive dashboard and wants one like it.
The difference matters, because real-time analytics costs significantly more to build and maintain than analytics with a delay. Before making that decision it is worth understanding exactly what you are paying for.
What the real difference is
Analytics with a delay - batch processing - means: data is collected over a period, processed, and results are available after an hour, overnight, or after a day. It is predictable, inexpensive to maintain, and simple to debug.
Real-time analytics - streaming - means: data is processed as it arrives, and results are available within seconds or minutes. This requires streaming infrastructure, a separate processing architecture, and a different approach to errors and reprocessing.
The cost difference is a multiple. The complexity difference in operations is also a multiple.
When real time is justified
There are two types of situations where latency actually costs money.
The first type - operational decisions. When the result of the analytics feeds a decision that must be made right now, and any delay makes that decision pointless. Examples: fraud detection at the moment of a transaction, real-time load balancing, production equipment monitoring with immediate response.
The second type - competitive advantage. When the speed of receiving information is itself a source of value for the customer or for operations. Trading platforms, marketplaces with dynamic pricing, operational control centres.
In both cases there is a concrete answer to the question: what exactly stops working if data arrives with a 15-minute delay? If there is no answer - real time is probably not needed.
Where it becomes expensive theatre
I see real-time analytics as decoration when:
- a dashboard updates every minute but is looked at once a day;
- data arrives in real time but decisions are made based on weekly summaries;
- the streaming system is maintained by one person who cannot be replaced;
- the system is expensive to run but nobody can name a specific decision that it actually accelerated.
In these cases a 30-minute or hour delay would have changed nothing - except the cost of the system.
How to make the decision
Three questions that help choose the right approach:
- What specific decision is made from this data, and how quickly must it be made?
- What happens if the delay is 15 minutes? An hour? A day? At what threshold does latency start costing money?
- Who will maintain the streaming infrastructure if we build it, and how?
If the first question produced a clear answer with concrete numbers - streaming is justified. If not - it is worth starting with batch processing, which can be improved later, rather than overpaying for speed that is not needed.
Most analytics tasks in an ordinary business are handled perfectly well with a delay. This is not a compromise. It is the right tool for most tasks.