Near real-time: when a business actually needs streaming, and when 15 minutes is fine
Not every online analytics setup makes economic sense. A look at when streaming data is justified and when it is an expensive illusion of speed.
A few times a year I hear roughly the same thing from clients: "we need real-time." Sometimes this is an accurate description of a real requirement. More often it is a slogan covering something else - a desire to see fresh numbers, a sense of control, or admiration for how large online platforms operate.
Before designing any streaming pipeline, I always ask one question: what exactly changes in the decisions or actions taken if data refreshes every second instead of every fifteen minutes?
Where latency actually costs money
There are classes of problems where every second of delay has a measurable price:
- fraud detection on payment transactions: blocking at 200 ms versus 15 minutes is a fundamentally different outcome;
- equipment monitoring on a production floor, where a failure needs to be caught before it becomes an incident;
- algorithmic trading, where millisecond order matters directly to the bottom line.
That is a short list. And notice: in every case there is not just "we want to see numbers faster" - there is a specific action that must happen immediately.
Where a 15-minute delay is not a compromise but the norm
Most management tasks live in a different time frame:
- today's sales dashboard: an executive looks at it a few times a day, and three-minute-old data changes nothing;
- warehouse stock report: half-hourly updates are sufficient for most operations;
- CRM funnel analytics: yesterday morning's data is normal, not a limitation.
Here a 15-minute or even hourly delay is not a weakness of the system. It is an appropriate answer to the actual need.
What the business pays for genuine real-time
Streaming architecture is not just a different tool - it is a different project economics. The batch model it replaces has its own failure modes and operational logic:
- more complex infrastructure: you need components that process data in motion, not in batches;
- harder to maintain data quality: errors and duplicates propagate instantly;
- higher monitoring demands: a broken pipeline goes unnoticed until someone looks at a number and senses something is wrong;
- more expensive to support: specialists are fewer and cost more.
None of that is an argument against streaming. It is an argument for an honest calculation: is the speed gain worth the costs that follow?
The middle option that gets overlooked
Between "nightly batch" and "real-time stream" there is a wide space - mini-batches, 5-15 minute refresh cycles, event-driven delivery without a heavy streaming stack. For most problems the optimum lives right there: data fresh enough, architecture still manageable.
I have seen projects where "moving to real-time" meant shifting from a daily batch to one every ten minutes. That was the right call and it solved the problem - without rewriting the entire infrastructure.
Questions to answer before choosing an architecture
Before framing a task as a streaming problem, it is worth working through:
- What decision cannot be made with 15-minute-old data but can be made with second-old data?
- Who specifically will react to a change in the data - a person or an automated system?
- If an automated system - do we already have the reaction logic built, or does that need to be built too?
- How will we know when the stream breaks and data stops arriving?
- Who will maintain this system in a year, after the person who built it has moved on?
If there is no concrete answer to the first question, the task almost certainly does not need streaming architecture. And that is good news: it means it can be done faster, cheaper, and more reliably.