Streaming vs batch: when real-time is justified and when nightly batch is the honest choice

Whenever the conversation turns to data processing architecture, the word "streaming" appears fairly quickly. It sounds serious, it is associated with sophisticated systems, and engineers often lean toward it by default.

I have nothing against stream processing. But I have seen enough projects where streaming was chosen not because the business pays for low latency, but because it felt like the right architectural decision. That distinction matters.

What actually differs

Batch processing accumulates events over a period and processes them together. A nightly report recalculation, a morning export to an analytics system, a weekly metrics run - classic examples. Simple operational model, predictable load, understandable failure behaviour.

Stream processing handles each event as it arrives, with minimal latency. Data moves continuously, and the system responds in real time or close to it. The question of when a business actually needs near-real-time data and when 15-minute intervals are fine is the same question framed from the business side rather than the architecture side.

Technically, the second approach is more complex: you have to think about event ordering, about what to do with late-arriving data, about system state between events, about delivery guarantees. Operating such a system is more expensive and more demanding.

The central question: who pays for latency

Before choosing an architecture, I usually ask one question: what specifically goes wrong in the business if data arrives with a one-hour delay? A four-hour delay? The next morning?

If the answer is "the analyst gets the report at lunch instead of in the morning" - latency costs nothing. Batch is perfectly adequate, and there is no reason to pay with complexity for speed that nobody needs.

If the answer is "we will not catch a fraudulent transaction in time" or "the customer will see a wrong balance at the moment of payment" - latency costs real money. Here streaming is justified, and the infrastructure complexity is paid for by a concrete business value.

The problem is that this question is rarely asked. Instead, people discuss technologies. The argument that faster computation changes something for the business is real - but the key word is "changes something": you still have to identify what.

When batch is the honest choice

Most analytics tasks in mid-sized companies do not require real-time data. Financial reports, sales metrics, operational indicators, marketing analytics - all of this works on data with a delay of a few hours, and that is fine.

Batch here is not an "outdated" approach. It is an approach with a simpler operational model, more predictable load, and more understandable failure behaviour. If the system went down overnight and did not run - that is visible and clear. If a streaming pipeline quietly starts dropping events - that can go unnoticed for a long time.

The rule is simple: if data with a few-hour delay does not break the business process - batch is more honest and cheaper to maintain.

When streaming is justified

Streaming makes sense where latency creates a real problem for the business or the customer.

A few signs that latency actually costs money:

• a decision must be made before the user's or system's next action;
• an event loses value if processed more than a few minutes later;
• accumulating events without processing creates a risk - financial, operational, or reputational;
• the user sees the consequences of the delay and reacts to it.

Fraud detection, real-time inventory management, production line monitoring, in-session personalisation - streaming solves a concrete problem in these cases.

A hybrid approach is often more honest than picking one

In practice, many systems use both approaches depending on the task. Transactions are processed in real time for validation and execution, but aggregates and reports are built in batch - because there is no need to see them immediately.

This is not architectural inconsistency. It is a sensible distribution of complexity to the tasks where that complexity is warranted.

A practical filter

Before choosing a data processing architecture, answer four questions:

1. What specifically breaks in the business process if data arrives with a one-hour delay?
2. Who - an employee or a customer - suffers from that delay, and exactly how?
3. Does the team have experience operating streaming systems, or would this be the first time?
4. What is the total cost of ownership of a streaming system compared with batch, including people and incidents?

If the first two questions have no concrete answer - the architecture choice is premature. First, understand what the business is actually paying for.