Event-driven architecture and data contracts: why data is no longer a by-product
Moving from synchronous integrations to event-driven architecture changes how data is treated. Data contracts become a first-class engineering artefact.
For a long time, data in systems was treated as a by-product. Business logic runs, records are written to the database, analytics will sort it out later. Integrations were built synchronously: one service calls another, gets a response, continues.
This model worked while there were only a handful of systems. When there are more than ten, synchronous integrations become a web of dependencies. Changing one service breaks another. One service going down stops the entire chain. Development slows - every change requires coordination between teams.
What event-driven architecture is
The answer to this problem is event-driven architecture. Instead of service A synchronously calling service B, service A publishes an event: "such-and-such happened." Everyone interested is subscribed and processes the event independently. Nobody waits for anyone else.
This changes the nature of dependencies. Service A does not know who will process its events. Service B does not depend on service A being available right now. Systems become more independent in their lifecycle.
In 2019 this architecture stops being exotic territory for large companies. Tools for streaming event processing - such as Apache Kafka - are mature enough, and they are starting to be used at mid-size companies. The conversation has shifted from "should we?" to "how do we do it properly?"
Why data contracts become critical
Here begins the main problem that is often underestimated when moving to event-driven architecture.
In the synchronous world, if service B changes its response format, service A notices immediately - the integration breaks visibly. The error can be found and fixed.
In the event-driven world the situation is different. Service A publishes events to a stream. Service B reads that stream. If service A changes the event structure - a field is renamed, a required field is added, a type is changed - service B does not know. It continues reading the stream and silently processes incorrect data. The error does not appear immediately and does not appear where the change was made.
A data contract is an explicitly recorded agreement about what an event looks like: which fields, what types, what is required, what is optional, what each field means semantically. This is not documentation for its own sake. It is an engineering artefact that protects against silent failures.
What changes for teams
Moving to event-driven architecture with data contracts changes how teams work.
Teams start thinking about the data they produce as a public interface. Just as an API cannot be broken without warning, the format of an event cannot be changed arbitrarily. Versioning is needed, along with backward compatibility or an explicit migration path.
This requires a discipline that not all teams are ready for straight away. But it is precisely this discipline that makes event-driven architecture work, rather than being a source of another category of non-obvious errors.
Questions for assessing maturity
If you are considering a move to event-driven architecture or are already using it:
- Are event formats explicitly documented - or does each consumer figure them out from the producer's source code?
- Is there a process for versioning events when their structure changes?
- Who is responsible for backward compatibility when a contract changes?
- How are event processing errors discovered - in real time, or after bad data has accumulated?
- Do event consumers have the ability to replay history when an error is discovered?
Event-driven architecture is not just a technical choice. It is a decision about how teams agree on data. And it is exactly that part which determines whether the architecture works or creates a new set of problems.