Accident investigation needs a digital trace: what to capture before it happens

In the air gap illusion with AI inside I agreed with an industry author's observation that IT and security specialists are not mandatory members of the commissions that investigate accidents in Russia's power sector. Since 1 January 2026 the 2009 decree he referred to has been replaced by government decree 1489 of 29 September 2025. I read the new rules with one question: did a computer cause of an accident find a place in them. The short answer is no.

The rules became broader. Incidents of two categories are now investigated alongside accidents. The commission must preserve oscillograms of the fault, event recorder files, parameter files and recordings of operator conversations. Electronic materials are kept for at least seven years; the investigation takes 20 days, extendable to 45 or 90. Among the causes a commission must consider are failures of dispatch and process control systems and incorrect operation of protection and automation. But a cyber attack, malware, an altered configuration, tampered data at the input of the control system are not mentioned. An IT or security specialist is still not a mandatory member of the commission.

This piece is not about the power sector. It is about any plant where a production line has a control system, and the control system has a log, a configuration and, increasingly, a model. Readers outside Russia will recognise the pattern from their own sector rules: accident investigation procedures written for mechanical and electrical causes, decades before software became the cause. The question is the same everywhere: what has to be preserved at the moment of the accident so that the investigation finds the cause rather than appointing the culprit.

How an investigation closes on the operator

The pattern I have seen more than once goes like this. The line stops or breaks. A commission assembles, looks at the oscillograms and the conversation recordings, interviews the operator. The operator confirms seeing a reading on the screen and acting by the procedure, or not acting because the reading looked normal. The commission records: human error or equipment failure. Corrective measures: additional training, replacement of the unit.

What the pattern does not consider. Why the reading on the screen looked the way it did. Whether it matched what came from the sensor. Whether the controller's configuration changed in the last month and who changed it. Whether a vendor's remote access session was active at the moment of the accident. Which firmware version the controller carried and whether it matches the one in the passport. If a predictive maintenance or optimisation model sits on the line, what recommendation it issued and on what inputs.

None of these questions is asked, and not because the commission is incompetent. They are not asked because by the time of the investigation there is nothing left to answer them with. The controller's log wrapped around after two days. Nobody captured the configuration, and the current one is no longer the one from before the accident, because the unit was replaced. The remote session was not logged. The model retrained on schedule the night after the accident, and its state at the moment of the decision was not kept.

I wrote about this from the other side in 2012, in who decides in the first 30 minutes of a failure: in the first half hour everyone is busy restoring, and the evidence is destroyed by the restoration itself. NotPetya in 2017 showed the same effect at corporate scale: a cyber attack turned into pure operational loss precisely because it went unrecognised as an attack for so long.

Three layers of trace a commission needs

If the purpose of an investigation is to find the cause rather than the culprit, three layers of data must be available at the moment of the accident. Decree 1489 covers only the first.

Layer one: what happened to the equipment. Oscillograms, event recordings, operating parameters, protection trips, conversation recordings. This is what the power sector has stored for decades, and the new rules made it a requirement. At a plant outside the power sector the equivalent lives in the SCADA historian and is usually kept.

Layer two: what state the control system was in. The configuration of controllers and SCADA at the moment of the accident and the history of its changes. Firmware versions with checksums. Event logs of controllers, engineering workstations and SCADA servers, including logins, set-point changes, project downloads. The remote access log: who, when, from which address, doing what. Network logs at the boundary of the process segment. This layer is not mentioned in the rules, and it is exactly what separates "equipment failure" from "somebody changed a set-point on Friday evening".

Layer three: what the decisions were based on. For the operator that is the screen as seen, meaning the history of values on the HMI, not only from the sensors. For a model it is the model version, the data set it was trained on, the inputs at the moment of the decision and the decision itself with its rationale. For an agent it is the pair of intention and action and the log of tool calls I described in observability for a swarm of AI agents. This layer is needed to tell human error from model error, and model error from tampered data at the model's input.

The difference between the layers is fundamental. The first answers "what broke". The second answers "what changed before that". The third answers "why the decision was what it was". A commission with only the first layer will always arrive at equipment failure or human error, because no other cause is visible to it.

What the regulator already requires, even where decree 1489 is silent

A gap in the investigation rules does not mean a gap in requirements. Three documents close it from other directions, and what you store is worth checking against them.

FSB order 547 of 25 December 2025, in force since 30 January 2026, sets the procedure for reporting computer incidents at critical infrastructure objects to the national coordination centre. Notification within a day, faster for significant objects. To report an incident you first have to see it, and to answer the centre's follow-up questions you have to describe it: what, when, at which object, with which indicators. That cannot be done from the "what broke" layer.

The FSTEC methodological document of April 2026, which I covered in section 3.18 and AI in the loop, requires logs to be kept for at least a year, every prompt and response of an AI system to be logged, and the integrity of configurations, firmware and model weights to be controlled. Formally for state systems, by direction for critical infrastructure. These are layers two and three, written down as measures.

Article 274.1 of the Criminal Code, which I covered in 187-FZ nine years on, applies to a company's own staff for breaches of operating rules that caused harm. When the trace was not preserved, the operator is the only person whose actions are documented, in the conversation recordings. Preserving layers two and three protects not only the investigation but the people who worked the line.

What to do before the accident: the practical minimum

The list below is what I propose to plants with a production line and five people in IT. It does not require a SIEM or a staff forensic analyst.

1. Scheduled configuration snapshots. Once a day, automatically, an export of controller configurations, SCADA projects and set-points into a store with checksums and version history. Any difference between two snapshots is an event that must land in the log with who changed it and when. For most platforms that is a script and a repository, not a product.

2. A firmware register with hashes. For every controller: firmware version, checksum, installation date, who installed it. A match check once a month and after any vendor maintenance. This closes the question "is the firmware the one in the passport".

3. Logs sent out and kept long. Events from controllers, engineering workstations and SCADA go to a separate log server outside the process segment, where the ring buffer will not overwrite them and restoration will not erase them. Retention one year; for critical infrastructure objects check the requirements for your category. The remote access log goes there too: every vendor or contractor session with start, end, address and list of actions.

4. Screen history, not just sensor history. The historian records the values the operator saw on the HMI, including calculated and filtered ones, with the same timestamps as the raw readings. A discrepancy between the two is the main piece of evidence in a telemetry tampering case.

5. The model's state at the moment of the decision. If a model sits on the line, every decision that reaches the operator or a set-point is stored with the model version, the training set identifier, the input vector and the output. Retraining does not overwrite previous versions; they are kept with dates. For agents, the tool call log with inputs and outputs.

6. A "freeze" button. A procedure for the first thirty minutes after an accident: who presses it, what exactly is frozen. The snapshot of configurations and logs is taken before restoration begins, not after. That is one line in the response plan and one person who knows it is theirs.

7. A security specialist on the commission by internal regulation. Decree 1489 does not require it, but nothing stops you adding one by local policy. At a plant without its own security staff that is an external specialist under contract, called by the same procedure as the vendor's representative.

Steps one to three are two or three months of work for the IT department with outside help at the start. Four and five depend on whether a model sits on the line. Six and seven are paper and one meeting.

Who signs

A separate question that gets lost in technical discussions: whose data is this, and who is responsible for it existing. If IT owns the configuration snapshots, process engineers own the historian, the security service owns the remote access log and the contractor who deployed the model owns the model, then at the moment of the accident nobody is responsible, and each of the four will point at another.

The working arrangement I propose: one owner of the digital trace at the enterprise, usually the chief engineer or the production director, because the trace protects their line and their people. They do not keep the logs themselves, but they sign the regulation that states what is kept, where, for how long and who checks. Once a quarter they receive a report: snapshots are taken, hashes match, logs arrive, the model is versioned. It is the same logic as in who makes the decisions in security incident response: responsibility must be assigned before the event, not during it.

Honest caveats

• I read decree 1489 as an engineer, not as a lawyer. The composition of commissions and the list of causes come from the text and from reviews, and if the energy ministry's sector documents add IT competence requirements for commissions, I will gladly acknowledge it. At the time of writing I did not find them.

• A digital trace does not replace the investigation. It makes it possible. A commission with all three layers of data may conclude "human error", and that will be a justified conclusion rather than the only one available.

• Storing everything is expensive; storing what matters is cheap. I am not proposing full packet capture of the process segment. Configuration snapshots, firmware hashes, event logs and screen history take gigabytes a year, not terabytes. It becomes expensive when it is unplanned.

• For categorised critical infrastructure objects the list is longer. There the regulator sets the logging and reporting requirements, and this minimum has to be checked against them, not substituted for them.

In short

• Since 1 January 2026 accidents and incidents in Russia's power sector are investigated under decree 1489. It requires oscillograms and conversation recordings to be kept, but a security specialist on the commission and a cyber attack among the causes are not in it.
• An investigation closes on the operator not because the commission is incompetent but because at the moment of the accident there is nothing to answer "what changed" and "why the decision was what it was".
• Three layers of trace are needed: what happened to the equipment, what state the control system was in, what people and models based their decisions on.
• Layers two and three are already required by FSB order 547 and the FSTEC methodology, if from other directions. Article 274.1 of the Criminal Code makes preserving them a protection for your own people.
• The practical minimum: configuration snapshots, a firmware register with hashes, logs sent out and kept for a year, screen history, the model's state at the moment of the decision, a freeze procedure, a security specialist on the commission by internal regulation.
• One owner of the digital trace at the enterprise, usually the chief engineer, signs the regulation and receives a quarterly report.

If you have had an accident where it never became clear why the operator saw what they saw, or you want the next one investigated from data rather than testimony, that is exactly the task I start with. Feel free to get in touch; the first conversation commits you to nothing.