Blog / explainability

Explainability, Traceability, Observability: Three Words That Are Not Synonyms

As more of the security decision moves into software, the operative question stops being 'did we detect it' and becomes 'can I defend this conclusion'. Three different failure modes of trust, and why passing one tells you nothing about the other two.

Published

9 min

An incident review, six weeks after the fact. The detection fired, the response ran, the host was isolated inside a few minutes, and nobody argues the outcome was wrong. Then someone from risk asks a question the room has not prepared for: on what basis did the platform reach that conclusion, and how would we show that basis to a regulator?

The analyst who closed the case has left for another employer. The alert summary is still in the record, well written and plausible: the endpoint exhibited credential-access activity consistent with an information stealer. What it does not say is which observations produced that reading, whether the credential-access step was observed or inferred, or what the sensor missed during the twenty minutes in question.

Everyone in the room believes the conclusion. Nobody can defend it.

The same thing plays out in shift handoffs, and in MSSP handbacks where a client asks the provider to justify a containment action taken on their behalf. As the security decision moves into software, the burden of proof moves with it.

Three words get used for the answer, usually as though they were interchangeable. They are not. They answer three different questions, they fail in three different ways, and a platform can satisfy one completely while failing the other two.

Generated explanations solve one third of this

Almost every serious security platform now produces natural-language explanations. An alert arrives, a language model reads the surrounding context, and the analyst gets a paragraph describing what appears to have happened.

This is genuinely useful. It compresses triage, gives a junior analyst a starting hypothesis instead of a wall of fields, and makes an incident legible outside the SOC. Anyone calling generated explanations decoration has not worked a queue at 3am.

But notice what one is. A description produced after the detection, from whatever context was available when it was requested. An account of the conclusion, not a record of how the conclusion was formed. Ask which evidence supported the second clause of its third sentence and you are usually out of luck. Ask what the sensor failed to capture and you get silence, because absence rarely reaches the context window at all.

That is not a flaw in the technique. It is a scope boundary.

Explainability: why does the system believe this?

Explainability answers the question a human asks first. Something has been flagged. What is the reasoning?

When it is missing, you get a verdict with a score attached. Severity high, confidence 0.87, technique T1555. The analyst's job becomes reverse-engineering the tool's opinion rather than assessing the activity, and the fastest path to closure is to trust the number or ignore it. Both guesses.

When it is present, the reasoning is stated in terms of the activity rather than the detection machinery. Not "rule 4471 matched", but: a process running under a service account read a browser credential store it has never touched before, wrote an archive to a directory it does not normally write to, then opened an outbound connection to a destination first seen nine minutes earlier.

A design choice sits underneath. An explanation can be generated downstream of an opaque detection, or the representation the detection runs on can carry explanation-relevant structure in the first place. We built our platform the second way. Telemetry becomes structured behavior before detection, and that structure carries what an explanation needs: who or what acted, what it can do, what it did, what changed immediately, what remained, and how certain we are about each. Explanation then reads a structure that already exists.

That is an architectural fact, not an outcome claim. It does not by itself make analysts faster. It makes the explanation checkable.

Traceability: show me exactly where that belief came from

Explainability gives you a story. Traceability lets you verify it.

When it is missing, the explanation is unfalsifiable. It reads well and there is no way to test it. Six weeks later, at the incident review, that is exactly where the room found itself.

When it is present, every link in the chain can be walked in both directions:

Telemetry → Evidence → Behavior → Detection → Decision → Response

and, more usefully, backwards, with each step carrying its own question:

Response       Why was this action taken?
   ↑
Decision       Which conclusion justified it?
   ↑
Detection      Which behavior supported that conclusion?
   ↑
Behavior       Which evidence supports that behavior?
   ↑
Evidence       Which raw telemetry produced that evidence?
   ↑
Telemetry      What did the sensor actually emit, and when?

The reverse walk is the one that matters under scrutiny, and the one most architectures cannot perform, because the links were never materialised. Events were correlated into an alert and the correlation was never kept as a relationship. You can see the inputs and the output. The join is missing.

This is not audit logging

Say it plainly, because the word invites the wrong reading. An audit log records who clicked what and when. This analyst opened the case at 14:02, this playbook executed at 14:06, this host was isolated at 14:07. Necessary, universal, and not what is being described here.

Security reasoning traceability records which evidence supported which behavior, which behavior supported which detection, and which detection supported which decision. Provenance over the reasoning, not over the interface.

The difference shows up the moment someone challenges the conclusion rather than the process. An audit log proves the isolation was authorised and executed correctly. It says nothing about whether it was justified. Only one of those survives a question about the finding itself.

In our platform the chain is carried structurally: events have identifiers, behaviors reference the source events that produced them, and threat alerts reference the behaviors that support them. That provenance is exposed through the interface and the API, not merely reconstructable in principle.

Observability: what did we see, what did we infer, and what did we never see?

The third word is the one most likely to be misread, because engineering already owns it. Platform observability concerns metrics, traces and logs from your own systems. This is a different sense, and the more precise name for it is evidence awareness.

The question concerns the boundaries of what the platform knows. Which parts of a behavior were directly observed? Which were inferred from their effects? Which parts of the timeline are simply dark, because a sensor was missing, an agent was down, or an event was dropped under load?

When it is missing, absence of evidence gets read as evidence of absence. One of the most consistent failure modes in incident response, and it runs in both directions. A team concludes no lateral movement occurred when the truth is that none was visible. Another concludes exfiltration happened when what they hold is an archive being written and a gap where the network record should be.

When it is present, the platform marks those cases in the record itself, before anyone asks: this was directly observed, this is inferred from a connection record with no payload visibility, a four-minute gap exists in process telemetry on that host. FACE-EO, the research representation our platform is built on, treats observability as one of its dimensions rather than a caveat bolted on afterwards. Partial observation and telemetry loss become things the model can express, instead of things that quietly degrade into confident-looking output.

The same incident, through all three lenses

An illustrative case. A signed remote-management utility appears on a finance workstation, installs a service, reads a browser credential store, writes an archive to a temporary directory, and opens an outbound connection to a host first seen that morning.

Through the explainability lens. A tool with administrative capability, running where it has no operational reason to exist, performed credential access followed by staging and egress. The behaviors share an execution context and a cause-and-effect ordering, which is why they are one incident rather than three unrelated events.

Through the traceability lens. The credential-access behavior references the process and file-access events that produced it, the staging behavior references the file-write events, the detection references those behaviors by identity, and the containment decision references the detection. Walk it backwards from the isolation action and you land on named telemetry with timestamps, not on a summary.

Through the observability lens. The credential read and the archive write were directly observed. The egress is partially observed: a connection record exists, payload inspection does not, so the volume transferred is unknown rather than zero. Process telemetry has a gap between 09:14 and 09:18 during an agent restart, so the model carries the possibility of unobserved activity in that window instead of implying nothing happened.

Same incident, three answers to three questions. Remove any one and something specific breaks.

Why the three compound

They are not a hierarchy and they do not substitute for each other.

Explainability without traceability is a story you cannot check. It sounds right, it may well be right, and there is no procedure for establishing which. Fine for triage, inadequate for anything contested.

Traceability without observability is a chain that hides its own gaps. Every link resolves, every reference is intact, and the completeness of the chain quietly implies completeness of the evidence. A chain built from partial telemetry looks exactly like one built from complete telemetry, unless the representation can say otherwise.

Observability without explainability is honest but unusable. A careful account of what was and was not seen, with no reading of what it means, hands the analyst qualified facts and a judgment call.

Together they support a different kind of statement: here is what we believe, here is the chain that produced it, here is the boundary of what we could see.

This gets more important as autonomy increases, not less

The intuition runs the other way, so state it directly. As systems take on more of the decision, the volume of decisions rises and the number of humans reviewing each falls. That is the point of the shift. It also means the decisions that do get reviewed are reviewed under pressure, after something has gone wrong, by people who were not present when they were made.

Under those conditions a fluent summary is not evidence. What holds up is a structure that can be walked, questioned and bounded, by a person or by another system. The properties that let an analyst defend a conclusion at review are the ones that let an automated workflow judge whether it has grounds to act at all.

That is the argument behind how our platform is built. Telemetry becomes structured behavior first, and detection, investigation and response reason over that structure. Explanation is a reading of it, traceability is the ability to walk it, observability is the record of its edges.

Understand the behavior behind the threat, and you can defend what you concluded about it.

What we are not claiming

Precision about our own claims is part of the argument. We do not claim: