A run that can explain itself

For years, “explainable software” meant source code, logs, and perhaps an audit table. A deterministic program could still be difficult to understand, but the intended path from input to output was written somewhere.

Agentic software complicates that path. A model interprets a goal, selects tools, reads changing information, revises a plan, and may act outside the application. The exact wording of its reasoning is neither a stable program nor reliable evidence. A successful final answer can hide a dangerous route. A failed answer can follow every policy correctly and stop for the right reason.

N17Q led me to a practical definition: software can explain its work when a person can reconstruct what it knew, what authority it had, what it attempted, what changed, where uncertainty remained, and how to intervene.

That definition is an engineering contract, not a request for the model to narrate confidently.

Preserve evidence, not hidden thought

I did not need private model reasoning to operate a dependable workflow. I needed the observable evidence admitted into decisions.

For each run, N17Q recorded:

• The user goal and constraints.
• The policy and tool catalogue in force.
• Resource identities, versions, and digests.
• Tool requests, validated arguments, and receipts.
• Structured decisions such as selected plan, evidence admission, and action request.
• Human approvals and modifications.
• State transitions, retries, budgets, and terminal outcome.

Model-generated summaries could help a reviewer navigate this record, but they were not the record. A summary that said “tests passed” linked to the test receipt, workspace revision, and scope. If the receipt showed only a targeted test, the interface did not promote it to universal success.

Explanation began with durable facts.

Authority should be visible before action

Many agent interfaces emphasized what the system could do and hid why it was currently allowed to do it.

N17Q made authority part of the action preview. A tool call named the capability grant, workflow state, user or policy that granted it, destination, and approval requirement. The model could select only from tools already admitted to the current state.

I used narrow capabilities:

• Read a named repository.
• Write within one sandbox workspace.
• Run an approved command family with limits.
• Propose a pull request.
• Publish only after scoped evidence and human approval.

“Repository access” was too broad to reason about. Read, local write, external publish, secret access, and administrative change carried different consequences.

The interface showed these boundaries before execution, not only afterward in an audit log. Control is most useful while a decision can still change.

Uncertainty needs durable state

Earlier systems had repeatedly taught the same lesson. T04P distinguished connected from current. X8B6 distinguished pending from conflicted. Z29C distinguished failure from unknown outcome. K81R distinguished fluent synthesis from answerable evidence.

Agents concentrate all of those uncertainties.

A tool timeout might leave an external action running. A retrieved document might be stale. Two sources might disagree. A model might produce a plan within policy but beyond the remaining cost budget. A human approval might refer to a superseded revision.

N17Q did not compress these into success and failure. Terminal and intermediate states included:

completed
completed-with-unverified-output
blocked-by-policy
blocked-by-missing-evidence
awaiting-approval
outcome-unknown
budget-exhausted
superseded
failed

These states were not primarily for observability dashboards. They determined safe recovery. An outcome-unknown external action could not be retried like a command confirmed not to have started.

An explainable run needs more than a chronological log. Reviewers need to test whether a different policy, model, or tool result would have changed the outcome.

N17Q stored tool outputs by digest and represented external effects with receipts. A recorded run could be replayed in a sandbox using fixed resources and simulated side effects. I could evaluate a new model against the same conditions or step through the original state transitions.

Z29C had limits. A live website, nondeterministic service, or human judgment could not be perfectly reconstructed. The trace named those gaps instead of filling them with fabricated fixtures. Reproducibility was scoped evidence, not a universal claim.

This made incident review more productive. “The agent behaved strangely” could become “the stale search result was admitted because the freshness policy lacked a maximum age.” The remedy might be retrieval policy, tool metadata, orchestration, or interface—not another paragraph in the prompt.

Agent runs consume time, model tokens, tool calls, external API quota, and human attention. Unbounded effort can look like persistence while quietly becoming a denial-of-service mechanism.

Each N17Q workflow declared budgets by resource and risk. Reading another file was cheap. Executing untrusted code required a sandbox and time limit. Repeating a failed external action required receipt analysis. Requesting a person's approval consumed attention and could not be used as the default branch for every ambiguity.

When a budget ended, the agent produced a handoff containing achieved state, admitted evidence, unresolved questions, and the smallest useful next decision. It did not hide exhaustion behind a generic error or continue with reduced scrutiny.

Budgets forced product teams to decide what mattered. A workflow optimized only for task completion would always ask for more. A responsible product also optimized for bounded consequence.

Intervention is a state transition

“Human in the loop” often meant a modal with Approve and Reject. Real intervention included changing scope, correcting evidence, removing a tool, editing a plan, supplying missing context, or stopping remaining work while preserving completed results.

N17Q represented intervention as a new event and state transition. The original proposal remained visible. The human change gained authorship and rationale. Subsequent actions used the new revision, making approval invalidation automatic when the underlying plan changed.

This protected both people and the system. A reviewer did not become the invisible author of everything the agent had done. The trace showed which decision belonged to whom.

The product also made stopping safe. Canceling a run prevented future steps but did not claim to undo completed external actions. The interface listed those actions and available compensations.

One universal trace view made everyone work too hard. The person who requested a report wanted to know whether it was complete and what remained. A reviewer needed artifact changes, evidence, and consequence. An operator needed attempts, clock domains, connection state, and recovery. A maintainer needed schemas and reducer versions.

N17Q derived these views from the same durable objects. They differed in detail, language, and eligible data; they did not tell different stories. A completed effect in the user view linked to the same intent and receipt an operator inspected. A protected policy reason remained hidden without turning the denial into a vague error.

Explanation became projection with controlled scope, not one enormous log labeled transparent.

Identity is the grammar of explanation

Every useful sentence in a run account needed a stable subject.

Which workspace changed? Which artifact passed review? Which consequence did three adapter attempts pursue? Which source supported the claim? Which approval applied to which bytes? Which model saw which evidence package?

N17Q assigned identities before transitions that could introduce ambiguity. Proposals, normalized candidates, effect intents, attempts, receipts, snapshots, checks, decisions, and account revisions stayed distinct and linked.

This sounded like bookkeeping until the first lost response. Without semantic effect identity, “retry” could mean another transport attempt or another external consequence. Without snapshot identity, “tests passed” could refer to yesterday's bytes. Explanation became possible because the nouns stopped moving underneath the verbs.

An agent trace records what the system proposed, invoked, and observed. It does not automatically prove what happened outside it.

N17Q's fixtures maintained a simulated world with resources and events independent of adapter responses. Live-like connections supplied receipts and status observations with explicit limits. Workspace snapshots measured local state independently of command summaries.

Evaluation compared trace, product ledger, and observed world. A provider could report success while the expected resource was absent. A model could claim failure while the resource existed after a lost response. The contradiction stayed visible.

Software cannot explain its work by quoting only itself. It needs evidence from the boundary it claims to have changed.

Explanation begins before execution

An audit log is late. By the time it reveals an overly broad destination or unrecoverable effect, the product may have crossed the only boundary where a different choice was possible.

N17Q used the same explanatory state to assemble previews: normalized consequence, exact artifact, current authority, audience, data flow, postconditions, uncertainty, and recovery limits. The person could inspect what the product believed it was about to do.

Approval then bound to that representation. If a material fact changed, the preview became stale and execution stopped.

The most valuable explanation was prospective. It improved the decision instead of merely documenting the aftermath.

Explanations must survive model changes

A long task could compact context, switch providers, restart workers, or outlive the process that began it. If its account depended on one model remembering a coherent story, continuity was fragile.

N17Q stored product state outside provider sessions and recompiled context from it. The successor model received a bounded narrative plus exact state for unresolved effects, approvals, budgets, candidate revision, and verification. It could reconsider a plan without receiving fresh authority.

Every model segment remained attributable in the trace. The user still saw one task organized around goal and artifacts.

Durability turned explanation from conversational memory into an application property.

Explanations need a ceiling

Evidence did not make every conclusion certain. A source might be stale, a receipt incomplete, a policy input unavailable, or a provider's behavior undocumented.

N17Q represented not assessable, outcome unknown, contradictory, and partially observed as first-class states. The final account could explain why one result seemed likely, but deterministic status did not rise above the strongest receipt or source.

Model graders received evidence packages and could cite, qualify, or abstain. They could not turn plausibility into a completed effect or passing invariant.

An accountable explanation includes what the system cannot establish. Certainty is not a writing style.

A successful response is transport evidence. A receipt is a product-shaped account of what the response and later observations support.

N17Q receipts bound effect identity, normalized consequence, connection, provider reference, observation time, postcondition, and residual uncertainty. Local commands produced artifact and test receipts tied to sealed snapshots. Reads produced source observations with scope and completeness.

Downstream steps declared which receipt strength they required. A notification could not claim a resource existed while the create intent remained merely accepted or unknown.

This created a chain from request to consequence to final prose. When one link was missing, the workflow paused instead of asking language to bridge the gap.

A stop condition is an explanation

Capable systems are often judged by how many obstacles they can route around. N17Q became safer when it could say exactly why another attempt would not add evidence or would risk another consequence.

Unknown outcomes had query budgets and retention windows. Repeated denials had semantic progress limits. Missing fixtures failed closed. Exhausted policy evidence produced indeterminate state. Long-running tasks stopped when authority changed.

The handoff preserved work, current state, relevant receipts, unavailable actions, and the smallest useful next decision. Stopping did not flatten the run into failure.

It showed the boundary the system had reached and protected everything already learned on the way there.

Recovery belongs in the original explanation

The first approval views emphasized intended result and added an Undo label where a provider exposed deletion. That was reassuring and usually incomplete.

N17Q explained whether interruption could prevent execution, how outcome would be observed, how long provider identity remained useful, what compensation could change, what residual exposure would remain, and who had recovery authority. These facts came from versioned, tested contracts.

When something went wrong, the operator saw the same plan the reviewer had considered. Compensation became another effect with its own authority and receipt; it did not erase history.

Explaining recovery before action made consent more informed and incidents less improvisational.

Data can be perfectly captured and effectively hidden. A green score above a hard safety failure, an alphabetical diff that buries a deleted test, or a spinner for unknown outcome changes how people interpret state.

N17Q organized views by consequence and decision. Validity preceded qualitative scores. Verification-sensitive files received hierarchy. Current state was distinct from historical attempts. Critical limitations appeared before approval controls and in the accessible reading order.

Raw evidence remained available beneath derived views. The interface did not replace it; it helped a person allocate attention.

Information architecture became an operational control because understanding shaped the next action.

Explanation must be testable

I could not trust the architecture merely because its event model looked rigorous. Fixtures deliberately lost responses, delayed visibility, changed policy, exhausted budgets, removed tests, compacted context, switched models, and contradicted receipts.

Hard invariants inspected the trace and simulated world. Qualitative graders reviewed explanation from evidence packages. Account checks compared final claims with artifacts, effects, and verification. The test harness accepted multiple safe trajectories and rejected polished invalid ones.

Historical incidents became minimized scenarios. Contract drift produced new cases. The system had to earn the same explanatory claim after every change.

Explainability moved from aspiration to regression surface.

Capturing everything forever would make the system expensive and invasive. Tool results could contain personal data, repository content, provider diagnostics, and reusable identifiers.

N17Q stored minimal normalized facts, bounded encrypted evidence where justified, content digests, provenance, and retention classes. Large artifacts and raw payloads could expire while tombstones preserved what was deleted and which claims became unassessable. Views projected data by role and purpose.

This meant an old run could eventually lose detail. The account said so. It never filled the gap with a generated reconstruction and called it history.

Responsible explanation includes selective, declared forgetting.

Explanation should be composable

A high-level task could contain research, transformation, verification, review, and delivery. Each subworkflow produced its own artifacts and receipts. Flattening them into one completion flag hid which boundary had failed.

N17Q composed outcome accounts through declared predicates. A verified report could exist while delivery remained blocked. A retrieved source could be fresh while one claim stayed unsupported. Parent status depended on exact child outcomes, not the confidence of their summaries.

This let small, well-explained components form a larger accountable workflow. It also localized repair: rerun the stale verification or reconcile the unknown effect without repeating completed research.

Composability turned explanation into an interface between parts of the system.

The measure of a trace was not how impressively it described the past. It was whether someone could make a better next decision.

N17Q reports paired every blocker or uncertainty with eligible actions: inspect a source, narrow scope, rematerialize an artifact, approve exact consequence, query an existing intent, preserve work, compensate, or stop. Unavailable actions stayed visible only where understanding required them and could not be invoked.

This avoided two extremes: a technical dump with no path forward and a polished recommendation detached from evidence. The explanation showed why each option was safe now.

Accountability became operational when it helped a person intervene without reconstructing the entire run.

The final answer became a view, not a verdict

At finalization, N17Q sealed local state and assembled a deterministic outcome account from task predicates, artifacts, checks, effects, denials, and unresolved evidence. The model received that account and wrote a concise explanation.

Claim checks prevented prose from exceeding the evidence. Coverage checks prevented material omissions. If generation failed, a plain renderer still produced the state. Late receipts created a new account revision and marked the earlier narrative superseded rather than silently editing it.

The assistant remained the best component for synthesis and human language. It stopped being the sole authority on whether its work had succeeded.

That division made both the prose and the product more dependable.

Explanation reveals where design debt lives

When a run could not answer a simple operator question, the missing explanation often exposed a deeper architectural gap.

If I could not say whether an action had happened, effect identity or receipt semantics were weak. If I could not say which bytes passed, snapshots and verification links were missing. If I could not say why a capability was offered, policy and catalogue provenance were incomplete. If I could not replay safely, environment boundaries leaked.

The repair was rarely “log more.” It was to represent the missing relationship in product state and make the corresponding transition enforceable.

Explanation became a way to discover what the software did not yet know about itself.

The project grew by narrowing its claims

N17Q began as a personal exploration of agent traces. Each difficult case made the design more conservative: distinguish proposal from action, intent from attempt, transcript from run, approval from authority, accepted from completed, cancelled from unknown, and narrative from account.

Those distinctions added states and reduced ambiguity. They also made the system more capable in a practical sense. Work could pause and resume, change models, move environments, recover from lost responses, and offer useful handoffs without repeating consequences.

Maturity did not come from claiming the system could handle everything. It came from knowing which claim each piece of evidence supported and stopping at that boundary.

It is tempting to add auditability after the workflow works. By then, important identities have been discarded, tool outputs overwritten, and policy decisions reduced to booleans.

Designing for explanation changed N17Q at the foundation:

• Every intent, attempt, workspace, tool call, approval, and external action had a stable identity.
• Immutable events preserved sequence; projections served the interface.
• Content digests linked claims to exact resource versions.
• Policy results carried reasons and versions.
• Side effects returned receipts.
• Retries formed lineage rather than impersonating the original attempt.
• UI states followed the same model operators used during recovery.

These choices cost storage and implementation effort. They reduced the much larger cost of operating capable software through guesswork.

Capability is not maturity

The industry's attention naturally follows what models can do next. By 2026, agent tooling made it increasingly practical to give models files, shells, network access, reusable skills, and long-running context. Capability expanded faster than many products' ability to explain failure.

Mature software does not act at every opportunity. It knows the boundary of its evidence, the boundary of its authority, and the recovery semantics of its actions. It can stop without losing its work and continue without repeating consequences.

The through-line from a responsive catalogue in 2012 to N17Q was not a particular framework. It was a widening definition of interface. Layout rules explained how content adapted to a screen. Freshness explained whether a dashboard described the present. Receipts explained whether an operation had happened. Evidence explained whether an answer deserved belief.

Software that can explain its work extends that craft to systems capable of acting on our behalf. The explanation is not an apology after something goes wrong. It is part of how the system earns permission to act at all.