Prompt logs help me debug agents

I used to think of prompt logs as product analytics. Store what somebody asked, notice common requests, and improve the prompt later. That works while the model is mostly returning text. Once an agent starts retrieving documents and calling tools, the prompt becomes part of the incident record. So do the retrieved context, tool arguments, tool results, model version, prompt version, and all the quiet instructions my application inserted along the way.

Normal HTTP logs still tell me whether /agent/run returned 200, which is nice and almost useless when the agent confidently opened the wrong ticket. The route worked. Eight seconds of decisions happened inside it, and the status code has no idea which one went bad.

A healthy route can contain a bad run#

This log is useful for an ordinary API endpoint:

{
  "route": "/api/deploy",
  "status": 200,
  "duration_ms": 842
}

The equivalent agent log leaves most of my questions unanswered:

{
  "route": "/agent/run",
  "status": 200,
  "duration_ms": 8421
}

During those eight seconds, retrieval may have returned old docs, the model may have picked the wrong tool, or the tool may have failed in a way the model happily ignored. A system prompt could have changed yesterday. One field could be projectId in the rendered prompt and project_id in the schema. The HTTP layer is healthy through all of this, so I need a trace shaped around the run rather than the request.

For a simple run, trace tr_123 might contain these steps:

1. Received user request.
2. Rendered prompt template v4.
3. Retrieved docs.
4. Called the model.
5. Parsed a tool call.
6. Called github.search_issues.
7. Called the model again.
8. Returned the answer.

Every step needs timing and enough metadata to connect it to the next one. A few steps may need redacted text when metadata cannot explain the failure. For model calls, I want a small record shaped roughly like this:

{
  "model": "some-model-id",
  "prompt_version": "triage-v4",
  "input_tokens": 1820,
  "output_tokens": 410,
  "tool_choice": "github.search_issues"
}

This data feels like paperwork while everything works and becomes extremely interesting the moment somebody asks why the agent did a weird thing. I would rather collect it during the boring runs than discover I needed it after the bad one.

Reconstruct what the model saw#

A prompt template only tells part of the story. It may look harmless:

You are a release assistant.
Use this context:
{{context}}

The bug can live entirely inside {{context}}. Retrieval may have supplied stale docs, truncation may have removed the important paragraph, or an earlier message may have pushed a useful instruction out of the model’s view. I want a way to reconstruct what the model actually saw, but raw prompts do not belong in the normal log stream. They can contain source code, secrets, customer data, and very personal text pasted by accident. A restricted store with short retention gives me an escape hatch without turning every dashboard into a privacy incident.

Behavior can also move when the model, prompt, tool schema, retrieval index, or application changes. I log those versions together so yesterday’s good run and today’s strange one can be compared without guessing:

{
  "model": "provider/model-version",
  "prompt_template": "support-triage",
  "prompt_version": "v12",
  "tool_schema_version": "2026-02-22",
  "retrieval_index": "docs-1842",
  "app_commit": "abc123"
}

Most dashboards only need metadata like this:

{
  "event": "agent_step",
  "trace_id": "tr_123",
  "step": 4,
  "kind": "tool_call",
  "tool": "github.create_issue",
  "allowed": true,
  "duration_ms": 631,
  "result": "success"
}

Raw prompts and responses get much stricter treatment, similar to sensitive request bodies in any serious backend. Access stays limited, retention stays short, and opening one should have a reason. I want to explain a bad run without guessing while avoiding the cursed version where debugging convenience turns into storing everybody’s private text forever. If an agent can make decisions, the decision path belongs in observability, with the sensitive parts kept behind a much heavier door.