Prompt logs are observability data

I used to think prompt logs were mostly product analytics.

You store what users asked, see what they want, and improve the prompt. Normal product stuff.

But for agents, prompt logs are closer to request traces.

If the agent makes a weird decision, the prompt is part of what happened. The tool calls are part of what happened. The retrieved context is part of what happened too.

Normal HTTP logs miss that.

the normal log is too thin#

This is useful for a normal API:

{
  "route": "/api/deploy",
  "status": 200,
  "duration_ms": 842
}

For an agent, this log is too thin:

{
  "route": "/agent/run",
  "status": 200,
  "duration_ms": 8421
}

What happened inside those 8 seconds?

Maybe retrieval returned stale docs. Maybe the model picked the wrong tool. Maybe the tool failed and the model ignored it. Maybe the system prompt changed yesterday. Maybe one field was named projectId but the tool wanted project_id.

The wrapper route can be healthy while the agent is doing nonsense inside it.

I want a small trace#

For a simple agent run, I would like to see something like:

trace tr_123
  receive user request
  render prompt template v4
  retrieve docs
  call model
  parse tool call
  call tool github.search_issues
  call model again
  return answer

Each step should have timing. Some steps should have metadata. Some may need redacted text.

For model calls, I would log things like:

{
  "model": "some-model-id",
  "prompt_version": "triage-v4",
  "input_tokens": 1820,
  "output_tokens": 410,
  "tool_choice": "github.search_issues"
}

I would log this because when someone asks “why did it do that?”, I need more than vibes.

rendered prompt matters#

The template gives only part of the story.

You are a release assistant.
Use this context:
{{context}}

The bug may be inside {{context}}.

Maybe the retrieved docs were stale. Maybe the context got truncated. Maybe a previous message pushed the important part out. Maybe the rendered prompt had two instructions fighting each other.

So for serious debugging, I want a way to reconstruct what the model saw.

Raw prompts still should stay out of normal logs. They can contain secrets, source code, customer data, and all kinds of accidental personal stuff.

It probably needs a restricted store, short retention, redaction, and boring access rules.

version everything#

Agent behaviour can change because the model changed, the prompt changed, a tool schema changed, or the retrieval index changed.

So I would log versions:

{
  "model": "provider/model-version",
  "prompt_template": "support-triage",
  "prompt_version": "v12",
  "tool_schema_version": "2026-02-22",
  "retrieval_index": "docs-1842",
  "app_commit": "abc123"
}

This feels boring until the agent starts doing something strange and you need to know what changed yesterday.

keep raw text separate#

Most dashboards only need metadata:

{
  "event": "agent_step",
  "trace_id": "tr_123",
  "step": 4,
  "kind": "tool_call",
  "tool": "github.create_issue",
  "allowed": true,
  "duration_ms": 631,
  "result": "success"
}

Raw prompts and raw responses are different. They are useful, but sensitive. I would treat them more like request bodies in a serious system: not in every log line, not visible to everyone, and not kept forever without a reason.

my current view#

Prompt logs carry production debugging information for agent systems. They are part trace, part audit trail, part debugging record.

The hard part is logging enough to understand a bad run without storing everyone’s private text forever.

So I would start with metadata, versions, trace ids, and careful storage for raw content.

If the agent can make decisions, the decision path needs to be debuggable.