Runtime Observability
Can you see what agents did?
Review traces, approvals, replay, audit export, failure response, and retention.
Step 1 of 4
Trace
What gets traced?
Pick fields captured for each run.
What this checks
Trace
If a field is not captured when the agent runs, nobody can answer questions about it later. Identity, tool calls and approval decisions matter most.
- Prompt / instruction
- User and agent identity
- Tool calls
- Tool inputs and outputs
- Approval decisions
- PR, ticket, config, or doc diffs
- Webhook / chat trigger data
- Token and cost usage
Controls
A trace you can read is a start. Replay, an undo plan and an audit export are what let you act on it.
- Approval queue
- Execution report
- Replay / rerun from trace
- Audit export
- Rollback / undo plan
- Eval or policy checks
Failure
When a run fails or is denied, someone has to find out before a customer does. Pausing the agent is the strongest response.
- No explicit response
- Notify requester
- Alert admin / owner
- Create incident
- Pause agent or workflow
- Record policy denial
Retention
Incident reviews and audits often happen months later. Records that expire in a week cannot answer them.
- Not retained
- 7 days
- 30 days
- 1 year
- Customer-managed retention
How to read your result
- Blind run
- You would learn what an agent did from its side effects. Capture identity and tool calls first.
- Reviewable run
- Runs can be read after the fact, but not replayed, undone or exported. Add those before agents write to shared systems.
- Auditable runtime
- Runs are traced, controllable and kept long enough to answer an audit.
Questions
What should an AI agent trace record?
At minimum: who started the run, which agent ran, each tool call with its inputs and outputs, and every approval decision. Without the identity and the approvals, a trace cannot settle who was responsible.
How long should agent run records be kept?
Long enough to cover your incident review and audit windows. For most companies that is a year or more, which rules out tools that keep traces for days.
Can you undo what an agent did?
Only if each action was recorded in enough detail to reverse it. An undo plan built from the record, reviewed before it runs, is safer than reverting by hand.