AI agent, tool-use, and evaluation analytics

An AI agent is not one model call. It is a workflow containing a user goal, planning, model turns, tool calls, retries, approvals, and a terminal outcome. Measure those layers separately or a cheaper model call can look like an improvement while the complete task becomes slower and less reliable.

GraphJSON is useful for compact, structured agent outcomes and trends. It is not a prompt store, trace viewer, model gateway, or evaluation engine. Keep full prompts, outputs, tool arguments, and sensitive retrieved context in a system designed and approved for that purpose.

For single-generation adoption and cost, start with LLM product and cost analytics.

Define the grains#

Attach run_id to every lower-level fact. Do not reuse a tool-call ID across retries.

Emit one terminal run event#

The run outcome is the primary analytical fact:

{
  "event": "agent_run_completed",
  "event_id": "evt_run_01J...",
  "run_id": "run_01J...",
  "account_id": "acct_42",
  "user_id": "usr_91",
  "agent_name": "support_resolution",
  "agent_version": "2026-07-26.3",
  "goal_type": "resolve_ticket",
  "outcome": "succeeded",
  "completion_source": "user_confirmed",
  "duration_ms": 18420,
  "model_calls": 4,
  "tool_calls": 3,
  "tool_failures": 1,
  "approval_count": 1,
  "input_tokens": 12840,
  "output_tokens": 1840,
  "estimated_cost_micros": 8240,
  "schema_version": 1
}

Use bounded values for goal_type, outcome, and completion_source. Possible outcomes:

succeeded
failed
abandoned
cancelled
timed_out
handed_off
blocked_by_policy

Do not infer success from the agent reaching its last step. Prefer an application-owned outcome, user confirmation, verified state change, or versioned evaluation.

Model tool calls#

Emit a terminal event for each attempt:

{
  "event": "agent_tool_call_completed",
  "event_id": "evt_tool_01J...",
  "run_id": "run_01J...",
  "step_id": "step_7",
  "tool_call_id": "call_9_attempt_2",
  "logical_tool_call_id": "call_9",
  "tool_name": "refund_lookup",
  "tool_version": "v2",
  "attempt": 2,
  "outcome": "succeeded",
  "duration_ms": 731,
  "result_size_bucket": "small",
  "error_class": null,
  "schema_version": 1
}

Do not copy tool arguments or responses by default. Record allowlisted metadata such as tool name, version, outcome, duration, and a safe error class. Keep attempt identity separate from logical call identity so retry rates remain measurable.

Track human approval#

For agents that can propose consequential actions:

{
  "event": "agent_approval_resolved",
  "event_id": "evt_approval_01J...",
  "run_id": "run_01J...",
  "approval_id": "approval_3",
  "action_type": "issue_refund",
  "decision": "approved",
  "reviewer_role": "support_lead",
  "wait_ms": 42000,
  "policy_version": "refund-v5",
  "schema_version": 1
}

The application or policy system remains authoritative for permission and execution. GraphJSON should receive an analytical copy after the decision is fixed.

Measure task success, not activity#

Start with:

goal achievement rate
= mature runs with a verified successful outcome
  / all eligible mature runs

Then place beside it:

• abandonment and handoff rate
• median and p95 end-to-end duration
• model calls per successful run
• tool calls and retries per successful run
• approval rate and approval wait time
• cost per successful run
• repeated-goal rate after apparent success
• user correction or undo rate

A longer run with more tool calls is not necessarily worse if it solves a more complex goal. Segment by stable goal type and difficulty class defined before the outcome.

Detect loops and retry amplification#

Run-level distributions reveal orchestration problems:

SELECT
  JSONExtractString(json, 'agent_version') AS version,
  count() AS runs,
  quantileExact(0.50)(
    JSONExtractInt(json, 'tool_calls')
  ) AS p50_tool_calls,
  quantileExact(0.95)(
    JSONExtractInt(json, 'tool_calls')
  ) AS p95_tool_calls,
  countIf(
    JSONExtractInt(json, 'tool_calls') >= 20
  ) AS high_tool_call_runs
FROM agent_events
WHERE
  JSONExtractString(json, 'event') = 'agent_run_completed'
  AND timestamp >= now() - INTERVAL 14 DAY
GROUP BY version
ORDER BY version

Define explicit safety ceilings in the agent runtime. Analytics can reveal loops, but it must not be the only mechanism that stops them.

Calculate cost per successful task#

Store money in integer micros or another documented minor unit:

SELECT
  JSONExtractString(json, 'agent_version') AS version,
  countIf(
    JSONExtractString(json, 'outcome') = 'succeeded'
  ) AS successful_runs,
  sum(
    JSONExtractInt(json, 'estimated_cost_micros')
  ) AS total_cost_micros,
  round(
    total_cost_micros / nullIf(successful_runs, 0),
    2
  ) AS cost_micros_per_success
FROM agent_events
WHERE
  JSONExtractString(json, 'event') = 'agent_run_completed'
  AND timestamp >= now() - INTERVAL 30 DAY
GROUP BY version

Document whether cost includes model calls only or also retrieval, tool providers, compute, and human review. Reconcile model cost with provider billing rather than treating estimates as invoices.

Publish evaluation results as versioned facts#

Run evaluations outside GraphJSON, then publish compact outcomes:

{
  "event": "agent_evaluation_completed",
  "event_id": "evt_eval_01J...",
  "evaluation_id": "eval_01J...",
  "run_id": "run_01J...",
  "suite": "support_resolution_holdout",
  "suite_version": "v8",
  "evaluator": "deterministic_and_human",
  "evaluator_version": "2026-07-20",
  "score": 0.92,
  "passed": true,
  "failure_category": null,
  "evaluated_at": "2026-07-26T20:00:00Z",
  "schema_version": 1
}

Keep the evaluation dataset, rubric, raw evidence, and reviewer records in the evaluation system. A score without a suite version and evaluator version cannot be compared safely over time.

Separate:

• offline regression suites
• sampled production review
• user feedback
• deterministic business outcomes
• model-graded results

Do not mix them into one unlabeled “quality score.”

Compare versions with mature runs#

Agent behavior changes with:

• model
• system prompt
• tools and tool schemas
• retrieval source
• orchestration code
• policy
• UI and user population

Assign an immutable agent_version that identifies a reproducible bundle, or attach the individual versions separately. Compare equivalent goal types over settled windows. Use randomized exposure when you need a causal product decision.

Protect content and identity#

Default-deny:

• prompts and model outputs
• retrieved documents
• tool arguments and results
• secrets and authorization headers
• private messages
• complete error bodies
• model chain-of-thought

Use opaque IDs to retrieve approved evidence from a restricted system during an investigation. Apply retention by collection, restrict dashboard splits, and suppress small customer segments.

Build the dashboard#

Recommended sections:

1. eligible runs, successful outcomes, and abandonment
2. latency and cost per successful goal
3. tool selection, failure, and retry rates
4. approval frequency and wait time
5. loops, ceilings, and policy blocks
6. quality by agent, model, tool, and evaluator version
7. user corrections, handoffs, and repeated goals
8. event completeness and cost reconciliation

Launch checklist#

• Runs, steps, generations, tools, approvals, and evaluations have separate IDs.
• Every lower-level fact carries the stable run ID.
• One terminal run event records the mature product outcome.
• Success comes from a verified outcome, not the last agent step.
• Tool retries preserve attempt and logical-call identity.
• Runtime ceilings stop loops independently of analytics.
• Cost is stored in a documented integer unit and reconciled.
• Evaluation suite and evaluator versions are immutable.
• Offline, production, user, and model-graded evidence stay distinct.
• Prompts, outputs, tool payloads, and secrets are excluded by default.
• Version comparisons use compatible goals and mature windows.
• GraphJSON is not presented as a trace viewer or evaluation engine.