Background jobs, queues, and workflow analytics

Background work has at least three clocks:

enqueue time → start time → terminal time
     queue wait      run time
     └──────── end-to-end time ────────┘

If instrumentation records only “job finished,” it cannot distinguish a slow worker from a long queue. If every retry is counted as a new job, reliability and throughput become misleading.

Keep the queue or workflow engine authoritative for current execution state. Send compact lifecycle facts to GraphJSON for analysis, dashboards, and supported alerts.

Define job, attempt, and workflow identity#

One job may have several attempts. One workflow may fan out into many jobs.

Do not create a new job_id when retrying the same logical work. Do create a new attempt_id.

Recommended lifecycle#

job_enqueued
job_started
job_attempt_failed
job_retry_scheduled
job_completed
job_failed
job_cancelled

job_attempt_failed is not terminal when another retry remains. job_failed means the logical job exhausted its policy or reached an unrecoverable condition.

Avoid emitting a vague job_processed event that could mean success, failure, or merely receipt.

Use a stable contract#

Terminal job event:

{
  "event": "job_completed",
  "event_id": "job:export_91:terminal:completed",
  "job_id": "export_91",
  "workflow_id": "account_export_42",
  "job_type": "generate_account_export",
  "queue": "exports",
  "account_id": "acct_7",
  "enqueued_at": 1785081600,
  "started_at": 1785081632,
  "completed_at": 1785081719,
  "queue_wait_ms": 32000,
  "run_ms": 87000,
  "end_to_end_ms": 119000,
  "attempt_count": 2,
  "outcome": "success",
  "worker_release": "2026.07.26.3",
  "schema_version": 2
}

Send completed_at as the GraphJSON request timestamp for terminal-event analysis. Keep the other times as integer Unix seconds and durations as integer milliseconds.

Do not send job arguments or results by default. They often contain customer content, storage paths, signed URLs, prompts, or credentials.

Emit attempts only when they support a decision#

Attempt event:

{
  "event": "job_attempt_failed",
  "event_id": "attempt:export_91:2:failed",
  "job_id": "export_91",
  "attempt_id": "export_91:2",
  "attempt_number": 2,
  "job_type": "generate_account_export",
  "queue": "exports",
  "error_class": "object_store_timeout",
  "retryable": true,
  "run_ms": 30102,
  "worker_release": "2026.07.26.3"
}

Use reviewed error classes, not raw exception messages. Preserve stack traces in the diagnostic system built for them.

High-volume systems may keep only the terminal event plus aggregate attempt counts. Retain per-attempt events when retry behavior, provider failure, or worker release diagnosis justifies their cost.

Instrument at the durable boundary#

Recommended sequence:

business transaction
  → write job or outbox record
  → enqueue
  → emit job_enqueued from durable state

worker claims message
  → record attempt start
  → perform work
  → commit result and terminal state
  → emit terminal analytical event

Do not emit job_completed before the result commits. If analytical delivery fails after the commit, a durable outbox can retry without repeating the business work.

Decompose latency#

Calculate:

queue_wait_ms = started_at - enqueued_at
run_ms = completed_at - started_at
end_to_end_ms = completed_at - enqueued_at

Interpretation:

Use percentiles and a denominator. One extremely slow job can distort an average.

SELECT
  JSONExtractString(json, 'job_type') AS job_type,
  count() AS completed_jobs,
  quantile(0.5)(JSONExtractInt(json, 'queue_wait_ms')) AS queue_wait_p50,
  quantile(0.95)(JSONExtractInt(json, 'queue_wait_ms')) AS queue_wait_p95,
  quantile(0.95)(JSONExtractInt(json, 'run_ms')) AS run_p95
FROM job_events
WHERE
  JSONExtractString(json, 'event') = 'job_completed'
  AND timestamp >= toUnixTimestamp(now() - INTERVAL 24 HOUR)
GROUP BY job_type
ORDER BY completed_jobs DESC

Calculate logical success, not attempt success#

For terminal events:

SELECT
  JSONExtractString(json, 'job_type') AS job_type,
  count() AS terminal_jobs,
  countIf(JSONExtractString(json, 'outcome') = 'success') AS succeeded,
  countIf(JSONExtractString(json, 'outcome') = 'failed') AS failed,
  if(terminal_jobs = 0, NULL, succeeded / terminal_jobs) AS success_rate
FROM job_events
WHERE
  JSONExtractString(json, 'event') IN (
    'job_completed',
    'job_failed',
    'job_cancelled'
  )
  AND timestamp >= toUnixTimestamp(now() - INTERVAL 7 DAY)
GROUP BY job_type

Deduplicate terminal event_id when retries can repeat analytical delivery. Do not divide successful attempts by all attempts and call it job success rate.

Measure retry cost#

Track:

• jobs requiring at least one retry
• attempts per completed job
• time added by backoff
• terminal recovery rate by error class
• retry exhaustion rate
• repeated retry by release or dependency

A high eventual success rate can conceal a costly retry storm. Show attempt_count and end-to-end latency beside terminal outcome.

Bound retries. A permanent validation error should dead-letter quickly, while a short provider outage can use exponential backoff with jitter.

Detect stuck work with snapshots#

The absence of a terminal event does not identify an in-progress job inside GraphJSON reliably; recent start events may still be running, delayed, or outside retention.

Have the authoritative queue monitor emit a periodic snapshot:

{
  "event": "queue_state_measured",
  "measurement_id": "exports:1785081900",
  "queue": "exports",
  "measured_at": 1785081900,
  "ready_jobs": 184,
  "running_jobs": 12,
  "oldest_ready_age_seconds": 941,
  "oldest_running_age_seconds": 613,
  "dead_letter_jobs": 3,
  "consumer_count": 8,
  "monitor_version": 2
}

Create a collection Single Line visualization over oldest_ready_age_seconds, dead_letter_jobs, or another bounded numeric field for a native alert.

Keep the authoritative stuck-job action in the queue system. GraphJSON should not retry, cancel, or unlock work.

Instrument scheduled jobs#

For scheduled work, include:

{
  "schedule_id": "daily-account-rollup",
  "scheduled_for": 1785024000,
  "started_at": 1785024300,
  "schedule_delay_ms": 300000,
  "window_start": 1784937600,
  "window_end": 1785024000
}

Monitor both terminal outcome and freshness. A daily job that never enqueues produces no failure event, so an independent scheduler monitor must detect the missing expected run.

Do not use execution time as the data window. Scheduled calculations should follow Derived events and scheduled rollups.

Model fan-out workflows#

For a parent workflow:

account export
  ├── query records
  ├── render files
  ├── write archive
  └── notify requester

Carry a stable workflow_id across every child. Record a separate terminal workflow event after the orchestrator determines the business outcome.

Do not infer workflow success by counting child success events unless the expected child set is known. Dynamic fan-out needs:

• expected child count or manifest
• terminal child count
• failed and cancelled child count
• workflow completeness rule

Build the operating dashboard#

Recommended sections:

1. enqueue and terminal throughput
2. logical success and failure by job type
3. queue-wait, run, and end-to-end percentiles
4. retry and exhaustion rates
5. backlog, oldest age, and dead letters
6. release and dependency breakdown
7. last expected scheduled run
8. customer impact by authorized account segment

Use SLOs, error budgets, and alerts when the workflow has a formal reliability objective.

Validate failure paths#

Exercise:

• successful first attempt
• retryable failure then success
• permanent validation failure
• retry exhaustion
• worker crash after business commit
• analytical timeout after terminal commit
• duplicate queue delivery
• delayed schedule
• missing consumer
• cancellation before and during execution

Verify that each logical job has one selected terminal outcome and that no failure path records success early.

Launch checklist#

• Job, attempt, and workflow IDs have distinct meanings.
• Terminal outcome is emitted only after the result commits.
• Retries retain the logical job ID.
• Queue wait, run time, and end-to-end time use explicit units.
• Raw arguments, results, and exceptions are excluded.
• Queue snapshots come from the authoritative system.
• Missing scheduled runs have an independent monitor.
• Duplicate delivery cannot inflate logical success.
• Dashboards show denominators and latency percentiles.
• The queue system—not GraphJSON—owns retries and current state.

For human or business processes with explicit states, approvals, work in progress, rework, service clocks, and terminal outcomes, continue with Workflow and state-machine analytics.