Observability & tracing model

When a generation fails at 2am you want exactly one question answered: “show me the full trace for generation X.” The 2.0 backend is built so that question is always answerable. Every service — Go and Python — speaks OpenTelemetry (OTel), exports traces, metrics and logs to a self-hosted SigNoz over OTLP, and stamps a trace_id onto every log line so a span and its logs are one click apart. This page explains the model: one bootstrap shape, a trace that starts in the browser, how the trace survives the async queue hop, and how errors are forced onto the span instead of hiding in the logs.

One bootstrap shape, copied across the fleet

There is exactly one telemetry init, mirrored everywhere:

• Go calls telemetry.Init(ctx, "core-database") — core-database/internal/telemetry/telemetry.go, core-mcp/internal/telemetry/telemetry.go.
• Python calls telemetry.setup("core-gateway-consumer"). The canonical helper lives in soundverse-py/src/soundverse/telemetry.py and is mirrored into each service as app/core/telemetry.py (the service copies add a record_error helper — see below).

Both wire up all three signals and export them over OTLP/HTTP. The whole exporter stack is switched entirely by environment — names only below, never values:

Env var NAME	Purpose
OTEL_EXPORTER_OTLP_ENDPOINT	The SigNoz OTLP ingest URL. The master switch — if unset, no exporters are installed at all.
OTEL_EXPORTER_OTLP_HEADERS	Comma-separated key=value list carrying the static bearer that Caddy gates SigNoz ingest with.
OTEL_EXPORTER_OTLP_PROTOCOL	Transport/encoding selector read by both bootstraps (the fleet defaults to OTLP/HTTP).
OTEL_SERVICE_VERSION / GITHUB_SHA	Sets service.version on every span so SigNoz can group/compare by release. GITHUB_SHA arrives as a Docker build-arg; an explicit OTEL_SERVICE_VERSION wins.
ENVIRONMENT	Becomes the deployment.environment resource attribute (local / staging / prod).
LOG_LEVEL	debug|info|warn|error (default info). Gates both stdout and the OTLP log bridge, in both languages.
OTEL_HEARTBEAT_SECONDS	Interval for the telemetry.heartbeat liveness canary (default 60, 0 disables).

OTLP env is blanket-injected by the shared deploy template (the STAGING_/PROD_ prefix-strip described in Environment & service discovery), so the wiring is uniform in code across the fleet. When logs are “missing” for a service in SigNoz, it is almost always a stale ACA revision predating the pipeline, not a code gap.

Graceful degradation is the point

When OTEL_EXPORTER_OTLP_ENDPOINT is empty, both languages skip exporter setup and fall back to trace-correlated JSON logs on stdout — no connection errors, no retries. That is why local runs and the test suite never reach for a collector. You don’t comment telemetry out to run offline; you just leave the endpoint unset.

Trace ↔ log correlation, on every record

This is the single most useful property. Every log line carries the active span’s trace_id and span_id:

• Go — a custom trace handler reads the span context off ctx and adds trace_id / span_id to each slog record; a fanout handler writes every record to both stdout JSON and OTLP.
• Python — LoggingInstrumentor patches the log-record factory and the JSON formatter renames the fields to trace_id / span_id / service, while a LoggingHandler ships the same record to OTLP.

So in SigNoz you pivot from a slow span straight to its logs, and from a stdout line straight to its trace, by the same id.

What gets spanned automatically

• otelhttp is the outer handler on the Go data plane, so the inbound traceparent is extracted and seeds the server span before any inner handler or DB call runs — one RPC is one connected trace.
• pgx and Redis calls get child spans under that server span, so you see exactly which query or cache op was slow. core-database is the service that gets the error path right for free — otelpgx auto-records query errors onto the span.
• The propagator is set explicitly to a composite of W3C TraceContext + Baggage in both languages, so cross-service propagation and the queue-hop round-trip (below) behave identically everywhere.

Flush on SIGTERM, or lose your last spans

Batch processors hold spans in memory before export. The Go Init returns a shutdown(ctx) func and the Python helper exposes shutdown(); both force-flush every provider. They must run on graceful drain so the final traces/logs before an ACA replica scales down aren’t dropped. core-mcp and core-database both defer shutdown after in-flight RPCs finish.

The trace starts in the browser

The trace does not begin at the gateway — it begins in the user’s browser. soundverse-saas-2.0/lib/otel/browser.ts starts a WebTracerProvider (service soundverse-web) that captures document load, user interactions, fetch/XHR, and injects traceparent onto same-origin /api/* calls — the seam that connects the browser trace to the BFF.

Browser spans never go straight to SigNoz. They are exported to a same-origin proxy, app/api/otel/[...signal]/route.ts, which forwards them to the ingest endpoint and attaches the bearer server-side. Two deliberate reasons: the SigNoz ingest token must never reach the browser (it is read from OTEL_EXPORTER_OTLP_HEADERS inside the route), and same-origin avoids CORS + NSG changes so SigNoz stays locked to the ACA outbound IP.

Two enrichments make RUM searchable:

• A JourneySpanProcessor stamps session.id (a per-tab id) and the current ui.route onto every span. Filter SigNoz by one session.id and you replay a user’s whole journey in order.
• A chat generation is wrapped in a single agentone.run span via runGenerationSpan(...). It is force-sampled — a ForcedRootSampler always exports it, overriding the head-sampling ratio in NEXT_PUBLIC_OTEL_SAMPLE_RATIO (default 0.25) — because its trace_id is shown to the user, and a displayed id whose trace was dropped is useless.

The trace badge — support's best friend

runGenerationSpan hands the trace_id to the UI synchronously the instant a run starts, rendered as a subtle diagnostic chip next to the generation (styled like the timestamp, not an alarm). A user reporting “my song failed” can quote that id; you paste it into SigNoz and see the entire run. The same id is propagated to the backend and persisted on the message (via the existing metadata_json field), so display, propagation and storage are all the one id — it survives a page reload on pending and failed cards. See Frontend architecture.

Surviving the async queue hop

The hardest boundary to trace is the async task queue: the gateway commits a task row to Postgres, a worker claims it later, and auto-instrumentation cannot see across a queue. This used to mint a fresh, disconnected trace id. It no longer does — the W3C context rides the task row itself:

1. Inject (gateway). core-gateway-consumer/app/enforcement/pipeline.py sets w3c_traceparent=telemetry.inject_traceparent() on the QueueTaskRequest, capturing the live span context.

2. Extract (worker). soundverse-py/src/soundverse/tools/fleet.py calls telemetry.extract_context(task.w3c_traceparent) and starts the task.run CONSUMER span with it as the parent. A blank/legacy traceparent simply yields a root span — graceful degradation for rows queued before the field existed.

3. The trace id reconstructed from that persisted traceparent is the same id surfaced in chat — so browser → gateway → worker is one connected trace in SigNoz.

Two different ids, two different jobs

The RequestContext.trace_id field on internal RPCs is a correlation/audit id (see core-gateway-consumer/app/core/rpc.py), not the OTel trace. The real trace travels in w3c_traceparent on the task row. Don’t confuse them. See The task queue for the row lifecycle.

Errors land on the trace, not just the logs

A failed RPC that only logs an error is invisible in a trace view — the span ends green and the trace you open to debug a failure looks successful. The systemic finding from the 2026-06 observability audit was exactly this: the happy path was well instrumented, but errors landed in logs, not spans, so the most-traversed gateway/core-mcp server span ended UNSET for every mapped error. The auto-instrumentation does not hook the error path (the gateway aborts via context.abort_with_status(), which the grpc-aio OTel proxy doesn’t intercept; core-mcp’s MCP transport returns HTTP 200 even for tool failures).

The fix is built on the shared ErrorDetail proto contract (soundverse-proto/proto/soundverse_proto/common/v1/error.proto — an ErrorCode enum + ErrorDetail message carrying request_id and trace_id), plus a small record_error(exc, *, message=…, mark_error=…) helper called at central choke points so individual handlers can’t forget:

• The gateway’s ErrorInterceptor (core-gateway-consumer/app/grpc_server/interceptors/errors.py) calls record_error at the catch site and stamps the current trace_id (formatted 032x) onto every ErrorDetail. Crucially, expected business rejections (rate-limited, not-found, insufficient balance) are recorded with mark_error=False — the cause stays inspectable on the span, but error dashboards aren’t flooded by normal rejections.
• core-mcp records on outbound RPC errors via a client interceptor and marks the tool-call span ERROR on err / res.IsError — see core-mcp.
• The browser force-samples uncaught errors (they were dropped when no span was active) and marks failed fetches ERROR; the agentone.run span is marked ERROR on a handled backend/stream failure, so the force-sampled trace the user opens no longer reads green.

The trace_id in the chat UI shipped with two real bug fixes

Surfacing the persistent trace_id in chat was paired with root-causing the failure that motivated it: an unguarded Redis xread that surfaced as an opaque internal error (now mapped to a descriptive retryable ERROR_CODE_UPSTREAM_UNAVAILABLE), and a max_retries=0 on gateway-queued tasks that made them unclaimable (retry_count < max_retries is 0 < 0) so the run stuck at planning… forever. The gateway and core-mcp now set a non-zero max_retries at enqueue.

Money-path & backlog metrics

Custom metrics make a leaked hold or a backing-up pipeline visible in SigNoz instead of a billing complaint:

• core-gateway-consumer/app/core/metrics.py defines settlement_failure_total and refund_failure_total (low-cardinality: tool_id / status only). The worker fleet records a generation_latency_seconds histogram and a token_usage_total counter.
• core-database/internal/telemetry/dbmetrics.go registers two observable gauges off the Postgres pool — generation_queue_depth (queued tasks awaiting a worker, by tool) and unsettled_task_age_seconds (age of the oldest terminal task whose reservation the reconciler hasn’t settled). Best-effort: logged-and-skipped on a query error, never fatal.

Money-path non-fatal failures are marked with a filterable attribute (settlement.failed / refund.failed) rather than ERROR status, so they don’t inflate the error rate. See The token ledger & money path and Billing reconciliation.

Reading one trace, end to end

Under a single trace id you should expect these spans, by tier:

flowchart LR
  B["Browser RUM<br/>agentone.run (force-sampled)"]:::frontend
  BFF["Next.js BFF<br/>instrumentation.ts"]:::frontend
  GW["core-gateway-consumer<br/>resolve → price → reserve → queue"]:::gateway
  DB["core-database<br/>otelhttp + pgx / Redis children"]:::data
  Q[("generation.tasks<br/>w3c_traceparent on the row")]:::data
  W["worker fleet<br/>task.run (CONSUMER) + agent.* steps"]:::worker
  SIG["SigNoz (OTLP)"]:::external

  B -->|traceparent on /api/*| BFF
  BFF -->|otel-interceptor injects onto gRPC| GW
  GW --> DB
  GW -->|QueueTask + w3c_traceparent| Q
  Q -->|ClaimNextTask| W
  B -.OTLP.-> SIG
  BFF -.OTLP.-> SIG
  GW -.OTLP.-> SIG
  DB -.OTLP.-> SIG
  W -.OTLP.-> SIG

  classDef frontend fill:#6366f1,color:#fff,stroke:#4338ca
  classDef gateway  fill:#0ea5e9,color:#fff,stroke:#0369a1
  classDef data     fill:#8b5cf6,color:#fff,stroke:#6d28d9
  classDef worker   fill:#10b981,color:#fff,stroke:#047857
  classDef external fill:#f59e0b,color:#111,stroke:#b45309

The BFF is the one hop auto-instrumentation can’t see: instrumentation.ts extracts the inbound traceparent and lib/grpc/otel-interceptor.ts re-injects it onto the outbound gRPC call to the gateway — without it the trace would dead-end at the BFF. Inside the worker, core-tool-agent adds agent.* spans for its own reasoning/tool steps (the second billing pipeline in the agent + MCP path).

Build provenance. Each span carries service.version, and core-database’s /healthz also reports build metadata for the service and the generated proto-go stubs it was compiled against (commit, ref, and the stub source_commit) — so when a trace looks wrong you can confirm exactly which build produced it, and whether it was on a stale proto. See the proto → deploy cascade.

Known gaps honest debt

Most of the worst holes flagged by the 2026-06 infra review are closed (the queue-hop trace, OTel in the worker fleet, the money-path counters). What remains:

• Metrics exist; SLOs and alerts do not. Planned We emit settlement_failure_total, refund_failure_total, generation_latency_seconds and token_usage_total, but nothing is wired to page on them and there is no p99-time-to-settle target. Counters nobody watches are half the job.
• Always-export-on-error needs the tail-sampling collector. In code, not deployed Error visibility today relies on force-sampling at known roots (agentone.run, uncaught-error spans). A trace that errors downstream after being head-sampled out can still be lost. The intended fix — a tail-sampling collector that keeps any trace containing an ERROR regardless of head ratio — exists as config (soundverse-proto/infra/signoz/otelcol-tail.yaml) but is not deployed.
• Generation workers have no /healthz. The Go services expose one; the worker fleet’s liveness is a short-TTL Redis heartbeat key ({env}:worker:active:{worker_id}) that nothing currently alerts on. Workers do push OTLP, so a hung-but-alive worker is partly observable.

See Known limitations for the full risk register.

Related

• Observability in practice — the SigNoz query runbook (EE v5 API, Viewer key, LOG_LEVEL)
• Request lifecycle — the generation path a trace follows
• The agent + MCP path — where the agent.* spans come from
• core-mcp — the second error/billing choke point
• Glossary — trace_id, traceparent, OTLP, force-sample