Observability in practice: SigNoz, logs, traces

Every service in the fleet — Go and Python — speaks OpenTelemetry and exports traces, metrics and logs to a self-hosted SigNoz over OTLP, stamping a trace_id onto every log line. This page is the operator’s side of that: how to query SigNoz from your shell, how to tell whether a service is even exporting, how to turn the volume up with LOG_LEVEL, and where the raw container logs live when the pipeline itself is suspect.

For the why (the OTel bootstrap, browser RUM, the queue-hop traceparent carrier, record_error, force-sample-on-error) see the Observability & tracing model architecture page. This page assumes it.

Where telemetry goes

All backend services and the web app export OTLP to one SigNoz instance (traces + metrics

• logs in ClickHouse) on a dedicated Azure VM, fronted by Caddy for TLS and a static bearer-token gate — SigNoz’s receiver has no built-in auth. The whole setup lives in soundverse-proto/infra/signoz/ (Caddyfile, docker-compose.override.yaml, OBSERVABILITY.md). SigNoz is your primary pane of glass — search by trace id, by session.id, or by service.name + service.version.

The attribute reference is in-repo

soundverse-proto/infra/signoz/OBSERVABILITY.md is the canonical list of span/log attributes to filter on, the metrics inventory, and the debugging playbook. Read it before building a query.

Query SigNoz from your shell

The UI is at https://signoz.soundverse.ai, but you can hit the API directly with a read-only Viewer key for scripted debugging. The deployment is v0.129.0 (EE) and exposes the v5 query API; the key travels in the SIGNOZ-API-KEY request header.

1. Load the creds. The endpoint and key live outside any git repo in ~/.config/soundverse/signoz.env (mode 600). Source it — it exports SIGNOZ_URL and SIGNOZ_API_KEY into your shell:

   source ~/.config/soundverse/signoz.env

   Names, never values

   The key value stays in that file. Never paste it into a doc, a memory, a commit, or a shell one-liner that ends up in history. npm run check:secrets fails the docs build if a real token appears here.

2. Confirm reachability / version. A plain GET works for the version endpoint:

   curl -s "$SIGNOZ_URL/api/v1/version" \
     -H "SIGNOZ-API-KEY: $SIGNOZ_API_KEY"

3. Pull RED metrics for the service list. This one is a POST, and the timestamps have a trap:

   curl -s "$SIGNOZ_URL/api/v1/services" \
     -H "SIGNOZ-API-KEY: $SIGNOZ_API_KEY" \
     -H "Content-Type: application/json" \
     -d '{"start":"<ns>","end":"<ns>","tags":[]}'

Timestamps are STRINGS in nanoseconds

The services body wants start/end as string nanosecond epochs ("1720224000000000000"). Passing them as JSON numbers returns a 400 unmarshal error; using GET returns the SPA HTML instead of JSON. Both are easy to mistake for an outage.

Worker “error rate” is mostly benign

In the services list, the tool-worker services (core-tool-sansaarm, core-tool-agent, core-tool-media, core-tool-stitch, and any template-core-tool) can show a near-100% “error rate”. That is almost always benign long-poll deadline / NotFound spans on the task-claim path being marked errored — not real failures. Confirm against real task.run span status before treating it as an incident. A genuinely high error rate on the web service or core-billing is worth a real look.

The heartbeat canary: is this service even exporting?

The failure mode that hides in plain sight is a service whose logs simply aren’t in SigNoz — usually a stale ACA revision that predates the OTLP env, not a code bug (the export wiring is uniform across the fleet). The tell is a liveness canary: every healthy service emits an INFO log line telemetry.heartbeat service=<name> on a fixed interval. Group SigNoz Logs by service.name; the absence of a heartbeat is the signal that a service isn’t exporting.

The knobs that govern it are plain env-var names (never commit the values):

Env var	Purpose
OTEL_EXPORTER_OTLP_ENDPOINT	The master switch. Unset → no exporters installed at all; the service falls back to trace-correlated JSON on stdout with no connection errors. That is why local runs and tests need no collector.
OTEL_EXPORTER_OTLP_HEADERS	The ingest bearer header the collector gates on (resolved from an org secret). If the endpoint is set but this is empty, the bootstrap logs a WARN.
OTEL_HEARTBEAT_SECONDS	Heartbeat cadence. Default 60; 0 disables the canary.
LOG_LEVEL	debug | info | warn | error, default info. Gates both stdout and the OTLP log bridge (see below).
OTEL_SERVICE_VERSION / GITHUB_SHA	Sets service.version on every span so you can group by release; an explicit OTEL_SERVICE_VERSION wins over the build-arg SHA.
ENVIRONMENT	Becomes the deployment.environment resource attribute (local / staging / prod).

The canary is wired identically in both languages — Python in soundverse-py/src/soundverse/telemetry.py (which covers all three tool workers and the agent via the fleet), Go in core-mcp/internal/telemetry/telemetry.go and the matching core-database module.

Four Python telemetry files, not one

The shared soundverse-py telemetry helper is duplicated into core-identity, core-storage, and core-gateway-consumer as app/core/telemetry.py (the gateway depends on soundverse-proto only, not soundverse-py, so it can’t import the shared one). Any telemetry change — a new knob, a suppressed logger — must touch all four Python files plus the two Go modules, or services drift apart.

Turn the volume up: LOG_LEVEL

LOG_LEVEL gates both the stdout handler and the OTLP→SigNoz bridge in lockstep, so setting LOG_LEVEL=debug on a staging revision gives you a full play-by-play in SigNoz, not just on the container’s stdout.

Go

parseLevel(os.Getenv("LOG_LEVEL")) feeds a leveled wrapper around the otelslog handler — otelslog has no level option of its own, so this wrapper is what keeps SigNoz log volume tied to the level. Default info.

Python

_configure_logging honours LOG_LEVEL on the root logger and the OTLP handler. It also runs _suppress_noisy_loggers(), which pins chatty third-party loggers (httpx, httpcore, urllib3, azure, grpc, openai, replicate, …) to WARNING — unless LOG_LEVEL=debug, where the operator explicitly wants everything.

Why the suppression exists

Without it, a bare INFO root logger let httpx log every outbound request — e.g. core-tool-sansaarm flooding one HTTP Request … 200 OK line every ~2s while polling its provider for a live preview. Harmless, but it drowns SigNoz. LOG_LEVEL=debug un-mutes them when you actually need them.

Reading a single tool call: verbose core-mcp logging

core-mcp is the second billing pipeline, and its tool-call handler used to be near-silent. It now binds a request-scoped logger the moment a call arrives — slog.With("call_id", …, "tool", …, "operation", …, "tool_id", …, "user_id", …, "workspace_id", …, "project_id", …) in core-mcp/internal/toolcall/handler.go — and logs every milestone against it: tool call received, tool call rate-limited, tool unpriced, task queued, completion, and generation failures. Filter SigNoz Logs by a single call_id and you reconstruct the whole tool call end to end.

That covers the classic “chat is stuck on planning” triage: search the call_id (or the displayed trace_id) and the reason is usually one log line away — tool unpriced (no pricing row → NotFound; see Add pricing to a tool), tool call rate-limited, or an upstream RPC error.

What to search by

SigNoz filtering is only as good as the attributes on the record. The high-value ones, stamped fleet-wide:

Attribute	Where it’s set	Use it to
trace_id	every log line, both languages	pivot log ↔ trace; it’s the id shown in the chat trace badge
session.id	every browser span (per-tab)	replay one user’s whole journey in order
task.id	gateway, worker task.run, BFF stream	tie the generation trace to the live-read trace
user.id	gateway generation RPCs, worker task.run	“everything user X did”
service.name + service.version	every span/log	scope to one service and one release
call_id	core-mcp tool-call logs	reconstruct a single tool call

The chat UI surfaces the trace_id as a small trace badge next to each generation (the agentone.run root span is force-sampled so a displayed id is never a dropped trace). A user quoting “my song failed” can hand you that id; paste it into SigNoz for the entire run — browser → BFF → gateway → core-database → worker.

Tail raw container logs

When telemetry export itself is suspect, go under it to the container’s stdout (the same trace-correlated JSON, just not via ClickHouse). The resource group is rg-core:

Follow one Container App

az containerapp logs show -n core-gateway-consumer-staging -g rg-core --follow

Multi-replica: list, then stream each

az containerapp replica list -n core-storage-staging -g rg-core \
  --query "[].name" -o tsv

To read the effective env/secret refs on a running revision without leaking values, see Inspect a running service’s env & secrets.

Metrics that exist (and the alerting that doesn’t)

The money path and the queue are instrumented, so a leaked hold or a backing-up pipeline is visible in SigNoz rather than surfacing as a billing complaint:

Metric	Emitted by	Answers
settlement_failure_total, refund_failure_total	core-gateway-consumer/app/core/metrics.py (low-cardinality)	is settlement leaking?
generation_latency_seconds, token_usage_total	the worker fleet	how slow / how much?
generation_queue_depth	core-database DB-gauge, by tool_id	is the pipeline backing up?
unsettled_task_age_seconds	core-database DB-gauge	oldest terminal task the reconciler hasn’t settled

Planned Alerting and SLOs are not wired yet. These counters emit, but nothing pages on refund_failure_total > 0 and there is no time-to-settle target. See Known limitations for the planned golden-metric set. The billing reconciler is the current backstop.

In code, not deployed A tail-sampling collector (otelcol-tail.yaml, wired into the SigNoz docker-compose.override.yaml + Caddyfile) is designed to keep any trace that contains an error, is force-sampled, or is slow — closing the “downstream error on a head-sampled-out trace” gap. It’s configured in-repo; confirm it’s actually live on the SigNoz VM before relying on it.

Generation workers have no /healthz

The Go services expose /healthz, but the tool-worker fleet does not. A worker’s liveness is a 30s-TTL Redis heartbeat at {env}:worker:active:{worker_id} (refreshed every 15s), which nothing currently alerts on. Workers do push OTLP, so a hung-but-alive worker is partly observable via the telemetry.heartbeat canary — but don’t reach for a worker health endpoint that isn’t there. The Redis key convention is in Make targets & Redis keys.

Related

• Observability & tracing model — the architecture behind this page
• Inspect a running service’s env & secrets — reading ACA env safely
• Troubleshooting FAQ — the failures you’ll actually hit
• Billing reconciliation — the settle backstop the money-path metrics watch
• Known limitations — the missing SLOs and the tail-sampling rollout