Troubleshooting FAQ

Every entry below is a real incident from this codebase. When something breaks, resist reading code at random. First work out which service owns the error string — the shape of the string tells you the language and the service that produced it — then pull the trace id and tail that container.

First, get oriented

Most failures arrive as an opaque string in a chat bubble or a gateway response. Provenance is your fastest diagnostic:

Error shape	Owner	Meaning
query scan failed: no rows in result set	core-database (Go row.Scan via dbErrToConnect)	The SQL ran but returned zero rows
unknown or inactive tool	core-gateway-consumer (Python)	A row is missing in generation.tools
UNAUTHENTICATED: invalid internal authorization	core-database	Internal bearer mismatch
TypeError: Descriptors cannot be created directly	any Python service (import time)	protobuf / OpenTelemetry version skew
Not enough segments	core-identity	An opaque Logto token where a JWT was expected

Every error response also carries a trace id (see Observability in practice). Grab it first, then tail the owning service.

The two-command reflex

Reproduce the failure, copy the trace id, then tail the owning app and search for that id: az containerapp logs show -n <app>-staging -g rg-core --follow. Nearly every entry below was found this way. To confirm a var is actually injected (the STAGING_/PROD_ prefix is stripped at deploy), see Inspect a running service.

Boot & import failures

TypeError: Descriptors cannot be created directly — a Python service exits at import

Pythonall Python services

Cause. A protobuf / OpenTelemetry version skew. soundverse-proto’s generated _pb2.py are protobuf-7 gencode with a hard runtime floor of protobuf>=7.35.1, but every released opentelemetry-proto caps protobuf<7.0. Forced onto protobuf 7, uv back-tracked the OTLP exporter to the 2022-era 1.11.1, whose pre-3.20 descriptor stubs the protobuf-7 C runtime rejects — so the process dies importing opentelemetry/proto/common/v1/common_pb2.py.

Fix. In the affected repo’s pyproject.toml, raise the exporter floor and override (not constrain) the protobuf cap, then re-lock and rebuild:

pyproject.toml

# dependencies
opentelemetry-exporter-otlp-proto-http>=1.42.1

[tool.uv]
override-dependencies = ["protobuf>=7.35.1"]

An override is required to defeat OTel’s <7.0 cap; opentelemetry-proto 1.42.x imports cleanly under protobuf 7.35.1. Never cap protobuf<7 — that kills soundverse-proto imports outright.

Per-repo, and NOT inherited

Each Python service needs its own copy of the override — it does not come from soundverse-py or template-core-tool. Re-running ./update-proto.sh (a uv sync --upgrade-package soundverse-proto) can silently re-resolve the exporter back to 1.11.1 and regress this. Verify the OTLP HTTP exporter still imports after any re-lock.

core-database exits at startup / schema init didn’t run

Gopostgres

Cause. core-database runs no migrations — it expects the schema to already exist and pings the database at startup. The one-time schema load (compose/postgres/00-prelude.sql then 10-load-schemas.sh, which mounts the per-domain schema.sql files from soundverse-proto) runs only on an empty data volume. If the volume already had a half-loaded schema, init is skipped and core-database can’t find its tables.

Fix. Check Postgres health and the init log with make logs s=postgres. If init half-ran, force a clean reload with make db-reset (destructive — it wipes and reloads). See the Docker quick start.

Build fails cloning soundverse-*

build

Cause. A missing or expired GitHub token. Every backend image pulls the private soundverse-* dependencies through a BuildKit secret sourced from GH_TOKEN.

Fix. Put a valid GH_TOKEN (scopes repo, read:packages) in your .env, or run gh auth login, then make doctor to confirm. Never commit the value — env-var name only.

SyntaxError: multiple exception types must be parenthesized — a worker won’t boot

Pythonworker

Cause. A Python-2-style except ValueError, KeyError: in a tool module. Because every tool is imported through ALL_TOOLS at boot, one bad module takes the whole worker (and any test that collects it) down with it.

Fix. Parenthesize the tuple — except (ValueError, KeyError): — and run make check. The strict ruff + pyright gate catches this class of error before it ships.

Auth & connectivity

UNAUTHENTICATED: invalid internal authorization — workers crash-loop on RegisterTool

Go + Pythonworker

Cause. The whole backend authenticates internal gRPC/Connect calls with a bearer keyed off the env var INTERNAL_RPC_SECRET; core-database rejects a mismatch as UNAUTHENTICATED. Historically soundverse-py read the wrong name (INTERNAL_AUTH_SECRET), never saw the injected value, and fell back to a dev sentinel — so every RegisterTool / GetActiveTools 401’d and the worker crash-looped.

Fix. Confirm STAGING_INTERNAL_RPC_SECRET / PROD_INTERNAL_RPC_SECRET exist at the GitHub org level (the deploy strips the prefix) and match what core-database verifies against. Locally it is one value in your .env and must be identical for every service. soundverse-py now accepts the canonical name with the legacy name as an alias:

soundverse-py/src/soundverse/config.py

internal_rpc_secret: str = Field(
    default=_DEV_SECRET_SENTINEL,   # the "unconfigured" marker
    validation_alias=AliasChoices("INTERNAL_RPC_SECRET", "INTERNAL_AUTH_SECRET"),
)

Not enough segments — every login 401s after the Logto migration

Pythoncore-identity

Cause. The IdP is self-hosted Logto (migrated off Zitadel). Logto only mints a JWT access token when the frontend requests an API resource; otherwise it returns an opaque token. core-identity validates locally against the JWKS, so an opaque token has no dots to split and PyJWT raises Not enough segments — /v1/auth/validate returns a false verdict and the gateway rejects the call.

Fix. Have the SaaS app request the API resource (set AUTH_LOGTO_RESOURCE) and set core-identity’s IDP_AUDIENCE to that same indicator — the token’s aud, not the client id. Two adjacent symptoms live here too: an issuer mismatch (IDP_ISSUER must be the exact Logto issuer) and a wrong audience. Profile enrichment is separate — Logto’s /oidc/me rejects the resource-scoped JWT, so enrichment runs through the Management API (LOGTO_M2M_CLIENT_ID / LOGTO_M2M_CLIENT_SECRET). Full setup: Configure Logto.

gRPC UNAVAILABLE … Connection reset by peer — logs show a resolved IP on :443

Pythonworker

Cause. A plaintext h2c dial into a TLS-only Azure Container Apps ingress on :443. Staging/prod ingress for core-database and core-storage is HTTP/2 with allowInsecure: false — it resets any cleartext connection. The IP in the log is just the resolved FQDN; it is not a raw-IP misconfiguration.

Fix. The per-service TLS flags default to true for staging/prod and must stay on there. Locally, where services run cleartext, set CORE_DATABASE_USE_TLS=false and CORE_STORAGE_USE_TLS=false in your .env. These flags live in four services and are not auto-derived — flip them together with the ingress. See Environment & service discovery.

ConnectError on POST http://localhost:8080/mcp — a worker can’t reach core-mcp

Pythonworker

Cause. The worker read an uninjected CORE_MCP_GRPC and fell back to its localhost:8080 default. Staging publishes the address only as CORE_MCP_GRPC (from the org var, prefix stripped) — ACA has no bare container-name DNS.

Fix. Ensure CORE_MCP_GRPC is injected. soundverse-py derives the MCP URL from it, choosing the scheme by port (a :80 suffix means http, otherwise https):

soundverse-py/src/soundverse/config.py

scheme = "http" if self.core_mcp_grpc.rsplit(":", 1)[-1] == "80" else "https"
self.core_mcp_url = f"{scheme}://{self.core_mcp_grpc}/mcp"

Set the full-URL override CORE_MCP_URL only as a local escape hatch.

Nothing generates

The AI Tools panel renders empty

gateway + worker

Cause. The panel is registry-only and schema-driven: the grid lists only the active tools returned by GetActiveTools, and each form is built from that tool’s input_schema_json. There is no curated fallback — if the registry is empty the panel is empty. The two usual root causes:

• No worker registered — nothing wrote active rows to generation.tools, or registration 401’d (see the INTERNAL_RPC_SECRET entry above).
• ENVIRONMENT mismatch — this is not a workspace bug. GetActiveTools filters by is_active and environment only; core-mcp drops all tools when GetActiveTools(env) returns empty. If the fleet registered tools under a different ENVIRONMENT than the gateway/core-mcp query (a common drift is one service on local while the DB rows say staging), the panel goes empty.

Fix. Confirm at least one worker booted, then compare environments:

SELECT name, environment, is_active FROM generation.tools;

against each service’s ENVIRONMENT var — gateway, core-mcp, and every worker must agree. The panel also filters out agent sub-tools by design (see Wire the AI Tools panel).

{"code":"NotFound","error":"query scan failed: no rows in result set"} — chat fails the instant a tool runs

Gocore-database

Cause. The tool is registered (its generation.tools row exists) but has no pricing row. The pricing lookup (FetchResolvedPricing, a bare SELECT … LIMIT 1 with no COALESCE) returns zero rows → the Go row.Scan error you see. Rate limits never hit this because their query wraps everything in COALESCE(…,0).

Fix. Declare pricing on the BaseTool; the fleet upserts it on registration (UpsertToolConfig → UpsertToolLicensePricing), idempotently, so editing the value and restarting the worker re-applies it. Free is not the absence of pricing — it must be an explicit zero-cost row:

pricing = (LicensePricing(license=LICENSE_ROYALTY_FREE, cost_base=0),)

Not the same as 'unknown or inactive tool'

The Go no rows error means the tool is registered but unpriced. The gateway’s Python unknown or inactive tool means the generation.tools row itself is missing — a registration failure, not a pricing one. Full recipe: Add pricing to a tool.

Chat stuck on “planning…” — a task is never picked up

Go + Pythonworker

Cause. An unclaimable task — e.g. max_retries=0, so a failed task is terminal rather than re-queued — or a worker that died mid-task with nothing to re-queue it, or an unguarded Redis xread on the event stream.

Fix. A stale-task reaper (ListStaleTasks) re-queues abandoned tasks and the stream auto-reconnects on reload; ensure max_retries>0 so retries actually run. To diagnose, inspect the row — its status and claim_expires_at tell you whether it was ever claimed or has gone stale:

SELECT id, status, retry_count, max_retries, claim_expires_at
FROM generation.tasks ORDER BY created_at DESC LIMIT 20;

The gateway also now catches the pricing NotFound and renders a FAILED message with a real reason instead of an infinite spinner. See Task queue on Postgres and Billing reconciliation.

Agent sub-tool calls hang until CallTimeout — events never arrive

Go + PythonRedis

Cause. A Redis logical-DB split. core-mcp publishes task event streams on one Redis DB index while the workers read from another; pub/sub and keys are scoped per-DB, so events are invisible across the gap and the caller waits out its timeout.

Fix. The DB index lives only in the /N path of the REDIS_ADDR URL (e.g. …:6379/1), fleet-wide. Remove any stray REDIS_DB override so every service lands on the same DB — there is intentionally no separate *_REDIS_DB knob anywhere; it is ignored. See the Redis key conventions.

Frontend quirks

The @-mention picker (or drag-drop) only shows recent files

JavaScriptsaas-2.0

Cause. The mention index (useMentionIndex in soundverse-saas-2.0/src/agentone/mentionIndex.js) was built as a fast autocomplete cache, not a complete index — it fetched exactly one listLibrary page (newest-first) per composer mount and never paginated. Once a library grew past that page, browsing the “Library” folder only reached the newest files, and a drag-drop whose name missed the cache silently inserted plain @name text instead of a real id-bearing chip.

Fix. The index now backfills subsequent pages in the background via the existing cursor, and drag-drop falls back to a whole-library search when the name misses, upgrading the placeholder to a real chip once resolved. If it recurs, check whether the cached list has been re-capped to page one before assuming a new bug — this class of defect returns whenever a “cache the first page” pattern is reused for a feature that implicitly promises completeness.

When the symptom isn’t on this list

1. Reproduce the failure and capture the trace id from the error response.

2. Tail the owning service — az containerapp logs show -n <app>-staging -g rg-core --follow — and search for that id.

3. Confirm the var is actually injected (the STAGING_/PROD_ prefix is stripped at deploy):

   az containerapp show -n <svc>-staging -g rg-core \
     --query "properties.template.containers[0].env"

4. Check proto/SDK versions line up across Go and Python — a contract skew is the root cause of several entries above.

5. When in doubt, the DB idempotency keys mean re-running settlement or config upserts is safe.

Related

• Observability in practice — pull traces and logs from SigNoz
• Inspect a running service — read effective ACA env without leaking values
• Add pricing to a tool — fix the NotFound / unpriced-tool trap
• Configure Logto — the API-resource / JWT gotcha in full
• Known limitations — the standing risk register