core-storage

The file plane. A pure grpc.aio (asyncio) service that stores content-deduplicated blobs in Azure Blob Storage, mints short-lived signed download URLs, transcodes a CMAF/HLS streaming rendition off every audio master, and ingests live audio into an HLS preview that finalizes to VOD. Workers and the gateway never touch Azure directly — they upload and mint through this service via soundverse-py.

This page is the service field guide

For the why — content-addressing, the FLAC master → CMAF/HLS pipeline, the same-origin stream proxy, the master paywall — read Storage & media plane. This page covers what core-storage is, how to run it, its files, its env, and the traps.

Where it sits

core-storage is the StorageService — all the blob/transcode business logic. It is itself a client of core-database’s StorageDataService, the only thing that opens a Postgres connection. So a blob write is two hops: core-storage puts bytes in Azure, then calls core-database to record the blobs / files / blob_variants / live_sessions rows. Every upstream call carries Authorization: Bearer $INTERNAL_RPC_SECRET.

StorageService vs StorageDataService — don't merge the channels

Two different services share the storage.v1 proto package. StorageService (this repo, Python) does the work; StorageDataService (core-database, Go) owns the SQL. The gateway must route them to separate channels — see the core-database field guide. Sending both onto one channel is a real bug we have already hit.

What it does

1. Content-deduplicated uploads. Every upload is hashed twice: the raw sha256 of the bytes (client integrity check + storage path), then a visibility-scoped db_blob_hash = sha256("$raw_hash:$container"). The scope means the same bytes in public vs private get distinct blob rows. On a GetBlobByHash hit the existing blob URL is reused; on a NOT_FOUND miss the bytes are uploaded and InsertBlob runs.

2. Three entry points.

   • UploadFile — client-streamed chunk messages (the metadata frame first, then bytes; server verifies client_sha256_hash).
   • UploadFromUrl — the service fetches a remote URL and re-hosts it (used for provider re-hosting, e.g. generated album art).
   • IngestStream — a bidi live-to-VOD stream (below).

3. A CMAF/HLS streaming rendition per audio master. After a master commits, an ffmpeg pass renders a seekable VOD rendition — an fMP4 init.mp4, .m4s media segments, and a VOD audio.m3u8 — under a deterministic prefix streaming/{master_hash}/{role}/. It is rendered in the background (see the latency trap below) and resolved lazily at play time under role STREAMING_VARIANT_ROLE (streaming_aac).

4. A lazy single-file download. Role DOWNLOAD_VARIANT_ROLE (download_m4a) renders a faststart AAC/M4A from the master on first download only, cached as a variant and single-flighted per master (concurrent first-downloads spawn one ffmpeg pass, not N). The master role serves the lossless master blob directly.

5. Signed reads. CreateDownloadLink mints a short-lived signed URL (default TTL from DOWNLOAD_LINK_DEFAULT_TTL). For an HLS rendition it signs the whole rendition at serve time (see the SAS trap) rather than a single blob.

6. Live-to-VOD ingest + GC. IngestStream captures a worker’s live byte stream into an ephemeral HLS preview and, on finalize, promotes the supplied final master to a permanent blob. A background reconciler garbage-collects abandoned preview sessions.

Only the Azure backend exists. STORAGE_PROVIDER accepts azure; the factory in app/core/storage/factory.py raises for anything else. s3 / mock are placeholder comments, not working backends.

Run locally

1. Point at an Azure Blob endpoint — set AZURE_CONNECTION_STRING at Azurite locally, or rely on the azure-identity managed-identity credential chain in the cloud.

2. Bring up core-database first (core-storage opens a channel to it on boot) and set CORE_DATABASE_GRPC; locally set CORE_DATABASE_USE_TLS=false.

3. Sync and run:

   run core-storage

   uv sync
   uv run python -m app.main

It listens on APP_PORT (default 80) and registers the native gRPC Health service — there is no HTTP /healthz (that is core-database’s shape). app.main also lifts the gRPC message ceiling to MAX_MESSAGE_BYTES (~100 MiB) so large UploadFile frames aren’t rejected with ResourceExhausted, and starts the live-session reconciler as a background task.

Provision the containers once

The private / public / live containers are otherwise assumed to pre-exist. Run ensure-storage-containers.sh once against a fresh account, or bring up the compose azurite-init step — see the docker quick start.

In staging the service is deployed externally reachable over TLS :443 (the container’s target_port is 80); the shared deploy template terminates TLS. See deploy a service.

Key files

• Directorycore-storage/

  • Directoryapp/

    • main.py boot: core-database channel, StorageServicer, health, reconciler
    • Directorygrpc_server/

      • Directoryservicers/

        • storage.py the whole StorageService — upload, transcode, sign, live ingest
      • reconciler.py background GC of stale live-preview sessions
    • Directorycore/

      • config.py every setting (containers, TTLs, transcode/ingest knobs)
      • Directorystorage/ Azure backend + the CloudStorageBackend interface + factory

        • …
      • Directorytranscode/ ffmpeg → CMAF/HLS + faststart M4A renditions

        • …
      • Directorylive/ ffmpeg live segmenter (byte stream → HLS-TS preview)

        • …
  • ensure-storage-containers.sh idempotent container provisioning

Key env vars

Names only — never commit values.

Var	Purpose
INTERNAL_RPC_SECRET	Bearer presented on every core-database call.
CORE_DATABASE_GRPC / CORE_DATABASE_USE_TLS	Upstream core-database address; TLS defaults on (set false locally).
APP_PORT	Listen port (default 80).
MAX_MESSAGE_BYTES	gRPC send/receive ceiling (default ~100 MiB) for large upload frames.
STORAGE_PROVIDER	Only azure is implemented; the factory raises otherwise.
AZURE_CONNECTION_STRING / AZURE_ACCOUNT_NAME	Azure auth (or the managed-identity chain).
AZURE_PRIVATE_CONTAINER / AZURE_PUBLIC_CONTAINER / AZURE_LIVE_CONTAINER	The three containers: private assets, public assets, ephemeral live previews.
STREAMING_VARIANT_ROLE / STREAMING_AUDIO_BITRATE / HLS_SEGMENT_SECONDS	CMAF/HLS rendition role, bitrate, target fMP4 segment length.
DOWNLOAD_VARIANT_ROLE / DOWNLOAD_LINK_DEFAULT_TTL	Lazy single-file download role, and the default signed-link TTL.
LIVE_SESSION_TTL_SECONDS / LIVE_SEGMENT_SECONDS / INGEST_MAX_CONCURRENT / RECONCILER_INTERVAL_SECONDS	Live preview lifetime, segment length, concurrent-ingest cap, GC cadence.

core-storage does not use Redis

The onboarding deck lists a REDIS_ADDR here — that is stale. core-storage has no Redis dependency: the live-session reconciler drives everything through core-database RPCs (ListStaleLiveSessions → delete_prefix → UpdateLiveSessionStatus), and config.py declares no Redis setting.

Live-to-VOD ingest

IngestStream is a bidirectional stream. The caller (typically the song-gen worker) sends a start frame, then live_chunk bytes, then either inline final_chunk frames or a final_url to fetch:

1. Preview. Bytes are fed to an ffmpeg live segmenter that writes an HLS-TS preview (live.m3u8 + .ts segments) into the live container. Each segment is signed and the playlist is rewritten to absolute signed URLs; the preview URL is emitted early so the UI can play while generation continues.

2. Finalize. The supplied final master is promoted to a permanent blob + files row. The live-preview MIME is decoupled from the master MIME — an MP3 live stream can finalize to an audio/flac master — then the preview is deleted and the session marked finalized.

3. GC backstop. If a worker dies mid-stream, the reconciler (reconciler.py) sweeps every RECONCILER_INTERVAL_SECONDS, deletes orphaned preview blobs, and marks the row expired.

The preview is best-effort: INGEST_MAX_CONCURRENT bounds concurrent segmenters and the service load-sheds the preview (serving without one) rather than starving ordinary uploads or failing the generation.

The traps that will actually bite you

The `live` container must exist or you get no preview URL

The live container is not created at runtime by default. On a fresh account the first PUT live/<session>/seg_00000.ts returns 404 ContainerNotFound, the ffmpeg pump dies, on_playlist never fires, and no streaming URL is ever emitted during generation (the master still finalizes fine — it promotes on a separate path). There is now a best-effort ensure_container self-heal in _start_live_preview, but the durable fix is to run ensure-storage-containers.sh (or the compose azurite-init) so the container pre-exists.

Two different songs must never dedup to one HLS playlist

An HLS playlist carries only relative segment filenames + durations — not the audio — so two different masters with the same segment layout produce byte-identical playlists. If the streaming variant were content-addressed on the playlist bytes alone, the second song would resolve its relative segment URIs against the first song’s directory and serve the wrong audio. The variant hash therefore folds in the master hash — streaming_variant_db_hash(raw_playlist_hash, container, master_db_hash) — so same-master re-renders stay idempotent while distinct songs never collide. See storage.py.

An HLS rendition is signed at serve time, and needs the CORS rule

A single Azure blob SAS can’t authorize the playlist’s sibling segments. So CreateDownloadLink (detecting the playlist MIME) reads the stored playlist, signs the #EXT-X-MAP init + every segment, writes a signed.m3u8, and returns that. hls.js then fetches the absolute signed segments cross-origin, which only works if the account-wide storage CORS rule is live for the app origin. The commit order that makes this safe is segments first, playlist last — the variant anchors on the playlist blob, so a crash mid-render leaves orphan segments but never a half-rendition a reader can see.

The HLS transcode is backgrounded — playback falls back to the master

The ~40–80 s CMAF/HLS transcode of a ~5-minute master used to run synchronously inside finalize, so it blocked the worker tool’s terminal “done” and became user-visible latency. It now runs as a fire-and-forget background task (re-fetching the committed master by hash, single-flighted per master). Until the rendition lands, CreateDownloadLink(role=streaming_aac) returns NotFound and (re-)kicks the render; the BFF falls back to the directly-playable master, so playback works either way and a later play picks up the finished HLS.

Empty library after reload — the ListFiles NULL-tags bug

Symptom: uploads show, then the library reads back empty after reload. Root cause is in the ListFiles SQL annotation (in soundverse-proto, served by core-database’s generated StorageDataService handler, not in this repo): an empty repeated tags field binds as SQL NULL, so CARDINALITY(NULL)=0 is NULL and the AND-chain drops every row. Fixed by wrapping in COALESCE(...). Filed here because it looks like a core-storage bug but lives on the data-plane side of the split.

Tests

pytest with asyncio_mode=auto (pyproject.toml). The suite stubs the storage layer, so no real Azure account is needed — the transcode, live-segmenter, ingest, and streaming-variant-hash paths all run against fakes.

Related

• Storage & media plane — the architecture and the why
• core-database — the StorageDataService this service calls
• Add a tool worker — workers upload via ctx.upload_file, never Azure directly
• TaskContext API — the ctx.* upload / streaming surface
• Environment variable catalog — every env name in one place
• Deploy a service — pushing core-storage to staging/prod