core-migration

core-migration is the engine that moves the old Soundverse 1.0 platform into 2.0. It is a Python queue-style worker — no ingress, no HTTP — that reaches out to the legacy prod database, the old public blob bucket, and core-database, and converges the legacy world into the 2.0 schemas one idempotent batch at a time. It carries two phases in one repo:

• Phase 1 — user + billing cutover. Mint 2.0 identities and grant carried-over tokens for the legacy user base (~1.24M rows). Opt-in; off by default.
• Phase 2 — library content re-ingest. Download each mapped user’s old library files, hash-dedup them into the 2.0 private bucket, attribute them to the 2.0 identity, and label them legacy. On by default.

Deploy status — wired, but functionally inert

The worker is feature-complete and the staging deploy workflow is wired (a push to staging deploys it as a no-ingress worker; the image defaults to MODE=worker). But the migration does nothing yet: its mapped-only reader joins migration.legacy_user_map, and that table has zero rows on prod. The upstream users+billing map is never populated because the deployed core-database does not serve MigrationService — a Connect probe of MigrateLegacyUser returns HTTP 404 (its committed go.mod pins a stale soundverse-proto-go whose registry predates the migration registration). See the blocker. Prod is not deployed.

The two phases

Both phases live behind independent env flags, so a deploy can run either, both, or (the safe default) only Phase 2.

Phase	Flag (env name)	Default	What it does	Backing SQL
1 · user + billing cutover	RUN_USER_CUTOVER	off	Loops migration.bulk_migrate_batch() — mints identities + grants carried-over tokens for the dormant legacy tail. Idempotent + resumable; logs progress (dormant users aren’t watching).	migration.bulk_migrate_batch() in bulk-cutover.sql
2 · library re-ingest	RUN_FILE_WORKER	on	Continuous re-ingest of every mapped user’s not-yet-migrated files into the 2.0 private bucket, with per-owner progress notifications.	migration.legacy_file_map spine (below)

Phase 1 mints identities and spends the ledger — treat it as a switch, not a knob

Turning on RUN_USER_CUTOVER writes to identity and the token ledger for ~1.24M users. It must only run after the prod pre-flight (the dead airbyte replication slot pinning WAL is resolved, and migration.* is excluded from the publication) or a bulk write can exhaust Azure storage. It is deliberately gated as an env var on the Container App, not a code path you flip casually.

Two run shapes: MODE=job vs MODE=worker

The same image runs either shape, selected by the MODE env var (default worker):

• MODE=job — one pass over the eligible rows, then exit. The original Azure Container Apps Job shape: re-run or shard to converge. Entry point app/main.py run().
• MODE=worker — the always-on service (the deployed default). Runs the optional cutover once, then loops the file re-ingest forever, consuming a Redis login-wake and emitting progress notifications. Entry point app/worker.py run_worker().

The per-file pipeline

Every file flows through the same idempotent pipeline in app/migrator.py. The claim mints a deterministic new_file_id, so every downstream write is keyed on a stable id and a crash anywhere resumes cleanly on the next pass — no duplicate library row.

flowchart TD
  PT["prod public.* legacy tables<br/>+ migration.legacy_user_map"]
  OB["old public bucket<br/>(soundversebuckets)"]
  M["core-migration worker<br/>claim → download+hash → blob → file → attrs → commit"]
  SDS["core-database · StorageDataService<br/>GetBlobByHash / InsertBlob /<br/>UpsertAttributes / Lyrics / Provenance"]
  FM["migration.legacy_file_map<br/>claim + commit (direct SQL)"]
  SF["storage.files insert<br/>(direct SQL: id + preserved created_at)"]
  NB["2.0 private bucket<br/>soundversecore/private"]

  PT -->|"mapped-only reader (JOIN legacy_user_map)"| M
  M -->|"claim: mint new_file_id"| FM
  M -->|"download + sha256"| OB
  M -->|"dedup (GetBlobByHash) / InsertBlob"| SDS
  M -->|"upload bytes"| NB
  M -->|"file row"| SF
  M -->|"attributes / lyrics / provenance"| SDS
  M -->|"finish: status=migrated"| FM

  classDef external fill:#f59e0b,color:#111,stroke:#b45309
  classDef worker fill:#10b981,color:#fff,stroke:#047857
  classDef data fill:#8b5cf6,color:#fff,stroke:#6d28d9
  class PT,OB,NB external
  class M worker
  class SDS,FM,SF data

1. Claim. Insert (or refresh) the migration.legacy_file_map row for (source_table, old_file_id). new_file_id is minted by the table DEFAULT (storage.uuid_generate_v7) on first claim and preserved on re-claim. A returned status of migrated means a prior pass finished this row → skip.

2. Download + hash. Stream the old blob to a temp file (bounded by MAX_FILE_BYTES / DOWNLOAD_TIMEOUT_SEC) and compute the raw sha256. A 404 / dead URL is recorded terminal as dead_url; an over-cap file as skipped_too_large — so neither is re-fetched every pass.

3. Blob (dedup-then-upload). Compute the content address db_blob_hash = sha256(raw:'private') (scoped_blob_hash). GetBlobByHash → reuse the existing blob on a hit; otherwise upload_file to Azure and InsertBlob. This is the same content-dedup contract as the rest of the storage plane.

4. File row (direct SQL). Insert storage.files with the deterministic id, the preserved legacy created_at, visibility = PRIVATE, and non-empty tags (ON CONFLICT (id) DO NOTHING).

5. Side tables. UpsertAttributes (the display / legacy / audio / image / video namespaces, set_by='migration'), plus UpsertFileLyrics / UpsertFileProvenance where the source has them.

6. Commit. Mark the map row migrated with the resolved hash, blob URL, size, and mime.

Why a dedicated job, not UploadFromUrl

core-migration does not call core-storage’s UploadFromUrl. That RPC’s InsertFile is non-idempotent (random id, no ON CONFLICT), hard-codes created_at = NOW() (which would clump the whole legacy library at the top of the sort), and its streaming-variant path would pin shared-prod core-storage under the load. The direct pipeline controls the id and created_at, and is re-runnable by construction.

The legacy_file_map spine

migration.legacy_file_map is the idempotency + dedup + coverage ledger for Phase 2. Its DDL lives in the migration proto’s schema.sql with a prod-apply twin at scripts/db/migration/add-legacy-file-map.sql (the DB-role grants runbook is in that file’s header).

• Primary key (source_table, old_file_id) — exactly one map row per legacy file.
• new_file_id minted at claim → the stable storage.files.id shared by the blob, file, and attribute writes.
• Terminal statuses — migrated, dead_url, skipped_too_large, error. The reader excludes the first three, so coverage converges as passes advance. The reconciliation report groups the map by (source_table, status).

The attribution gate: mapped-only, read-only

The reader (app/core/db.py) bakes the attribution gate into the SQL: it JOINs migration.legacy_user_map and enumerates a file only when its owner already has a 2.0 identity — a row with an allowed status (ALLOWED_USER_STATUSES, default migrated,held_enterprise) and a non-null new_user_id. This makes the content phase strictly downstream of the user phase:

• Read-only against the map. Enumerating a file triggers none of migrate_legacy_user()‘s side effects (token grants, spend blocks, airbyte). Dormant users’ files are simply not read — they get re-picked on a later pass once they log in and map.
• Connectors-first sequencing. The users+billing phase itself is sequenced behind the 2.0 payment rails: the plan builds the core-billing Stripe / PayU / Apple / Google connectors and the Logto Apple sign-in connector first, so the ~3,300 live consumer subscriptions port cleanly and Apple users have a 2.0 login path before their data moves.

The user_id crux — migrated files must carry the new UUID

2.0 library reads scope on caller.user_id = the identity UUID (LibraryService.ListLibrary → ListFiles(user_id=...)). So the migrated storage.files.user_id must be legacy_user_map.new_user_id (the new UUID), never the legacy char(24) id. Lifting-and-shifting the old id verbatim would orphan every migrated file. The reader resolves the UUID via the join; do not shortcut it.

Data-plane split

core-migration deliberately routes some writes through core-database and does others as direct SQL:

Path	Operations	Why
core-database StorageDataService (gRPC, internal-auth)	GetBlobByHash, InsertBlob, UpsertAttributes, UpsertFileLyrics, UpsertFileProvenance	Keep the generated SQL + Redis cache-invalidation centralized in the one Postgres-credentialed service.
Direct psycopg pool (prod DSN)	Read legacy public.* + legacy_user_map; legacy_file_map claim/commit; storage.files insert	The insert must control the id (minted at claim) and the preserved legacy created_at — neither of which the InsertFile RPC exposes.

The direct pool is autocommit, single-statement, so a crash always leaves a clean, resumable state.

Scope, labels, and sources

Locked decisions: all media (audio + image + video); DNA and derivative/copy tables excluded; everything ingests private — old isPublic / isStaffPick are preserved as labels, not bucket placement.

Ten SourceSpecs in app/sources.py map each legacy file-bearing table to 2.0 labels (6 audio: audios, user_uploaded_audios, user_recorded_audios, ai_studio_generated_audios, standalone_tts_generations, standalone_sfx_generations; 2 image: user_uploaded_images, banana_messages; 2 video: user_uploaded_videos, videos). Each spec encodes the exact legacy SQL identifiers — old unquoted camelCase folded to lowercase (audiourl, userid), but standalone_* kept quoted "userId" / "createdAt", and videos is keyed by task_id, not _id.

Labelling is uniform: storage.files.tags is always ['legacy', <media>, <source_key>, …] (never empty — that dodges the empty-tags → NULL insert bug), and full-fidelity metadata (title, old visibility flags, bpm, dimensions, etc.) lands in storage.attributes. See Add durable per-file metadata for the attribute model this rides on.

Progress notifications & the login-wake

In MODE=worker, Phase 2 runs two coordinated loops:

• File loop — a full re-ingest pass over all mapped users, then, when idle, a BRPOP on the Redis wake list {env}:common:migration:wake. core-identity LPUSHes a user_id there right after that user’s tokens migrate at login, so their library is swept immediately instead of at the next idle sweep. Best-effort: with no REDIS_ADDR the loop falls back to timed idle sweeps.
• Progress reporter — rolls up legacy_file_map per recently-touched user and UpsertNotifications a library_migration row (dedup key library_migration:{user_id}) via core-database’s NotificationDataService. That is what surfaces “Importing your library…” → “Your library is ready” in the app’s notification center. Both loops are wrapped so a notification hiccup never breaks the migration.

Operating the worker

All knobs are env vars (names only — never commit values). Scale-out is by sharding: run N replicas with the same SHARD_COUNT and distinct SHARD_INDEX (mod(abs(hashtext(old_file_id)), shard_count) == shard_index). The legacy_file_map claim ON CONFLICT is the real double-processing guard — sharding is throughput, not correctness.

Env name	Purpose
MODE	worker (always-on, default) or job (one pass, exit)
RUN_USER_CUTOVER / RUN_FILE_WORKER	Enable Phase 1 / Phase 2
CUTOVER_BATCH_SIZE / CUTOVER_SLEEP_SEC	Phase 1 batch size + breather between batches
DATABASE_DSN	Direct prod Postgres DSN (pooler)
CORE_DATABASE_GRPC / CORE_DATABASE_USE_TLS	core-database StorageDataService endpoint
INTERNAL_RPC_SECRET	Shared internal-auth bearer for core-database + notifications
AZURE_ACCOUNT_NAME / AZURE_PRIVATE_CONTAINER / AZURE_CONNECTION_STRING	Destination private bucket (empty connection string → managed identity)
REDIS_ADDR / REDIS_PASSWORD	Login-wake list (empty → timed sweeps only)
SHARD_COUNT / SHARD_INDEX / CONCURRENCY / MAX_DOWNLOADS_PER_SEC	Scale-out + source-bucket throttle
SOURCES / MEDIA_TYPES / USER_ID	Restrict to source tables / media lane / a single owner
DRY_RUN / LIMIT / BATCH_FETCH	Enumerate-only, bounded cohort, keyset page size
MAX_FILE_BYTES / DOWNLOAD_TIMEOUT_SEC	Per-file download guards

Safe first pass

Start bounded: DRY_RUN=true with a small LIMIT on a cohort of ~100 mapped users, then validate the result through LibraryService.ListLibrary before opening the throttle. Run large videos in their own lane (MEDIA_TYPES=video, lower CONCURRENCY).

The blocker: the map is empty

The whole migration is currently inert, and the fix is upstream, not here:

1. migration.legacy_user_map has 0 rows on prod. No user has ever been migrated, so the mapped-only reader enumerates nothing.

2. Root cause: core-database doesn’t serve MigrationService. A Connect probe of MigrateLegacyUser returns 404 while TokenService at the same host/auth returns 200. The deployed core-database’s committed go.mod pins a stale soundverse-proto-go whose RegisterAllDBServices predates the migration (and DNA) registration — core-database auto-registers from that registry, so a stale pin = missing routes.

3. Fix: bump core-database’s go.mod / go.sum to a soundverse-proto-go that registers MigrationService, rebuild, redeploy, and re-probe. Then apply the legacy_user_map / legacy_file_map DDL + grants on prod. The JIT login path in core-identity (ingest.migrate_legacy) is already wired and is secondary — the 404 is the hard blocker.

Until then, the read-only Legacy 1.0 → 2.0 bridge (/api/compat/*) is what actually surfaces pre-2.0 data to users — a separate, already-shipping path that reads the old backend live rather than migrating it.

Key files

• Directorycore-migration/

  • Directoryapp/

    • main.py MODE=job entry — one pass, then exit
    • worker.py MODE=worker entry — cutover + file loop + reporter
    • runner.py sharded, keyset-paged pass driver
    • migrator.py the per-file pipeline (claim → … → commit)
    • sources.py the 10 SourceSpecs + label builders
    • notifications.py library_migration progress emitter
    • Directorycore/

      • config.py all env knobs
      • db.py direct-SQL layer (reader gate + claim/commit + file insert)
  • Dockerfile defaults ENV MODE=worker
  • Directory.github/workflows/ deploy-aca-{staging,prod}.yml (no ingress)

    • …

Related

Legacy 1.0 → 2.0 bridgeThe read-only /api/compat layer that serves old data today.

core-identityJIT provisioning + the login migrate_legacy call and Redis wake.

core-databaseServes StorageDataService — and must serve MigrationService.

core-billingThe payment rails the user cutover is sequenced behind.

Storage & media planeThe content-dedup + private-bucket contract the re-ingest reuses.

Add durable per-file metadataThe attributes model the migrator writes into.