Skip to content
SoundverseSoundverse 2.0 — Engineering Docs

Add a field or RPC to a data service

The data tier is codegen-driven: there is no hand-written CRUD. A custom protoc plugin reads SQL annotations off each RPC and emits a Connect handler that runs that SQL against a pgx pool. So “add a column” means edit the proto + the schema, then regenerate — never write a Go handler by hand.

Steps

Section titled “Steps”

  1. Edit the .proto. Open the right file (e.g. proto/soundverse_proto/generation/task/v1/task.proto). Tag each row field with a column, and give a new RPC its SQL:

    task.proto
    string status = 7 [(soundverse_proto.db.v1.db_column) = "status"];
    

    rpc GetTask(GetTaskRequest) returns (GetTaskResponse) {
      option (soundverse_proto.db.v1.target_db) = "prod";        // which pool
      option (soundverse_proto.db.v1.sql_query) =
        "SELECT id, environment, tool_id, status "
        "FROM generation.tasks WHERE id = NULLIF($1, '')::uuid LIMIT 1";
      option (soundverse_proto.db.v1.require_internal_auth) = true;
    }

    For a Postgres ENUM column, use the db_enum_text field option (send the lowercase label, cast in SQL).

  2. Mind the $N rule. The request’s first field is always the shared soundverse_proto.common.v1.RequestContext, and it consumes no placeholder. So $1 is the first scalar field after the context. In QueueTaskRequest, context is field 1, environment is field 2 → $1; priority (field 18) is bound last as $17. Off-by-one here silently binds the wrong column.

  3. Update the sibling schema.sql. The DDL lives beside the proto (e.g. generation/task/v1/schema.sql), not in core-database — core-database runs no migrations. The compose Postgres init loads these schema.sql files read-only from the soundverse-proto submodule. Your RETURNING/SELECT column order must line up with the order of the db_column-annotated fields.

  4. Regenerate, in this exact order (mirrors ci.yml):

    in soundverse-proto/
    # build the custom DB plugin into ./bin first
    cd cmd/protoc-gen-soundverse-db && go build -o ../../bin/protoc-gen-soundverse-db . && cd ../..
    buf generate                                        # Go, Python(+stubs), Swift, Kotlin
    buf generate --template buf.gen.connect.yaml        # Connect handlers + Connect-ES TS
    buf generate --template buf.gen.soundverse-db.yaml  # SQL-backed Go DB bindings
    cd cmd/gen-db-registry && go run . && cd ..          # aggregate db_registry.go

  5. Merge to staging. staging-release.yml regenerates and publishes soundverse-proto-go@staging and soundverse-proto-python@staging (plus Swift / Kotlin / Web). See the deploy cascade for what happens next.

  6. Bump the Go consumers by hand. In core-database (and core-mcp if it calls the RPC) run ./update-proto.sh (it does go get …-go@staging && go mod tidy), commit, and redeploy. The regenerated registry/db_registry.go mounts the new handler via RegisterAllDBServices(mux, dbPools, redisClient, internalSecret, cacheKeyPrefix).

  7. Python consumers auto-update. A PR to pick up the new SDK is opened automatically.

The four things that must agree

Section titled “The four things that must agree”

For a row to round-trip correctly, these four must line up — the plugin scans field order:

  1. the sql_query column list (SELECT/RETURNING),
  2. the db_column annotations on the message fields,
  3. the schema.sql table definition,
  4. the $N request-field binding order.
Section titled “Related”