Run proto codegen locally

soundverse-proto is the source of truth for every contract in the fleet and a custom code-generation pipeline. After you edit a .proto (or its sibling schema.sql), you regenerate to turn the annotated contract into per-language SDKs plus ready-to-run, SQL-backed Go Connect handlers. This page is the exact ordered recipe — it mirrors .github/workflows/ci.yml, so if these commands pass locally, CI will too.

This regenerates; it does not publish

Everything lands under gen/, which is gitignored — the output is published to the per-language repos (soundverse-proto-{go,py,web,swift,kotlin}), never committed here. Regenerating locally is for editing and verifying the contract. For how the regenerated stubs actually reach the services, see Refresh proto stubs and the proto → deploy cascade.

TL;DR — the whole pipeline

Four codegen passes plus one registry program, in this exact order (the DB plugin must exist before the pass that invokes it):

regenerate everything (run from the soundverse-proto repo root)

# 1. Build the custom DB plugin into ./bin (required before step 4)
cd cmd/protoc-gen-soundverse-db && go build -o ../../bin/protoc-gen-soundverse-db . && cd ../..

# 2. Standard SDKs: Go, Python (+ mypy stubs), Swift, Kotlin
buf generate

# 3. Connect Go handlers + Connect-ES TypeScript clients
buf generate --template buf.gen.connect.yaml

# 4. SQL-backed Go DB bindings (uses the plugin built in step 1)
buf generate --template buf.gen.soundverse-db.yaml

# 5. Aggregate registry (scans the generated *.soundverse_db.go files)
cd cmd/gen-db-registry && go run . && cd ../..

Prerequisites

The dev environment is a Nix flake + direnv: .envrc runs use flake, and flake.nix provides buf, go, python311, and mypy-protobuf. Without Nix, install the three things the pipeline actually shells out to:

• buf — drives all three buf generate passes.
• Go — builds the custom DB plugin and runs the registry program (CI pins Go 1.26; anything >= 1.22 works).
• mypy-protobuf — buf.gen.yaml references the local plugins protoc-gen-mypy / protoc-gen-mypy_grpc, which mypy-protobuf supplies. pip install mypy-protobuf and make sure they’re on PATH, or step 2 fails.

What each pass emits

Three of the four passes are plain buf generate runs against a different template; the last is a Go program. Every template is a real file in the repo root.

#	Template / command	Emits
1	go build the DB plugin → ./bin/	The local protoc-gen-soundverse-db binary step 4 depends on.
2	buf.gen.yaml (default)	gen/go, gen/python (+ mypy .pyi stubs), gen/swift, gen/kotlin.
3	buf.gen.connect.yaml	Connect Go handlers (connectrpc/go) into gen/go, and Connect-ES TS clients (bufbuild/es, protobuf-es v2) into gen/web.
4	buf.gen.soundverse-db.yaml	One *.soundverse_db.go per DB-backed service, via ./bin/protoc-gen-soundverse-db.
5	cmd/gen-db-registry (go run .)	gen/go/soundverse_proto/registry/db_registry.go — the aggregate RegisterAllDBServices mount point.

Steps

1. Build the DB plugin first. ordering trap buf.gen.soundverse-db.yaml declares local: ./bin/protoc-gen-soundverse-db — a binary that does not exist until you build it. Build it into ./bin/ before step 4:

   cd cmd/protoc-gen-soundverse-db && go build -o ../../bin/protoc-gen-soundverse-db . && cd ../..

   bin/ is gitignored, so this is a per-checkout step. Skip it (or leave a stale binary after editing the plugin) and step 4 either fails to find the plugin or emits outdated handlers.

2. Generate the standard SDKs. buf generate with no --template uses the default buf.gen.yaml and emits Go, Python (with mypy stubs), Swift, and Kotlin:

   buf generate

3. Generate the Connect layer. This produces the Connect Go handler scaffolding and the browser TypeScript clients the SaaS BFF consumes:

   buf generate --template buf.gen.connect.yaml

4. Generate the SQL-backed Go DB bindings. This is the pass that invokes the plugin from step 1. For every RPC annotated with sql_query / target_db / db_column, it writes a complete *.soundverse_db.go Connect handler that runs the SQL against a pgx pool, scans rows into the response message, and applies the Redis cache options. It fails fast — a bad annotation aborts generation with an error, so a green run means your contract is internally consistent:

   buf generate --template buf.gen.soundverse-db.yaml

5. Generate the aggregate registry. gen-db-registry is not a plugin — it’s a Go program that walks the generated *.soundverse_db.go files with go/ast and emits registry/db_registry.go:

   cd cmd/gen-db-registry && go run . && cd ../..

   That file exposes one function consuming services call to mount every DB-backed handler:

   func RegisterAllDBServices(
       mux *http.ServeMux,
       dbPools map[string]*pgxpool.Pool,
       redisClient redis.Cmdable,
       internalSecret string,
       cachePrefix string,
   ) error

   It also emits build-provenance accessors (Version(), SourceCommit(), SourceRef()) fed by the PROTO_GO_VERSION / PROTO_GO_SOURCE_COMMIT / PROTO_GO_SOURCE_REF env vars. Those are optional locally — unset, they fall back to GITHUB_SHA / GITHUB_REF_NAME and finally to "unknown". CI sets them; you don’t need to.

Verify it the way CI does

CI gates on formatting and lint before it generates, and unit-tests the two codegen programs separately. Run the same checks locally:

buf format -d --exit-code   # format check (diff + non-zero exit on drift)
buf lint                    # STANDARD ruleset, PACKAGE_VERSION_SUFFIX excepted

# Golden + unit tests for the codegen programs (needs only Go — the golden
# test runs against a committed FileDescriptorSet fixture, no proto compile)
( cd cmd/protoc-gen-soundverse-db && go test -race ./... )
( cd cmd/gen-db-registry         && go test -race ./... )

buf breaking is intentionally off

Compatibility checks are disabled during the startup phase — the contract may reshape freely, so buf generate will happily regenerate a breaking change. Don’t rely on the proto repo to catch a breaking edit; that’s on you and the consumers. See the proto → deploy cascade for how a change lands downstream.

What you touch and what you produce

• Directorysoundverse-proto/

  • Directoryproto/ the contracts you edit (.proto + sibling schema.sql)

    • …
  • buf.gen.yaml pass 2 — standard SDKs
  • buf.gen.connect.yaml pass 3 — Connect Go + web TS
  • buf.gen.soundverse-db.yaml pass 4 — DB handlers (local plugin)
  • Directorycmd/

    • Directoryprotoc-gen-soundverse-db/ the custom protoc plugin (build first)

      • …
    • Directorygen-db-registry/ pass 5 — aggregate registry program

      • …
  • Directorybin/ built plugin binary lands here — gitignored

    • …
  • Directorygen/ all generated output — gitignored, published never committed

    • …

The README is stale — trust the code

The repo’s README.md still describes an earlier “proto contracts only” vision from before the DB codegen layer existed. The authoritative descriptions are CLAUDE.md and ci.yml.

Related

• Add a field or RPC to a data service — the edit that makes you run this
• Refresh proto stubs after a contract change — pull the regenerated SDKs into consumers
• Drive the proto → deploy cascade — publish → consumer bump → deploy
• How the DB codegen plugin works — what step 4 actually emits
• The contract: proto as source of truth — why the proto is authoritative
• Proto contract reference — the DB annotations and buf templates