CI/CD deploy pipeline

Deployed Every service in the fleet deploys the same way: push to a long-lived branch, a thin per-service workflow calls one shared reusable template, and that template builds an image to ACR and creates-or-updates an Azure Container App. The clever part — and the part that most often trips people up — is that environment variables and inter-service URLs are wired by naming convention, not by hand. Read this before you add a service or wonder why one service “can’t find” another.

This page is the how it works map. For the day-to-day runbook (triggering, watching, rolling back) see Deploy a service and Roll back a deployment.

The pipeline at a glance

flowchart LR
  DEV[push to staging or prod branch]:::worker
  WF["per-service caller<br/>deploy-aca-staging/prod.yml"]:::worker
  TMPL["shared template<br/>service-template deploy-aca-template.yml"]:::worker
  IMG[docker build and push image]:::worker
  ENV["parse env: collect STAGING_/PROD_<br/>secrets + vars, strip prefix"]:::worker
  ACR[(ACR soundversegeneral)]:::external
  ACA["az containerapp<br/>create or update"]:::external
  FQDN["publish FQDN as<br/>ENV_SERVICE_GRPC org variable"]:::external

  DEV --> WF --> TMPL
  TMPL --> IMG --> ACR --> ACA
  TMPL --> ENV --> ACA
  ACA --> FQDN

  classDef worker   fill:#10b981,color:#fff,stroke:#047857
  classDef external fill:#f59e0b,color:#111,stroke:#b45309

Branch → environment model

There are two long-lived deploy branches: staging and prod. Every deployable repo carries two workflow files, each only a handful of lines:

Trigger	Caller file	Calls
push: staging	.github/workflows/deploy-aca-staging.yml	service-template/.github/workflows/deploy-aca-template.yml@staging
push: prod	.github/workflows/deploy-aca-prod.yml	the same template pinned @prod

The per-service caller is intentionally dumb — it picks the environment, names the app, declares ingress/port, and forwards all org secrets with secrets: inherit. Here is the real staging caller for the consumer gateway (core-gateway-consumer/.github/workflows/deploy-aca-staging.yml), in full:

deploy-aca-staging.yml

jobs:
  deploy-staging:
    uses: soundversegit/service-template/.github/workflows/deploy-aca-template.yml@staging
    with:
      github_environment: "staging"
      service_name: "core-gateway-consumer-staging"
      dockerfile_path: "Dockerfile"
      enable_ingress: true
      is_external: true
      target_port: 80
      pass_github_token_to_docker: true
    secrets: inherit

The prod caller is the same shape — it just sets github_environment: "prod", service_name: "...-prod", and pins @prod. Because the engine lives in one repo (service-template), a fix to the deploy logic reaches every service on its next push; nothing is vendored or copy-pasted.

One naming exception

soundverse-saas-2.0 names its files deploy-staging.yml / deploy-prod.yml (no -aca-) and runs the deploy behind a quality gate that reuses its own ci.yml lint/typecheck job.

A worker has no ingress

Background workers (the whole tool fleet) set enable_ingress: false and omit the port — see core-tool-agent/.github/workflows/deploy-aca-staging.yml and the copy-me template-core-tool. They get no FQDN and accept no inbound traffic; they reach Redis, core-database, and core-storage as clients. To wire a brand-new repo in, see Add a service to the pipeline.

What the shared template does

All the real work lives in service-template/.github/workflows/deploy-aca-template.yml. One pass through it, in order:

1. Mint a private-repo token. Uses the GitHub App creds GO_PUBLISH_APP_ID / GO_PUBLISH_APP_PRIVATE_KEY via actions/create-github-app-token, so the build can clone the private git deps (the soundverse-proto-* SDKs, soundverse-py). Falls back to GH_READ_REPO_TOKEN if the App isn’t configured; if neither exists the job fails fast.

2. Build & push the image. docker/build-push-action tags soundversegeneral.azurecr.io/<service>:<sha> and :latest. The token is handed to BuildKit as secrets: "gh_token=…" — consumed by a --mount=type=secret in the Dockerfile — so it is never baked into an image layer. The GHA layer cache is keyed per <service>-<environment>.

3. Parse env vars & secrets. A short Python step reads toJSON(secrets) and toJSON(vars) and rewrites them into Azure flags by prefix — the convention detailed below.

4. Create or update the app. If az containerapp show finds nothing, it runs az containerapp create (with ingress flags, --min-replicas 1 --max-replicas 10, secrets, env). Otherwise it runs az containerapp secret set + az containerapp update (new image + env) and toggles az containerapp ingress enable/disable.

5. Apply a readiness probe (optional). Only when health_check_path is set. az containerapp create/update has no probe flag, so the step reads the live template JSON, injects an HTTP readiness probe into every container, and re-applies via --yaml. Today only core-identity uses this (/healthz).

6. Publish the FQDN. If the app has ingress, it fetches the deployed FQDN and writes it back as a GitHub org variable — this is how services discover each other (next section).

The env-injection convention

There is no per-service .env in the cloud. Configuration lives as GitHub org-level Variables and Secrets, and the template selects and renames them by prefix. The rule, implemented in the “Parse Environment Variables & Secrets” step:

• The job computes a prefix from github_environment: STAGING_ or PROD_.
• Every Variable named STAGING_<NAME> is injected as <NAME> (prefix stripped) — a plain env var.
• Every Secret named STAGING_<NAME> becomes an ACA secret <name-kebab> and is injected as <NAME>=secretref:<name-kebab> — the value never appears in the container spec, only a reference.
• A few keys are always ignored: GITHUB_TOKEN, AZURE_CREDENTIALS, GO_PUBLISH_APP_ID, GO_PUBLISH_APP_PRIVATE_KEY, GH_READ_REPO_TOKEN.

So an org variable STAGING_REDIS_ADDR arrives in the container as REDIS_ADDR; the secret STAGING_INTERNAL_RPC_SECRET arrives as INTERNAL_RPC_SECRET via a secretref. Swap the prefix to PROD_ and the same variable set produces the prod config — that is the entire “environment” mechanism.

Service discovery rides the same convention. Step 6 writes the app’s FQDN back as an org variable named <ENV>_<SERVICE>_GRPC (the trailing _STAGING/_PROD is stripped from the service name first). External apps are published as fqdn:443, internal ones as fqdn:80. So deploying core-database-staging sets STAGING_CORE_DATABASE_GRPC, which is then injected into every other service as CORE_DATABASE_GRPC on its next deploy. The full convention — including local TLS-off flags — is on Environment & service discovery; the fleet-wide names live in the env-var catalog.

A name mismatch fails silently

The contract is purely by string. If a service reads a different env name than what is published for the same thing, it gets nothing and quietly falls back to a code default — no error, just wrong behaviour. This is the class of bug behind the historical INTERNAL_AUTH_SECRET vs INTERNAL_RPC_SECRET and REDIS_URL vs REDIS_ADDR drifts. Cross-check the name your code reads against the published org variable.

There is no GitHub environment: block

The deploy job does not declare a GitHub environment: block — config comes from org-level Variables/Secrets, not environment-scoped ones. A token with only read:org cannot enumerate them, so the only reliable way to see what a running service actually got is to read the live container: see Inspect a running service. Injected secrets show as a secretRef, never a value — by design.

The proto release cascade

A proto change deploys through its own pipeline, not this one, and the two shapes differ. Python consumers auto-bump in two hops: a soundverse-proto release bumps soundverse-py, whose moving main fires soundverse_updated; each worker’s update-private-deps.yml listens for that dispatch (plus a daily 06:00 cron safety net), runs uv lock --upgrade-package soundverse, and opens a PR on branch automation/update-private-deps — merge it once CI is green, which triggers the deploy above. Go services (core-database, core-mcp) have no auto-bump; their repository_dispatch: [proto_updated] trigger is commented out, so they are advanced manually with go get …@staging. The full mechanics — and why the prod cascade needs hand-coordination — are on Drive the proto → deploy cascade.

Deploy & roll back

A push (or fast-forward merge) to staging/prod runs the workflow on push. For an off-cycle run, dispatch it and watch it from the repo:

off-cycle staging deploy

gh workflow run deploy-aca-staging.yml --ref staging
gh run watch

To roll back, either re-deploy a known-good commit (a fresh image + update) or pin ACA ingress traffic to a previous revision. The step-by-step runbooks are on Deploy a service and Roll back a deployment.

Deploy in dependency order

When a contract change spans several services, deploy producers before consumers — typically core-database first, then the services that call it. Revision-based rollback only restores the image; it does not undo an org-variable change, so prefer re-deploying a good commit when config also drifted.

Related

• Deploy a service (staging & prod) — the day-to-day runbook
• Roll back a deployment — revision pinning + redeploy
• Environment & service discovery — the injection convention in depth
• Add a service to the pipeline — wire a new repo in
• Inspect a running service — read effective env without leaking values
• Drive the proto → deploy cascade — how a proto change propagates