Set per-tool rate limits (and how tiers scale them)

Rate limits are the sibling of pricing: the same declarative class attribute, the same fleet-upserts-on-boot flow, the same three-level scope cascade. The machinery is live and enforced today — a Redis fixed-window counter shared byte-for-byte between core-mcp and the consumer gateway — but no production tool sets a limit. Every tool in the fleet declares rate_limit = None (unlimited), so the gate is wired and dormant.

Live, but no tool uses it yet

This page is how you turn it on for a tool, how “different limits for different tiers” works — a per-plan multiplier by default, or a manual per-tier override when you need an exact number — where users see their usage, and the sharp edges before you ship.

The one-sentence model

You declare one base limit per tool. The caller’s subscription plan multiplies it by billing.plans.rate_limit_multiplier. Free = ×1, so the number you write is the free-tier allowance, and paid tiers scale up from it proportionally.

Where the limit lives

You declare it as a class attribute on your tool, next to pricing and credentials. On every boot, after RegisterTool, WorkerFleet._apply_tool_config (fleet.py) upserts it — only if it’s non-None — via the idempotent UpsertToolRateLimits RPC. Your code is the source of truth: edit the value, restart the worker, it re-applies. This is step 3 of the config sequence (UpsertToolConfig → UpsertToolLicensePricing → UpsertToolRateLimits → UpsertToolCredentials).

The RateLimit dataclass is tiny — see config.py:

soundverse/tools/config.py

@dataclass(frozen=True)
class RateLimit:
    """Per-config rate limits. ``0`` means unlimited for that window."""
    limit_per_hour: int = 0
    limit_per_day: int = 0

And the default on BaseTool (base.py) is rate_limit: RateLimit | None = None — which is why every un-configured tool is unlimited: the fleet skips the upsert entirely when it’s None, so no row is written.

Turn it on

Declare a RateLimit with an hourly and/or daily window. Both are request counts, not tokens.

app/tools/my_tool.py (class attributes)

from soundverse.tools import RateLimit

class MyTool(BaseTool[MyInput, MyOutput]):
    model = "my-model"
    operation = "generate"
    rate_limit = RateLimit(limit_per_hour=20, limit_per_day=100)
    # ...pricing, credentials, etc.

That’s the whole tool-author surface. Restart the worker and the fleet writes the row. The number you pick is the free-tier allowance (free multiplier is 1.0); a Creator user gets ×2, a Pro user ×5, and so on (see the tier table).

Two DB constraints will reject a bad RateLimit

generation.tool_config_rate_limits (schema.sql) enforces:

• each window must be > 0 (chk_rate_limits_positive), and
• limit_per_day >= limit_per_hour (chk_rate_limits_day_gte_hour).

Set a window to 0 to mean “unlimited for that window” (the fleet coerces 0 → NULL on the way in). You cannot express limit_per_hour=50, limit_per_day=10 — the upsert fails the check.

Do they actually work? Yes — one shared counter, two enforcers

Rate limiting is enforced in two services, and they are the same limiter on purpose.

Consumer gateway (today's real path)

core-gateway-consumer enforces inline in the generation pipeline — step 1 of resolve tool → rate-limit → price → reserve → QueueTask. It’s not an interceptor; it runs inside pipeline.py via the Redis limiter in ratelimit.py (explicitly “a port of core-mcp/internal/ratelimit”). A breach raises RateLimited → RESOURCE_EXHAUSTED / ERROR_CODE_RATE_LIMITED. This path queues straight to the worker fleet — it does not call core-mcp — so for consumer traffic this is the single enforcement point.

core-mcp (agent / MCP path)

core-mcp enforces the identical check inline in toolcall/handler.go, step 1 of Handler.Handle, using internal/ratelimit/ratelimit.go. Same windows, same key scheme, same rollback. This is the gate for the agent/MCP tool-call path.

Studio gateway (NOT wired)

core-gateway-studio has a CONVERSION_LIMITS_BY_PLAN table (FREE 5rpm, CREATOR/PRO 30rpm, MAX 200–400rpm) in routers/conversion/config.py — but nothing imports it. The whole conversion package is dead code (the only export anyone uses is REDIS_LISTENER_ENABLED). Studio applies no rate limiting. Don’t be misled by that config file.

Both live enforcers write the same Redis keys, a deliberate cross-language contract (keys.common(env, ...) in Python, keys.CommonKey(env, ...) in Go):

the shared key shape — {env}:common:...

rl:tool:{user}:{workspace}:{project}:{tool_id}:h     # tool, per hour
rl:tool:{user}:{workspace}:{project}:{tool_id}:d     # tool, per day
rl:model:{user}:{workspace}:{project}:{model}:h      # model, per hour
rl:model:{user}:{workspace}:{project}:{model}:d      # model, per day

Four counters per call — a tool dimension and a model dimension, each in an hour and a day window. Each is a fixed-window INCR with a first-hit EXPIRE; a window that would exceed its limit triggers a DECR rollback of everything touched, so a rejected call burns no quota. A window whose limit is 0 is skipped (unlimited).

Shared bucket, not two buckets

Because the keys match, core-mcp and the consumer gateway draw down one bucket. A user who spends their hourly quota via the agent path has less left on the consumer path. This is intended — the alternative (a tier suffix or a per-service prefix) would split counters and double a user’s real quota. Don’t add either.

Fail-OPEN when Redis is absent

Both limiters short-circuit to allowed when their Redis handle is nil/unset. core-mcp no-ops if REDIS_ADDR is unset; the consumer gateway no-ops if its Redis client failed to build. In a real deploy Redis is required for the event stream so it’s always present — but if you’re testing “why isn’t my limit firing?”, check Redis is actually wired first.

How tiers scale it

By default a tier just applies a scalar multiplier to your base limit at enforcement time (generation.model_rate_limits is multiplier-only — models have no per-tier override). For an exact per-tier number, add a plan_code override row on generation.tool_config_rate_limits — see Manual per-tier overrides above.

The multiplier lives on billing.plans.rate_limit_multiplier, seeded in core-billing (plans.py):

Plan family	rate_limit_multiplier	Effective limit if base = 20/hr
free	1.0	20/hr
creator	2.0	40/hr
pro	5.0	100/hr
max_x1 … max_x5	10 / 20 / 30 / 40 / 50	200 … 1000/hr

The gateway resolves the caller’s plan via GetEffectiveEntitlements (fail-closed to free on any error/timeout, 2s deadline — entitlements.py) and scales each resolved limit:

core-gateway-consumer/app/enforcement/pipeline.py

def _scale(n: int, mult: float) -> int:
    return 0 if n <= 0 else max(1, math.ceil(n * mult))
# ...
mult = ent.rate_limit_multiplier or 1.0

Two consequences fall straight out of that formula:

• 0 stays 0. If your base limit is unlimited (0), no multiplier makes it finite — every tier is unlimited. To limit anyone, you must set a non-zero base.
• The default is strictly proportional. With just a base you get base × {1, 2, 5, 10…50}. When you need a tier to break that ratio — “free = 5/hr, creator = 50/hr, pro = unlimited” — declare a manual override for that tier (next section).

Manual per-tier overrides

When the multiplier ladder doesn’t fit, pin an exact limit for specific plans with rate_limit_overrides — a map keyed by exact plan_code, alongside the base rate_limit:

app/tools/my_tool.py (class attributes)

from soundverse.tools import RateLimit

class MyTool(BaseTool[MyInput, MyOutput]):
    rate_limit = RateLimit(limit_per_hour=5, limit_per_day=20)   # base = the free-tier number
    rate_limit_overrides = {
        "pro":    RateLimit(limit_per_hour=500, limit_per_day=5000),
        "max_x5": RateLimit(0, 0),   # unlimited for the top tier
    }

Semantics:

• An override is used verbatim for callers on that exact plan_code — it replaces the base × multiplier result (no multiplier is applied on top).
• Plans without an entry fall back to base × their multiplier as before. Above, creator still gets 5 × 2 = 10/hr; only pro and max_x5 are pinned.
• An override of RateLimit(0, 0) means unlimited for that tier.
• Keys are exact plan codes — free, creator, pro, max_x1 … max_x5 — so you can give each max level a different number. (There’s no “whole max family” key.)
• rate_limit (the base) must be set for overrides to be meaningful — the base is what every un-overridden tier scales from.

Under the hood each override is its own tool_config_rate_limits row keyed (config_id, plan_code); the resolver prefers an exact plan_code match over the base row and flags it so the enforcer skips the multiplier. Same declare-and-restart flow as everything else on this page.

Removing an override leaves a stale row

The fleet only upserts. Deleting an override from code does not delete its DB row — the old limit lingers. Clear it with DeleteToolRateLimits (or a reconcile step) if you retire a tier override.

Where users see it

Users get a Settings → “Rate limits” tab (in the same modal as Manage your plan / Appearance) listing every tool with its hourly and daily usage for their plan — used / limit meters with a resets-in countdown, and an “Unlimited” badge for tools they aren’t limited on. It’s backed by GenerationService.GetRateLimitUsage, which reuses the exact resolver + counter keys the enforcers use, so the numbers shown are the numbers enforced.

Scope cascade & model-level limits (beyond the tool class)

Two things the RateLimit class attribute does not cover, both resolved by FetchResolvedRateLimits (which unions tool-config and model limits, picks the highest-priority scope row, and coalesces a missing value to 0 = unlimited):

• Scope overrides — tool_config_rate_limits rows carry a config_scope (platform | workspace | project | user). Your class attribute writes the platform row. A narrower row (e.g. one workspace) overrides it for those callers. Those are set via the same RPC with a non-empty workspace_id/project_id/user_id, i.e. an ops/DB action, not the tool class.
• Per-model limits — generation.model_rate_limits caps a model across whatever tools use it (keyed environment/model/tool_type/scope/…), via UpsertModelRateLimits. Useful for a shared expensive backend model. Also an ops action — there’s no tool-class field for it. The resolver returns tool and model limits independently, and both are enforced (the four counters above).

Rate limit ≠ token cost

These are different mechanisms enforced in sequence in the same pipeline. Rate limiting is a requests-per-hour/day Redis counter; pricing/reserve is a token-balance deduction. A blocked call is RESOURCE_EXHAUSTED (rate) vs FAILED_PRECONDITION (insufficient funds). Setting a token price does not rate-limit, and vice-versa.

Checklist

1. Decide the free-tier allowance. The base number you write is what a free user gets; paid tiers scale up by the multiplier ladder. Work backwards from the free experience.

2. Declare rate_limit = RateLimit(limit_per_hour=N, limit_per_day=M) on the tool class, with N, M > 0 and M >= N. Use 0 on a window only to mean “unlimited for that window”.

3. Restart the worker. The fleet re-upserts the declared config on boot; the resolved number is then fetched per call — no separate seed step, no cache to flush for the limit value.

4. Confirm Redis is wired in the target env (REDIS_ADDR for core-mcp; the shared client for the gateway). No Redis → the limiter fails open and nothing is enforced.

5. Need a per-model cap or a workspace/user override? That’s UpsertModelRateLimits / scoped UpsertToolRateLimits — a DB/ops action, not a class attribute.

6. Need arbitrary per-tier numbers (not a fixed multiple)? Add rate_limit_overrides keyed by exact plan_code — an override is used verbatim (no multiplier); un-overridden tiers still scale from the base.

Related

• Add pricing to a tool — the sibling declarative config; same upsert flow, different mechanism (tokens, not requests)
• Add a tool worker — the full tool recipe this slots into
• The token ledger & money path — the reserve/settle mechanism rate limiting sits alongside
• core-mcp — the agent-path enforcer and the shared Redis key contract