Function Registry

The function registry powers the eval policies and protocols. It handles argument declaration, CLI integration, and external module:function lookup.

For how policies and protocols fit into an eval, see Protocols and Policies.

Defining a Policy or Protocol

The eval registries, policies and protocols in protocols/eval/registry.py, declare configurable parameters directly as keyword arguments. The registry synthesizes a BaseModelPython args class from the function signature and keeps it on the function's FunctionInfo (defined in registry.py): each parameter's annotation and default become a field's type and default, and a pydantic Field(description=...) default documents it. Policies live in src/control_tower/protocols/eval/policies/ and protocols in src/control_tower/protocols/eval/protocols/.

from pydantic import Field

from control_tower.protocols.eval.blue import Protocol
from control_tower.protocols.eval.registry import protocols


@protocols.register(name="trusted-monitoring")
def trusted_monitoring(
    monitors: Monitors = Field(description=MONITORS_ARG_DESCRIPTION),
    sus_threshold: float | None = Field(default=None, description="Act at or above this score."),
    blocks: int = Field(default=0, ge=0),
) -> Protocol:
    """Ask the monitors about each action; at or above `sus_threshold`, block up to `blocks` times, then audit."""
    return TrustedMonitoring(monitors=monitors, sus_threshold=sus_threshold, blocks=blocks)

A policy is registered the same way with @policies.register(...) and returns a Policy; it also declares attempts_main_task, attempts_side_task and attempts_sneaky, which trajectory conversion records on every trajectory it produced. Registration is automatic via the decorator: the registry imports every non-underscore module under its discovery package on first lookup.

Naming Convention

By default, the registry strips the _<identifier> suffix from the function name and converts underscores to hyphens. Each registry passes its own identifier (see registry.py):

No Python function name ends in _eval-policy, so an eval policy or protocol is named after its function unless you pass name="...": honest_policy would otherwise be honest-policy.

The docstring's first sentence becomes the description shown in help output.

Passing Arguments from the CLI

# Policies use -p (short for --policy-arg)
ct run eval --policy sleep -p min_s=3 -p max_s=4 --task-file simple

# Protocols use -bp (short for --protocol-arg)
ct run eval \
  --policy attack \
  --protocol trusted-monitoring \
  -bp 'monitors=[simple]' \
  -bp sus_threshold=70 \
  --task-file simple

Hyphens in keys are converted to underscores automatically. Unknown keys raise a validation error. Models are not policy arguments: an eval binds them to roles with --model-role (see Model Configuration). ct run control-eval selects two policies from the same registry, with --honest-policy and --attack-policy, whose arguments go through --honest-policy-arg / -hp and --attack-policy-arg / -ap, and one protocol with --protocol / --protocol-arg; see Control Evals.

Discovering Available Arguments

ct protocols policies           # List all policies
ct protocols policies -v        # List with all arguments
ct protocols policies attack    # Show args for a specific policy
ct protocols protocols          # List all protocols
ct protocols models                   # List model aliases

Cost Tracking

CONTROL_TOWER_PRICING selects the rate source:

• file (default) loads a generated JSON table (~/.cache/control-tower/model_rates.json, or CONTROL_TOWER_MODEL_RATES).
• api loads the deployment's pricing collection over GET /api/pricing and needs a viewer token (ct login or CONTROL_TOWER_API_TOKEN).
• off loads nothing. Model calls still run and bill, but every one except mockllm/model's is recorded as unknown cost, so use it only for runs that make no priced calls.

With file or api, the table loads at Task construction, and a missing or empty table fails the eval before any sample starts. A model the loaded table has no row for stamps unknown cost, not $0. Inspect's mockllm/model calls no provider, so it always costs $0, whatever table is loaded.

uv run python scripts/generate_model_rates.py writes the file: vendors' own prices from LiteLLM's price map under bare names, and OpenRouter's catalog under each model's full openrouter/... name, which a lookup tries before the bare name. An OpenRouter row is the default route's price, so a run pinned to an endpoint that prices differently needs a manual row under the same key. A re-run keeps rows marked "manual": true and models the source has stopped publishing.

External Functions

Any CLI flag that accepts a policy or protocol name also accepts a module:function path, following the same convention as uvicorn app:factory. This lets you use policies and protocols defined in your own project without modifying control-tower.

ct run eval --policy my_project.policies:my_attack --task-file simple

The function must have the same shape as a registered one (keyword arguments, returning a Policy or Protocol). External environments also work via --env my_project.envs:my_env_config, and --extra-src puts a directory of experiment code on the path without installing it. A : in --env always means a module reference; on --ec2, name the directory holding its package with --extra-src (see "Shipping external code to workers" in EC2 Sandbox).

Quick Reference