Evaluation Runs

agent_runs is the run-management layer for Control Tower. A run is the metadata container around one evaluation batch:

  • run header metadata (run_id, name, policy, task set, tags, uploader, timestamps)
  • optional eval-log URL
  • associated eval samples exposed as trajectories by the viewer

The ct runs command group uploads eval logs as runs, lists and inspects run headers, and downloads eval archives.

Uploading Results

# Upload during eval
ct run eval --policy attack --env web_scraping --main-task crawl_depth --side-task visit_malicious_website

# Upload with tags for easier filtering
ct run eval --policy honest --env web_scraping --main-task crawl_depth --tag "experiment-1" --tag "baseline"

# Upload a previously saved eval
ct runs make logs/my-eval.eval

Uploads need a per-user API token: run ct login to cache one, or set CONTROL_TOWER_API_TOKEN. The CLI uploads through the viewer's HTTP API, so it needs no MONGOURI or AWS keys. See Database and the HTTP API reference.

Converting Old Runs

An eval log names its format in eval_version. Control Tower reads task results only in the current format, so a command that loads an older log (a replay, ct run monitor) converts it in memory first, and an upload refuses a log that is not current. ct runs convert writes the converted copy to upload; the source run and its log are left as they were.

# One converted log per source: a local .eval, a run id or a run URL
ct runs convert <run_id> logs/old.eval

# An old honest run and an old attack run as one control-eval log
ct runs convert --honest <honest_run> --attack <attack_run>

# Upload each converted log as a new run
ct runs convert <run_id> --upload

Each major version behind is one step, applied in order:

  • A log from before eval_version is rebuilt as a ct run eval log with its monitor verdicts in the monitor Score, keeping its policy name, task set and tags and gaining the legacy-run tag. Its samples get new ids. A log whose samples record no environment or tasks is too old to convert.
  • Below 3.0, a one-policy control_tower/control_eval log becomes a control_tower/eval log, a sample's main_task and side_task become the main_tasks and side_tasks lists, and its submitted arguments are keyed by task id.
  • Also below 3.0, the linux_scorer Score becomes workspace_diff, one Score per task and tasks, with the header's scorers and metrics recomputed. A verdict that decided nothing scores failed when the log records a scorer timeout or crash, and stays unscored when the environment was not ready or the log gives no cause. A live side task that passed gets its landed_at from the transcript, and a side task that names no failure_type is delayed.

--honest or --attack alone converts one run and says which stream its samples belong to. An honest run's samples lose their side tasks, which the honest policy never saw, and those tasks' Scores; both are kept under the sample's converted_from metadata. A run whose policy is an honest one but whose samples carry side tasks is refused until one of the two flags says which it is. --honest and --attack together merge the two runs into one control-eval log holding both streams, whose header keeps the metadata the two runs agree on and names both policies. The runs must share their protocol, white settings and models, and an honest and an attack sample cannot share an id and epoch. A converted log carries no audit Score; add one with a post-hoc audit.

A converted log is written to its own file under --output-dir (logs/converted by default), and uploads as a new run of new trajectories. Its header's converted_from lists the runs it came from, and each sample's converted_from.run names its own. The command prints one JSON line per converted log, {"converted_from": [...], "eval": "<path>", "run_id": "<new run id, with --upload>"}, and skips a log that is already current.

Listing Runs

ct runs list                      # List recent runs
ct runs list --limit 10           # Limit results
ct runs list --tag baseline       # Filter by tag
ct runs list --format json        # Output as JSON for scripting

Inspecting a Run

ct runs info <run_id>             # Show detailed info
ct runs info <run_id> --format json

Downloading Runs

ct runs download-eval <run_url_or_id>  # Download a run's eval log
ct runs download-eval <s3_eval_url>    # Download an eval log by s3:// URL

ct runs download-eval fetches a run's eval log through the viewer's /api/runs/{id}/eval endpoint when given a viewer run URL or run ID, and otherwise fetches an s3:// eval log URL directly, which needs AWS read access. Either way it caches the log under data/eval-logs/. See Downloading from the Viewer for the URLs the viewer exposes.

To download every sample bundle associated with a run, pass its viewer URL to ct runs download-traj.

ct runs download-traj https://viewer.example/runs/<run_id>