How to preview a coding benchmark run before spending on model calls

Before launching a model run, use the --dry-run option to preview a benchmark plan. In Weave Router's public weave-bench harness, this option resolves the task manifest and displays commands and configurations for comparison arms. The plan command itself has no network impact and does not initiate any benchmark jobs.

Key points include:

- The --smoke option selects a smaller task set, but it does not execute any actual runs. Using --smoke with --dry-run will initiate jobs, which could incur costs.

- To set up the benchmark CLI, clone the public Weave Router repository, navigate to the bench directory, and export the necessary environment variables.

- Run the following command to generate a plan: uv run --locked --extra $HARBOR_EXTRA --extra tap weave-bench run terminal-bench-4 --arms router,luna --smoke --dry-run

- The plan output provides information on the selected task, manifest size, pinned versions, task name, and the exact harbor run command for each arm. It does not include any actual model results or costs.

- Double-check the plan before removing the --dry-run flag:

- Confirm the task selection and distinguish between the total manifest tasks and the smoke-subset tasks.

- Record the pinned versions of Harbor, Codex, and the benchmark.

- Verify the comparison arms and their configurations.

- Note the expected environment variable names for a real run.

- Remember that the smoke flag only limits task selection and does not prevent model calls.

- Once you're ready to execute, follow the repository's run instructions and ensure you provide the required credentials in your environment. Keep track of the run ID, source revision, task manifest, any failures, and the final report. Do not treat the dry-run plan as actual benchmark evidence.