Custom Tasks and Scoring

rogue-arena-inspect launch runs a ready-made example task. To change the agent, its prompt, or its tools, write your own Inspect task. This page covers the parts specific to Rogue Arena. The package README has the full reference.

Start from the examples

The example tasks ship inside the package: eval_example.py, team_example.py, team.yaml, and the eval-example.eval.yaml run config. To find them, run:

python -c "import importlib.resources as r; print(r.files('inspect_rogue_arena') / 'examples')"

Copy one out, edit it, and run it with --task:

rogue-arena-inspect launch --eval <id> --task my_task.py --model <provider>/<model>

Write your own task

Give the task the Rogue Arena sandbox and scorer:

from inspect_ai import Task, task
from inspect_ai.agent import react
from inspect_ai.dataset import Sample
from inspect_ai.util import SandboxEnvironmentSpec

from inspect_rogue_arena import RogueArenaSandboxConfig, rogue_arena_hops, rogue_bash


@task
def my_eval() -> Task:
    return Task(
        dataset=[Sample(input="...your objective...", target="scored by Rogue Arena")],
        solver=react(tools=[rogue_bash()]),
        scorer=rogue_arena_hops(),
        sandbox=SandboxEnvironmentSpec(
            "rogue-arena", RogueArenaSandboxConfig(eval="<id>")
        ),
    )

rogue_bash() is the shell tool the examples use. It tells the model its default timeout and runs long commands in the background, so a slow command finishes and a brief network error doesn’t end the run.

Each sample in the dataset gets its own trial, on its own copy of the range.

Tip

Tell the model what proves each hop. Rogue Arena counts a key-file hop only when the planted key appears in the agent’s command output or in a file it downloads. There’s nothing to submit, so if the objective doesn’t say to find and read the key files, the model may get there and still score nothing.

Settings

Set these in RogueArenaSandboxConfig, or with the environment variable. Settings in the task win over environment variables.

Setting	Environment variable	Default	What it does
eval	ROGUE_ARENA_EVAL	required	The eval ID, or a name that matches exactly one eval.
concurrency	ROGUE_ARENA_CONCURRENCY	2	The most trials running at once. Each one holds a deployment slot.
max_trials	ROGUE_ARENA_MAX_TRIALS	10000	The most trials the run can start. Set it to at least samples × epochs, plus retries.
max_runtime_seconds	ROGUE_ARENA_MAX_RUNTIME_SECONDS	3600	Each trial’s time limit, counted from the claim, including build time. From 60 to 86400.
max_wait_seconds	ROGUE_ARENA_MAX_WAIT_SECONDS	900	How long a trial may take to become ready.

If you set concurrency in a task, pass the same number to inspect eval as --max-sandboxes. The README lists the remaining settings.

Scoring with rogue_arena_hops()

The scorer closes the trial and reads the score Rogue Arena returns. It never checks a hop itself.

• Value: hops passed divided by total hops, counted under the eval’s scoring mode. The answer reads like 2/4 hops.
• Metadata: trial_id, hops_passed, hops_total, reached_step, scoring_mode, outcome, terminal_reason, timed_out and per_hop.

per_hop.passed is each hop’s raw check result, not whether the hop counted. In sequential mode, a hop after the first failed step doesn’t count even if its check passed. Use hops_passed and reached_step for the count.

How the trial ended	Result
The agent finished or gave up	Scored.
It ran out of time	Scored normally, with timed_out: true.
An infrastructure error, a cancel, or no score	Not scored. The sample errors and stays out of the average.

By default, Inspect stops the whole run on the first sample error. To keep going, pass --no-fail-on-error, or a threshold such as --fail-on-error 0.1. --retry-on-error N starts a fresh trial for each retry, so leave room in max_trials.

Python client

RogueArenaClient covers the whole eval API, for scripts that don’t use Inspect:

import asyncio

from inspect_rogue_arena import RogueArenaClient


async def show_evals() -> None:
    client = RogueArenaClient.from_env()
    for entry in await client.list_evals():
        print(entry.environment_id, entry.name, entry.hop_count, entry.scoring_mode.value)


asyncio.run(show_evals())

It reads ROGUE_ARENA_BASE_URL and ROGUE_ARENA_TOKEN from the environment. The README lists every method.