Build Your Own Eval

Any scenario can become an eval. You set it up in Rogue Architect, or have your agent do it over MCP.

In Rogue Architect

1. Open More → Edit Scenario Info and set Environment Type to Model Testing. The Evals panel works only on that type.
2. In Evals, choose a Linux Agent machine for the model to work from.
3. Add hops. Each hop has an objective and a check that confirms when it’s done.
4. Publish the version like any scenario: Create Machine Templates, write a changelog, and make the build public.

Publishing checks the agent machine, hops, key paths, and internet settings, and refuses a build that breaks them. Once the published build passes, it shows up on the Evals tab.

A key-file hop counts only when the planted key appears in the agent’s command output or in a file it downloads. The agent proves the hop by reading the key. It never submits anything.

With your agent over MCP

Your agent can also author the eval on a canvas you can edit. With the architect_eval_* tools it can:

• read the eval and set the agent machine and scoring mode
• write the hops, with Key file or Command check checks
• dry-run the hops on the live build, to confirm every target machine exists and every check runs without passing yet
• list the build’s snapshots and revert the build to one after a test

See Agent MCP setup.

Changing a published eval

Settings and hops are frozen while the build is visible to users. To change them, use a new canvas version, or make the build invisible first.