The agent bench

How an external agent plays a scenario or starts New Game toward a goal with no human at the keyboard, and how the run is scored. The bench is the bench subcommand of the game binary, built with --features debug, and lives in crates/nova_bench with one TypeScript relay in tools/nova_bench/pi/.

                 referee protocol                 process channel
  +-----------+  (unix socket, JSONL)  +------------+  (stdin/stdout JSONL)  +----------------+
  |  agent    | <--------------------> | nova_bench | <--------------------> |  nova-protocol |
  |  (pi, a   | observe/act/page/finish|  (referee) |  tick / input / aim ...|  --norender    |
  |  script)  |                        |            |  <- snapshot           |  --channel step|
  +-----------+                        +-----+------+                        +----------------+
                                             |
                                 audit.jsonl, score.json, log

Three processes, one referee. The agent never touches the game and the game never sees the agent. nova_bench sits between them: it owns the clock, condenses what the agent sees, logs every line in both directions, and scores the run from the game’s own snapshots. An agent cannot grade itself, because the referee reads the world, not the agent’s report.

The game side is the process channel (nova_channel, see Architecture) in --norender --channel step mode: the world moves only inside a step, so an agent may think for a minute and its observation never goes stale.

nova_bench depends on no nova_* crate. It speaks JSON over pipes and a socket, so its tests link neither Bevy nor the game.

Running a play

cargo run --features debug bench play <scenario> --agent baseline
cargo run --features debug bench play <scenario> --agent pi --model gpt-5.6-luna --thinking low
cargo run --features debug bench play <scenario> --agent 'cmd:python3 my_agent.py'
cargo run --features debug bench play --session new-game --world-seed 424242 --seed 7 \
  --agent pi --goal 'Explore the open world and report what you find'

<scenario> is an installed scenario id, or a path ending in .ron loaded as a loose content file. --session new-game takes no scenario. It starts at the real main menu; the agent clicks New Game, enters --world-seed in the setup modal and clicks Create. New Game requires both --world-seed (the u32 world-generation seed) and --seed (the separate u64 gameplay RNG seed), plus --agent pi or --agent cmd:.... The combat-only baseline agent cannot navigate the menu. A scenario cannot be combined with New Game, and --world-seed is not accepted for fixture plays. New Game uses the base bundle’s open_world scenario; naming open_world as a scenario does not create the required world session.

The bench ships six fixtures under crates/nova_bench/scenarios/:

• hunt.content.ron: one player gunship, one hostile inline raider parked 2600 m ahead, an objective and both outcomes. The scenario scores itself.
• range.content.ron: an open range with no objective and no victory. A 600 m planetoid 7 km ahead, two rocks, a nav beacon, and an unarmed hostile derelict support navigation, orbit, parking, and PDC goals.
• slingshot.content.ron: an offset gravity well, a distant exit mark, and a hostile pursuer that wakes after 45 seconds support gravity-assist, evasion, collision, and route-cheese goals.
• arsenal.content.ron: a player hull authored inline with a railgun, two torpedo bays and two PDC mounts, plus a carvable rock, two inert targets and a late picket, supports mixed weapons, asteroid carving, the command shell, cheat, and protocol-abuse goals.
• docking.content.ron: the docking_approach example as a fixture. A tender with a port on its bow and an inert spar 120 m ahead, turned out of square, support docking and RCS goals; me.docking carries the approach.
• docking_warship_tender.content.ron: the catalog Line Warship and an inert damaged frame tender, their collars 115 m apart and 12 degrees out of square, support docking captures between real multi-section hulls.

These six fixtures are bounded by --ticks and --deadline; a reader grades the end block of the score and any recorded footage. The red-team rules are in crates/nova_bench/scenarios/README.md, and the goal deck itself is scripts/bench-plays.sh, which runs any of twelve named plays with its goal, seed and budgets already set:

nix develop -c scripts/bench-plays.sh --list
nix develop -c scripts/bench-plays.sh sommelier shell cheat

For these fixtures, --seed pins the world, not the run: the model is the other half and takes a different route every time. The script pins the setup so two plays of one goal are comparable, which is as close to reproducible as an agent play gets.

New Game’s initial observation has no player ship. On the main menu the agent’s view includes the visible named UI targets so it can find the New Game button, seed field and Create button. The agent must replace the modal’s prefilled seed; merely appending digits does not select the requested world. Menu actions use the same pointer, text and key gestures as other UI actions. The referee checks the world’s actual session seed before gameplay scoring. A missing player at menu start is expected; a wrong seed or scenario fails setup, and a run that ends before setup completes does not count as a completed play. Menu-backdrop ships do not count as combat kills. The audit retains menu steps and the launch target; replay starts at the menu and checks world-seed identity as well as the final flight state. A free-mode goal has no authored Victory: judge the world state and the score, not the agent’s report alone.

The run directory holds audit.jsonl, score.json, game.log, agent.log (a cmd: agent’s stdout and stderr) and profile/, an empty settings profile the child game is pointed at so a play never reads or writes yours. The command prints the run directory and the score table on stdout and exits non-zero when the game or agent process fails, or New Game setup fails or remains incomplete. A Defeat alone does not make it fail.

The referee protocol

JSONL request and reply over the unix socket named by NOVA_BENCH_SOCKET. One request per connection, one line each way.

A malformed request gets {"error": "..."} and the run continues. Every view carries over, ended_by and budget_left; once over is true, act moves nothing. New Game setup actions use the same turn, tick and deadline budgets as flight.

A gesture is one object with one verb. The referee stamps the tick, so an agent’s vocabulary is relative and a transcript is replayable.

Gestures in one act land on the same tick, in order. A held input stays held across acts until released.

One act cannot drive both readings of a shared key. targeting.radar_hold and targeting.radar_clear are one key read two ways - a short press clears, a long one searches - so an act carrying both is refused with a message that says to split it. Nothing downstream could report that: a press has no verdict to give.

Splitting it is not enough on its own. The referee checks the act against its held set too, so driving one reading while the other is still held is refused the same way: the key is down for the first reading, and the second would release that same source behind the referee’s back. The release goes in its own act, before the act that drives the other reading.

The pairs are not a table in the bench. nova_input already models the relation as ActionBinding::follows, the channel publishes it as inputs.shared, and the referee checks the act against whatever the world reported. A game that declares a second shadow action is guarded without an edit here, and one that drops a shadow stops being guarded for a rule that no longer holds.

The view

The raw snapshot is a probe artifact: every hull plate, every ordnance record. The referee condenses it to what a pilot perceives, in meters, meters per second and degrees:

• tick, game_seconds, game_state, world_seed (null until New Game creates a world), over, ended_by, budget_left, game_errors; ui.targets lists visible named pointer targets, only on the main menu and while the Tab interface is open (ui.pause is Interface).
• objectives, outcome (null until Victory or Defeat), comms (the last twelve radio lines).
• me: id, position_m, speed_mps, velocity_bearing_deg, turn_rate_dps, health, cargo (capacity_g, used_g and free_g in grams, credits, and items as {item, count} in the game’s item order, empty for an empty hold), weapons_hot, combat_lock, travel_lock, radar (the acquisition dwell while the radar gesture is held: the candidate, the dwell target, dwell_secs of dwell_needed and the dwell_fill a player watches on the ring, itself null when no dwell is running; null otherwise), autopilot (engaged: action, target and phase, or null; completed: the last action that finished), docking (docked, the connection a clamp holds the hull to, and pair: the approach to the nearest port on the travel lock, as an offset and a bearing in the hull frame with the three capture gates and eligible), gravity_well (the dominant well’s id), sections (the bridge, the drives, each mount with its weapon: kind, ammo, on_target, firing), hull_plates as {total, damaged, lost} and withheld_capabilities (what the scenario has not granted yet).
• contacts: every other ship with allegiance (Enemy, Player, Neutral), distance_m, bearing_deg, closing_mps, health, defeated, weapons_hot, ai_target.
• beacons with distance_m and bearing_deg.
• canisters: every live cargo canister, with a numeric runtime id unique within the run (not stable across revisions), distance_m and bearing_deg relative to the player’s ship, and ordered stacks of {item, count}. Distance and bearing are absent without a player ship.
• bodies: the asteroids and planets, in three tiers. near and in_the_way carry the full record - kind, radius_m, distance_m to the centre, surface_m to the surface, bearing_deg, closing_mps - and in_the_way adds why. The kind says whether a body can be destroyed: every Asteroid can, no Planet can. Everything else is one summary per {distance band}.{bearing sector} group with a key, a count and the nearest surface, farthest surface and largest radius in it; observe {"expand": ["<key>"]} opens one back up and expanded lists the keys that actually opened, because a key can go stale as the ship moves. See Why the bodies are tiered.
• ordnance as {inbound, outbound} counts.
• objective_log: every objective card posted and completed, in order.
• cinematic (only while a scene plays) as {playing, skippable}, and cheats_marked (only when the run armed cheats).
• inputs.live (registry wire names that are not locked out, never dynamic section.<id> wires), inputs.held, and inputs.shared: the pairs of wire names that read one physical key. The game declares that relation (ActionBinding::follows) and the channel publishes it, so the referee refuses an act driving both sides of a pair without the bench holding a table of its own.
• commands (answers to command gestures).

bearing_deg is [azimuth, elevation] from the nose: azimuth positive to starboard, elevation positive up. Registry wire names come from the live set in the view. For section.<id> weapon triggers, use an ID in me.sections[] whose weapon is not null. For a mining emitter, use the ID of a section whose class is Mining and check alive and disabled for working status. Section selectors do not appear in inputs.live or the registry bindings inspector. Section responders can also be thrusters.

There is no refused list, because an input has no verdict to give. Every gate a pilot meets - needs a lock, needs a well, needs the stance, needs range - fires above the input layer, so a press that changed nothing looks exactly like one that did. The channel’s applied block is an ECHO of the lines a frame consumed, {line, input, phase, tick}, and the agent reads the effect off the world: autopilot.engaged, combat_lock, radar.dwell_fill, weapons_hot, bearing_deg, objectives. A wire line the game could not parse is still an error, in game_errors, and always carries a message.

The raw snapshot carries a mission block (objectives, outcome, comms, the objective log, the cinematic and the cheat mark), beacons, bodies, and canisters (runtime ID, engine-unit position and velocity, ordered stacks), plus each ship’s autopilot, radar, cargo and gravity_well; see nova_probe::capabilities::snapshot.

Mining and cargo

Mining uses a per-section hold: find a me.sections[] record with class: "Mining", check its alive and disabled status, then press section.<id> using that record’s id. For the Line Warship’s mining_beam, send {"press": "section.mining_beam"} for a bounded attempt and then {"release": "section.mining_beam"}. The selected emitter must deploy before it can pulse toward a travel-locked rock; reach and aim are checked from its face, not the ship’s centre. The agent view has no carve result or intake position. It does show all live canisters, their contents, and their distance and bearing from the player’s ship. A newly visible ore canister shows ore in the world, not its source or pickup; a canister’s disappearance alone can also mean it was destroyed. The intake takes a whole canister on trigger contact when hold mass permits; only an increase in the relevant me.cargo.items count proves pickup. Jettison (undocked) and Buy and Sell (docked to a trader) are Inventory pane buttons, driven with pointer gestures on the ui.targets the interface shows. Read every result off me.cargo.

An agent finds registry wires in inputs.live; it gets section selectors from me.sections[].id for weapons (weapon not null) or mining (class is Mining). Dynamic section.<id> wires are absent from that live list and from bindings, which inspects registry actions only. For example, bindings flight.main_drive reports that registry action’s keys and whether its context is live; bindings alone lists every registry action by its wire name. bind and bind reset take the bare registry name, such as main_drive. Neither command binds a section wire.

Why the bodies are tiered

A scenario can field a hundred rocks. Enumerated in full they cost more tokens than the rest of the view together - measured over the first tutorial victory, bodies was about 14200 characters of a 20800-character view, two thirds of every observation, over a run that spent 1.18 M input tokens.

Enumeration is also the wrong shape. What a pilot needs about a rock is not that it exists at 8 km: it is what is in the line of sight, what blocks the radar, what is on the GOTO path, and what the ship is about to hit. So each body is classified ALONE, as a pure function of that body and the ship’s frame - no clustering, no seed, no iteration, no memory of the last view - so two observations of an unmoved world group identically and an agent can reason about a group across turns. The bands are a fixed ladder and the sectors are the four quarters, so a rock changes group only when it crosses an edge.

Expansion is for detail, NEVER for safety: anything that can end the run is already in near or in_the_way.

Agents

• baseline: an in-process Rust policy. It waits for a hostile, raises the weapons, holds the radar until a combat lock stands, closes to 2500 m and holds every mount’s trigger while the lock stands. Zero tokens; it proves the referee, the score and the audit, and is what a CI check would run.
• pi: spawns pi --mode rpc with the built-in tools off and the relay extension on, sends one prompt (the pilot manual in crates/nova_bench/src/manual.md, the scenario, the goal, the first view; the game knowledge behind it sits in crates/nova_bench/src/pages/, read on demand through the page tool) and consumes the RPC event stream: assistant text, thinking, tool calls and usage land in the audit. NOVA_BENCH_PI names the binary when pi is not on the path. If pi settles with the run still open it is nudged once; a second settle ends the run as agent_exit.
• cmd:<argv>: any process that speaks the referee protocol. It gets NOVA_BENCH_SOCKET in its environment; its stdout and stderr go to agent.log. The run ends when the process exits.

The run ends when the referee says so

Whichever comes first: the scenario declares an outcome; the tick, turn or wall-clock budget runs out; the agent calls finish; the agent process exits. ended_by names it. After that the game gets EOF (the channel’s clean exit) and pi gets an abort.

The score

score.json, from the snapshot stream only:

There is no composite number. A composite is policy; the table is the record.

The audit

audit.jsonl, one object per event with a wall-clock t, flushed per line:

• run_start, run_end (the reason and the score)
• channel_out (every line sent to the game), channel_in (every snapshot, condensed; --audit-raw keeps the full one beside it)
• agent_request, agent_reply (the referee protocol, verbatim)
• agent_text, agent_thinking, agent_tool, agent_usage (pi)
• refusal, note

Replay

cargo run --features debug bench replay <run dir>/audit.jsonl [--record <dir>]

Replay feeds the audit’s channel_out lines to a fresh game under the recorded launch target and RNG seed, including any New Game menu actions, and compares where the world ended with the audit’s last view. New Game replay also compares the authoritative world seed. Two plays of one seed are not byte-identical once rounds fly: how many land drifts between processes. The verdict has three grades:

With --record the replay is the agent’s run as a movie, stitched the same way a play’s is.

The movie

--record <dir> on a play or replay passes the directory to the game’s channel recorder. The offscreen assembly uses the render stack without an OS window. It routes cameras and UI picking to one image; the shared nova_autopilot LoopRecorder requests that image in Last for each stepped tick. The directory must be empty (or absent): recording refuses to overwrite prior contents. Frame zero is the state after tick one. The recorder keeps <dir>/frame_%06d.png, starting at frame_000000.png, and writes <dir>/bench.jsonl with per-frame sampled VoiceMix SFX, copied sound samples, <dir>/audio.f32le (staged stereo PCM, retained), and <dir>/bench.webm with Opus SFX. The SFX mix is independent of live --mute output.

The channel’s step clock advances by 16,667 microseconds per tick; video playback is nominally 60 fps, one frame per tick, regardless of how long the agent thinks between acts. After the game exits, the bench draws the run’s action rail over those PNGs and encodes <dir>.mp4, muxing the recording’s audio. The bench prints the movie path, frame count and length after the score table and notes them in the audit. If the face is unavailable, the bench makes a plain movie with the same audio. An older bare-PNG directory with no bench.jsonl or bench.webm can still be made into a silent movie with bench movie. A new recording with missing expected audio, sample bytes, frames, failed capture, or failed encoding/muxing is an error, not a silent success; ffmpeg is required for recording and movie creation.

Recording needs a GPU and a display. Run the bench from the dev shell (nix develop), which is where the Vulkan loader finds the driver, on an X display: your own, or an Xvfb (DISPLAY=:99). A play without --record needs neither. Inspect the recorded frames and listen to the movie before claiming the rendered picture or sound is correct.

The action rail

cargo run --features debug bench movie <run dir>/audit.jsonl --frames <dir>

bench movie remakes the movie of frames already on disk, so the overlay can be changed without replaying anything. --out names the file (default <frames>.mp4), --font the face, and --plain leaves the frames untouched by the rail. Both paths mux bench.webm audio when present; if bench.jsonl says a new recording exists but its audio is missing, they fail.

The rail is composited in the bench, not in an ffmpeg filter graph: image decodes the PNGs, ab_glyph sets the game’s own terminal face over them, and ffmpeg is handed finished RGB frames on stdin. Layout that lived in a filter string could not be read or tested.

A header strip carries the identity of the run - scenario, agent, seed, clock, tick and turn - so a clip lifted out of a movie still says where it came from. Under it a transcript of the last six lines, newest at the bottom, older ones fading:

> act    aim camera.camera_rotate [-80, 0] x21
> say    EXIT is 3.3 degrees off the nose at 7.27 km. With weapons lowered,
         I'll hold radar to acquire it as a travel lock.
> act    press targeting.radar_hold
> think  Planning lock release and navigation

One lane per kind of line, each its own colour: goal (once, at the head), think, say, look (observe and page, which spend no tick), act, nova for a command shell line, cheat, and err in red for a refusal.

The cheat lane is the world’s word, not the shape of a command. The first snapshot that carries cheats_marked puts CHEATS ARMED: this run is marked on the rail in amber, and from that frame on the header says CHEATED and turns amber too. A clip cut from a marked run cannot be read as clean, and a cheat the game does not mark does not colour the rail - which is the point of the “cheat without getting marked” red-team goal.

Placement is arithmetic, not a guess. The recorder captures one frame per stepped tick and numbers from zero, so frame = tick - 1. channel_in carries the tick; agent prose does not, because the world is frozen while the model thinks, so it lands on the tick of the snapshot it was answering. Rows that would share one frame are spread 21 frames apart - a first turn can produce ten rows, and all ten on frame zero is a wall of text. That spread is why the opening rows can trail the world by a second or two; the rail is a log of the run, not a caption track.

A short run can talk for longer than it flies. When the spread rows would run past the last frame, the gap tightens until they fit, and if even a one-frame gap is too tight the movie holds the last frame the game drew - up to 30 s - while the remaining rows scroll past. The header clock freezes with the picture, so a held tail never claims ticks the run did not take.

Measured on the hunt fixture, seed 7

Three pi plays of one seed, one row each; the third was recorded and stitched into a 20.7 s movie at 1280x720. The first play’s opening act was malformed (two verbs in one gesture); the relay returned the parse error as a tool error and the model corrected itself on the next call. Both audits replay close.

Measured on the range fixture, seed 7

The range has no objective, so the goal comes from --goal and the reader grades it from the end row. All plays --ticks 12000 --deadline 900; the pi rows are gpt-5.6-luna with low thinking.

The first orbit play approached at 572 m/s, tapped the orbit helm as the well appeared and dove into the surface in Burn; pi stopped talking with the bridge and the main drive gone. The manual now gives the speed band (60 ticks of drive is 80 m/s, 240 is 320, the helm takes a ring from 320 but not from 550) and the same model held a ring at 2.1 km on the next play. The first destroy play locked the derelict in 100 ticks, then burned to 240 m/s, tapped Stop at 2 km, overshot at 157 m, kept the guns on it to 4578 hp while it receded, and burned to 1650 m/s trying to get back. The scripted probe closes at 80 m/s, stops at 1.6 km and holds the triggers: the hauler breaks up and leaves contacts after 56 game seconds. With the fight steps rewritten around range control, pi closed at 160 m/s, tapped Stop at 1.96 km, settled 1.1 km off and broke the hauler up in 900 ticks of fire.

Not built yet

A --ui tui renderer on the same event bus, and bench run, a sweep of plays over a scenario set with an index.json, an HTML report and a --baseline diff release over release.