Terminal benchmarks often stop at files, commands, and exit codes. Interactive software has a second truth: the project must import, launch, accept controlled input, change runtime state, and produce the intended behavior. GameForgeBench evaluates coding agents on that complete loop.
The benchmark target is a coding agent, not a player bot. Evaluator-controlled probes provide runtime evidence; the agent is judged on engineering work, debugging, behavior, and regression resistance.
How It Differs From Related Benchmarks
SWE-bench treats a resolved GitHub issue as the task and checks whether the repo's existing test suite passes after the patch. The reward is binary and the environment is a layered Docker image built from the original repo. It tests general software engineering but has no engine runtime.
JAMER (arxiv 2606.19830) generates Godot projects from theme keywords and measures structural completeness (SCS) and behavioral alignment (BAS) using headless Godot runs. It uses Claude Code as the agent harness with model-swapping. Its metrics are continuous rather than binary and the tasks are generative rather than repair.
GameCraft-Bench (arxiv 2606.17861) covers 140 Godot tasks across 15 game families and judges agent output by replaying recorded gameplay through a multimodal LLM rubric judge. The top frontier agent reached 41.5% on its scale.
GameForgeBench differs on three points. First, every task is an existing-project repair inside a real engine (Unity, Godot, Minecraft, Roblox, C++), not a generative or pure-function task. Second, the reward is determined solely by task-specific runtime behavior checked by a deterministic native probe — file presence and project parse are supporting gates only and cannot produce a passing reward on their own. Third, the scoring design is explicit: binary reward (1 when all required behavior checks pass, 0 otherwise) plus an optional post-hoc diagnostic score, with no LLM judge in the scoring path.
Open Release
The corpus contains 81 normalized executable candidates spanning Godot, Unity, Roblox, Minecraft, Unreal, Web, and generic engine projects. Tasks are released in batches as their Docker verifiers and oracle/no-op controls are hardened for public use. More tasks are coming soon.
Each task is a self-contained contract with a starter project, a natural-language request, a reproducible environment, protected files, a reference solution, and executable acceptance checks.
task.toml: task metadata, schema version, timeouts, and resource limits.
instruction.md: starting condition, requested outcome, constraints, and acceptance criteria without revealing the patch.
environment/: pinned engine, operating system packages, project dependencies, and clean runtime entrypoint.
tests/: benchmark-owned verification executed outside the candidate implementation.
solution/: maintainer reference implementation used to prove solvability; never shown to the agent.
Tasks must not depend on a contributor's local absolute path, private package registry, undisclosed asset, network download at evaluation time, or interactive GUI step.
To audit the open-source package layout and current Harbor controls:
Only tasks with a current Harbor 0.22.0 oracle/no-op run record are counted as reproducible.
How To Run The Example
Requires Godot 4.6.1 for local inspection and Docker for the hermetic verifier.
cd release/tasks/godot-battle-status-bars-gloss
godot --headless --path . --editor --quit
The task README contains the reference-workspace command and the Docker invocation. The verifier observes the running Godot project and writes a machine-readable result with per-check diagnostics.
Evaluation Model
GameForgeBench separates four layers of evidence:
Artifact: required files, schemas, bindings, and protected-workspace integrity.
Engine: import, parsing, scene loading, and native startup.
Behavior: ordered events, state transitions, boundaries, persistence, and lifecycle behavior.
Diagnostics: invariant-level results explaining the outcome.
This separation distinguishes a plausible file edit from a project that actually behaves correctly when the engine runs it.
The scoring design is explicit and simple: each check is binary (pass or fail), and the primary Harbor task reward is 1 only when all required checks pass, otherwise 0. An optional post-hoc diagnostic score in [0, 1] can be computed from passed_checks / total_checks for near-miss analysis, but it is never substituted for the primary reward. File presence and project parse checks are supporting gates only — a passing reward cannot be obtained from them alone.
Task Quality Bar
A strong task has a narrow user-visible objective, a believable engineering problem, enough surface area for inspection and repair, deterministic evidence, and a meaningful regression boundary. It should reward understanding the project rather than string matching a known patch.
Reviewers reject tasks that can pass by deleting the gameplay loop, replacing the project with a stub, weakening the verifier, hard-coding the probe sequence, or relying on screenshots alone.
How To Contribute
Fork the repository and create a branch for one task.
Add a directory under release/tasks/<task-name>/ using the format above.
Make tests/test.sh executable and include a reference solution.
Run a clean Docker build, the verifier, and the reference solution from a clean checkout.
Open a pull request describing the runtime behavior and regression boundary.
Public Scope
The public repository is the inspectable trust layer: task contracts, starter projects, verifier design, reference examples, and reproducible reports. Private evaluation variants, collected agent trajectories, and service credentials are not published as benchmark answers.
Contributors
GamePhanesStudio maintains the benchmark, task format, runtime adapters, and evaluation tooling. Contributions are welcome through issues, discussions, and pull requests.
License
MIT.
Citation
@software{gameforgebench2026,
author = {GamePhanesStudio},
title = {GameForgeBench},
year = {2026},
url = {https://github.com/GamePhanesStudio/GamePhanes},
license = {MIT}
}
