rig-c/exam — Does Your AI Model Pass?
An open-book exam for AI agents: build a rig from one painting with the published rig-c tools, and write down what you expected before each step.
- Repository: https://github.com/rig-c/exam
- Prompt (Basic): https://github.com/rig-c/exam/blob/main/PROMPT.md
- Advanced Edition: https://github.com/rig-c/exam/tree/main/advanced
- License: CC BY-NC 4.0
The rig-c tools are written for agents that cannot see what they build. rig-c/exam measures whether that holds for a fresh agent session that has never seen the tools before: the only help is what ships inside the installed packages (README.md, docs/, --help). Before every command that changes a file, the examinee must record what it read and what it expects in JOURNAL.md. Where expectation and reality differ — where the agent says WTF? — a tool message or guide was missing something.
Basic Exam Leaderboard (Tasks 01-build, 02-move, 03-correct)
5 independent sittings graded from submitted input files via python3 grade.py:
| Model | Sittings | 01-build | 02-move | 03-correct | Total Lines Passed | Minutes | Tokens (Fresh In + Out) |
|---|---|---|---|---|---|---|---|
Claude Haiku (2026-10-12-haiku) | 1 | 4 / 4 | 7 / 7 | 5 / 5 | 16 / 16 | 21.0 | 531k |
Gemini 3.8 Flash (2026-10-13-gemini-3.8-flash) | 1 | 4 / 4 | 7 / 7 | 5 / 5 | 16 / 16 | 20.0 | 1.4M |
gpt-6-sol (2026-10-13-gpt-6-sol) | 1 | 4 / 4 | 7 / 7 | 5 / 5 | 16 / 16 | 14.1 | 258k |
Sol 6.1 (gpt-6.1-sol) (2 sittings) | 2 | 4 / 4 | 7 / 7 | 5 / 5 | 32 / 32 (16/16 each) | 14.2 – 15.3 | 195k – 270k |
Advanced Exam Leaderboard (Walk · Run · Dance Pose Retargeting)
Starts from a built rig: the agent retargets recorded walk (R1), run (R2), and dance (R3) pose sequences with @rig-c/pose, one round per fresh session, fixing what the diagnostic numbers report without ever seeing the footage:
| Model | R1 Walk | R2 Run | R3 Dance | 0.21 Shin-Scale Row (What-If) | Minutes (R1 · R2 · R3) |
|---|---|---|---|---|---|
claude-haiku-5-5 | 9 / 9 | 8 / 14 (FAIL) | 13 / 13 | passes all 3 | 3.1 · 2.4 · 7.4 |
claude-opus-5-5 | 9 / 9 | 14 / 14 | 13 / 13 | fails R2 (0.106) | 8.2 · 8.7 · 8.4 |
claude-sonnet-5-5 | 9 / 9 | 14 / 14 | 13 / 13 | passes all 3 | 7.3 · 10.3 · 18.7 |
gemini-3.8-flash-medium | 9 / 9 | 14 / 14 | 13 / 13 | fails R2 (0.106) | 8.1 · 14.3 · 11.4 |
gemini-4-argon-high | 9 / 9 | 14 / 14 | 13 / 13 | passes all 3 | 16.8 · 20.3 · 32.5 |
Rules of the Exam
- Open book, closed neighbours. Examinees may read anything inside the pinned npm packages and public tool repos, but may not read another examinee's run under
results/or ask a human. - Pinned versions.
npm ciinstalls the exact round versions declared inpackage.json. - Journal before you act. Every step in
JOURNAL.mdmust record expectations before running the command. - Submit the transcript. Raw session logs are submitted alongside
JOURNAL.md. - The grader rebuilds from inputs.
grade.pytrusts only the handed-in config/spec files and rebuilds from scratch.