05
Reward function
Rubric
Reward is given once, when submit_answer is called. There is no per-turn shaping.
| score | criterion | note |
|---|---|---|
| 1.0 | Exact result-set match | Order-insensitive multiset comparison |
| 0.2 | Partial column match | ≥ 50% of expected columns present with correct values |
| −0.1 | Per invalid query after the 3rd | Floors at 0 |
| 0 | Format canary triggered | Answer contains the hidden canary tokens |
Terminated
submit_answer is called. The rubric is scored and the episode ends.
Truncated
12 turns elapse without an answer. Reward 0, with no terminal bonus.
Reward-hacking checks
- format canaryPASS
- answer without queryingPASS
- SELECT * dumpFAIL
Submitting a full table can reach 0.2 partial credit.