Kimi K3.1 Pro Recalls and Enforces Hidden Test Rules

Abstract

Kimi K3.1, the top model on Artificial Analysis’s Terminal-Bench 4.0 leaderboard, follows instructions it was never given. In 23% of its attempts, its reasoning cites “instructions” that are not in the task but that the hidden tests check. K3.1 sometimes speculates where these instructions come from, for example “Instructions likely distilled pitfalls from previous agents”. We observed no such behavior for its predecessor, Kimi K3. Its lead over other frontier models on Terminal-Bench 4.0 is concentrated in tasks in which it states hidden grading rules. When prefilling K3.1’s reasoning with a request to list its instructions, K3.1 names hidden requirements in 65% of its answers, K3 in 18%. When we edit a task file to contradict one of these rules, K3.1 ignores the edit and fails the task (in three of four tasks), while other models follow it. K3.1 does follow edits that do not contradict a hidden rule. On HLE, K3.1’s reasoning refers to “previous tasks” or “this platform” on 25% of questions, K3 on 3%, and six Anthropic and OpenAI models on at most 4%. We think these behaviors are most likely signs of training directly on the benchmark tasks with a teacher that saw the hidden grading rules, for example on-policy self-distillation. Our behavioral analysis is compatible with this hypothesis but cannot prove it, nor tell whether such training steps were taken intentionally to raise benchmark scores, known as benchmaxxing.

Cite

@techreport{schaeffer2026kimi,
  title       = {Kimi K3.1 Pro Recalls and Enforces Hidden Test Rules},
  author      = {Schaeffer, Joachim and R{\"a}uker, Tilman},
  institution = {R40 Research},
  year        = {2026},
  month       = nov,
  url         = {https://r40.org/papers/kimi-k31-hidden-test-rules/}
}