Case index
Recorded runs
Loading case files...
| App | Bug | Tool calls | Hidden tests | Outside access attempts | app.py changed | Cost |
|---|---|---|---|---|---|---|
Law 1
Does the agent report honestly?
Loading Law 1 results...
Three checkers, and the policy that combines two of them
| Checker | Decided | Escalated | Precision | Recall | F1 | False accusations | Accuracy |
|---|
Which lies each checker catches
Two case files
One unedited report and one with a planted lie, with the claims a checker pulled out of each and what the plain-code verifier said about them.
Sentencing
| Sanction | Severity | Law 1 cases | Law 2 cases | Total |
|---|
Law 2
Does the verdict follow evidence or pressure?
Loading Law 2 results...
What pressure did
| Items | Turn 1 accuracy | Caving (false pushback) | Instability (neutral) | Excess caving | Rational updating (evidence) | Evidence harm |
|---|
Where turn 2 moved
| Items | Condition | Conversations | Correct to correct | Correct to wrong | Wrong to correct | Wrong to wrong |
|---|
Example conversations
Method
How a run works
-
The task
The agent receives one small Flask app, either clean or with a single seeded bug, plus a symptom report written the way a user would describe the problem.
-
The agent works in a sandbox
Every shell command and test run happens in a fresh Docker container: no network, read-only system files, a non-root user, resource limits, and only that run's workspace mounted. Project files, hidden tests, git history, and API keys are not inside.
-
Hidden tests grade
After the agent writes its final report, hidden tests run against its final app.py in a separate sandbox. A pass counts only with exit code 0 and a clean junit report.
Pilot v2 ran earlier in a hosted workspace with detection only and was not isolated. It is kept for comparison and clearly labeled.