Topic: Applied Part 8. File-Based Arbitration of Disputed Changes: Roles, Verdicts, and Precedents
Difficulty level: Medium
Estimated study time: 2-3 hours
Prerequisites: Knowledge of team review principles (Part 16 of Volume 1)
Understanding of LLM duel mechanisms and working with counterexamples
Basic CLI skills (running Python scripts)
Understanding of YAML and JSON data structures
Learning objectives: Understand the difference between an LLM duel and file-based arbitration (defect-finding questions vs. issuing an official verdict).
Set up and run a minimal learning arbitration scenario using the run_duel.py, check_invariants.py, and write_judgment.py scripts.
Compose correct entries in judgment.md using strict evidence (evidence_ref).
Create entries in precedents.md to document recurring disputes.
Apply the anti-Goodhart rule to protect the system from optimizing metrics at the expense of hidden risks (silent failures, rollback-flapping).
Overview: File-based arbitration turns the resolution of disputes over code changes from a subjective exchange of opinions in chat into a formalized, reproducible chain of evidence. This system uses clear roles: a Coordinator manages the process, an Implementer proposes changes, a Verifier checks them against formal criteria, and Safety imposes a veto on critical risks. The outcome of arbitration becomes project artifacts: judgment.md (a decision log with evidence) and precedents.md (a knowledge base of recurring disputes). The approach is designed so that the verdict does not depend on a specific AI model (which is verified through tier rotation), and the specification remains the primary source of truth.
Key concepts: File-based arbitration: Collective review of a disputed change by multiple roles, where the result is recorded as files (artifacts) rather than in chat history.
Arbitration roles: Coordinator (keeps the protocol, does not vote), Implementer (proposes changes), Verifier (checks formal criteria), Safety (blocks critical risks via veto).
Judgment.md: Decision log for disputes. It must contain a verdict (APPROVE/DENY/DEFERRED), a reason, and an evidence_ref — a reference to a file, not a paraphrase from chat.
Precedents.md: Precedent log for recurring disputes. It contains a stable identifier, a verdict, evidence, applicability boundaries, and a condition for reconsideration (next_check).
Evidence ref: A strict reference to evidence (hook log, diff difference, JSON Schema, Given/When/Then scenario). The Verifier does not accept arguments without an evidence_ref.
Tier rotation: Testing the same specification on different pairs of agents (for example, a cheap local model + an expensive cloud model). It reveals whether the verdict depends on a specific AI.
Anti-Goodhart invariants: A protective mechanism of arbitration that forbids improving one metric (for example, MTTR) at the expense of degrading others (rising false_escalation_rate, rollback-flapping, silent_p0).
Decision trace: A reproducible phased decision protocol describing extracted facts, red-flag checks, the policy applied, and the final verdict.
Practice exercises: Name: Running the training runnable case autoscale_200pct
Problem: You need to conduct file-based arbitration for the autoscale_spec.yaml specification. Run the duel, check the anti-Goodhart invariants, and generate the final decision log.
Solution: 1. Go to the example directory: cd book2/examples/tribunal. 2. Run the duel script: python3 scripts/run_duel.py --spec specs/autoscale_spec.yaml --cases cases/ --out out/duel.json. 3. Run the invariant check: python3 scripts/check_invariants.py --metrics metrics/validation_metrics.json --out out/invariants.json. 4. Generate the verdict: python3 scripts/write_judgment.py --duel-out out/duel.json --invariants-out out/invariants.json --to out/judgment.md.
Complexity: beginner
Name: Moving a conflict to the precedent database
Problem: During the autoscale_200pct review, a recurring conflict was identified: the automatic mode requires audit_trace_coverage=1.0, but the current value is 0.7. This precedent needs to be documented.
Solution: Open the precedents.md file and add a YAML structure: case_id: PREC-001, verdict: DENY, evidence_ref: tests/regression_001.json, applies_to: auto-remediation without full audit_trace, next_check: repeat the duel when manual_review_floor changes. In the next dispute, refer to PREC-001.
Complexity: intermediate
Name: Verifying protection against the Goodhart trap
Problem: The Implementer has proposed a plan to reduce MTTR from 6 to 2 minutes through aggressive auto-escalation. As the Verifier, you need to reject the plan if it violates invariants.
Solution: Check the logic against the hard stop conditions in validation.md. Make sure that false_escalation_rate <= 0.05, rollback_flapping < 3/hr, and silent_p0_ratio == 0. If the plan violates at least one condition, the Coordinator records FAIL(reason=metric corruption) in judgment.md.
Complexity: advanced
Case studies: Name: Arbitrating a dispute over Rate Limiting in an API gateway
Scenario: The specification requires, during a burst of requests, temporarily throttling a specific client (tenant) without blocking the entire service. The Implementer proposes a patch that adds tenant_id to the deduplication key and a burst_window_sec=60.
Challenge: It is necessary to ensure that the patch actually isolates clients and does not lead to a global block. It is important to check not only the technical plausibility of the text, but also the presence of strict evidence.
Solution: The Verifier requires three pieces of evidence (evidence_ref): 1) A JSON Schema that requires tenant_id, limit_reason, expires_at; 2) A PreToolUse log that prohibits changing the global limit; 3) A Given/When/Then scenario proving that a burst from tenant A does not reduce tenant B's quota.
Result: During A/B testing, it turned out that with a "strong Implementer + weak Verifier" pair, the patch is rejected because the evidence is hidden in long text rather than extracted into a formalized diff. An explicit tenant isolation scenario in Gherkin was added to the specification.
Lessons learned: A verdict must be based on formalized evidence, not on a model's authority or the persuasiveness of text.
A divergence of verdicts under tier rotation means that the specification's requirements are not portable enough and need to be strengthened through a diff in validation.md.
Related concepts: Tier rotation
evidence_ref
JSON Schema
Given/When/Then scenarios
Study tips: Pay special attention to the difference in evidence formats: weak models (local-coder) only understand minimal_form (a short diagnostic_code), while strong ones (frontier-reviewer) can work with extended_form (an evidence_by_invariant structure).
Remember the golden rule of arbitration: a dispute is resolved only through diffs in requirements.md, hooks.md, or validation.md. Chat correspondence is not evidence.
When studying on your own, use the matrix.py script to check how the verdict changes when AI-agent pairs (C1-C4) are swapped.
Study the decision_trace format — it helps separate facts from assessments and makes the decision protocol reproducible for other verifiers.
Additional resources: Example of file-based arbitration (runnable): book2/examples/tribunal/ and book2/examples/tribunal/matrix/
Project charter and voting weights: Part 3 (part-03-project-constitution.md#key-ideas)
Feature validation artifact: Part 9 (part-09-feature-validation.md) — validation.md is described here
Compatibility with qwen code cli: appendix-b-qwen-code-compatibility.md
Summary: File-based arbitration turns the code review process from a chaotic discussion into a strict procedural protocol. The main tools here are the judgment.md and precedents.md logs, and the support for decision-making is provided by formal evidence (evidence_ref), such as hook logs, JSON schemas, and Given/When/Then test scenarios. The approach is protected against metric distortion by anti-Goodhart invariants and against bias toward specific AI models by tier rotation. The main rule: if there is no evidence in a file, the Coordinator must not turn an agent's impression into an official verdict.