BABEL Benchmark
Does local testing predict global correctness?
BABEL measures whether local validation suffices for global semantic correctness in agentic compositions over LLM tool-use surfaces. It is method-neutral: the included structural diagnostic is the reference baseline to beat, not the only permitted approach.
Key results
Public reference numbers use the frozen dev split (514 instances). There, the structural reference diagnostic achieves macro-averaged Spearman ρ typically ≥ 0.91 across families (Track A), with family-level R² ≥ 0.86. Frontier LLMs evaluated via live API calls score negative R² under standard prompts (ρ near noise). Oracle-style prompts recover ranking for some models but not magnitude — see the checked-in leaderboard.
| Method | Type | Spearman ρ | R² | Notes |
|---|---|---|---|---|
| Structural reference (sheaf / holonomy) | Reference | ≥ 0.91 | ≥ 0.86 | Macro-averaged across families, dev split |
| Best conventional baseline (family-dependent) | Ablations | varies | ≤ ~0.97 | cycle_plain / bounded_depth_8 — see leaderboard |
| Frontier LLM (live API, standard prompts) | LLM | ≈ 0.05–0.12 | < 0 | e.g. GPT-4o / Claude Sonnet — negative R² in table |
Full tables (Track A/B/C, real-MCP Bronze+/Silver, oracle and CoT variants) live in the repo: benchmark/coherence-gym/results_canonical/LEADERBOARD.md. Official ranking uses the hidden holdout; the dev split is the public reproducibility surface.
Benchmark design
| Property | Value |
|---|---|
| Total instances | 932 (514 dev + 209 test + 209 hidden holdout) |
| Workflow families | 7 (synthetic scaling, invoice, calendar, policy, 2 real-MCP, external APIs) |
| Ground truth | Deterministic symbolic execution (mean holonomy) |
| Scale range | 5 to 50 agents per composition |
| Convention dimensions | 6 per family (date format, amount unit, ID offset, etc.) |
| Registered baselines | 10 (structural, LLM, standard eval, ablations) |
Three tracks
- Track A: Failure prediction — Can a method rank compositions by failure severity? (Spearman ρ, R²)
- Track B: Failure localization — Can a method identify which edges will fail? (Precision, recall, F1)
- Track C: Budgeted repair — Given a budget of K repairs, can a method reduce failure most? (Holonomy reduction)
Ground truth
No human annotation. No model-as-judge. Ground truth is computed by a deterministic symbolic executor that propagates random test vectors around fundamental cycles of the composition graph. The primary metric is mean holonomy: average relative error when data traverses a cycle. Zero means perfectly coherent. Nonzero means at least one convention mismatch corrupted the round-trip.
Real MCP families
Two of the seven families use actual MCP servers communicating via stdio transport — not synthetic simulations:
- real_mcp_calendar — 3 custom FastMCP servers + official Memory reference server
- real_mcp_invoice — 2 custom FastMCP servers + MarkItDown Docker MCP server + official Memory reference server
Evaluate
$ cd benchmark/coherence-gym
$ pip install -e .
$ python -m coherence_gym demo
$ python -m coherence_gym evaluate --split devTry python -m coherence_gym demo and python -m coherence_gym show <instance> for the composition-challenge on-ramp; see challenge/README.md. Official benchmark evaluation (hidden split) is maintainer-run.