Results

Benchmark results & proofs

Benchmark evaluations and OpenAI research problems formally verified with Turing Agent.

Evaluation

Benchmark results

Theorem formalization is measured on the 143 problems of Formal TCS Bench (statements from recent TCS papers). A submission succeeds only if it compiles and passes independent semantic review. Proof formalization uses 248 theorems from the PFR and TestingLowerBounds (TLB) projects in RLMEval; success means a Lean proof with no sorry. Each model is run as direct generation, inside Codex CLI, inside Claude Code, and inside Turing Agent. All numbers are pass@1 with one final submission per problem.

Pass@1 (%). Underlined: best in column. PFR covers additive combinatorics and entropy methods (Polynomial Freiman–Ruzsa); TLB covers information-theoretic lower bounds for hypothesis testing.

Bar charts of pass@1 results for GLM-5.3 and DeepSeek-V4-Pro.
Pass@1 results on Formal TCS Bench (compilation and success) and RLMEval (PFR, TLB, total) for both models.

With GLM-5.3 and DeepSeek-V4-Pro, Turing Agent reaches 57.34% and 61.54% success on Formal TCS Bench — 16.78 and 32.17 points above Claude Code, the strongest baseline — with compilation rates of 97.20% / 99.30%. The gap between compiling and being faithful is exactly why semantic review is evaluated separately.

On RLMEval it exceeds 82% overall with both models: 82.66% with GLM-5.3 (best of all methods, 2.42 points above Claude Code) and 82.66% with DeepSeek-V4-Pro, 8.06 points above Codex CLI and within 4.03 points of Claude Code.

OpenAI's 722 math manuscripts

OpenAI research problems

We highlight 3 problems from OpenAI's research collection, with Lean-kernel-checked proofs produced by Turing Agent using DeepSeek-V4-Pro and DeepSeek-V4.1-Flash.

Funnel from 722 manuscripts to 121 TCS problems, of which 22 were not fully verified in Lean; three examples verified by Turing Agent are highlighted.
Task selection. 121 of the 722 problems are TCS-related; 22 of those had not been fully verified in Lean; three highlights verified by Turing Agent are shown below. These highlighted cases involve reconstructing missing arguments, introducing and verifying new auxiliary lemmas, and coordinating long dependency chains — up to more than a hundred verified auxiliary claims.
Write-ups

Problem explorer