Overview
Parsimony is an open-source benchmark for comparing how much code AI coding models add. It complements benchmarks centered on solve rate and cost by measuring patch size, reflecting the concern that unnecessarily large changes can make a codebase harder to maintain.
Models are ranked by the mean net syntax units added on tasks they solved, with lower results placed first. The benchmark also presents all-measured results that include failed attempts.
Measurement Method
Parsimony applies each published patch to the task code and parses both versions of every changed file. Instead of counting lines, it counts syntax units such as calls, names, and literals, then calculates additions minus removals.
The analysis excludes tests, documentation, and generated files, and currently measures Python only. Pass or fail status comes from published SWE-bench Verified results; Parsimony does not rerun the tests.
Comparison Tools
- A leaderboard displays rank, mean net syntax units, solved-task measurements, solve counts, and measurement coverage.
- Bar and scatter charts support single-metric views or comparisons between two metrics.
- Shared-task ratios include conditional 95% ranges and counts for the underlying comparisons.
- Charts can be downloaded as PNG files or copied as images.
- Users can load benchmark JSON in the browser to view their own code-size results with the same chart and table.
Interpreting Results
A smaller patch is not automatically a better patch: syntax-unit totals measure size rather than readability or design quality. A model can rank highly by deleting code, while failed attempts that make no changes can pull an all-measured average toward zero.
Comparisons also require attention to coverage. Each model may have a different set of solved and measurable tasks, missing measurements are not treated as zero, and large patches can strongly affect ratios. Agent configuration, harness choice, retry policy, peer selection, and reasoning effort can further limit model-only conclusions.
