A new open-source benchmark called Parsimony asks a question that most AI coding leaderboards leave out: how much extra code does a model add when it fixes a software problem? According to the project's page, which carries a publication timestamp of 14 October 2026, existing benchmarks focus on whether a model solves a task and how much it costs to run. They do not penalise excess code, even though bloated changes can make a codebase harder to maintain over time. Parsimony is designed to fill that gap by measuring how much code AI models add.
Why the size of an AI patch matters
When a coding agent fixes a bug, the result is a patch: a set of changes to the existing files. A patch that solves the problem but adds hundreds of unnecessary lines can be a poor outcome for the developers who have to read, review and maintain it later. Parsimony argues that a solve rate alone hides this cost.
The idea is simple. A model that reaches the same result with less new code leaves a cleaner project behind. Over months of use, that difference can affect how easy a codebase is to change, debug and extend.
A focus on net change, not line count
Parsimony does not count lines. The project's page says that counting lines would reward cramming many statements into each line, so the benchmark counts coding units instead. These are syntax elements such as statements, function calls, comparisons, names and literals.
Deleted code also matters. The benchmark subtracts removed units from added units to produce a net figure. A patch that replaces a long block with a shorter one therefore scores better than one that only adds new code.
How Parsimony measures code
The method starts with the patches that other benchmarks already publish. Parsimony applies each patch to the task's original code, then parses both the old and new versions of every changed file. According to the page, nothing is executed during this step. The tool analyses the code's structure rather than running it.
Some files are excluded from the count. The page says that tests, documentation and generated files are left out, and that only Python code is currently measured. This keeps the focus on the logic that a model actually writes into a project.
Where the pass and fail results come from
Parsimony does not run the tests itself. Pass and fail results come from SWE-bench's published results, and the page states that the tests are not rerun. This means the size measurements are layered on top of results that already exist, rather than being generated by a new test run.
- Patches come from benchmarks that are already published.
- Each patch is applied and both file versions are parsed, with nothing executed.
- Tests, docs and generated files are excluded.
- Solve status is taken from SWE-bench's published results.
What the leaderboard shows
The leaderboard ranks models by mean net units added on the tasks they solved, with the lowest figure at the top. A separate column, labelled net units added (mean), also includes failed attempts, so readers can see how much code a model writes even when it does not succeed. The page also reports how many solved attempts could be measured.
Parsimony includes an outside reference point, the Artificial Analysis Intelligence Index. Scores are matched by model and reasoning effort. The page notes that this index does not affect the ranking, so it serves only as context for how capable each model is in general.
Independent runs shown separately
Some models appear faded on the chart. According to the page, these were run by its author on the first three to ten DeepSWE Python tasks, using pi for OpenAI models and Claude Code for Anthropic models. Their results were graded with each task's own tests. The page says these faded entries are not verified and are not ranked.
The author also ran GPT-5.6 Sol both ways. The page reports that the author's run added 0.88 times the code of the published run. The author presents this as one comparison point rather than a general finding.
Limits readers should keep in mind
Parsimony itself makes a clear caveat: a smaller patch is not automatically a better one. Unit counts measure size, not readability or design quality. A short solution can still be hard to follow, and a longer one can be clearer.
There is also a link between size and success. The page reports that models which write more code also tend to solve more tasks, with Spearman correlations between 0.33 and 0.60 on the boards that include 26 or more models. This means that a low code count could partly reflect a model that solves fewer problems, so the two measures should be read together.
Finally, every model in the comparison uses the same agent, mini-SWE-agent. The page notes that the agent still influences how much code gets written, so results may shift with a different tool.
What comes next
Parsimony also offers a way for others to add their own code-size results from different benchmarks. Users can download an example file, fill in their figures and open it to generate the same chart and table. The page says the file stays in the user's browser.
For developers choosing AI coding tools, the project offers a useful reminder that a model's success rate is only part of the picture. How much code it leaves behind may matter just as much over the life of a project. Readers should still treat any single ranking as one input among several, especially since the benchmark currently covers Python only.
Related: Reduce Claude Code Token Costs With optimAIzr CLI · Anthropic's OSS Scanner Automates Security Checks for Open Source
Comments (0)