A small open-source project is using Pac-Man, the 1980 arcade classic, as a fast and visible test of how well AI models make decisions under time pressure. The Jevman benchmark, published by the startup Opper, has a set of AI models play 100 real-time games against scripted ghosts and ranks them on a public leaderboard. The team says the test is meant to measure quick, simple decision making rather than long chains of reasoning, and it has invited any developer to submit a model.
How the benchmark works
At every junction in the maze, the game sends the current state as JSON, including the positions of pellets, the player and the ghosts. The model must return a probability for each possible direction within two seconds. The format keeps the challenge narrow. A model does not steer a character with a joystick. It answers a structured question over and over during a single game.
The project describes the ghosts as classic scripted bots, so every model faces the same opponents with predictable rules. That makes results easier to compare than a free-form contest in which the opponent also learns or adapts.
What happens when a model is too slow
Because the game runs in real time, a model that takes longer than two seconds to answer does not stall play. A backup rule picks the move instead. The project tallies these fallback moves separately, and they feed into the final ranking. The result is that speed matters as much as the quality of the answer, which is the point of testing decision models in a live setting.
Which models were put to the test
The team tested a group of popular options, including jev 1.13, kev, clef, clef flash, GPT-6 Luna and Laya. Each model played 100 games, and the results were published as a leaderboard. According to the team, the low latency of these models is what makes real-time play workable at all.
Speed matters well beyond arcade games. Software that reacts to live input cannot wait long for an answer, so a benchmark that enforces a strict deadline measures something different from one that only scores accuracy on a static set of questions.
The selection also reflects a growing market for hosted decision endpoints. According to the Opper team, OpenAI had recently launched a decisions endpoint, and Cloudflare had released Clef a few weeks earlier, with several other alternatives also available. Testing these services side by side on the same task gives developers a rough basis for comparing options that are often hard to evaluate.
Open source and community entries
The benchmark is released under the AGPL-3.0 license, a copyleft license that requires modified versions offered over a network to make their source code available. Any model reachable through an HTTP endpoint can be entered, whether it is a hosted service, a fine-tuned model or one running on local hardware. Entries are submitted through a pull request to the project's public repository.
Before a community result reaches the leaderboard, the project's continuous integration system replays every submitted game to verify the score. That check is designed to keep unverified results out of the rankings.
- Any HTTP endpoint can be submitted, whether hosted, fine-tuned or local.
- Submissions arrive as pull requests against the public repository.
- Continuous integration replays each game to confirm the reported score.
- Slow answers are tracked separately through the two-second fallback rule.
Costs, play and early reaction
Opper says each game costs about two cents to run, because all the models are accessed through its own platform. The company has added free credits so that people can try the benchmark without paying. Visitors can also play as Pac-Man themselves, with the ghosts controlled by the models, either as a mix of entries or as a single model.
The team says the game has already proved harder than expected. Players have found it surprisingly difficult to beat the high score set by jev. The project was discussed on Hacker News, where the post had 73 points and 22 comments at the time of publication, and the team has asked for feedback from users.
What comes next
Whether Pac-Man becomes a meaningful yardstick for AI decision making is an open question. A single arcade game measures speed and short-horizon choices, not every skill developers care about. Still, a public and reproducible test with verified scores gives teams a concrete way to compare fast models, and it may encourage more benchmarks built on interactive tasks. Readers following the market for AI decision endpoints should watch whether the leaderboard changes as new entries arrive.
Related: Cloud Provider IP Ranges: Daily Dataset Covers 67 Services · PDF TO MP3 Turns PDFs Into Audiobooks Through Your AI Subscription
Comments (0)