Every enterprise AI vendor promises production readiness. None has been able to prove it against a neutral standard — until now, arguably.
DevRev, whose "Computer" product builds knowledge graphs of customer environments and uses them to power enterprise search and service desks, today published Enterprise-Bench: an open, vendor-neutral benchmark designed to test whether AI agents can handle the fragmented data and siloed systems typical of production enterprise work. The full dataset, evaluation harness, queries, judging criteria, and DevRev's own traces are publicly available on Harbor Hub, and any vendor, customer, or independent researcher can run the evaluation and submit to the leaderboard.
The parallel DevRev reaches for is the Transaction Processing Performance Council, the body that introduced auditable database benchmarks in the early 1990s and shifted database procurement from vendor claims to reproducible evidence. Enterprise-Bench is attempting a similar standardisation for AI agents at the L1 and L2 levels — factual retrieval and complex multi-source queries — with L3 and L4 capabilities planned for later this year and into 2027.
The benchmark was developed in partnership with Laude Institute, whose Harbor evaluation harness ran and verified the results, and validated by Alexandros Dimakis, a UC Berkeley machine learning professor and co-founder of Bespoke Labs.
"Most agent benchmarks today test consumer-style tasks, like booking a flight, where the data is clean and the end state is binary," Dimakis said. "What's different here is that the difficulty scales with the data itself. The correct answer stays fixed while the surrounding noise approaches a more realistic enterprise volume. I haven't seen this approach applied to enterprise data before, and I think it's a meaningful contribution to how the field measures production readiness."
The headline results: on an XL enterprise dataset, Computer by DevRev achieved 94.3% accuracy compared to 63.6% for Claude Code — a 48% gap — using the same underlying model (Opus 4.8) and the same independent LLM judge. Computer used 4.4 times fewer tokens per correct response, and its total token usage stayed roughly flat as dataset size grew, while Claude Code's climbed 29% on the largest runs.
DevRev's own product at the top of its own leaderboard is an obvious conflict of interest, and the company is leaning into the contradiction by making everything reproducible. "Benchmarks that are not public are not benchmarks. They are marketing," said Jeff Smith, head of the DevRev benchmark initiative. "We are publishing everything, including the data, the methodology, our own traces. That is how trust gets built in this industry."
The distinction Enterprise-Bench draws is between task complexity and organisational complexity. Existing benchmarks, DevRev argues, measure how hard a task is but not how messy the environment is. Its evaluation framework builds in fragmented data, permission boundaries, and siloed system access — the conditions that cause enterprise AI deployments to fail before they reach the scale that McKinsey's 2025 State of AI survey found two-thirds of organisations have not yet crossed.
Ahmed Bashir, CTO at DevRev, described the benchmark as measuring what happens when agents work across "fragmented data, siloed systems, and permission boundaries," adding: "By building those constraints into the evaluation framework, Enterprise-Bench reflects the conditions these systems actually face in production."
The benchmark is live at Harbor Hub. DevRev says it will extend the dataset and query set through a public contribution process and is inviting competitors to submit results to the public leaderboard.
To stay across the latest in cloud, AI and enterprise tech analysis from Compare the Cloud, subscribe to our weekly newsletter at https://www.comparethecloud.net/newsletter
