DevRev has launched Enterprise-Bench, an open benchmark for testing AI agents in enterprise environments. The framework is designed to provide a neutral standard for assessing production use.
The benchmark compares AI systems on accuracy, efficiency and safety in conditions that reflect fragmented data, siloed systems and permission boundaries inside large organisations.
In the first published tests, DevRev said its own product, Computer, outperformed Anthropic's Claude Code. On the XL enterprise dataset, Computer recorded 94.3% accuracy against 63.6% for Claude Code, while using about 5,598 tokens per correct response compared with about 24,461.
According to DevRev, Computer's token usage stayed broadly flat as the dataset grew, while Claude Code's total token use rose 29% on the largest runs. Both systems were tested on identical L1-L2 tasks with the same data, the same model family and the same independent LLM judge.
DevRev argues that this makes system design and data retrieval architecture, rather than the underlying model alone, the main point of difference in the results.
Industry gap
The launch comes as businesses continue to experiment with AI tools but remain cautious about wider deployment across operations. DevRev cited McKinsey research showing that nearly two-thirds of organisations have not started scaling AI across the enterprise, and that no more than 10% in any given business function are scaling AI agents.
Enterprise-Bench was developed with Laude Institute, whose Harbor evaluation harness was used to run and verify the tests. The work was also validated by Professor Alexandros Dimakis of the University of California, Berkeley, who is also a Co-Founder of Bespoke Labs and a DevRev board member.
DevRev has published the dataset, methodology, results and evaluation traces, and said vendors, customers and researchers can run the benchmark and submit scores to a public leaderboard.
The framework is intended to mirror the complexity of enterprise systems rather than the simpler consumer tasks that dominate many existing agent benchmarks. Those tests often focus on activities such as booking travel or completing a single, cleanly defined workflow.
By contrast, Enterprise-Bench starts with factual retrieval and complex multi-source queries under what DevRev describes as organisational complexity. That includes cases where the right answer must be found across multiple systems while respecting access controls and audit requirements.
"What Enterprise-Bench adds to the benchmark community is a way to measure organizational complexity, not just task complexity," said Ahmed Bashir, Chief Technology Officer at DevRev.
"In real enterprises, AI systems have to work across fragmented data, siloed systems, and permission boundaries. By building those constraints into the evaluation framework, Enterprise-Bench reflects the conditions these systems actually face in production," Bashir said.
Testing method
The first release covers L1-L2 tasks, which span factual retrieval and more complex queries across several sources. DevRev described this as an early stage in a broader grading system modelled in spirit on levels of autonomous driving.
Professor Dimakis said the approach differs from common benchmark designs because the difficulty rises with the volume of surrounding information rather than from a change in the correct answer itself.
"Most agent benchmarks today test consumer-style tasks, like booking a flight, where the data is clean and the end state is binary," said Professor Alexandros Dimakis of the University of California, Berkeley.
"What's different here is that the difficulty scales with the data itself. The correct answer stays fixed while the surrounding noise approaches a more realistic enterprise volume. That is a much harder and more useful test of whether an agent's accuracy actually holds up. I haven't seen this approach applied to enterprise data before, and I think it's a meaningful contribution to how the field measures production readiness," Dimakis said.
DevRev said the benchmark assesses three measures it believes are often missing from existing tests: whether an answer is correct and traceable to a verifiable source, whether cost rises with question complexity or data volume, and whether permission boundaries are respected.
All submitted results must include traces alongside scores, according to DevRev. The company added that an independent LLM judge verifies outcomes against published criteria.
Public challenge
Releasing the full dataset and evaluation process is also a direct challenge to AI vendors that make competing claims about agent performance without publishing comparable evidence. DevRev drew a parallel with the role industry benchmarks played in database markets when suppliers moved from broad performance claims to auditable tests.
"Benchmarks that are not public are not benchmarks. They are marketing," said Jeff Smith of the Office of the CTO at DevRev.
"We are publishing everything, including the data, the methodology, our own traces. That is how trust gets built in this industry, and that is how, as vendors, we build better systems," Smith said.
DevRev, founded in 2020, sells AI software under the Computer brand. The company says the product builds a knowledge graph from a customer's existing systems while keeping permissions intact, and uses that structure for enterprise search, employee service desks and customer support.
The business is led by Co-Founder and Chief Executive Officer Dheeraj Pandey, a former Co-Founder and Chief Executive Officer of Nutanix, and Co-Founder Manoj Agarwal, a former Senior Vice President of Engineering at Nutanix. DevRev said it operates globally across eight offices.