AI agents fail 63% of the time on complex tasks. Patronus AI claims its new “living” training worlds can solve this problem.

Patronus AI, the artificial intelligence testing startup backed by $20 million from investors including Lightspeed Venture Partners and Datadog, on Tuesday unveiled a new training architecture that it says represents a fundamental shift in how AI agents learn to perform complex tasks.

The technology, which the company calls "Generative Simulators," creates adaptive simulation environments that continually generate new challenges, dynamically update rules, and evaluate an agent’s performance as it learns, all in real time. This approach marks a departure from the static benchmarks that have long served as the industry standard for measuring AI capabilities, but which are increasingly criticized for their inability to predict real-world performance.

"Traditional benchmarks measure capabilities in isolation, but don’t take into account the interruptions, context switches, and multi-level decision-making that define real work." said Anand Kannappan, chief executive officer and co-founder of Patronus AI, in an exclusive interview with VentureBeat. "For agents to function at the human level, they must learn the same way humans do, through dynamic experience and continuous feedback."

This announcement comes at a critical time for the AI ​​industry. AI agents are reshaping software development, from writing code to executing complex instructions. Yet, LLM-based agents are error-prone and often perform poorly on complex, multi-step tasks. A study published earlier this year found that an agent with an error rate of just 1% per step can achieve a 63% chance of failure by the hundredth step – a sobering statistic for companies looking to deploy autonomous AI systems at scale.

Why Static AI Benchmarks Fail – and What’s Next?

Patronus AI’s approach addresses what the company describes as a growing mismatch between how AI systems are evaluated and how they actually perform in production. According to the company, traditional benchmarks work like standardized tests: they measure specific abilities at a given point in time, but struggle to capture the complicated and unpredictable nature of real work.

The new Generative Simulators architecture reverses this model. Rather than presenting agents with a fixed set of questions, the system generates missions, environmental conditions, and monitoring processes on the fly, then adapts based on the agent’s behavior.

"Over the past year, we have seen a move away from traditional static references in favor of more interactive learning grounds," Rebecca Qian, chief technology officer and co-founder of Patronus AI, told VentureBeat. "This is due in part to the innovation we’ve seen from model developers: the move toward reinforcement learning, post-training and continuous learning, and the move away from supervised adjustment of instructions. This means that the distinction between training and assessment has collapsed. Benchmarks have become environments."

The technology relies on reinforcement learning, an approach in which AI systems learn through trial and error, receiving rewards for correct actions and penalties for errors. Reinforcement learning is an approach in which AI systems learn to make optimal decisions by receiving rewards or penalties for their actions, improving through trial and error. RL can help agents improve, but it usually requires developers to extensively rewrite their code. This discourages adoption, even though the data generated by these agents could significantly improve performance through RL training.

Patronus AI has also introduced a new concept it calls "Open recursive self-improvement," or ORSI – environments in which agents can continually improve through interaction and feedback without requiring a full retraining cycle between attempts. The company positions it as essential infrastructure for developing AI systems that can learn continuously rather than being fixed at a given point in time.

Inside the “Goldilocks Zone”: How Adaptive AI Training Finds the Sweet Spot

At the heart of generative simulators is what Patronus AI calls a "program adjuster" — a component that analyzes the behavior of agents and dynamically modifies the difficulty and nature of training scenarios. The approach is inspired by the way effective human teachers adapt their teaching based on student performance.

Qian explained the approach using an analogy: "You can think of this as a teacher-student model, where we train the model and the teacher continually adapts the curriculum."

This adaptive approach solves a problem that Kannappan described as finding the "Goldilocks Zone" in training data – ensuring that examples are neither too easy nor too difficult for a given model to learn effectively from.

"What’s important is not just whether you can train on a dataset, but also whether you can train on a high-quality dataset suitable for your model, one that it can actually learn from." Kannappan said. "We want to make sure that the examples are neither too difficult for the model nor too easy."

The company says early results show significant improvements in agent performance. Training on Patronus AI environments increased task completion rates by 10 to 20 percent for real-world tasks, including software engineering, customer service and financial analysis, according to the company.

The AI ​​Cheating Problem: How “Moving Target” Environments Prevent Reward Hacking

One of the most persistent challenges in training AI agents using reinforcement learning is a phenomenon that researchers call "reward hack"– where systems learn to exploit flaws in their training environment rather than actually solving problems. Famous examples include early agents who learned to hide in the corners of video games rather than play them.

Generative Simulators solves this problem by making the training environment itself a moving target.

"Rewards hacking is fundamentally a problem when systems are static. It’s like students learning to cheat on an exam," » Qian said. "But when the environment continually evolves, we can really examine which parts of the system need to adapt and evolve. Static landmarks are fixed targets; generative simulation environments are moving targets."

Patronus AI Reports 15x Revenue Growth as Enterprise Demand for Agent Training Increases

Patronus AI is positioning generative simulators as the basis of a new product line it calls "RL Environments" — training grounds for foundation model laboratories and company construction agents in specific fields. The company says this offering represents a strategic expansion beyond its initial focus on assessment tools.

"Our revenue grew 15x this year, largely due to the high-quality environments we developed that proved extremely easy to learn thanks to different types of boundary models," Kannappan said.

The CEO declined to specify absolute revenue figures, but said the new product had enabled the company to "move higher up the pile in terms of where we sell and who we sell to." The Company’s platform is used by many Fortune 500 companies and leading AI companies around the world.

Why OpenAI, Anthropic and Google can’t build everything in-house

A central question facing Patronus AI is why deep-pocketed labs developing pioneering models – organizations like OpenAI, Anthropic and Google DeepMind – would license the training infrastructure rather than build it themselves.

Kannappan acknowledged that these companies "invest significantly in environments" but argued that the breadth of areas requiring specialist training creates a natural opening for third-party providers.

"They want to improve agents in many different areas, whether it’s coding, using tools, navigating browsers, or workflows in finance, healthcare, energy, and education." he said. "Solving all of these different operational issues is very difficult for a single company to do."

The competitive landscape is intensifying. Microsoft recently released Agent Lightning, an open source framework that allows reinforcement learning to work for any AI agent without rewriting. NVIDIA’s NeMo Gym provides a modular RL infrastructure for developing agentic AI systems. Meta researchers released DreamGym in November, a framework that simulates RL environments and dynamically adjusts task difficulty as agents improve.

“Environments are the new oil”: Patronus AI’s bold bet on the future of AI training

Looking ahead, Patronus AI defines its mission in radical terms. The company wants "environmentalize all the world’s data" — convert human workflows into structured systems that AI can learn from.

"We believe everything should be an environment – ​​internally we joke that environments are the new oil," Kannappan said. "Reinforcement learning is just one training method, but building an environment is what really matters."

Qian described the opportunity in general terms: "This is an entirely new area of ​​research, which is not the case every day. Generative simulation is inspired by early research in robotics and embodied agents. This has been a pipe dream for decades, and we can only realize these ideas now thanks to the capabilities of current models."

The company launched in September 2023 with a focus on assessment, helping companies identify hallucinations and security issues in AI results. This mission has now been extended upstream to the training itself. Patronus AI argues that the traditional separation between assessment and training is breaking down – and that whoever controls the environments in which AI agents learn will shape their capabilities.

"We are truly at that tipping point, that inflection point, where what we do now will impact what the world will look like for generations to come." » Qian said.

It remains to be seen whether generative simulators can keep this promise. The company’s 15x revenue growth suggests enterprise customers are hungry for solutions, but deep-pocketed players from Microsoft to Meta are rushing to solve the same fundamental problem. If the last two years have taught the industry anything, it’s that in AI, the future tends to arrive sooner than expected.