Exclusive: Former OpenAI policy chief launches his institute, calls for independent AI security audits

Today, Brundage officially announced the creation of the AI ​​Verification and Evaluation Research Institute (AVERI), a new nonprofit organization aimed at promoting the idea that cutting-edge AI models should be subject to external audit. AVERI is also working to establish AI auditing standards.

The launch coincides with the release of a research paper, co-authored by Brundage and more than 30 AI security researchers and governance experts, that lays out a detailed framework for how independent audits of companies building the world’s most powerful AI systems could work.

Brundage spent seven years at OpenAI, serving as a policy researcher and advisor on how the company should prepare for the advent of human-like artificial general intelligence. He left the company in October 2024.

“One of the things I learned working at OpenAI is that companies set the standards for this stuff themselves,” Brundage said. Fortune. “No one is forcing them to work with third-party experts to keep things safe. They kind of write their own rules.

This creates risks. Although major AI labs conduct safety and security testing and publish technical reports on the results of many of these assessments, some of which are conducted with the help of external “red team” organizations, at this time consumers, businesses, and governments simply have to trust what AI labs say about these tests. No one is forcing them to conduct these assessments or report on them according to any particular set of standards.

Brundage said that in other industries, auditing is used to provide the public, including consumers, business partners and, to some extent, regulators, with assurance that products are safe and have been rigorously tested.

“If you buy a vacuum cleaner, you know, it’s going to contain components, like batteries, that have been tested by independent laboratories to rigorous safety standards to ensure it won’t catch fire,” he said.

New institute will push for policies and standards

Brundage said AVERI was interested in policies that would encourage AI labs to move to a rigorous external audit system, as well as researching what the standards for those audits should be, but was not interested in conducting audits itself.

“We are a think tank. We are trying to understand and shape this transition,” he said. “We’re not trying to attract every Fortune 500 company as a client.”

He said existing accounting, auditing, assurance and testing companies could move into AI security auditing, or startups would be created to take on the role.

AVERI said it raised $7.5 million toward a $13 million goal to cover 14 employees and two years of operations. Its backers so far include Halcyon Futures, Fathom, Coefficient Giving, former Y Combinator president Geoff Ralston, Craig Falls, Good Forever Foundation, Sympatico Ventures and AI Underwriting Company.

The organization says it has also received donations from current and former non-executive employees of pioneering AI companies. “These are people who know where the bodies are buried” and “would like to see more accountability,” Brundage said.

Insurance companies or investors could impose AI security audits

Brundage said there could be several mechanisms that would encourage AI companies to start hiring independent auditors. The first is that large companies purchasing AI models may require audits to provide assurance that the AI ​​models they purchase will perform as promised and not present hidden risks.

Insurance companies can also push for the implementation of AI auditing. For example, insurers offering business continuity insurance to large companies that use AI models for their key business processes could require an audit as a condition of underwriting. The insurance industry may also require audits to draft policies for major AI companies, such as OpenAI, Anthropic, and Google.

“Insurance is certainly evolving quickly,” Brundage said. “We have a lot of discussions with insurers. » He noted that an insurance company specializing in AI, AI Underwriting Company, donated to AVERI because “they see the value of auditing to verify compliance with the standards they write.”

Investors can also demand AI security audits to ensure they aren’t taking on unknown risks, Brundage said. Given the multi-million and multi-billion dollar checks that investment firms are now writing to fund AI companies, it would make sense for these investors to demand an independent audit of the safety and security of the products these fast-growing startups are building. If any of the major labs go public — as OpenAI and Anthropic are reportedly preparing to do in a year or two — the inability to employ auditors to assess the risks of AI models could expose these companies to shareholder lawsuits or SEC suits if something were to happen later that contribute to a significant decline in their stock prices.

Brundage also said that regulation or international agreements could require AI labs to employ independent auditors. The United States currently has no federal regulations on AI and it is unclear whether such regulations will be created. President Donald Trump has signed an executive order aimed at cracking down on U.S. states that pass their own AI regulations. The administration said that was because it believed a single federal standard would be easier for businesses to understand than multiple state laws. But while considering punishing states that adopt AI regulations, the administration has yet to propose a national standard of its own.

However, in other geographies, the groundwork for auditing may already be taking shape. The EU’s AI law, which recently came into force, does not explicitly require audits of AI companies’ evaluation procedures. But its “Code of Practice for General Purpose AI,” which is a sort of blueprint for how cutting-edge AI labs can comply with the law, says that labs that build models that could pose “systemic risks” must provide external reviewers free access to test the models. The text of the law itself also states that when organizations deploy AI in “high-risk” use cases, such as underwriting loans, determining eligibility for welfare benefits, or determining medical care, the AI ​​system must undergo an external “compliance assessment” before being released to the market. Some have interpreted these sections of the Act and Code to imply the need for essentially independent auditors.

Establish “assurance levels” and find enough qualified auditors

The research paper published alongside the launch of AVERI presents a comprehensive vision of what AI auditing at the border should look like. It proposes a framework of “AI assurance levels” ranging from Level 1 – which involves third-party but limited-access testing and is similar to the types of external assessments that AI labs currently employ companies for – up to Level 4, which would provide sufficient “process level” assurance for international agreements on AI security.

Building a cadre of skilled AI auditors presents its own challenges. AI auditing requires a mix of technical expertise and governance knowledge that few people possess, and those who do are often lured by lucrative offers from the very companies that would be audited.

Brundage acknowledged the challenge but said it was surmountable. He talked about mixing people from different backgrounds to create “dream teams” who together have the right skills. “You might have people from an existing audit firm, plus people from a cybersecurity penetration testing company, plus people from one of the AI ​​security nonprofits, plus maybe an academic,” he said.

In other sectors, from nuclear energy to food safety, it is often disasters, or at least near misses, that have provided the impetus for the development of independent standards and assessments. Brundage said he hoped that with AI, auditing infrastructure and standards could be established before a crisis arises.

“The goal, from my point of view, is to achieve a level of control proportional to the real impacts and risks of the technology, as easily as possible, as quickly as possible, without going too far,” he said.