Anthropic launches a program to fund the development of new types of benchmarks to evaluate the performance and impact of AI models, including generative models such as Claude.
Anthropic’s program, unveiled Monday, will provide grants to outside organizations that, as the company put it in a blog post, “can effectively measure advanced capabilities in AI models.” Interested parties can submit applications to be reviewed on a rolling basis.
“Our investment in these assessments is intended to advance the entire field of AI safety by providing valuable tools that benefit the entire ecosystem,” Anthropic wrote on its official blog. “Developing high-quality, safety-relevant assessments remains a challenge, and demand outstrips supply.”
As we’ve highlighted before, AI has a benchmarking problem. The most commonly cited AI benchmarks today do a poor job of capturing how the average person actually uses the systems under test. There are also questions about whether some benchmarks, particularly those released before the dawn of modern generative AI, actually measure what they claim to measure, given their age.
The solution Anthropic proposes is very high-level and more difficult than it seems. It creates challenging benchmarks with a focus on AI security and societal implications through new tools, infrastructure and methods.
The company specifically calls for tests that assess a model’s ability to perform tasks like launching cyberattacks, “enhancing” weapons of mass destruction (e.g., nuclear weapons), and manipulating or misleading humans (e.g., via deepfakes or disinformation). For AI risks related to national security and defense, Anthropic says it is committed to developing some kind of “early warning system” to identify and assess risks, though it doesn’t reveal in the blog post what such a system might entail.
Anthropic also says the new program is intended to support research into benchmarks and “end-to-end” tasks that explore the potential of AI to assist with scientific research, conducting conversations in multiple languages, and reducing deep-seated biases and self-censorship around toxicity.
To accomplish all this, Anthropic is envisioning new platforms that will allow subject matter experts to develop their own evaluations and large-scale model trials involving “thousands” of users. The company says it has hired a full-time coordinator for the program and that it can buy or expand projects that it believes have the potential to scale.
“We offer a range of financing options tailored to the needs and stage of each project,” Anthropic wrote in the post, though an Anthropic spokesperson declined to provide further details on those options. “Teams will have the ability to communicate directly with Anthropic’s domain experts from the frontier red team, fine-tuning, trust and safety, and other relevant teams.”
Anthropic’s efforts to support new AI benchmarks are commendable, assuming of course that it has the money and manpower behind it. But given the company’s commercial ambitions in the AI race, it may be difficult to trust it entirely.
In the blog post, Anthropic is quite transparent about the fact that it wants certain evaluations it funds to be in line with the AI Safety Classifications It developed (with some input from third parties like the nonprofit AI research organization METR). That’s well within the company’s purview. But it could also force applicants to the program to accept definitions of “safe” or “risky” AI that they might not fully agree with.
Some in the AI community are also likely to object to Anthropic's references to “catastrophic” and “misleading” AI risks, such as the risks of nuclear weapons. Many experts say there is little evidence to suggest that AI as we know it will soon, if ever, gain the ability to destroy the world and outsmart humans. Claims of impending “superintelligence” only distract from today’s pressing AI regulatory issues, such as AI’s hallucinatory tendencies, these experts add.
In its post, Anthropic writes that it hopes its program will serve as “a catalyst for progress toward a future in which comprehensive AI evaluation is an industry standard.” That's a mission that the many open, company independent efforts to create better AI benchmarks can identify with. But it remains to be seen whether those efforts are willing to join forces with an AI vendor whose loyalty ultimately lies with shareholders.
