/

METR Says Anthropic’s AI Roughly Doubled Its Own Pace. The Proof Is Redacted

An outside evaluator says Anthropic’s model progress likely doubled in pace, mostly thanks to AI. Anthropic blacked out most of the evidence METR found persuasive.

By

·

Written with AI assistanceHow we reportCorrectionsReport an error

Two men standing in a modern office lobby with wooden A-frame seating booths and a potted tree, during a UK government visit to Anthropic in San Francisco

An outside evaluator spent 11 weeks inside Anthropic’s data and came back with a number the AI safety world has been dreading for years: Anthropic’s models are probably getting better about twice as fast as they used to, and AI itself is probably doing most of the accelerating. The nonprofit METR published that assessment on Nov. 5, alongside a separate review saying Anthropic’s own public risk report likely understated how fast things were moving.

The twist is that the evidence METR found most persuasive is blacked out. Anthropic had the right to redact non-public information, and METR says it used that right on “a majority of the evidence we found most informative.” So the headline finding arrives with a strange instruction attached: trust the auditor, because you can’t check the work.

Key points

  • METR estimates that as of October 2026, Anthropic is improving its models on well-specified tasks at roughly 2x the March 2024 to October 2025 rate, and says it would not be surprised by anything from 1.3x to 3x.
  • On the one published measure, an internal benchmark index called AECI, METR estimates a 1.9x acceleration since October 2025, with a 95% confidence interval of 1.5x to 2.8x.
  • METR guesses that about 80% of the current acceleration comes from Anthropic’s own use of AI, and says it is confident faster growth in training compute is not the cause.
  • Anthropic co-founder Sam McCandlish says the company redacted heavily because the details are commercially sensitive and could hand rivals “a roadmap,” and says Anthropic will publish its own analysis in the coming weeks.

What METR actually found

METR is a nonprofit that evaluates frontier AI agents, known for research tracking how long a task AI agents can finish on their own. For this project, four METR researchers (2.5 full-time equivalents, per the report) worked from Aug. 12 to Oct. 29 with internal Anthropic material: benchmark scores for unreleased models, pretraining and reinforcement-learning metrics, compute data, staff surveys, code and pull-request counts, and about 15 interviews. METR says it took no payment, worked on Anthropic laptops, and used free Anthropic tokens mostly for data analysis.

Its conclusions, in METR’s own hedged terms:

  • Pace: Anthropic is “likely” improving model capabilities on well-specified tasks at about twice the rate it did between March 2024 and October 2025. METR stresses “high uncertainty” and a plausible range of 1.3x to 3x.
  • Cause: a majority of today’s acceleration, “perhaps ≈80%,” is likely due to AI, mostly as an assistant to human researchers rather than as an autonomous researcher. METR says non-AI factors likely played a large role in late 2025.
  • Compute: METR says it saw “convincing evidence” that the speedup was not driven by faster growth in training compute. The evidence itself was redacted.
  • Brakes: METR found no clear bottleneck in algorithms, compute, experiment compute or data likely to bite in the next 12 months, while warning that its view on bottlenecks “changed several times.”

That last point is the one that should make people sit up. If AI speeds up AI research, better AI then speeds it up further, and nothing obvious stands in the way, the curve can bend fast. METR says there is “a significant risk that further capabilities progress could lead to even faster acceleration” and that it “cannot rule out a substantial further acceleration” over the next six months. Its announcement thread repeated the 2x estimate and the 1.3x to 3x range.

What “2x” means, and what it doesn’t

This is where most hot takes will go wrong, so here is the mechanism in plain English. METR measures acceleration as a ratio of growth rates. A 2x acceleration means a year now buys as much progress as two years did at the old pace. It does not mean Claude is twice as smart, twice as fast, or twice as useful. It means the slope of the improvement line roughly doubled.

The published evidence for that slope comes from AECI, Anthropic’s in-house version of the Epoch Capabilities Index. It folds many benchmark scores into a single capability number. METR fit a line with one break point, which landed on Oct. 7, 2025, and compared the slope before and after. Using that method, it says Anthropic made about as much benchmark progress in the last year as in the two years from October 2023 to October 2025, which it describes as roughly Claude 2.1 to Sonnet 4.5.

That 1.9x is sensitive to modeling choices, and METR shows its work here:

How METR varied the analysisEstimated acceleration
Standard AECI (main result)1.9x (95% CI 1.5 to 2.8)
Different assumptions about how hard the hardest benchmark tasks are (link functions)1.1x to 2.9x
Different sets of model snapshots1.9x to 2.2x
Different sets of benchmarks1.9x to 2.2x
Including all frontier labs’ models, not just Anthropic’sAbout 1.4x

Source: METR’s Nov. 5 assessment. The last row matters. Anthropic’s models trailed the industry frontier on AECI before late 2025, METR says, so part of the “acceleration” may be Anthropic catching up from a slow start rather than the field as a whole speeding up. METR flags both readings and does not pick one.

There is also a measurement problem looming. METR says an unreleased internal model it calls Model 4 scores above 50% on every one of the 15 AECI benchmarks it was tested on. When the yardsticks saturate, you lose the ability to tell how much taller the thing has grown.

The 12x Slack poll, and why to ignore it

One number in the report is going to travel far beyond its evidence: Anthropic staff, voting with emoji reactions in internal Slack polls, put their average self-reported speedup from the newest internal model at about 12x in October 2026. METR calls this “only very loose evidence,” notes it comes from an unrepresentative subset of staff, and points out that self-reports can overstate real productivity. Its own 2025 developer productivity study found early-2025 AI tools caused a slowdown.

More to the point, METR explains that individual speedup is not the same thing as acceleration. Keeping progress on a steady curve already requires ever more input because of diminishing returns. A researcher can feel 12x faster while the company’s overall slope only doubles. If you see “Anthropic engineers are 12x faster” in a headline this weekend, that is a Slack poll wearing a lab coat.

The interviews are more useful. METR had Claude sort 11 interviewees’ quotes by whether they supported a 2x-or-more speedup in their own area: 2 yes, 3 leaning yes, 4 unclear, 2 no. One researcher said they were “now only bottlenecked on ideas & compute, no longer bottlenecked on code.” Another guessed only 15% to 20% of the change was due to AI and said they did not believe in self-reported productivity gains.

Did Anthropic cross its own tripwire?

The companion review is where things get pointed. Anthropic’s Responsible Scaling Policy (version 3.4, per METR) treats a “dramatic acceleration” of AI progress, roughly double the prior rate and plausibly due to automated AI research, as an early-warning threshold. Anthropic’s August 2026 Risk Report said its models had not crossed it as of July 15.

METR agrees that catastrophic risk from those July models was low. But it says it is “plausible” the 2x threshold had already been crossed, and that the public report “understates the level of acceleration that was occurring.” Its evidence is a side-by-side of the public and private versions of Anthropic’s own report:

Anthropic’s public report saidWhat METR says the private material showed
Leading indicators show acceleration “by less than a factor of 2”The private figures imply 1.97 by METR’s calculation; METR’s own range was 1.3 to 2.8
Fairly confident the 2025 speedup came from non-AI factorsThe private report called it “plausible” that AI-attributable speedup exceeded 2x
Model 2 scored only about 1.5 AECI points above Mythos 5, weak evidence against a sustained speedupThe private report expected later Model 2 snapshots to do better, and METR says they did

Source: METR’s Nov. 5 Risk Report review. To be fair to Anthropic, METR also says that both its researchers and Anthropic’s relevant staff “lean toward” the threshold not having been crossed on July 15. The complaint is about confidence and wording. But “less than a factor of 2” for a number that rounds to 2 is exactly the kind of phrasing a safety framework is supposed to prevent.

The redaction fight

Anthropic’s answer came from co-founder Sam McCandlish, in a Nov. 5 post on X that introduced himself as the co-founder who leads much of Anthropic’s safety work. “Many of their conclusions match my own,” he wrote. “AI is speeding up our research, but it’s hard to say exactly how much.” On the blacked-out evidence: “We redacted a lot, and I accept that this makes it harder for others to check METR’s work,” adding that revealing what drives the speedup “risks handing out a roadmap for others to go faster.” In a follow-up reply, he argued that embedded evaluators let the public trust “the evaluator rather than just trust in us.”

That argument has some force. If the recipe for a 2x research speedup were public, every rival lab would copy it by Monday. But METR’s report carefully avoids endorsing it. A footnote says METR makes no comment on whether the redactions were justified, because it is not well placed to weigh Anthropic’s business interests against the public’s. METR also discloses that Anthropic tightened its redaction guidance between Oct. 27 and 29, which led the two sides to agree on a roughly five-day extension.

Two other process details deserve more attention than they are getting:

  • The compute-for-safety check was cut. The original contract included verifying how Anthropic splits compute between capabilities and safety work. METR says Anthropic changed the scope midway to drop it, before METR reached conclusions. Anthropic told METR it wanted to keep the engagement focused and would revisit it later.
  • This was not the “employee-like” access Anthropic announced. On Oct. 15, Anthropic said it had begun giving six METR employees access “roughly on par” with its own risk teams. METR says this project is separate, started in August, and never had that unmediated access. It also lacked query access to unreleased models.

And one detail is almost too on the nose: METR says Claude was involved in most of the data collection and analysis Anthropic handed over, wrote a significant share of the documents METR received, and did most of METR’s own analysis. METR says it checked against human interviews and saw no sign of systematic bias. Still, an audit of whether AI is accelerating AI, largely run by the AI in question, is a sentence nobody would have believed in 2023.

Why it matters

Whether AI can speed up its own development is the central question behind most serious fears about runaway AI, and until now outsiders have mostly argued about it from public benchmarks and vibes. Here, an independent group has published a judgment based on a frontier lab’s internal numbers, and mainstream outlets such as AP picked it up. Even with the redactions, “probably about 2x, mostly due to AI, no obvious brakes” is a meaningful data point coming from the skeptics’ side of the table: METR’s earlier work includes studies that found AI slowing developers down and benchmarks overstating progress.

Our read: the honest version of this story is narrower than the scariest headlines and bigger than Anthropic’s public report. The evidence supports faster progress on tasks that are easy to grade. It says much less about messy real-world work, which METR itself lists as a reason it could be overestimating. And it applies to one lab, measured on that lab’s own index, with that lab deciding what the public gets to see.

The catch for everyone else is that the auditing model is now the product. If “trust the evaluator” becomes the industry standard, the evaluators’ independence, funding, conflict policies and redaction terms matter as much as any benchmark. METR deserves credit for publishing its own conflict disclosure, including that one team member received and declined an Anthropic job offer during the project.

What to watch next

  • Anthropic’s own report. McCandlish promised an analysis of the speed of progress inside Anthropic “in the coming weeks,” including where it stands against its Responsible Scaling Policy. The key question is whether Anthropic now says the 2x threshold has been crossed, and what it commits to doing about it.
  • The compute verification. Anthropic told METR it would revisit the capabilities-versus-safety compute check. Whether it actually happens, and who does it, is a cleaner test of transparency than any benchmark.
  • The embedded team’s work. Anthropic says the six METR staff with employee-like access will publish risk assessments, and that evaluators at that tier can publish their views on whether redactions were justified, which this report pointedly did not do.
  • New yardsticks. With Model 4 above 50% on every AECI benchmark, watch for harder internal tests. Without them, the next estimate of acceleration will be even less certain than this one.
  • Other labs. Nobody has published a comparable inside look at OpenAI, Google DeepMind or xAI. If Anthropic is accelerating and the others are not being measured, that is a gap in the picture, not evidence they are slower.

Hype in. Signal out.

Featured image: Secretary of State Peter Kyle visits Anthropic in San Francisco, Oct. 25, 2024. Picture by Alecsandra Dragoi / UK Department for Science, Innovation and Technology, via Wikimedia Commons (original on Flickr), CC BY 2.0. Cropped to 16:9.