Automated Weak-to-Strong Researcher

Anthropic · Original post

A great piece — I read it twice in a row. Anthropic's research taste is as good as ever.

Preface

Where AI shines: it can systematically carry out simple experiments / research problems. "Simple" here means that humans can define the metric and quantify the outcome of the experiment.

The key bottleneck for alignment research is moving from proposing and executing ideas to designing evals: we should find the right metrics (data, models) that AARs can reliably hill-climb without overfitting.

Setting

Let AI do research on its own.

Goal: train a weak model (Qwen1.5 0.5B) on ground-truth data, then train a strong model (Qwen3 4B-base) on the weak model's labels — a classic weak-to-strong supervision setup. The core question: how should we use the strong model, the weak model, and the weak model's labels to end up with a strong model that is actually strong?

Domains: chat, code, math.

Setup: 9 × Opus 4.6. Each agent gets a different initial prompt and can propose ideas and run experiments on its own; a shared discussion space lets them exchange ideas and code.

Main findings

It works very well: weak labels recover almost all of the performance of ground-truth labels (93%).

How the agents behave

• Steering agents toward different initial directions helps.

  

  

• The AI runs ablations and strips away unnecessary complexity.

  

• The strategies the AI finds generalize to some extent.

  

Case studies: strategies found by the AI

• Method 1: Contrastive Consistency Search (exploiting contrastive structure in the strong model's internal activations) + an evolutionary strategy (random parameter search). PGR = 0.93
• Method 2: EM posterior. PGR = 0.78
• Method 3: label filtering in the strong model's embedding space. PGR = 0.75
• Method 4: MDL curriculum learning, based on how confident the weak model is in each label. PGR = 0.68
• Method 5: Epiplexity — directly measure how much the strong model can learn from the weak labels and keep what works. PGR = 0.62

Reward hacking

• Hack 1: exploiting quirks of the data
  • Math: the AI noticed that "the most frequent answer is usually correct" — no need to judge correctness at all, just count.
  • Code: the authors deliberately sampled solutions from several different LLMs to avoid stylistic bias. The AI still learned to cluster solutions by which model wrote them and found that some models had higher accuracy — cheating by "telling who wrote the code."
• Hack 2: brute force
  • Training is stochastic (different random seeds give different results). The AI tried many seeds → picked the best score → trained the next generation from the best model → picked the best score again → and so on.
• Hack 3: probing for the answers
  • Calling the API for "ablations" to leak the answers.
• Hack 4: running the code to get the answers
  • On the code-verification task, the AI simply wrote unit tests and ran the code — if it passes, it's correct. This bypasses both the weak and the strong model and gets the answer directly.

Tips for working with AI researchers

• Don't hard-code a workflow for the AI (e.g. propose ideas → generate plans → write code → run smoke tests → run full training → analyze results → repeat); give it autonomy instead.

  The core issue is that rigid steps prevent AARs from adapting its process to the idea at hand.

• Ambiguous research directions beat specific research ideas: overly specific guidance limits how widely the model explores.

• LM-driven iteration (the model decides what to explore next) > heuristic evolutionary search (score-based, trading off the current best strategies against less-explored ones).

  LM based iteration naturally benefits from general capability improvements in planning and experiment analysis, whereas heuristic search does not.

• Share knowledge through local files rather than a search API: share the codebase directly, much like GitHub.

  Local access, by contrast, lets AAR browse and discover relevant findings it would not have known to search for, an advantage analogous to why researchers reading broadly often find connections that targeted literature search miss.

Future work

• Dataset generalization
• Model scale generalization

• Deploying AARs in production is hard — real-world settings are not "clean."

  production settings involve more nuanced fuzzy rewards, such as whether the discovered idea is hardware-efficient and compatible with existing infrastructure.

• Faster, more scalable experiments run by agents

• Richer research logs: for model training, accumulating insights matters a lot.
• Research taste: may be becoming less important — simply scaling up diversity can also work.
• AI interpretability — legibility training.