Benchmark profile
WeirdML v2 (WeirdML)
A machine-learning engineering benchmark that tests whether LLMs can train models on novel datasets, write PyTorch code, and improve through iterative feedback.
How BenchLM shows WeirdML
BenchLM mirrors the top public WeirdML v2 rows from the official CSV. WeirdML tests whether models can do machine-learning engineering on 17 novel datasets, reporting average accuracy across tasks.
WeirdML is display only on BenchLM because the task is a specialized ML-agent evaluation with iterative feedback and execution constraints. BenchLM stores it as context rather than a weighted model-only benchmark.
Average accuracy on WeirdML — WeirdML v2
BenchLM mirrors the published average accuracy view for WeirdML. gpt-5.5 (xhigh) leads the public snapshot at 84.91% , followed by gpt-5.5 (high) (83.90%) and claude-opus-4.8 (xhigh) (82.89%). BenchLM does not use these results to rank models overall.
gpt-5.5 (xhigh)
OpenAI
gpt-5.5:xhigh
gpt-5.5 (high)
OpenAI
gpt-5.5
claude-opus-4.8 (xhigh)
Anthropic
claude-opus-4.8:xhigh
Average accuracy table (25 models)
ScoreThe published WeirdML snapshot places gpt-5.5 (xhigh) first at 84.91%. The third row is 2.02 points behind. The broader top-10 range is 9.46 points, so many of the published results sit in a relatively narrow band.
25 models have been evaluated on WeirdML. The benchmark falls in the External benchmark mirrors category. We keep external benchmark mirrors separate from the weighted global scoring system, so these results remain source-specific evidence. WeirdML is currently displayed for reference but excluded from the scoring formula, so it does not directly affect overall rankings.
About WeirdML
Year
2026
Tasks
17 novel ML engineering tasks
Format
Average accuracy across tasks
Difficulty
Novel dataset modeling and iterative debugging
WeirdML v2 evaluates models on 17 unusual ML tasks and reports average accuracy across tasks from the official CSV. BenchLM mirrors the top official rows as display-only ML-agent evidence.
BenchLM freshness & provenance
Version
WeirdML 2026
Refresh cadence
Quarterly
Staleness state
Current
Question availability
Public benchmark set
BenchLM uses freshness metadata to decide whether a benchmark should still be treated as a strong differentiator, a benchmark to watch, or a display-only reference. For the full scoring policy, see the BenchLM methodology page.
FAQ
What does WeirdML measure?
A machine-learning engineering benchmark that tests whether LLMs can train models on novel datasets, write PyTorch code, and improve through iterative feedback.
Which model leads the published WeirdML snapshot?
gpt-5.5 (xhigh) currently leads the published WeirdML snapshot with 84.91% average accuracy. BenchLM shows this benchmark for display only and does not use it in overall rankings.
How many models are evaluated on WeirdML?
25 AI models are included in BenchLM's mirrored WeirdML snapshot, based on the public leaderboard captured on WeirdML v2.
Choose a model with this week’s evidence
Join 2,000+ readers for ranking moves, pricing changes, and the claims that still need proof.
One email each week. Unsubscribe anytime.