Benchmark profile
Vals MedQA (MedQA)
Evaluating language model bias in medical questions.
Data verifiedHow BenchLM shows MedQA
BenchLM mirrors the public Vals AI MedQA leaderboard captured from https://www.vals.ai/benchmarks/medqa and updated by Vals on April 16, 2026. The snapshot preserves overall scores, uncertainty, latency, cost-per-test metadata, and task-level scores where Vals publishes them.
MedQA is display only on BenchLM. Vals proprietary or Vals-hosted aggregate views are useful context, but BenchLM does not use them as weighted ranking inputs or as a replacement for benchmark-native source records.
MedQA score on MedQA — April 16, 2026
BenchLM mirrors the published medqa score view for MedQA. O1 leads the public snapshot at 96.52% , followed by GPT-5.1 (96.38%) and Gemini 3.1 Pro Preview (96.37%). BenchLM does not use these results to rank models overall.
O1
OpenAI
openai/o1-2024-12-17
GPT-5.1
OpenAI
openai/gpt-5.1-2025-11-13
Gemini 3.1 Pro Preview
google/gemini-3.1-pro-preview
MedQA score table (95 models)
ScoreThe published MedQA snapshot places O1 first at 96.52%. The third row is 0.15 points behind. The broader top-10 range is 0.64 points, so many of the published results sit in a relatively narrow band.
95 models have been evaluated on MedQA. The benchmark falls in the External benchmark mirrors category. We keep external benchmark mirrors separate from the weighted global scoring system, so these results remain source-specific evidence. MedQA is currently displayed for reference but excluded from the scoring formula, so it does not directly affect overall rankings.
About MedQA
Year
2026
Tasks
Medical question answering
Format
Accuracy score
Difficulty
Medical knowledge and bias evaluation
BenchLM mirrors the public Vals AI MedQA leaderboard as display-only external evidence. The captured snapshot preserves overall scores, task-level scores where Vals publishes them, uncertainty, latency, and cost-per-test metadata. It is excluded from BenchLM weighted rankings.
BenchLM freshness & provenance
Version
MedQA 2026
Refresh cadence
Quarterly
Staleness state
Current
Question availability
Public benchmark set
BenchLM uses freshness metadata to decide whether a benchmark should still be treated as a strong differentiator, a benchmark to watch, or a display-only reference. For the full scoring policy, see the BenchLM methodology page.
FAQ
What does MedQA measure?
Evaluating language model bias in medical questions.
Which model leads the published MedQA snapshot?
O1 currently leads the published MedQA snapshot with 96.52% medqa score. BenchLM shows this benchmark for display only and does not use it in overall rankings.
How many models are evaluated on MedQA?
95 AI models are included in BenchLM's mirrored MedQA snapshot, based on the public leaderboard captured on April 16, 2026.
Choose a model with this week’s evidence
Join 2,000+ readers for ranking moves, pricing changes, and the claims that still need proof.
One email each week. Unsubscribe anytime.