Skip to main content
BenchLM
Advertise

Benchmark profile

NL2Repo

A repository-understanding benchmark that measures whether models can map natural-language requests onto the right code locations and system changes.

Data verified

Benchmark score on NL2Repo — July 23, 2026

BenchLM mirrors the published score view for NL2Repo. GLM-5.2 leads the public snapshot at 48.9% , followed by Ornith-1.0-397B (48.2%) and Qwen3.7 Max (47.2%). BenchLM does not use these results to rank models overall.

1Open

GLM-5.2

Z.AI

glm-5-2

48.9%
Overall 63.96Context 1M
2Open

Ornith-1.0-397B

DeepReinforce AI

ornith-1-0-397b

48.2%
Overall —Context 256K
3Closed

Qwen3.7 Max

Alibaba

qwen3-7-max

47.2%
Overall 72.84Context 1M
13 modelsCodingCurrentDisplay onlyUpdated July 23, 2026

Benchmark score table (13 models)

Score
1
GLM-5.2Z.AI · Open weight
48.9%
2
Ornith-1.0-397BDeepReinforce AI · Open weight
48.2%
3
Qwen3.7 MaxAlibaba · Closed
47.2%
4
Claude Opus 4.5Anthropic · Closed
43.2%
5
Qwen 3.6 Max (preview)Alibaba · Closed
42.9%
6
GLM-5.1Z.AI · Open weight
42.7%
7
MiniMax M3MiniMax · Open weight
42.1%
8
Qwen3.7 PlusAlibaba · Closed
41.1%
9
MiniMax M2.7MiniMax · Open weight
39.8%
10
Qwen3.6-27BAlibaba · Open weight
36.2%
11
Ornith-1.0-35BDeepReinforce AI · Open weight
34.6%
12
Qwen3.6-35B-A3BAlibaba · Open weight
29.4%
13
Ornith-1.0-9BDeepReinforce AI · Open weight
27.2%

The published NL2Repo snapshot places GLM-5.2 first at 48.9%. The third row is 1.7 points behind. The broader top-10 range is 12.7 points, so the table still separates the published systems.

13 models have been evaluated on NL2Repo. The benchmark falls in the Coding category. This category carries a 20% weight in BenchLM.ai's overall scoring system. NL2Repo is currently displayed for reference but excluded from the scoring formula, so it does not directly affect overall rankings.

About NL2Repo

Year

2026

Tasks

Natural language to repository tasks

Format

Repository understanding benchmark

Difficulty

System-level software comprehension

MiniMax cites NL2Repo as a system-level engineering benchmark that rewards deep understanding of complex repositories and their operational structure.

MiniMax M2.7: Early Echoes of Self-Evolution

BenchLM freshness & provenance

Version

NL2Repo 2026

Refresh cadence

Quarterly

Staleness state

Current

Question availability

Public benchmark set

CurrentDisplay only

BenchLM uses freshness metadata to decide whether a benchmark should still be treated as a strong differentiator, a benchmark to watch, or a display-only reference. For the full scoring policy, see the BenchLM methodology page.

FAQ

What does NL2Repo measure?

A repository-understanding benchmark that measures whether models can map natural-language requests onto the right code locations and system changes.

Which model scores highest on NL2Repo?

GLM-5.2 by Z.AI currently leads with a score of 48.9% on NL2Repo.

How many models are evaluated on NL2Repo?

13 AI models have been evaluated on NL2Repo on BenchLM.

Compare Top Models on NL2Repo

GLM-5.2 vs Ornith-1.0-397BOrnith-1.0-397B vs Qwen3.7 MaxQwen3.7 Max vs Claude Opus 4.5Claude Opus 4.5 vs Qwen 3.6 Max (preview)
Last updated: July 23, 2026 · BenchLM version NL2Repo 2026

Choose a model with this week’s evidence

Join 2,000+ readers for ranking moves, pricing changes, and the claims that still need proof.

One email each week. Unsubscribe anytime.