Model selection
Head-to-head comparisons across providers and sizes on your workload, weighing quality, latency and cost together.
Practice 03 — Model testing & evals
Benchmarks don't run your business. Your tasks do.
We build evaluation harnesses on your own data and success criteria, so choosing a model, changing a prompt or upgrading a version is a measured decision — not a vibe check.
deepswe · 113 tasksoct 2026
| model | harness | pass@1 | $/attempt |
|---|---|---|---|
| GPT-5.6 Luna | Copilot | 74.1% | $0.37 |
| Gemini 3.8 Flash | Copilot | 74.1% | $2.79 |
| Claude Opus 5 | Copilot | 71.2% | $5.66 |
| GPT-6 Astra | Copilot | 71.0% | $8.43 |
| Claude Opus 5.5 | Copilot | 70.8% | $11.98 |
| Claude Opus 5 | Claude Code | 64.4% | $13.49 |
| Auto | Copilot | 37.0% | $1.62 |
System, not model
Evaluate the system, not just the model.
A model's results depend on everything around it. The same model scored 71% in one agent harness and 64% in another, at less than half the cost. We test the pieces together and find the configuration that meets your quality, reliability, latency and cost targets on your own workload.
- model
- prompts
- reasoning effort
- tools
- agent harness
- environment
- verification
What we test
Golden datasets
Curated, versioned test sets labelled with your domain experts — the ground truth every change is measured against.
Graders you can trust
Code checks where possible, model-graded rubrics where needed — each calibrated against human judgment.
Agent & tool-use evals
Multi-step trajectory testing: did the agent call the right tools, recover from errors and stop when it should?
Safety & red-teaming
Adversarial suites for prompt injection, data leakage and policy violations before your users find them.
Production monitoring
Online evals and sampling that catch drift when inputs, models or upstream data change underneath you.
From our own bench
16 agent configurations. 7,571 attempts at real engineering tasks.
We ran today's frontier models through DeepSWE — 113 real-world software tasks, each attempted four or five times — and scored every attempt against its tests. The leaderboard wasn't the interesting part.
Same model, different harness, different result.
Claude Opus 5 passed 71.2% of attempts inside GitHub Copilot and 64.4% inside Claude Code — and cost 58% less per attempt ($5.66 vs $13.49).
The cheapest model tied for first.
GPT-5.6 Luna passed 74.1% at $0.37 per attempt — about 1/32 the cost of Claude Opus 5.5, which scored 70.8%.
One run isn't a verdict.
Luna scored 74.1% on one Copilot CLI release and 69.9% on the next, same model. A swing that size is within run-to-run noise here — which is why upgrade decisions need repeated trials, not a single score.
Results that hold up.
Where our configurations overlap Meta's published DeepSWE evaluation, our pass rates land within two points of theirs.
Release gates
Every AI update is a potential regression.
Models, prompts, tools and agent frameworks change constantly — often underneath you, when a provider ships a new version. We wire your eval suite into the release process, so every change is measured against a baseline from your own workload before it reaches users.
Repeated trials tell a real regression from run-to-run noise, so a hold means something and so does an approval.
- 01trigger
A change is proposed
model, prompt, tool, harness, routing
- 02CI
Run the eval suite
your golden set, repeated trials
- 03report
Compare to baseline
quality, reliability, cost, latency
- 04CI
Check for regressions
does anything that passed now fail?
- 05human
Approve or hold
against thresholds agreed up front
When teams call us
You can't answer “is it better?” with confidence.
- A new model launched and you don't know if switching is safe.
- Prompt tweaks fix one case and quietly break three others.
- Leadership wants proof before a wider rollout.
- Your AI spend is climbing and nobody knows what a cheaper model would cost you in quality.
- A regulator, auditor or customer is asking how you test.
Typical engagement
We build the harness alongside your team and wire it into CI, so evals keep running long after we've left.
What you get
- 01
An eval harness
Runs your tasks against any model, prompt or agent configuration, with repeated trials.
- 02
Golden sets
Versioned test cases labelled with your domain experts.
- 03
Release gates in CI
Every AI change measured against your baseline before it reaches users.
- 04
A model recommendation
The configuration that meets your quality, cost and latency targets — with the evidence.
Done when
A model or prompt change gets an approve-or-hold from your own pipeline, without us in the loop.