AI-CAD, graded
against parts
that exist.
Most AI-CAD benchmarks grade a model against other CAD. We grade it against parts that were designed, built and installed by people, then check the result on the physical part. Each run is scored by a panel of 25 independent graders, five personas on five different AI models, and by the designer. Every score and every failure gets published.
- What
- AI-CAD benchmark + lab notebook
- Since
- October 2026
- Graded runs
- 9 · on 6 physical parts
- Builder so far
- Claude · Autodesk Fusion, OpenCascade
- Scored by
- 5 personas × 5 models + the designer
- Rubric
- v2 · anchored levels, cap rules
- License
- CC BY-NC 4.0
- Status
- v0.1 preview
Reference part · DRS endplate
Serial 001 · Summer 2025on parts that were built
5 personas × 5 AI models
the first grade missed
tagged, dated, public
What it is.
Project Ground Truth is the Tracehaus lab for AI in CAD. An AI agent gets the job a designer would get: rebuild this part as editable CAD, or design one that fits here. Then we grade what it made against the part that already exists, the one somebody drew, built and bolted on.
Synthetic benchmarks check whether CAD code compiles, or how close a model lands to a scraped file. They can't tell you whether the hole is on the right side, whether the welder could reach the joint, or whether the part fits the car. Those are the mistakes that cost money, and they only show up against the physical part.
It isn't text-to-CAD. We use AI the way a studio would: as an assistant that cleans up scans, rebuilds geometry and drafts designs, with a person checking the work. The rubric scores and the failure logs are the real output. They show where AI-CAD breaks in ways a surface-deviation number can't.
Why a scanning studio. We measure real parts for a living. Applying a rubric to a physical object means knowing the normal deviation of how things get made, even within one batch, and where that overlaps with the AI's error.
Why cars. Everyone assumes a car is symmetric. It isn't, and manufacturing adds its own variation on top. That makes car parts a hard, honest test bed: the AI can't lean on a clean mirror plane.
What we've found.
Updated Nov 2026 · 9 runs- 0.114mm
- P95 surface deviation from the designer's file. By that number, a near-perfect rebuild.
- 2–3mm
- What nearest-surface deviation said about the hole, because the flange is only 5 mm thick.
- 19.2mm
- How far off the hole really was: mirrored across the centre line. A hole-centre check caught it.
- 25/25
- Panel graders who flagged it, once hole centres sat side by side in the evidence.
Accurate isn't the same as right.
An AI rebuild of our DRS wing endplate matched the original to 0.114 mm at P95, and its M6 mounting hole was mirrored, 19.2 mm off. Nearest-surface deviation reported it as a 2–3 mm error, because the flange is only 5 mm thick. A hole-centre check caught it, and calipers on the printed part confirmed it: 131 mm pivot to M6 on the original, 119.4 mm on the rebuild.
Run R007 · re-parametrize · Autodesk FusionFive model families caught what one grade missed.
We regraded that run with the full panel: five personas on five AI models, none of which saw the build or the answer. All 25 graders found the mirrored hole, each citing the same 19.2 mm offset. What made the difference was the evidence, not the graders: once the hole positions from the original and the rebuild sat side by side in a table, nobody could miss it. So that comparison now runs automatically in every build's own self-check, and the panel confirms it.
Panel pilot on R007 · Claude, GPT, Gemini, Kimi, DeepSeekThe fix held on a blind rerun.
We gave the same task to a fresh session that had never seen the first attempt or its mistake. Its new interface check failed midway on a slot end, which exposed a sketch bug, and it fixed the bug before writing its report. All four interfaces landed within 0.001 mm of the designer's original. On the panel, none of the 25 graders raised an interface problem, and dimensional accuracy moved from 2 to 3.
Run R008 · re-parametrize rerun · Autodesk FusionMore accurate every run. Not more editable.
Three runs of the same rebuild task, regraded the same way against the designer's native file: P95 surface deviation fell from 0.112 mm to 0.033 mm to 0.017 mm, and the last two got all four interfaces right. Editability didn't move. Half of the third run's combined edits broke the model, and the designer scored all three a 2. The agents keep getting closer to the shape and no closer to how a designer builds it.
Runs R007, R008, R009 · like-for-like regrade, Nov 15, 2026Perfect interfaces, wrong part.
Given only a brief and the interface dimensions, the AI hit every mounting point within 0.006 mm and designed a flat plate. The real part is a curled endplate that rises 200 mm off the flange and carries three ducts. Design intent scored 2 out of 5.
Run R006 · blind design-for-fit · Autodesk FusionThe AI designed for the scan, not for the weld.
Asked to design the cage-mount boxes for a Subaru strut tower, the AI used 3.3× the steel of the human design and planned stitch welds over gaps. The real boxes were drawn up on the spot with the welder: full-perimeter welds, built to spread the cage tube's loads into the tower. The AI never considered weld access.
Run R005 · blind design-for-fit · collaboration with Heliarc WorksCAD kernels fail silently.
A thicken returned a "valid" solid with a volume of −10,327 cm³. A sweep returned zero volume and no error. A boolean that met a surface exactly at its crest did nothing at all. None of them raised an exception. Volume checks and section renders caught every one.
Runs R002, R004 · OpenCascadeRegistration finds convincing wrong answers.
Fitting a hood back onto its engine bay, the optimizer found a clean-looking minimum 80 mm off; the latch position gave it away. On another car, ICP made the left/right symmetry check worse by sliding 48 mm. The lesson: check every registration against an independent measure.
Runs R003, R005Renders catch what deviation can't.
Under gloss lighting, a fitted hood surface rippled. The waviness sat inside a 1.26 mm RMS that looked fine on paper, and a mold plug would have carried it into every part. Rendering is now a scored stage, not decoration.
Run R001 · presentation fidelityThe designer saw what 25 graders didn't.
The designer scored design intent 1 out of 5 on all three endplate rebuilds. The panel gave 3 to 4, and the agent gave itself 3. His reason was the same every time: the agent rebuilt the curved wall as a stack of cross-sections, which isn't how the part was modelled. Manufacturability went the other way: the designer gave 5, the AI graders 3 to 4. Editability is the one score everyone agreed on, a 2.
Runs R007, R008, R009 · designer scores, Nov 15, 2026AI reports still claim more than the files show.
After scoring, each grader reads the agent's own report with its scores locked and lists every claim the files don't back. Every grader found some, 2 to 11 each. On the rerun, the report claimed a fix it hadn't made and a print setup that can't work: a part with a 350 mm straight edge, placed "diagonally" on a 350 mm build plate.
Runs R007, R008 · panel step 2
How we work.
Every run, same loopCapture a real part
Our own designs, parts we bought, or client parts run privately, each designed by a person and then built. We collect the scan, the designer's native CAD or drawings, and caliper readings, each tagged with its source.
Write the brief
What a designer would be handed, nothing more. On blind tasks and reruns the agent works in a fresh session that has never seen the answer, and every file it opens is logged.
Let the agent build
The agent drives real CAD (Autodesk Fusion, or OpenCascade in Python) and delivers an editable feature tree, not a mesh. Every decision and every build script is logged as it goes, failed ones included.
Measure it
Deterministic checks, run once: deviation both ways and per region, hole and slot centres against the designer's file, an editability sweep, volume checks, section renders.
Panel grades it
Five personas on five AI models score the judgment calls from an evidence packet, then check the agent's report with their scores locked. The designer settles every split.
Check it on the part
Print it, measure it, fit it. Log every failure, and rerun the same task when a newer model comes out.
| Task | The agent gets | Graded against | Status |
|---|---|---|---|
| Re-parametrizedead geometry → editable feature tree | A dumb STEP import with no history | The designer's native CAD and the printed part | In the set |
| Blind design-for-fitdesign a part without seeing the answer | A brief, plus interface dimensions or a sectioned scan of the space | The designer's CAD or fabrication drawings | In the set |
| Scan-to-CADraw 3D scan → parametric CAD | The scan and a short intake note | Scan deviation, measured critical dimensions, the physical part | Client runs · public set next |
How we score.
Rubric v2 · 1 to 5 per criterion| Criterion | The question |
|---|---|
| Dimensional accuracy | How far off is it, against the reference and against measured critical dimensions, relative to the scan's own noise and the part's own asymmetry? |
| Design intent | Does the feature tree reflect how the part was really designed? |
| Editability | Push one parameter (a wall, a hole spacing) to the edge of its plausible range. Is the geometry still valid? |
| Manufacturability | Could it be made by the process the real part was made with? |
| Assembly fit | Does the fabricated part fit its mate? Scored only after it's built, by a person, never from CAD. |
| Presentation fidelity | Do the renders match the as-built part without overstating it? |
Every level has a written anchor. Cap rules stop good statistics from hiding a bad part: a wrong interface caps dimensional accuracy at 2, however close the surface is. Process scores (honesty, efficiency) are recorded but kept out of the average.
Measured before judged. Deviation, interface checks, volume and the editability sweep are deterministic and run once. Only the judgment calls go to graders: intent, manufacturability, and what the numbers mean.
Every number has a source.
One scoring round is one roll of the dice, and repeating the same grader repeats its blind spots. So the judgment calls go to a panel. Five personas, each run on five different AI models: 25 graders per run. Every grader is a fresh session that never saw the build, the agent's self-score, the designer's score or the other graders.
Different personas look for different failures. Different models bring different habits. Neither removes bias on its own; together they make it measurable.
| Persona | Plays | Goes deep on |
|---|---|---|
| Fabricator | The welder, printer or installer making it tomorrow | Process, joining, load path |
| Designer | An industrial designer of this kind of part | Concept and design intent |
| Inspector | Metrology and QC | Interfaces, and what it was graded against |
| Successor | The engineer who inherits the file | Editability, parameter health |
| Skeptic | Red team | Claims vs files, silent failures |
25 of 25. Each grader was a fresh session with no view of the build, the answer or the other graders, and each cited the same 19.2 mm offset.
The build's own self-check, which had no hole-centre table, scored the same rebuild 4 for accuracy.
- Two steps per grader. Score from the evidence packet first. Then read the agent's report with the scores locked, and list every claim the files don't back.
- Evidence or it doesn't count. A score has to cite a file and a value, or it's left out of the median.
- Blind A/B on design tasks. Graders see the human design and the AI's as Design A and Design B, then guess which is which.
- Median, spread and flags. We publish the median and range per criterion. Any split of two points or more goes to the designer to settle, and the reason is logged.
- Calibrated against the designer. Each persona's and each model's offset from the designer is tracked and reported, not silently corrected.
- Leak-checked. Before a panel runs, the packet and prompts are checked for hints. The pilot found two and removed them.
| R007 · first rebuild | R008 · blind rerun | R009 · second rerun | |||||||
|---|---|---|---|---|---|---|---|---|---|
| Criterion | Self | Panel | Designer | Self | Panel | Designer | Self | Panel | Designer |
| Dimensional accuracy | 4 | 2 2–2 | 3 | 4 | 3 3–3 | 3 | 4 | 3 3–3 | 5 |
| Design intent | 3 | 3 3–4 | 1 | 3 | 4 3–4 | 1 | 3 | 4 3–4 | 1 |
| Editability | 3 | 3 3–3 | 2 | 3 | 3 1–3 · flagged | 2 | 2 | 2 1–3 · flagged | 2 |
| Manufacturability | 4 | 3 3–4 | 5 | 3 | 4 3–5 · flagged | 5 | 3 | 4 3–5 · flagged | 5 |
Panel = median of 25 graders (24 on R009), with the range underneath. Designer scores were given on Nov 15, 2026, with the mirrored hole known; R007's accuracy came down from 4 to 3 for it. Assembly fit: every grader abstained, as the rubric requires, because nothing has been fabricated.
The panel held R008's and R009's accuracy at 3 with a cap for work graded against an import rather than the designer's native file. Those runs were given the import on purpose, so the designer has waived the cap for this task; a panel rerun on accuracy is next. The editability split traces to rubric wording, and the anchor is being rewritten.
The runs.
Average score · self / panel / designer| Run | Part | Task | Agent · CAD | Self | Panel | Designer | Data |
|---|---|---|---|---|---|---|---|
| R009 | DRS wing endplateTracehaus design · printed PET-CF | Re-parametrizesecond blind rerun | Claude · FusionFable 5.2 | 3.00 | 3.25 | 3.25 | Public |
| R008 | DRS wing endplateTracehaus design · printed PET-CF | Re-parametrizeblind rerun of R007 | Claude · Fusion | 3.25 | 3.50 | 2.75 | Public |
| R007 | DRS wing endplateTracehaus design · printed PET-CF | Re-parametrize | Claude · Fusion | 3.50 | 2.75 | 2.75 | Public |
| R006 | DRS wing endplateTracehaus design · printed PET-CF | Blind design-for-fit | Claude · Fusion | 3.50 | – | 4.25 | Public |
| R005 | Cage-mount boxeswith Heliarc Works · welded steel | Blind design-for-fit | Claude · OpenCascade | 3.00 | – | 2.25 | Public |
| R004 | Off-road front bumperclient part | Scan-to-CAD | Claude · OpenCascade | 3.00 | – | 3.00 | Private |
| R003 | Engine bayclient part · case study | Scan-to-reference | Claude · Python | 3.67 | – | – | Private |
| R002 | Vented fiberglass hoodclient part · case study | Scan-to-CAD | Claude · OpenCascade | 3.40 | – | – | Private |
| R001 | Fiberglass hoodclient part · case study | Scan-to-CAD | Claude · OpenCascade | 3.40 | – | – | Private |
Self and designer: averaged over the criteria each run was scored on. Panel: the per-criterion medians for accuracy, intent, editability and manufacturability, averaged. "–" means not scored yet. Panel regrades of R005 and R006 are next.
Private runs: client geometry is never published. Their failure tags are, with the part left blank, so the failure statistics cover every run. Assembly fit isn't scored yet; none of the AI outputs has been fabricated.
Where AI-CAD breaks.
128 failures · 9 runsEvery run keeps a log of everything that went wrong: what broke, which iteration, whether it was fixed, and what caught it. A volume check, a section render, an editability sweep, a caliper, a grader. The log is the most useful thing we make. It's what a model would need to learn from, and what anyone grading AI-CAD should test for.
Each entry carries a tag, a family, a status (fixed, open, caught, disputed) and what caught it. The full log ships with the benchmark (07).
Geometry
Missed features, clipped corners, draft ignored on a molded part.
draft-ignoredKernel
Operations that return a valid-looking solid that's wrong.
thicken-invertedProcess
Parameters that only document a value, planes hard-coded instead of referenced.
params-not-drivingRegistration
Alignments that settle into a convincing false minimum.
off-data-penalty-biasIntent
Right numbers, wrong idea of what the part is for.
intent-missedFabrication
Designs that ignore how the part gets welded, printed or molded.
weld-strategy-missedRender
Surfaces that pass the numbers and fail under light.
surface-not-fairTooling
Analysis tools that fail without saying so.
section-tool-silentThe data.
CC BY-NC 4.0DRS endplate
A real, shipped part three ways: the designer's original STEP, a SimReady OpenUSD asset, and the ground truth SimReady has no field for: a measured mass, where every number came from, and how the AI rebuild scored.
TraceHaus/drs-endplate ↗Ground Truth bench
Three tasks on two built parts, with briefs, references and five graded runs: the AI's CAD, deviation reports, interface checks, self, panel and designer scores, the failure logs, plus machine-readable tables of runs and failures.
TraceHaus/ground-truth-bench ↗Run your model
Building an AI-CAD model? We'll run it on the same tasks, through the same checks and the same panel, and send you every score, every grader's reasoning and every failure.
Email the lab →- Public: Tracehaus's own designs, and parts we bought or have rights to. The cage-mount boxes are included with Heliarc Works' OK.
- Counts only: client parts. Their geometry stays private; their failure tags are in the statistics with the part left blank.
- Never: branded aftermarket parts.
Limits, and what's next.
- Small. Eight runs is a set of case studies, not a statistical benchmark. It grows as parts get built.
- One builder so far. Every build is Claude. The panel spans five model families; the builds don't yet.
- One human reference. The designer is still the only human grader, scoring their own parts. A second reviewer is planned.
- Models vary more than personas. On the panels so far, persona medians came out nearly identical; the spread came from the models. More runs will show whether the personas earn their place.
- Rubric wording drives splits. The widest disagreement so far came from two models reading the same anchor differently, not from the part.
- Text-only graders abstain on presentation, so that criterion rests on four models.
- Briefs leak. One brief was written by someone who'd seen the answer, and it showed in the wording. Future briefs get reviewed by someone who hasn't.
- Fit is untested until the AI outputs are fabricated.
- Open AI-CAD models on the same tasks: CAD-Coder, CAD-Recode and others.
- Panel regrades of R005 and R006, the two blind design tasks, with the human and AI designs shown as A and B.
- Three build attempts on every blind design task, each in a fresh session, all published.
- A panel rerun on accuracy for R008 and R009, now that the import cap is waived for this task.
- Simulation in the loop. Our CFD setup is validated on the NACA 0012 reference case: lift within about 1%, drag still 3–11% low and not yet grid-converged, so drag isn't quoted yet. FEA follows.
- Fabricate the AI's designs and score assembly fit on the real mate.
- Rerun every task as new models come out, against the same rubric.
Grade it against reality.
Building an AI-CAD model and want it run on parts that exist? Have a part you'd like in the set? Write to the lab. For commercial use of the data, ask us first.
Client and vehicle details withheld on private runs. Data released under CC BY-NC 4.0, attribution to Tracehaus LLC.