Record / Lab · Project Ground Truth

AI-CAD, graded
against parts
that exist.

Most AI-CAD benchmarks grade a model against other CAD. We grade it against parts that were designed, built and installed by people, then check the result on the physical part. Each run is scored by a panel of 25 independent graders, five personas on five different AI models, and by the designer. Every score and every failure gets published.

What
AI-CAD benchmark + lab notebook
Since
October 2026
Graded runs
9 · on 6 physical parts
Builder so far
Claude · Autodesk Fusion, OpenCascade
Scored by
5 personas × 5 models + the designer
Rubric
v2 · anchored levels, cap rules
License
CC BY-NC 4.0
Status
v0.1 preview
Render of the DRS wing endplate: a curled carbon-fibre-filled part with three ducts, two mounting bosses and an arc slotReference part · DRS endplate
Close-up of the endplate edge, stamped Designed by Tracehaus in California USA, Summer 2025, 001Serial 001 · Summer 2025
The part behind runs R006 to R009: a rear-wing DRS endplate, designed by Tracehaus, printed in PET-CF and shipped. Every AI build is graded against this file.Renders from the designer's CAD
9Graded runs
on parts that were built
25Graders per run
5 personas × 5 AI models
25/25Graders caught the hole
the first grade missed
128Failures logged
tagged, dated, public
01

What it is.

Project Ground Truth is the Tracehaus lab for AI in CAD. An AI agent gets the job a designer would get: rebuild this part as editable CAD, or design one that fits here. Then we grade what it made against the part that already exists, the one somebody drew, built and bolted on.

Synthetic benchmarks check whether CAD code compiles, or how close a model lands to a scraped file. They can't tell you whether the hole is on the right side, whether the welder could reach the joint, or whether the part fits the car. Those are the mistakes that cost money, and they only show up against the physical part.

It isn't text-to-CAD. We use AI the way a studio would: as an assistant that cleans up scans, rebuilds geometry and drafts designs, with a person checking the work. The rubric scores and the failure logs are the real output. They show where AI-CAD breaks in ways a surface-deviation number can't.

Why a scanning studio. We measure real parts for a living. Applying a rubric to a physical object means knowing the normal deviation of how things get made, even within one batch, and where that overlaps with the AI's error.

Why cars. Everyone assumes a car is symmetric. It isn't, and manufacturing adds its own variation on top. That makes car parts a hard, honest test bed: the AI can't lean on a clean mirror plane.

02

What we've found.

Updated Nov 2026 · 9 runs
The mirrored hole, run R007 Section through the flange of the DRS endplate, drawn from the designer's STEP file. The designer's M6 hole sits 131.0 mm from the flap pivot. The AI rebuild put it 19.2 mm away, mirrored across the part's centre line, 119.4 mm from the pivot. X 0 DESIGNER AI REBUILD 19.2 131.0 119.4 FLAP PIVOT Ø8.2 ARC SLOT · 11.99° DRS TRAVEL SECTION Z −5.5 · MM
Run R007Section through the flange, drawn from the designer's STEP. The rebuild matched the surface; its M6 hole landed on the other side of X 0. Pivot-to-hole distances measured with calipers on both printed parts.
One rebuild, four numbers
0.114mm
P95 surface deviation from the designer's file. By that number, a near-perfect rebuild.
2–3mm
What nearest-surface deviation said about the hole, because the flange is only 5 mm thick.
19.2mm
How far off the hole really was: mirrored across the centre line. A hole-centre check caught it.
25/25
Panel graders who flagged it, once hole centres sat side by side in the evidence.
  1. Accurate isn't the same as right.

    An AI rebuild of our DRS wing endplate matched the original to 0.114 mm at P95, and its M6 mounting hole was mirrored, 19.2 mm off. Nearest-surface deviation reported it as a 2–3 mm error, because the flange is only 5 mm thick. A hole-centre check caught it, and calipers on the printed part confirmed it: 131 mm pivot to M6 on the original, 119.4 mm on the rebuild.

    Run R007 · re-parametrize · Autodesk Fusion
  2. Five model families caught what one grade missed.

    We regraded that run with the full panel: five personas on five AI models, none of which saw the build or the answer. All 25 graders found the mirrored hole, each citing the same 19.2 mm offset. What made the difference was the evidence, not the graders: once the hole positions from the original and the rebuild sat side by side in a table, nobody could miss it. So that comparison now runs automatically in every build's own self-check, and the panel confirms it.

    Panel pilot on R007 · Claude, GPT, Gemini, Kimi, DeepSeek
  3. The fix held on a blind rerun.

    We gave the same task to a fresh session that had never seen the first attempt or its mistake. Its new interface check failed midway on a slot end, which exposed a sketch bug, and it fixed the bug before writing its report. All four interfaces landed within 0.001 mm of the designer's original. On the panel, none of the 25 graders raised an interface problem, and dimensional accuracy moved from 2 to 3.

    Run R008 · re-parametrize rerun · Autodesk Fusion
  4. More accurate every run. Not more editable.

    Three runs of the same rebuild task, regraded the same way against the designer's native file: P95 surface deviation fell from 0.112 mm to 0.033 mm to 0.017 mm, and the last two got all four interfaces right. Editability didn't move. Half of the third run's combined edits broke the model, and the designer scored all three a 2. The agents keep getting closer to the shape and no closer to how a designer builds it.

    Runs R007, R008, R009 · like-for-like regrade, Nov 15, 2026
  5. Perfect interfaces, wrong part.

    Given only a brief and the interface dimensions, the AI hit every mounting point within 0.006 mm and designed a flat plate. The real part is a curled endplate that rises 200 mm off the flange and carries three ducts. Design intent scored 2 out of 5.

    Run R006 · blind design-for-fit · Autodesk Fusion
  6. The AI designed for the scan, not for the weld.

    Asked to design the cage-mount boxes for a Subaru strut tower, the AI used 3.3× the steel of the human design and planned stitch welds over gaps. The real boxes were drawn up on the spot with the welder: full-perimeter welds, built to spread the cage tube's loads into the tower. The AI never considered weld access.

    Run R005 · blind design-for-fit · collaboration with Heliarc Works
  7. CAD kernels fail silently.

    A thicken returned a "valid" solid with a volume of −10,327 cm³. A sweep returned zero volume and no error. A boolean that met a surface exactly at its crest did nothing at all. None of them raised an exception. Volume checks and section renders caught every one.

    Runs R002, R004 · OpenCascade
  8. Registration finds convincing wrong answers.

    Fitting a hood back onto its engine bay, the optimizer found a clean-looking minimum 80 mm off; the latch position gave it away. On another car, ICP made the left/right symmetry check worse by sliding 48 mm. The lesson: check every registration against an independent measure.

    Runs R003, R005
  9. Renders catch what deviation can't.

    Under gloss lighting, a fitted hood surface rippled. The waviness sat inside a 1.26 mm RMS that looked fine on paper, and a mold plug would have carried it into every part. Rendering is now a scored stage, not decoration.

    Run R001 · presentation fidelity
  10. The designer saw what 25 graders didn't.

    The designer scored design intent 1 out of 5 on all three endplate rebuilds. The panel gave 3 to 4, and the agent gave itself 3. His reason was the same every time: the agent rebuilt the curved wall as a stack of cross-sections, which isn't how the part was modelled. Manufacturability went the other way: the designer gave 5, the AI graders 3 to 4. Editability is the one score everyone agreed on, a 2.

    Runs R007, R008, R009 · designer scores, Nov 15, 2026
  11. AI reports still claim more than the files show.

    After scoring, each grader reads the agent's own report with its scores locked and lists every claim the files don't back. Every grader found some, 2 to 11 each. On the rerun, the report claimed a fix it hadn't made and a print setup that can't work: a part with a 350 mm straight edge, placed "diagonally" on a 350 mm build plate.

    Runs R007, R008 · panel step 2
03

How we work.

Every run, same loop
  1. Capture a real part

    Our own designs, parts we bought, or client parts run privately, each designed by a person and then built. We collect the scan, the designer's native CAD or drawings, and caliper readings, each tagged with its source.

  2. Write the brief

    What a designer would be handed, nothing more. On blind tasks and reruns the agent works in a fresh session that has never seen the answer, and every file it opens is logged.

  3. Let the agent build

    The agent drives real CAD (Autodesk Fusion, or OpenCascade in Python) and delivers an editable feature tree, not a mesh. Every decision and every build script is logged as it goes, failed ones included.

  4. Measure it

    Deterministic checks, run once: deviation both ways and per region, hole and slot centres against the designer's file, an editability sweep, volume checks, section renders.

  5. Panel grades it

    Five personas on five AI models score the judgment calls from an evidence packet, then check the agent's report with their scores locked. The designer settles every split.

  6. Check it on the part

    Print it, measure it, fit it. Log every failure, and rerun the same task when a newer model comes out.

Task types
TaskThe agent getsGraded againstStatus
Re-parametrizedead geometry → editable feature treeA dumb STEP import with no historyThe designer's native CAD and the printed partIn the set
Blind design-for-fitdesign a part without seeing the answerA brief, plus interface dimensions or a sectioned scan of the spaceThe designer's CAD or fabrication drawingsIn the set
Scan-to-CADraw 3D scan → parametric CADThe scan and a short intake noteScan deviation, measured critical dimensions, the physical partClient runs · public set next
04

How we score.

Rubric v2 · 1 to 5 per criterion
CriterionThe question
Dimensional accuracyHow far off is it, against the reference and against measured critical dimensions, relative to the scan's own noise and the part's own asymmetry?
Design intentDoes the feature tree reflect how the part was really designed?
EditabilityPush one parameter (a wall, a hole spacing) to the edge of its plausible range. Is the geometry still valid?
ManufacturabilityCould it be made by the process the real part was made with?
Assembly fitDoes the fabricated part fit its mate? Scored only after it's built, by a person, never from CAD.
Presentation fidelityDo the renders match the as-built part without overstating it?

Every level has a written anchor. Cap rules stop good statistics from hiding a bad part: a wrong interface caps dimensional accuracy at 2, however close the surface is. Process scores (honesty, efficiency) are recorded but kept out of the average.

Measured before judged. Deviation, interface checks, volume and the editability sweep are deterministic and run once. Only the judgment calls go to graders: intent, manufacturability, and what the numbers mean.

Every number has a source.

MEASUREDcalipers or scale SCANfrom the 3D scan CADdesigner's native file STANDARDfrom a spec ASSUMEDa guess, never shown as measured CONFIRMEDa guess the designer confirmed DECISIONthe designer's call
The grading panel · in use since November 2026

One scoring round is one roll of the dice, and repeating the same grader repeats its blind spots. So the judgment calls go to a panel. Five personas, each run on five different AI models: 25 graders per run. Every grader is a fresh session that never saw the build, the agent's self-score, the designer's score or the other graders.

Different personas look for different failures. Different models bring different habits. Neither removes bias on its own; together they make it measurable.

ClaudeOpus 5.5 GPT6 Astra Gemini4 Argon KimiK3.1 Pro DeepSeekV4 Pro · text only
PersonaPlaysGoes deep on
FabricatorThe welder, printer or installer making it tomorrowProcess, joining, load path
DesignerAn industrial designer of this kind of partConcept and design intent
InspectorMetrology and QCInterfaces, and what it was graded against
SuccessorThe engineer who inherits the fileEditability, parameter health
SkepticRed teamClaims vs files, silent failures
R007 · who caught the mirrored hole
ClaudeOpus 5.5
GPT6 Astra
Gemini4 Argon
KimiK3.1 Pro
DeepSeekV4 Pro
Fabricatormakes it tomorrow
19.2mm
19.2mm
19.2mm
19.2mm
19.2mm
Designerdesign intent
19.2mm
19.2mm
19.2mm
19.2mm
19.2mm
Inspectormetrology, QC
19.2mm
19.2mm
19.2mm
19.2mm
19.2mm
Successorinherits the file
19.2mm
19.2mm
19.2mm
19.2mm
19.2mm
Skepticred team
19.2mm
19.2mm
19.2mm
19.2mm
19.2mm

25 of 25. Each grader was a fresh session with no view of the build, the answer or the other graders, and each cited the same 19.2 mm offset.

The build's own self-check, which had no hole-centre table, scored the same rebuild 4 for accuracy.

  • Two steps per grader. Score from the evidence packet first. Then read the agent's report with the scores locked, and list every claim the files don't back.
  • Evidence or it doesn't count. A score has to cite a file and a value, or it's left out of the median.
  • Blind A/B on design tasks. Graders see the human design and the AI's as Design A and Design B, then guess which is which.
  • Median, spread and flags. We publish the median and range per criterion. Any split of two points or more goes to the designer to settle, and the reason is logged.
  • Calibrated against the designer. Each persona's and each model's offset from the designer is tracked and reported, not silently corrected.
  • Leak-checked. Before a panel runs, the packet and prompts are checked for hints. The pilot found two and removed them.
Panel vs self-score vs designer
R007 · first rebuildR008 · blind rerunR009 · second rerun
CriterionSelfPanelDesignerSelfPanelDesignerSelfPanelDesigner
Dimensional accuracy42 2–2343 3–3343 3–35
Design intent33 3–4134 3–4134 3–41
Editability33 3–3233 1–3 · flagged222 1–3 · flagged2
Manufacturability43 3–4534 3–5 · flagged534 3–5 · flagged5

Panel = median of 25 graders (24 on R009), with the range underneath. Designer scores were given on Nov 15, 2026, with the mirrored hole known; R007's accuracy came down from 4 to 3 for it. Assembly fit: every grader abstained, as the rubric requires, because nothing has been fabricated.

The panel held R008's and R009's accuracy at 3 with a cap for work graded against an import rather than the designer's native file. Those runs were given the import on purpose, so the designer has waived the cap for this task; a panel rerun on accuracy is next. The editability split traces to rubric wording, and the anchor is being rewritten.

05

The runs.

Average score · self / panel / designer
Self-scorePanel medianDesigner
Average score, 1 to 5 · hover a mark for its value
R009DRS endplatere-parametrize, second rerun
R009: self 3.00, panel 3.25, designer 3.25
R008DRS endplatere-parametrize rerun
R008: self 3.25, panel 3.50, designer 2.75
R007DRS endplatere-parametrize
R007: self 3.50, panel 2.75, designer 2.75
R006DRS endplateblind design-for-fit
R006: self 3.50, designer 4.25
R005Cage-mount boxesblind design-for-fit
R005: self 3.00, designer 2.25
R004Off-road bumperscan-to-CAD
R004: self 3.00, designer 3.00
R003Engine bayscan-to-reference
R003: self 3.67
R002Vented hoodscan-to-CAD
R002: self 3.40
R001Fiberglass hoodscan-to-CAD
R001: self 3.40
RunPartTaskAgent · CADSelfPanelDesignerData
R009DRS wing endplateTracehaus design · printed PET-CFRe-parametrizesecond blind rerunClaude · FusionFable 5.23.003.253.25Public
R008DRS wing endplateTracehaus design · printed PET-CFRe-parametrizeblind rerun of R007Claude · Fusion3.253.502.75Public
R007DRS wing endplateTracehaus design · printed PET-CFRe-parametrizeClaude · Fusion3.502.752.75Public
R006DRS wing endplateTracehaus design · printed PET-CFBlind design-for-fitClaude · Fusion3.50–4.25Public
R005Cage-mount boxeswith Heliarc Works · welded steelBlind design-for-fitClaude · OpenCascade3.00–2.25Public
R004Off-road front bumperclient partScan-to-CADClaude · OpenCascade3.00–3.00Private
R003Engine bayclient part · case studyScan-to-referenceClaude · Python3.67––Private
R002Vented fiberglass hoodclient part · case studyScan-to-CADClaude · OpenCascade3.40––Private
R001Fiberglass hoodclient part · case studyScan-to-CADClaude · OpenCascade3.40––Private

Self and designer: averaged over the criteria each run was scored on. Panel: the per-criterion medians for accuracy, intent, editability and manufacturability, averaged. "–" means not scored yet. Panel regrades of R005 and R006 are next.

Private runs: client geometry is never published. Their failure tags are, with the part left blank, so the failure statistics cover every run. Assembly fit isn't scored yet; none of the AI outputs has been fabricated.

06

Where AI-CAD breaks.

128 failures · 9 runs

Every run keeps a log of everything that went wrong: what broke, which iteration, whether it was fixed, and what caught it. A volume check, a section render, an editability sweep, a caliper, a grader. The log is the most useful thing we make. It's what a model would need to learn from, and what anyone grading AI-CAD should test for.

Each entry carries a tag, a family, a status (fixed, open, caught, disputed) and what caught it. The full log ships with the benchmark (07).

Geometry

Missed features, clipped corners, draft ignored on a molded part.

draft-ignored

Kernel

Operations that return a valid-looking solid that's wrong.

thicken-inverted

Process

Parameters that only document a value, planes hard-coded instead of referenced.

params-not-driving

Registration

Alignments that settle into a convincing false minimum.

off-data-penalty-bias

Intent

Right numbers, wrong idea of what the part is for.

intent-missed

Fabrication

Designs that ignore how the part gets welded, printed or molded.

weld-strategy-missed

Render

Surfaces that pass the numbers and fail under light.

surface-not-fair

Tooling

Analysis tools that fail without saying so.

section-tool-silent
07

The data.

CC BY-NC 4.0
Interfaces in the designer's file
Where the R007 rebuild put the M6
3D view loads when you get here
The DRS endplate from the Hugging Face dataset, straight from the designer's STEP: M6 hole, flap pivot and DRS arc slot in chartreuse. The orange ring is where the R007 rebuild put the M6. Turn to the flange to see both.Display mesh: 0.05 mm chordal tessellation
What's public, and what isn't
  • Public: Tracehaus's own designs, and parts we bought or have rights to. The cage-mount boxes are included with Heliarc Works' OK.
  • Counts only: client parts. Their geometry stays private; their failure tags are in the statistics with the part left blank.
  • Never: branded aftermarket parts.
Read the case studies
08

Limits, and what's next.

Honest limits
  • Small. Eight runs is a set of case studies, not a statistical benchmark. It grows as parts get built.
  • One builder so far. Every build is Claude. The panel spans five model families; the builds don't yet.
  • One human reference. The designer is still the only human grader, scoring their own parts. A second reviewer is planned.
  • Models vary more than personas. On the panels so far, persona medians came out nearly identical; the spread came from the models. More runs will show whether the personas earn their place.
  • Rubric wording drives splits. The widest disagreement so far came from two models reading the same anchor differently, not from the part.
  • Text-only graders abstain on presentation, so that criterion rests on four models.
  • Briefs leak. One brief was written by someone who'd seen the answer, and it showed in the wording. Future briefs get reviewed by someone who hasn't.
  • Fit is untested until the AI outputs are fabricated.
Next
  • Open AI-CAD models on the same tasks: CAD-Coder, CAD-Recode and others.
  • Panel regrades of R005 and R006, the two blind design tasks, with the human and AI designs shown as A and B.
  • Three build attempts on every blind design task, each in a fresh session, all published.
  • A panel rerun on accuracy for R008 and R009, now that the import cap is waived for this task.
  • Simulation in the loop. Our CFD setup is validated on the NACA 0012 reference case: lift within about 1%, drag still 3–11% low and not yet grid-converged, so drag isn't quoted yet. FEA follows.
  • Fabricate the AI's designs and score assembly fit on the real mate.
  • Rerun every task as new models come out, against the same rubric.

Grade it against reality.

Building an AI-CAD model and want it run on parts that exist? Have a part you'd like in the set? Write to the lab. For commercial use of the data, ask us first.

Client and vehicle details withheld on private runs. Data released under CC BY-NC 4.0, attribution to Tracehaus LLC.