QualityRelay SAFE-MESH

Inspection agents that decide, part by part, whether a part is safe to ship, and keep deciding when the factory network fails.

ScienceClaw Hackathon 2026 (MIT LAMM × E14 Fund) · Team QualityRelay

🌐 Live demo	qualityrelay.vercel.app (digital factory, agent panel, CNC loop; all factory data SIMULATED)
🎬 Demo video	video
📊 Slides	slides
🧾 Every claim and its evidence	CLAIMS_AND_OPEN_WORK.md

---

Contents

1. Why this matters
2. The idea
3. What we built
4. Key result: the line keeps running when the network does not
5. Closing the loop: a line that fixes its own bad batch
6. Real parts, real measurements
7. How we kept it honest
8. What is real and what is simulated
9. Run it yourself
10. Repository map
11. Cross-team collaboration

---

1. Why this matters

Every part on a production line is a decision under uncertainty. Ship a bad part and it reaches a customer; hold a good one and you lose output.

Our test case is a slot and a tab that must fit together. Each part can pass its own tolerance, and the pair can still fail to fit, because the errors stack. Caliper, camera and CAD evidence about the same part also disagree regularly.

Getting these calls wrong is expensive:

• $2.3M lost per hour of unplanned downtime at a large automotive plant (Siemens, True Cost of Downtime 2024)
• 15–20% of sales typically spent on the cost of poor quality (ASQ)

Factories are now adding AI to inspection. The default design sends every raw camera frame to one central server that decides for everyone. That works until the uplink saturates or the server goes down; then evidence arrives late, parts wait, and the line slows or stops.

2. The idea: inspect like a good engineering team

A good engineering team does not send every raw measurement to the boss. Each engineer owns their data, asks a colleague only when they need a second opinion, and says "I'm not sure, hold this part" instead of guessing.

QualityRelay works the same way:

flowchart LR
    subgraph Stations["Station agents (each owns its evidence)"]
        CAL["Caliper agent<br/>critical width"]
        CAM["Camera agent<br/>photo per part"]
        VIS["Vision agent<br/>anomaly model"]
        ASM["Assembly agent<br/>pairs slot + tab<br/>LLM planner"]
    end
    CAL <-- "signed REQUEST_EVIDENCE /<br/>EVIDENCE_RESPONSE" --> ASM
    CAM <--> VIS
    VIS <--> ASM
    CAL --> GATE{"SAFE-MESH<br/>safety gate"}
    CAM --> GATE
    VIS --> GATE
    ASM --> GATE
    GATE -->|"evidence complete<br/>and consistent"| PASS["PASS"]
    GATE -->|"missing, stale, duplicated,<br/>conflicting or unverifiable"| HOLD["HOLD"]
    GATE -->|"out of tolerance"| REJECT["REJECT"]
    PASS --> HUMAN["Human approves<br/>every physical action"]
    HOLD --> HUMAN
    REJECT --> HUMAN

Loading

Three rules make it safe:

1. Evidence stays where it is produced. Stations exchange small, role-signed messages (about 4.7 kB per part), not raw images (about 4 MB).
2. Unsure means HOLD, never PASS. Evidence that is missing, stale, duplicated (one photo counted twice) or fails its signature check sends the part to HOLD.
3. A human approves every physical action. Agents recommend; people sort and sign off.

3. What we built

Component	What it does	Where
Peer protocol	Role-signed REQUEST_EVIDENCE / EVIDENCE_RESPONSE with access control, ownership and verification (signature, staleness, lineage); failed verification → HOLD	qualityrelay17/, P1 results
SAFE-MESH safety gate	Turns evidence into PASS / HOLD / REJECT; never PASSes on incomplete evidence	qualityrelay17/
LLM planners	OpenAI and Nebius models choose the next evidence only when a case is unclear; a live run of 30 parts cost $0.156; the website replays a recorded run for $0	LLM_PLANNERS.md
Digital factory	Event-driven line with printers or CNC machines, cameras, caliper chamber, review station, assembly and sorter; live agent panel	qualityrelay.vercel.app
Anomaly model	Trained on an NVIDIA L4 GPU: 0.948 image AUROC on the public MVTec screw benchmark; ready for our printed-part photos	ANOMALY_GPU_RUN_31OCT.md
Rare-event Monte Carlo	NVIDIA Warp, for low-probability failure scenarios	WARP_MC_AND_TUNING_31OCT.md
Physical bench	3D-printed slots and tabs, caliper chamber with ESP32 logger, fit tests, labeled photos	Section 6

The digital factory at qualityrelay.vercel.app: every agent decision can be opened and inspected.

4. Key result: the line keeps running when the network does not

We registered this test before writing the code (registration): our station agents against a central server that receives every raw frame, on the same simulated line, 20 seeds × 200 pairs.

Network condition	Central server (good assemblies/h)	Station agents	Escapes per 1,000 (both)
Fast link, 100 Mbit/s	195.7	195.8	1.04
Uplink 4 Mbit/s	104.4	195.8	1.04
Uplink 2 Mbit/s	43.2	195.8	1.04
Server down for 30 min	89.7	195.8	1.04

• Same safety. Both architectures ship 1.04 bad parts per 1,000; today's fixed rules ship 3.90.
• About 850× less network traffic. 4.7 kB of signed messages per part instead of about 4 MB of raw frames.
• Where it does not pay, and we say so. On a fast link, both make identical decisions (360 runs, zero differences): decentralization is not smarter with full information. With compressed frames, the central server recovers. The advantage is real where bandwidth is limited or links fail.
• Stated assumption: the local network between stations stays up. We have not yet tested station-to-station failures.

📄 Full results: P1_DECENTRALISATION_RESULTS.md · data: examples/results/p1_decentralisation/

Also when a sensor drops out. Against fixed rules in 15 simulated failure scenarios, the QualityRelay policy shipped fewer bad assemblies in all 15. It kept 95% more good output when 15% of camera readings went missing and 56% more when a camera was cut off by a network partition, because fixed rules hold everything while QualityRelay uses the evidence that is left. It kept throughput in only 4 of the 15 scenarios, and we report that too (SIM_18_1_RESULTS.md).

5. Closing the loop: a line that fixes its own bad batch

Inspection is only half the job. In our CNC scenario D01, the whole first batch comes out wrong (a 1.6 mm tool-offset error), and the agents correct the machine:

flowchart LR
    A["CNC machines cut<br/>slots and tabs"] --> B["Camera agent<br/>counts defects"]
    B -->|"6 defects in the<br/>last 12 parts"| C["Whole line stops"]
    C --> D["Caliper chamber measures<br/>every suspect part"]
    D --> E["chamber_measurements.csv<br/>one row per dimension"]
    E --> F["CNC agent computes<br/>G-code correction"]
    F --> G{"Operator<br/>approves?"}
    G -->|yes| H["Machine restarts"]
    H --> I["Good parts paired<br/>and assembled"]

Loading

• Result (simulated): seeds 0–2, 60 pairs: 0 escaped assemblies, 60/60 shipped good. No blank is cut while the line is stopped. Try it on the live demo (scenario D01). Details: CNC_DEMO_SCENARIO_D01.md
• Under printer drift, feeding inspection back to the printer raised good output from 154 to 195 parts per hour and cut escapes from 4.96 to 2.59 per 1,000 (INNOVATIONS_RESULTS_31OCT.md).
• No real CNC machine is controlled. The G-code is illustrative. The physical half of this loop today is our caliper chamber, below.

6. Real parts, real measurements

The simulation is only worth something if it connects to real parts. We printed and measured them.

Fit test: nominal, good, risk and defective slots × tabs.

Caliper chamber: each reading streams to a CSV row (screenshot from a demo run).

Specimen	Design width	Role
S-GOOD / T-GOOD	21.20 / 19.80 mm	comfortable clearance
S-NOM / T-NOM	21.00 / 20.00 mm	nominal
S-RISK / T-RISK	20.70 / 20.30 mm	each in tolerance, pair at the edge
S-DEF / T-DEF	19.40 / 21.60 mm	wrong size, must be rejected

• 8 printed specimens, CAD and STL in examples/cad/, print list in print_manifest.csv
• Caliper chamber: caliper → ESP32 → CSV logger, firmware in firmware/measurement_station/, procedure in MEASUREMENT_LOGGING.md
• Photo dataset: 88 photos of the 8 printed specimens, taken with our camera on a fixed white background and named by specimen ID and defect type (photos, capture guide)
• Human-confirmed sorting: the demo routes parts to PASS / HOLD / REJECT bins by hand; no actuator moves without a person (PHYSICAL_DEMO_OVERVIEW.md)

These are tagged PHYSICAL_CAPTURE. They show the measurement pipeline on real parts; they do not validate the factory-scale numbers above.

7. How we kept it honest

• Pre-registered. Scenarios, metrics and decision rules were written down before running: protocol registration (13 scenarios × 30 seeds × 1,000 parts) and P1 registration.
• Negative results kept. Our first reference policy was not better than conservative rules (protocol results); majority voting among agents was unsafe (21.7 unsafe passes per 1,000); with full information, decentralization adds messages, not insight (Gate A); an earlier cost-saving claim was withdrawn after we found unpriced messages.
• Every claim tagged SIMULATED, PHYSICAL_CAPTURE or EXTERNAL_DATASET, listed in CLAIMS_AND_OPEN_WORK.md.
• 550+ automated tests, including prompt-injection tests for the LLM planners.

8. What is real and what is simulated

Component	Status
Peer protocol, safety gate, digital factory, website	✅ Built and running
LLM planners (OpenAI, Nebius)	✅ Live runs; website replays a cached run
Network, failure and CNC-loop results	🧪 SIMULATED, pre-registered
Anomaly model	🧪 Trained on a public dataset (EXTERNAL_DATASET); 88 printed-part photos collected
Printed parts, caliper chamber, fit tests, 88-photo dataset	🔩 PHYSICAL_CAPTURE
Real CNC control, factory deployment, station-to-station link failures	⏳ Not done yet

9. Run it yourself

python -m pip install -r requirements-demo.txt
python -m pytest -q                                   # run the test suite
uvicorn qualityrelay17.api:app --host 127.0.0.1 --port 8000
# open http://127.0.0.1:8000

Or open the live demo. Photo capture and photo measurement need a local run.

10. Repository map

Path	What is inside
qualityrelay17/	Agents, peer protocol, safety gate, simulation engine, API
web/	The Studio website (digital factory, 3D Lab, agent panel)
docs/research/	Registrations, results, references
docs/engineering/	Physical bench, measurement, LLM planners, team handoffs
examples/results/	Raw result files for every experiment
examples/cad/	Specimen and sorter CAD/STL
firmware/	ESP32 measurement station and sorter
tests/	Test suite
README_START_HERE.md	Full engineering notes

Cross-team collaboration

We worked with two other ScienceClaw teams:

We are collaborating with SentiLab to develop and test an updated CNC G-code workflow based on QualityRelay’s inspection feedback. QualityRelay detects dimensional drift and creates a traceable, bounded correction; SentiLab helps translate that correction into revised G-code for the CNC workflow. The resulting workflow and documentation can be reused by both teams.

Our target is an automated manufacturing system that detects process drift, updates the CNC instructions, and verifies the resulting parts without routine human supervision. Safety is maintained through predefined correction limits, G-code validation, machine interlocks, sensor confirmation, automatic HOLD states, and rollback to the last verified program. The current CNC and G-code demonstration remains simulated; autonomous operation on a physical CNC still requires validation.

QualityRelay can collaborate with CoForm to close the loop between AI-generated design and physical manufacturing. CoForm provides verified CAD, tolerances, and print-ready geometry; QualityRelay compares printed parts against that design using cameras and appropriate metrology, detects manufacturing drift, and returns traceable feedback for automatic redesign or process correction. Together, we can demonstrate an end-to-end workflow from a user request to a verified physical part.

Built with

NVIDIA (L4 GPU training for the anomaly model; Warp for rare-event Monte Carlo) · OpenAI and Nebius (LLM planners) · Vercel (live demo) · 3D printing for the physical specimens

Team: Bakdaulet · adilnurgg · HyorinmaruK · Shuyan · Vinayak

---

All factory-scale numbers are simulated and pre-registered. Industry figures are cited from their sources; see docs/research/ for references.