Wiznet makers

1. Overview

kicad-happy-testharness is an open-source testing framework designed to ensure the reliability of kicad-happy, a KiCad-based PCB design analysis tool.

Automated PCB design analyzers face several challenges:

Changes to analysis logic can unintentionally break existing behavior.

New KiCad project types may introduce unforeseen edge cases.

Manual review alone is often insufficient to detect quality regressions early.

To address these issues, kicad-happy-testharness:

Uses real-world open-source KiCad projects as a testing corpus.

Implements a three-layer validation architecture consisting of:

Baseline Snapshots

Assertions

LLM Reviews

Automatically detects regressions and supports continuous quality improvement.

2. System Architecture

The test harness manages data on a per-repository basis and employs three independent validation layers that together form a continuous feedback loop.

      Open-Source KiCad Projects (Corpus)
                    ↓
         Run kicad-happy Analyzer
                    ↓
    ┌───────────────────────────────┐
    │     3-Layer Validation        │
    │                               │
    │ Layer 1: Baseline Snapshot    │
    │ Layer 2: Assertion Check      │
    │ Layer 3: LLM Review           │
    └───────────────────────────────┘
                    ↓
       Regression Detection
                    ↓
          Fix → Repeat

All reference data is stored under:

reference/{repo}/{project}/

allowing full Git-based version tracking.

3. Three Validation Layers

Layer 1 — Baseline Snapshot

The complete analyzer output is captured as a baseline snapshot and compared against future executions.

Tools

regression/snapshot.py — Generate baselines

regression/compare.py — Compare current output against baseline

This enables fast regression detection while filtering only the repositories that changed.

python3 regression/snapshot.py --repo {repo}
python3 regression/compare.py --repo {repo}
python3 regression/compare.py --all --only-changes

Layer 2 — Assertions

Assertions define outputs that the analyzer must always detect for specific projects.

Assertions are stored as JSON files under:

reference/{repo}/{project}/assertions/

Supported operators include:

exists

equals

range

min_count

contains_match

Since assertions are committed to Git, they provide permanent regression protection.

python3 regression/run_checks.py --repo {repo}
python3 regression/seed.py --repo {repo}

Layer 3 — LLM Review

Original KiCad files and analyzer summaries are bundled into review packets and independently evaluated by Claude.

This layer is particularly effective at identifying quality issues that are difficult to detect using deterministic checks.

Review findings are tracked using identifiers such as:

FND-00000001

and can later be promoted into permanent assertions.

python3 regression/packet.py --strategy random --count 5
python3 regression/findings.py list
python3 regression/findings.py promote FND-00000001

4. The Ratchet Model — Quality Only Moves Forward

The three validation layers do not operate independently; they form a feedback loop.

When an LLM review discovers a defect:

LLM Finding
      ↓
Recorded Finding
      ↓
Promoted to Assertion
      ↓
Automatically Verified in Every Future Run

Once a defect is discovered, it can never silently reappear.

This "Ratchet" model is the core principle of the framework.

5. Technology Stack

Core Components

Python 3 CLI-based tooling

KiCad project analysis

Git-based version control

Analysis Targets

Real-world open-source KiCad projects containing:

.kicad_pcb

.kicad_sch

related design artifacts

Data Storage

reference/{repo}/{project}/

Stores:

Baselines

Assertions

Findings

LLM Integration

Claude API

Independent quality assessment and review

Development Environment

Python 3

KiCad

Git

Connection to W5500

One of the goals of kicad-happy is to automatically identify design issues during the PCB design phase.

Many open-source hardware projects in the corpus contain Ethernet circuitry based on the WIZnet W5500, making them valuable real-world validation targets.

The W5500 is a Hardwired TCP/IP Ethernet controller connected to an MCU through SPI. PCB-level verification commonly involves checking:

Pin assignments

SPI bus connectivity

Decoupling capacitor placement

Power distribution integrity

Ethernet interface design rules

Automated analysis tools such as kicad-happy can help detect these issues early in both schematic and PCB layout stages.

6. Applications

Use Cases

Automated KiCad design verification

Open-source hardware quality assurance

Embedded networking circuit review

AI-assisted PCB analysis pipelines

CI/CD integration for hardware projects

Regression testing for PCB design tools

Value Proposition

The framework validates PCB analysis tools simultaneously through:

Automated baseline comparison

Explicit assertion verification

LLM-based quality review

This multi-layer approach enables continuous improvement of analyzer reliability while minimizing the risk of undetected regressions.

7. Conclusion

kicad-happy-testharness is a testing framework that validates PCB design analysis tools using a corpus of real-world open-source KiCad projects.

Through its three-layer validation architecture:

Baseline Snapshots

Assertions

LLM Reviews

and its Ratchet-based quality model, the framework:

Automatically detects regressions

Converts discovered defects into permanent test cases

Ensures quality improves in a single direction over time

The result is a practical QA platform that can contribute to higher reliability across the open-source KiCad hardware ecosystem.