---
title: "kicad-happy Test Harness"
url: "https://maker.wiznet.io/Sunny_/projects/kicad-happy-test-harness/"
markdown_url: "https://maker.wiznet.io/Sunny_/projects/kicad-happy-test-harness/md"
type: "UCC: User Created Content"
author: "Andrew Klofas"
author_url: "https://github.com/aklofas/kicad-happy-testharness"
editor: "WIZnet"
editor_url: "https://maker.wiznet.io/"
original_author: "Andrew Klofas"
original_url: "https://github.com/aklofas/kicad-happy-testharness"
published: "2026-05-29"
language: "en"
likes: 0
views: 294
comments: 0
source: "WIZnet Makers (https://maker.wiznet.io/)"
---

# kicad-happy Test Harness

> Test harness for kicad-happy skills and analyzers

Original author: Andrew Klofas (source: https://github.com/aklofas/kicad-happy-testharness)

## Article

## 1. Overview

**kicad-happy-testharness** is an open-source testing framework designed to ensure the reliability of **kicad-happy**, a KiCad-based PCB design analysis tool.

Automated PCB design analyzers face several challenges:

Changes to analysis logic can unintentionally break existing behavior.

New KiCad project types may introduce unforeseen edge cases.

Manual review alone is often insufficient to detect quality regressions early.

To address these issues, **kicad-happy-testharness**:

Uses real-world open-source KiCad projects as a testing corpus.

Implements a three-layer validation architecture consisting of:

Baseline Snapshots

Assertions

LLM Reviews

Automatically detects regressions and supports continuous quality improvement.

---

## 2. System Architecture

The test harness manages data on a per-repository basis and employs three independent validation layers that together form a continuous feedback loop.

```plaintext
      Open-Source KiCad Projects (Corpus)
                    ↓
         Run kicad-happy Analyzer
                    ↓
    ┌───────────────────────────────┐
    │     3-Layer Validation        │
    │                               │
    │ Layer 1: Baseline Snapshot    │
    │ Layer 2: Assertion Check      │
    │ Layer 3: LLM Review           │
    └───────────────────────────────┘
                    ↓
       Regression Detection
                    ↓
          Fix → Repeat
```

All reference data is stored under:

```plaintext
reference/{repo}/{project}/
```

allowing full Git-based version tracking.

---

## 3. Three Validation Layers

![](https://maker.wiznet.io/upload/ckeditor5/1064341221%5F1780073206%2Epng)

![](https://maker.wiznet.io/upload/ckeditor5/1064341221%5F1780073222%2Epng)

### Layer 1 — Baseline Snapshot

The complete analyzer output is captured as a baseline snapshot and compared against future executions.

**Tools**

`regression/snapshot.py` — Generate baselines

`regression/compare.py` — Compare current output against baseline

This enables fast regression detection while filtering only the repositories that changed.

```plaintext
python3 regression/snapshot.py --repo {repo}
python3 regression/compare.py --repo {repo}
python3 regression/compare.py --all --only-changes
```

---

### Layer 2 — Assertions

Assertions define outputs that the analyzer must always detect for specific projects.

Assertions are stored as JSON files under:

```plaintext
reference/{repo}/{project}/assertions/
```

Supported operators include:

`exists`

`equals`

`range`

`min_count`

`contains_match`

Since assertions are committed to Git, they provide permanent regression protection.

```plaintext
python3 regression/run_checks.py --repo {repo}
python3 regression/seed.py --repo {repo}
```

---

### Layer 3 — LLM Review

Original KiCad files and analyzer summaries are bundled into review packets and independently evaluated by Claude.

This layer is particularly effective at identifying quality issues that are difficult to detect using deterministic checks.

Review findings are tracked using identifiers such as:

```plaintext
FND-00000001
```

and can later be promoted into permanent assertions.

```plaintext
python3 regression/packet.py --strategy random --count 5
python3 regression/findings.py list
python3 regression/findings.py promote FND-00000001
```

---

## 4. The Ratchet Model — Quality Only Moves Forward

The three validation layers do not operate independently; they form a feedback loop.

When an LLM review discovers a defect:

```plaintext
LLM Finding
      ↓
Recorded Finding
      ↓
Promoted to Assertion
      ↓
Automatically Verified in Every Future Run
```

Once a defect is discovered, it can never silently reappear.

This "Ratchet" model is the core principle of the framework.

---

## 5. Technology Stack

### Core Components

Python 3 CLI-based tooling

KiCad project analysis

Git-based version control

### Analysis Targets

Real-world open-source KiCad projects containing:

`.kicad_pcb`

`.kicad_sch`

related design artifacts

### Data Storage

```plaintext
reference/{repo}/{project}/
```

Stores:

Baselines

Assertions

Findings

### LLM Integration

Claude API

Independent quality assessment and review

### Development Environment

Python 3

KiCad

Git

---

## Connection to W5500

One of the goals of **kicad-happy** is to automatically identify design issues during the PCB design phase.

Many open-source hardware projects in the corpus contain Ethernet circuitry based on the **WIZnet W5500**, making them valuable real-world validation targets.

The **W5500** is a Hardwired TCP/IP Ethernet controller connected to an MCU through SPI. PCB-level verification commonly involves checking:

Pin assignments

SPI bus connectivity

Decoupling capacitor placement

Power distribution integrity

Ethernet interface design rules

Automated analysis tools such as **kicad-happy** can help detect these issues early in both schematic and PCB layout stages.

---

## 6. Applications

### Use Cases

Automated KiCad design verification

Open-source hardware quality assurance

Embedded networking circuit review

AI-assisted PCB analysis pipelines

CI/CD integration for hardware projects

Regression testing for PCB design tools

### Value Proposition

The framework validates PCB analysis tools simultaneously through:

Automated baseline comparison

Explicit assertion verification

LLM-based quality review

This multi-layer approach enables continuous improvement of analyzer reliability while minimizing the risk of undetected regressions.

---

## 7. Conclusion

**kicad-happy-testharness** is a testing framework that validates PCB design analysis tools using a corpus of real-world open-source KiCad projects.

Through its three-layer validation architecture:

Baseline Snapshots

Assertions

LLM Reviews

and its Ratchet-based quality model, the framework:

Automatically detects regressions

Converts discovered defects into permanent test cases

Ensures quality improves in a single direction over time

The result is a practical QA platform that can contribute to higher reliability across the open-source KiCad hardware ecosystem.

---

Source: https://maker.wiznet.io/Sunny_/projects/kicad-happy-test-harness/