Mutual-TLS MQTT, Verified OTA and Device Identity on an ESP32-S3 with the WIZnet W5500

Project Introduction

sensor-station by Paul B. (Firebolt) is Ethernet-first ESP32 firmware for occupancy and people-counting systems. It is the base platform for stations that read mmWave, PIR or thermal sensors and report occupancy counts and sensor events to an MQTT broker.

The firmware provides what a fielded device needs:

• Provisioning over a Wi-Fi access point and/or Ethernet, with a built-in web UI.
• MQTT with configuration and certificates stored in NVS.
• OTA firmware updates with rollback.
• Room for device-specific sensor logic.

The author’s stated goal is firmware that “stays manageable after the board leaves the bench.” That means a field unit can be recovered, updated remotely, and trusted on the network.

The same firmware builds for two boards:

On both boards Wi-Fi is the primary data path and Ethernet is the fallback.

Security Architecture

Security runs through the whole design rather than being bolted on. It has six layers.

1. Mutual-TLS MQTT

Every station connects to the broker over MQTT over TLS (port 8883) with a client certificate. The Mosquitto broker is set to require_certificate true, so a device without a valid certificate cannot connect.

• TLS is handled by mbedTLS through Arduino-ESP32’s NetworkClientSecure. It runs over lwIP, so the same code works over Wi-Fi, W5500 Ethernet, or RMII Ethernet.
• The broker certificate must carry the broker’s IP address in its Subject Alternative Name. The author found that mbedTLS rejects a certificate whose CN matches but whose SAN has no IP entry (-9984, CERT_VERIFY_FAILED).
• The client ID and topic path come from the station’s hostname (sensor-XXXXXXXX, derived from its MAC address). This lets a broker ACL confine each station to its own topics.

2. Verified OTA Updates

OTA is triggered by an MQTT command and is designed around image integrity:

• HTTPS only. http:// firmware URLs are refused.
• SHA-256 required. The command carries the expected hash of firmware.bin. The device hashes the image it wrote and refuses to boot it if the hash doesn’t match. The docs are explicit that the hash, not the server certificate, is what protects the image. The download server’s certificate is deliberately not verified, so firmware can be hosted on a self-signed server, a public CA, or a CDN.
• Dual OTA partitions and rollback. New firmware goes to the inactive slot. A boot counter rolls back automatically after more than five failed boots, and the firmware checks the target slot before switching.
• Version control. Numeric semver comparison blocks accidental downgrades unless "force": true is set.

The trust chain is the TLS-authenticated MQTT command, which carries the hash, plus an on-device integrity check. A tampered or corrupted image cannot be booted, even if the file server is compromised.

3. Hardened Web UI and Certificate Handling

• HTTP Basic Auth protects the MQTT configuration page (/mqtt).
• Write-only certificate management. Certificates and keys can be uploaded through the web UI into NVS but can’t be read back out.
• An Ethernet-only deployment mode keeps the Wi-Fi radio off where it isn’t wanted.

4. Per-Device Identity: a Factory PKI (Draft Design)

The most forward-looking part of the repo is a draft design (docs/development/device-certificates.md) for giving every station its own identity at manufacture. It isn’t built yet.

• Keys are generated on the device. Over USB at the flash station, the device creates an ECDSA P-256 key pair in NVS and returns a CSR. The private key never leaves the chip.
• Two-tier CA. An offline root CA signs only intermediates. Each flash station or production line has an intermediate CA, ideally on a hardware token such as a YubiKey. The intermediate is constrained (pathlen:0) and rotated every two years.
• Certificate-only login. No username or password, and no per-device state on the broker. One ACL pattern covers the whole fleet.
• Portable. The same factory certificate works with Mosquitto + ThingsBoard CE and with AWS IoT Core, so changing platforms is a configuration change.
• No enrolment service on the network. Physical presence on the USB port authenticates the request.

5. Honest About Its Own Weak Spots

The repo openly lists the gaps it still has to close:

• A shared fallback identity. src/certs.h contains a client certificate and its private key, tracked in git. The test certificates in docs/certs/ include client.key. Any station without NVS certificates falls back to that shared identity. The factory PKI is meant to remove this.
• No certificate date checks on the device. ESP-IDF builds mbedTLS without CONFIG_MBEDTLS_HAVE_TIME_DATE by default, because many embedded devices boot without a reliable clock. Chain and signature checks still run. The broker does check the dates on the station’s certificate chain.
• Legacy stations. The ESP8266 stations already in the field use plain MQTT without TLS. This firmware is their replacement.

6. Engineering Process

Local git hooks block direct commits and pushes to main, so every change goes through a pull request. Security behaviour is documented next to the code, including the failed experiments.

WIZnet’s Role: the W5500, and a Lesson in Where TLS Lives

The W5500 is the wired Ethernet interface on the Waveshare ESP32-S3-POE-ETH, which the OTA documentation calls the main target. The way the project uses it is the most instructive part of the story.

First attempt: the WIZnet hardwired TCP/IP path. The original build used the Arduino Ethernet library, which runs TCP/IP inside the W5500. TLS came from SSLClient (BearSSL) layered on top. It never worked. EthernetClient exposes only connect(IPAddress, port), with no hostname variant, so BearSSL never received the hostname it needs to verify the server certificate. The author tried trust anchors only, IP SANs, a custom resolver, manual DNS, and mDNS, and all of them failed. They concluded that MQTTS was impossible on that path and removed it.

What works: the W5500 in MAC-raw mode. The same chip is now driven by Arduino-ESP32 3.x’s ETH library (ETH.begin(ETH_PHY_W5500, …)). In this mode the W5500 acts as an SPI Ethernet MAC and PHY, and lwIP and mbedTLS run on the ESP32-S3. The author’s bench test on 29 October 2026 used Ethernet only, with Wi-Fi never started:

• A mutual-TLS MQTT session stayed up and published every 10 seconds.
• Each TLS handshake took about 0.94 s.
• Free heap stayed at about 271 KB.

The test can be reproduced with src/samples/w5500_tls_test.cpp (env esp32s3-w5500-tls-test). The W5500 links at 100 Mbit/s, but the SPI bus limits throughput to a few Mbit/s, which is plenty for MQTT and OTA.

The author’s own conclusion is that the limitation was the WIZnet-offload library, not the W5500 chip. The W5500’s MACRAW socket mode lets a design keep a cheap, proven SPI Ethernet chip while running a full software stack, including TLS with hostname verification, on the host MCU.

Why the W5500 suits this design:

• It is a single SPI chip for wired Ethernet, so a PoE ESP32-S3 board needs no RMII-capable MCU.
• It gives the same security code path as Wi-Fi. NetworkClientSecure behaves identically on either interface, so mTLS and verified OTA work over a cable.
• It provides a deterministic wired fallback when Wi-Fi is down, which also serves as the recovery path for a unit that needs reprovisioning or an update.

Possible Applications

• Occupancy and people counting in offices, retail, libraries, and transit, using mmWave, PIR or thermal sensors.
• Smart building and facilities management: feeding real-time occupancy to HVAC, lighting, and space-booking systems over authenticated MQTT.
• Secure industrial IoT gateways: PoE-powered nodes that need mutual TLS, a hardware-unique identity, and safe remote updates.
• Fleet-scale IoT deployments: the factory-PKI design is a template for giving thousands of devices unique certificates without per-device broker configuration.
• Cloud-portable telemetry: the same device identity works with self-hosted Mosquitto/ThingsBoard or AWS IoT Core.
• Reference design for TLS over the W5500: a worked example of MAC-raw mode with lwIP and mbedTLS on the ESP32.

FAQ

Q: What is sensor-station?
It is ESP32 firmware for Ethernet-first sensor stations that report people-counting and occupancy data over MQTT. It includes provisioning, a web UI, mutual-TLS MQTT, and verified OTA updates.

Q: How does it use the WIZnet W5500?
On the Waveshare ESP32-S3-POE-ETH, the W5500 provides wired, PoE-powered Ethernet over SPI. It runs in MAC-raw mode, so the TCP/IP stack (lwIP) and TLS (mbedTLS) run on the ESP32-S3.

Q: Why not use the W5500’s built-in TCP/IP stack?
The project’s TLS stack needed the server hostname to verify the broker certificate. The Arduino EthernetClient API it used connects by IP address only, so verification always failed. This is a limit of that Arduino library combination, not of the W5500. TLS stacks that run on the MCU can use W5500 hardware sockets and still verify hostnames, but MAC-raw mode was the simplest route on the ESP32.

Q: Is TLS over the W5500 fast enough?
In the author’s bench test a mutual-TLS handshake took about 0.94 s, free heap stayed around 271 KB, and the session stayed up while publishing every 10 seconds. That is ample for MQTT telemetry and OTA.

Q: How are firmware updates protected?
Updates are triggered by an MQTT command over mutual TLS, downloaded over HTTPS only, and checked against a SHA-256 hash carried in the command before they can boot. Dual partitions and a boot-count watchdog roll back to the previous firmware after repeated failed boots.

Q: If the download server’s certificate isn’t verified, can’t an attacker swap the firmware?
No. The expected SHA-256 arrives through the authenticated MQTT channel, so a modified image fails the hash check and is never booted. The HTTPS layer adds confidentiality, and the hash provides integrity.

Q: How does each device get its own identity?
The planned factory process has each station generate its own ECDSA P-256 key over USB and receive a certificate signed by a rotated intermediate CA under an offline root. This is a draft design. Today, stations use certificates uploaded to NVS, falling back to a shared test certificate compiled into the firmware.

Q: Are there known security gaps?
The author documents three:

• A shared client certificate and private key committed to the repo, which acts as a fallback identity.
• No certificate date checks on the device, a deliberate ESP-IDF default.
• Legacy ESP8266 stations that still run plain MQTT.

The first and last are on the roadmap to fix.

Q: Which brokers and platforms does it work with?
Mosquitto with ThingsBoard Community Edition today. The certificate design is also compatible with AWS IoT Core.