1 Introduction

In this era of smart hardware and the Internet of Things, MicroPython and Raspberry Pi PICO are leading the new trend of embedded development with their unique advantages. MicroPython, as a streamlined and optimized Python 3 language, provides efficient development and easy debugging for microcontrollers and embedded devices.

 When we combine it with the WIZnet W5100S/W5500 network module, the development potential of MicroPython and Raspberry Pi PICO is further amplified. Both modules have built-in TCP/IP protocol stacks, making it easier to implement network connections on embedded devices. Whether it is data transmission, remote control, or building IoT applications, they provide powerful support.

 In this chapter, we will take WIZnet W5100S as an example to use MicroPython development method to perform MQTT loopback communication example.

2. Related network information

2.1 Introduction

MQTT is a standards-based message transfer protocol, or a set of rules for communication between machines. Smart sensors, wearables, and other Internet of Things (IoT) devices often need to send and receive data over resource-constrained, bandwidth-limited networks. These IoT devices use MQTT for data transmission as it is easy to implement and can transmit IoT data efficiently. MQTT supports message transmission from device to cloud and from cloud to device.

2.2 Working principle

MQTT communication includes the following four steps, of which the subscription and publishing steps may be multiple times:

1. The client connects to the MQTT server: The client establishes a connection with the MQTT server through the TCP/IP protocol. During the connection process, the client needs to provide the client identification (ClientId), as well as user name, password and other information to the server. If the connection is successful, the client and server will establish a session and start data transmission.

2. Subscription to topics: After establishing a connection, the client can subscribe to topics of interest in order to receive messages published on these topics. The subscription operation can be implemented through the SUBSCRIBE message in the MQTT protocol. The topic to be subscribed and the QoS (Quality of Service) level need to be specified in the message. The server will return a SUBACK message to indicate success or failure of the subscription.

3. Publish messages: After establishing a connection, the client can publish messages to the specified topic. The publishing operation can be implemented through the PUBLISH message in the MQTT protocol. The topic and message content to be published need to be specified in the message. The server will return a PUBACK message to indicate the success or failure of message publishing.

4. Disconnect: The client can actively disconnect from the MQTT server. Before disconnecting, the client needs to send a DISCONNECT message to the server. The server will return a DISCONNECT message, indicating that the connection was successfully disconnected.

2.3 Advantages

Lightweight and efficient: The implementation of MQTT on IoT devices requires minimal resources and therefore can be used even on small microcontrollers. For example, the smallest MQTT control message can be only two data bytes. MQTT message headers are also small, which optimizes network bandwidth.

Scalable: MQTT implementation requires minimal code and consumes very little power in operation. The protocol also has built-in capabilities to support communication with a large number of IoT devices. Therefore, you can implement the MQTT protocol and connect with millions of such devices.

Reliable: Many IoT devices connect via unreliable cellular networks with low bandwidth and high latency. MQTT has built-in capabilities that reduce the time it takes for IoT devices to reconnect to the cloud. It also defines three different quality of service levels to ensure reliability for IoT use cases - at most once (0), at least once (1), and exactly once (2).

Security: MQTT enables developers to easily encrypt messages and authenticate devices and users using modern authentication protocols such as OAuth, TLS1.3, customer-managed certificates, and more.

Good support: Many languages ​​such as Python provide extensive support for MQTT protocol implementation. Therefore, developers can quickly implement it in any type of application with minimal coding.

Publish-Subscribe Pattern: One-to-many communication can be easily achieved.

2.4 Application

Internet of Things M2M communication, Internet of Things big data collection: MQTT protocol can achieve efficient communication between devices and real-time collection of large-scale data.

Android message push, WEB message push: MQTT protocol can be used to implement real-time message push on Android and Web platforms.

Smart hardware, smart furniture, smart appliances: MQTT protocol can be used for the control and management of smart hardware, smart furniture, smart appliances and other equipment.

Internet of Vehicles communication, electric vehicle station pile collection: In the field of Internet of Vehicles, the MQTT protocol can be used for communication between vehicles and infrastructure, as well as data collection at electric vehicle charging stations.

Smart cities, telemedicine, and distance education: In the fields of smart city construction, telemedicine, and distance education, the MQTT protocol can achieve efficient communication between devices and real-time collection of large-scale data.

Electric power, oil and energy and other industry markets: In the electric power, oil and energy and other industries, the MQTT protocol can be used for remote monitoring and data collection of equipment.

In general, the MQTT protocol is widely used in the field of Internet of Things because of its lightweight, simple, open and easy-to-implement characteristics.

3. WIZnet Ethernet chip

WIZnet mainstream hardware protocol stack Ethernet chip parameter comparison

W5100S/W6100 supports 8-bit data bus interface, and the network transmission speed will be better than W5500.

W6100 supports IPV6 and is compatible with W5100S hardware. If users who already use W5100S need to support IPv6, they can be Pin to Pin compatible.

W5500 has more Sockets and send and receive buffers than W5100S

Compared with the software protocol stack, WIZnet's hardware protocol stack Ethernet chip has the following advantages:

Hardware TCP/IP protocol stack: WIZnet's hardware protocol stack chip provides a hardware-implemented TCP/IP protocol stack. This hardware-implemented protocol stack has better performance and stability than software-implemented protocol stacks.

No additional embedded system software stack and memory resources required: Since all Ethernet transmit and receive operations are handled by the independent Ethernet controller, no additional embedded system software stack and memory resources are required.

Resistant to network environment changes and DDoS attacks: Compared with software TCP/IP protocol stacks that are susceptible to network environment changes and DDoS attacks, hardware protocol stack chips can provide more stable Ethernet performance.

Suitable for low-specification embedded systems: Even in low-specification embedded systems, hardware protocol stack chips using WIZnet can show more efficient Internet application operating performance than high-specification systems using software TCP/IP protocol stacks.

4. MQTT communication example explanation and use

4.1 Program flow chart

4.2 Test preparation

Software:

Thonny

MQTTX

Hardware:

W5100S IO module + RP2040 Raspberry Pi Pico development board or WIZnet W5100S-EVB-Pico development board

Micro USB interface data cable

cable

4.3 Connection method

Connect to PC USB port via data cable

When using W5100S/W5500 IO module to connect to RP2040

RP2040 GPIO 16 <----> W5100S/W5500 MISO

RP2040 GPIO 17 <----> W5100S/W5500 CS

RP2040 GPIO 18 <----> W5100S/W5500 SCK

RP2040 GPIO 19 <----> W5100S/W5500 MOSI

RP2040 GPIO 20 <----> W5100S/W5500 RST

Directly connect to the PC network port through a network cable (or: both the PC and the device are connected to the switch or router LAN port through a network cable)

4.4 Related code

We open the mqtt.py file directly.

Step one: You can see that SPI is initialized in the w5x00_init() function. And register the spi-related pins and reset pins into the library. The subsequent step is to activate the network and use DHCP to configure the network address information. When DHCP fails, configure the static network address information. When the configuration is not successful, the information about the network address-related registers will be printed out, which can help us better troubleshoot the problem.

Step 2: Try to connect to the MQTT server. If the connection fails, enter the reset procedure.

Step 3: Subscribe to the topic, bind the message callback function, and start the timer to keep alive.

Step 4: Wait for the message to be received, and perform loopback processing if the message is received.

In addition, the umqttsimple.py library needs to be saved to the development board, otherwise it will cause running errors.

#mqtt.py file
from umqttsimple import MQTTClient
from usocket import socket
from machine import Pin,SPI,Timer
import network
import time
import json
​
#mqtt config
mqtt_params = {}
mqtt_params['url'] = 'test.mosquitto.org'
mqtt_params['port'] = 1883
mqtt_params['clientid'] = 'W5100S'
mqtt_params['pubtopic'] = '/W5100S/pub'
mqtt_params['subtopic'] = '/W5100S/sub'
mqtt_params['pubqos'] = 0
mqtt_params['subqos'] = 0
​
timer_1s_count =  0
tim = Timer()
client = None
​
"""
W5x00 chip initialization.
 
param: None
returns: None
​
"""
def w5x00_init():
    spi=SPI(0,2_000_000, mosi=Pin(19),miso=Pin(16),sck=Pin(18))
    nic = network.WIZNET5K(spi,Pin(17),Pin(20)) #spi,cs,reset pin
    nic.active(True)
    
    try:
        #DHCP
        print("\r\nConfiguring DHCP")
        nic.ifconfig('dhcp')
    except:
        #None DHCP
        print("\r\nDHCP fails, use static configuration")
        nic.ifconfig(('192.168.1.20','255.255.255.0','192.168.1.1','8.8.8.8'))#Set static network address information
    
    #Print network address information
    print("IP         :",nic.ifconfig()[0])
    print("Subnet Mask:",nic.ifconfig()[1])
    print("Gateway    :",nic.ifconfig()[2])
    print("DNS        :",nic.ifconfig()[3],"\r\n")
    
    #If there is no network connection, the register address information is printed
    while not nic.isconnected():
        time.sleep(1)
        print(nic.regs())
​
"""
Subscribe to the topic message callback function. This function is entered when a message is received from a subscribed topic.
 
param1: The topic on which the callback is triggered
param2: Message content
returns: None
​
"""
def sub_cb(topic, msg):
    topic = topic.decode('utf-8')
    msg = msg.decode('utf-8')
    if topic == mqtt_params['subtopic']:
        global client
        print("\r\ntopic:",topic,"\r\nrecv:", msg)
        client.publish(mqtt_params['pubtopic'],msg,qos = mqtt_params['pubqos'])
        print('\r\ntopic:',mqtt_params['pubtopic'],'\r\nsend:',msg)
    
"""
Connect to the MQTT server.
 
param: None
returns: None
​
"""
def mqtt_connect():
    client = MQTTClient(mqtt_params['clientid'], mqtt_params['url'], mqtt_params['port'],keepalive=60)
    
    client.connect()
    print('Connected to %s MQTT Broker'%(mqtt_params['url']))
    return client
​
"""
Connection error handler.
 
param: None
returns: None
​
"""
def reconnect():
    print('Failed to connected to Broker. Reconnecting...')
    time.sleep(5)
    machine.reset()
​
"""
1-second timer callback function.
 
param1: class timer
returns: None
​
"""
def tick(timer):
    global timer_1s_count
    global client
    timer_1s_count += 1
    if timer_1s_count >= 30:
           timer_1s_count = 0 
           client.ping()
​
"""
Subscribe to Topics.
 
param: client object
returns: None
​
"""
def subscribe(client):
    client.set_callback(sub_cb)
    client.subscribe(mqtt_params['subtopic'],mqtt_params['subqos'])
    print('subscribed to %s'%mqtt_params['subtopic'])
 
    
def main():
    global client
    print("WIZnet chip MQTT example")
    w5x00_init()
    
    try: 
        client = mqtt_connect()
    except OSError as e:
        reconnect()
        
    subscribe(client)
    
    tim.init(freq=1, callback=tick)
    
    while True:
        client.wait_msg()
        
        
    client.disconnect()
​
if __name__ == "__main__":
    main()

#umqttsimple.py file
import usocket as socket
import ustruct as struct
from ubinascii import hexlify
​
​
class MQTTException(Exception):
    pass
​
​
class MQTTClient:
    def __init__(
        self,
        client_id,
        server,
        port=0,
        user=None,
        password=None,
        keepalive=0,
        ssl=False,
        ssl_params={},
    ):
        if port == 0:
            port = 8883 if ssl else 1883
        self.client_id = client_id
        self.sock = None
        self.server = server
        self.port = port
        self.ssl = ssl
        self.ssl_params = ssl_params
        self.pid = 0
        self.cb = None
        self.user = user
        self.pswd = password
        self.keepalive = keepalive
        self.lw_topic = None
        self.lw_msg = None
        self.lw_qos = 0
        self.lw_retain = False
​
    def _send_str(self, s):
        self.sock.write(struct.pack("!H", len(s)))
        self.sock.write(s)
​
    def _recv_len(self):
        n = 0
        sh = 0
        while 1:
            b = self.sock.read(1)[0]
            n |= (b & 0x7F) << sh
            if not b & 0x80:
                return n
            sh += 7
​
    def set_callback(self, f):
        self.cb = f
​
    def set_last_will(self, topic, msg, retain=False, qos=0):
        assert 0 <= qos <= 2
        assert topic
        self.lw_topic = topic
        self.lw_msg = msg
        self.lw_qos = qos
        self.lw_retain = retain
​
    def connect(self, clean_session=True):
        self.sock = socket.socket()
        addr = socket.getaddrinfo(self.server, self.port)[0][-1]
        self.sock.connect(addr)
        if self.ssl:
            import ussl
​
            self.sock = ussl.wrap_socket(self.sock, **self.ssl_params)
        premsg = bytearray(b"\x10\0\0\0\0\0")
        msg = bytearray(b"\x04MQTT\x04\x02\0\0")
​
        sz = 10 + 2 + len(self.client_id)
        msg[6] = clean_session << 1
        if self.user is not None:
            sz += 2 + len(self.user) + 2 + len(self.pswd)
            msg[6] |= 0xC0
        if self.keepalive:
            assert self.keepalive < 65536
            msg[7] |= self.keepalive >> 8
            msg[8] |= self.keepalive & 0x00FF
        if self.lw_topic:
            sz += 2 + len(self.lw_topic) + 2 + len(self.lw_msg)
            msg[6] |= 0x4 | (self.lw_qos & 0x1) << 3 | (self.lw_qos & 0x2) << 3
            msg[6] |= self.lw_retain << 5
​
        i = 1
        while sz > 0x7F:
            premsg[i] = (sz & 0x7F) | 0x80
            sz >>= 7
            i += 1
        premsg[i] = sz
​
        self.sock.write(premsg, i + 2)
        self.sock.write(msg)
        # print(hex(len(msg)), hexlify(msg, ":"))
        self._send_str(self.client_id)
        if self.lw_topic:
            self._send_str(self.lw_topic)
            self._send_str(self.lw_msg)
        if self.user is not None:
            self._send_str(self.user)
            self._send_str(self.pswd)
        resp = self.sock.read(4)
        assert resp[0] == 0x20 and resp[1] == 0x02
        if resp[3] != 0:
            raise MQTTException(resp[3])
        return resp[2] & 1
​
    def disconnect(self):
        self.sock.write(b"\xe0\0")
        self.sock.close()
​
    def ping(self):
        self.sock.write(b"\xc0\0")
​
    def publish(self, topic, msg, retain=False, qos=0):
        pkt = bytearray(b"\x30\0\0\0")
        pkt[0] |= qos << 1 | retain
        sz = 2 + len(topic) + len(msg)
        if qos > 0:
            sz += 2
        assert sz < 2097152
        i = 1
        while sz > 0x7F:
            pkt[i] = (sz & 0x7F) | 0x80
            sz >>= 7
            i += 1
        pkt[i] = sz
        # print(hex(len(pkt)), hexlify(pkt, ":"))
        self.sock.write(pkt, i + 1)
        self._send_str(topic)
        if qos > 0:
            self.pid += 1
            pid = self.pid
            struct.pack_into("!H", pkt, 0, pid)
            self.sock.write(pkt, 2)
        self.sock.write(msg)
        if qos == 1:
            while 1:
                op = self.wait_msg()
                if op == 0x40:
                    sz = self.sock.read(1)
                    assert sz == b"\x02"
                    rcv_pid = self.sock.read(2)
                    rcv_pid = rcv_pid[0] << 8 | rcv_pid[1]
                    if pid == rcv_pid:
                        return
        elif qos == 2:
            assert 0
​
    def subscribe(self, topic, qos=0):
        assert self.cb is not None, "Subscribe callback is not set"
        pkt = bytearray(b"\x82\0\0\0")
        self.pid += 1
        struct.pack_into("!BH", pkt, 1, 2 + 2 + len(topic) + 1, self.pid)
        # print(hex(len(pkt)), hexlify(pkt, ":"))
        self.sock.write(pkt)
        self._send_str(topic)
        self.sock.write(qos.to_bytes(1, "little"))
        while 1:
            op = self.wait_msg()
            if op == 0x90:
                resp = self.sock.read(4)
                # print(resp)
                assert resp[1] == pkt[2] and resp[2] == pkt[3]
                if resp[3] == 0x80:
                    raise MQTTException(resp[3])
                return
​
    # Wait for a single incoming MQTT message and process it.
    # Subscribed messages are delivered to a callback previously
    # set by .set_callback() method. Other (internal) MQTT
    # messages processed internally.
    def wait_msg(self):
        res = self.sock.read(1)
#         self.sock.setblocking(True)
        if res is None:
            return None
        if res == b"":
            raise OSError(-1)
        if res == b"\xd0":  # PINGRESP
            sz = self.sock.read(1)[0]
            assert sz == 0
            return None
        op = res[0]
        if op & 0xF0 != 0x30:
            return op
        sz = self._recv_len()
        topic_len = self.sock.read(2)
        topic_len = (topic_len[0] << 8) | topic_len[1]
        topic = self.sock.read(topic_len)
        sz -= topic_len + 2
        if op & 6:
            pid = self.sock.read(2)
            pid = pid[0] << 8 | pid[1]
            sz -= 2
        msg = self.sock.read(sz)
        self.cb(topic, msg)
        if op & 6 == 2:
            pkt = bytearray(b"\x40\x02\0\0")
            struct.pack_into("!H", pkt, 2, pid)
            self.sock.write(pkt)
        elif op & 6 == 4:
            assert 0
        return op
​
    # Checks whether a pending message from server is available.
    # If not, returns immediately with None. Otherwise, does
    # the same processing as wait_msg.
    def check_msg(self):
#         self.sock.setblocking(False)
        return self.wait_msg()

4.5 Burning Verification

To test the Ethernet examples, the development environment must be configured to use a Raspberry Pi Pico.

Required development environment

Thonny

If you must compile MicroPython, you must use a Linux or Unix environment.

Step 1: Copy the program into Thonny and select the environment as Raspberry Pi Pico.

Step 2: Save the umqttsimple.py library file to the development board.

Step 3: Run the program, open MQTTX, connect to the same server, then subscribe to the development board's publishing topic, and the publishing topic is the development board's subscription topic.

Step 4: Send a message in MQTTX to observe the loopback test effect.

Note: Because MicroPython's print function enables stdout buffering, sometimes the content will not be printed out immediately.

5. Precautions

If you use WIZnet's W5500 to implement the examples in this chapter, you only need to burn the firmware of the W5500 and run the example program.