--- title: "Quality Assurance with TinyAutomator" url: "https://maker.wiznet.io/scarlet/projects/quality-assurance-with-tinyautomator/" markdown_url: "https://maker.wiznet.io/scarlet/projects/quality-assurance-with-tinyautomator/md" type: "UCC: User Created Content" author: "TinyAutomator" author_url: "https://www.hackster.io/446297/quality-assurance-with-tinyautomator-d590f9" editor: "WIZnet" editor_url: "https://maker.wiznet.io/" original_author: "TinyAutomator" original_url: "https://www.hackster.io/446297/quality-assurance-with-tinyautomator-d590f9" published: "2023-01-05" language: "en" tags: ["Automotive", "PLC", "AIoT"] hardware: ["WIZnet W5500"] likes: 18 views: 2820 comments: 6 source: "WIZnet Makers (https://maker.wiznet.io/)" --- # Quality Assurance with TinyAutomator > Computer Vision provides the relentless electronic eye that will watch thousands of components exit the production line in good shape Original author: TinyAutomator (source: https://www.hackster.io/446297/quality-assurance-with-tinyautomator-d590f9) ## Components - **WIZnet W5500** x 1 ([docs](https://docs.wiznet.io/Product/Chip/Ethernet/W5500)) WIZnet parts: W5500 ([Datasheet](https://docs.wiznet.io/Product/Chip/Ethernet/W5500/datasheet?utm_source=maker&utm_medium=project&utm_campaign=w5500), [product hub](https://maker.wiznet.io/products/w5500/)) ## Article ## **Story** ### **Motivation** The role of quality assurance is to ensure that the final product matches the company's quality standards. Factory end-products usually consist of assemblies of smaller sub-assemblies. By introducing additional nodes of QA in a production line, the overall efficiency of the facility increases. Before starting out building your own Quality Assurance solution, make sure to walk through our [Introduction to TinyAutomator tutorial](https://www.hackster.io/443075/edge-computing-with-tinyautomator-in-industrial-applications-8f3c2f)as it will give you a birds-eye perspective regarding the reasons why we picked it for our solution, what features it has and how to use its basic functions efficiently. ### **Our use case 🏭** In our case, we are monitoring an Injection molding machine that creates polypropylene fittings. Due to various reasons, those fittings may end up with various defects specific to injection molding, like flow lines, sink marks, warping, and others. In this tutorial, we will recognize the short filling of a certain fitting, as seen in the pictures below. Of course, this has to be eliminated from the packaging line, the machine has a basic weighting mechanism, but sometimes it cannot detect some defects on that alone. ![](https://hackster.imgix.net/uploads/attachments/1401850/1643190027879_ZsawZ0agPl.jpeg?auto=compress%2Cformat&w=740&h=555&fit=max) 1 / 2 ### **Hardware requirements 🧰** - **Industrial Shields RPI based PLC** - running a TinyAutomator instance ([Industrial Shields 012002000200)](https://www.industrialshields.com/shop/product/raspberry-plc-ethernet-21-i-os-analog-digital-plus-raspberry-pi-4b-2gb-ram-included-8gb-pslc-msd-w-linux-1917); - **Power supply** - 24V, 1.5A; - **M5Stack UnitV2** - The standalone AI Camera for Edge Computing (SSD202D) TinyML; - **WiFi-enabled relay module** - We've used a [NORVI IIOT ESP32 Based Industrial Controller](https://norvi.lk/industrial-esp32-norvi-iiot/)(NORVI-IIOT-AE02-V). ### **Computer vision 📷Setting up the M5stack UnitV2 camera** First thing first, power up your camera by connecting it to your PC via a USB-C cable. The driver installation varies, depending on your operating system and a thorough setup guide can be found in the [official documentation.](https://docs.m5stack.com/en/unit/unitv2) If you are using a Linux-based machine, there is no driver installation required as the camera is automatically recognized. Once the connection is successful, open your browser of choice and access **10.254.239.1**, the static IP of the camera, which will lead you to the control interface. The user interface gives you access to all the functioning modes of the UnitV2, as well as provides you with a continuous camera stream alongside the serial output of the camera. Once the camera is set up and the training is done, it can be powered through the USB-C cable and run independently, without it being connected to the PC. You can connect to the camera remotely using SSH. In the [M5Stack documentation](https://docs.m5stack.com/en/quick_start/unitv2/config) you can find details about how to access the device as root: ```plaintext ssh m5stack@10.254.239.1 //user: m5stack //pwd: 12345678 //user: root //pwd: 7d219bec161177ba75689e71edc1835422b87be17bf92c3ff527b35052bf7d1f ``` **The Online Classifier** For our application, we will be using the online classifier mode. While using the **Online Classifier**, you can train and classify the objects in the green target frame in real-time, and the feature values obtained from training can be stored on the device for the next boot. For reliable results, you need at least 100 good photos of the features you intend to classify, in all the possible positions. For best results, we recommend having good repeatability of the system that places the objects in front of the camera. **Training the model** ![](https://hackster.imgix.net/uploads/attachments/1401867/training1_b6Vq7F1Css.png?auto=compress%2Cformat&w=740&h=555&fit=max) Under the **Online Classifier** tab, you will notice a checkbox list. This allows us to define the features we wish to identify and train the model accordingly. In the middle of the screen, you can observe the live camera stream and the bounding box in which we will be placing the feature to be recognized and on the right side, you can see the serial output of the camera in JSON format. Because we are training a model from scratch, first click the **reset** button, rename the class with the feature you want to identify, and click the checkbox next to it. Next, place the object inside the green bounding box and press the **train** button. Congrats! You just recorded your first data point. Keep doing this until you have at least 100 good pictures with the feature. ![](https://hackster.imgix.net/uploads/attachments/1401868/training2_9Nrq0AX1sZ.png?auto=compress%2Cformat&w=740&h=555&fit=max) Next, click on the **add** button, rename the new class to something on the lines of **no_defect** and click on the checkbox next to it. Now, we will be training the model to recognize a proper object. Just like before, take at least 100 good photos. Next, click on the **add** button, rename the new class something on the lines of **no_defect** and click on the checkbox next to it. Now, we will be training the model to recognize a proper object. Just like before, take at least 100 good photos. ![](https://hackster.imgix.net/uploads/attachments/1401870/training3_EE8KS4nWn5.png?auto=compress%2Cformat&w=740&h=555&fit=max) Finally, we must take at least 50 good photos of the background against which the objects are presented. We strongly suggest that the background is a static one and if possible, provide a uniformly colored-background (something along the line of a big piece of cardboard). **Model execution** Once the training is done, click **Save and run** and the model is saved on UnitV2. If you wish to further add new samples to your model, simply click on the checkbox corresponding to the feature you wish to train and keep adding data points to it. When you are done, clicking on **save and run** will update your model. ![](https://hackster.imgix.net/uploads/attachments/1403654/def_1kdEQqZL45.png?auto=compress%2Cformat&w=740&h=555&fit=max) 1 / 2 Once the model is running on the UnitV2, the value corresponding to the **best_match** key is the result of the analysis. **Data gathering 📊** ![](https://hackster.imgix.net/uploads/attachments/1403655/qa_tutorial_yguPKyYfwE.png?auto=compress%2Cformat&w=740&h=555&fit=max) Fundamentally, the system we have devised for this use case employs the UnitV2 camera to monitor the production line and, in the case, if a defective item is detected, it sends a message via MQTT to TinyAutomator, where a task gathers the data stream, passes it through a set of rules and sends an MQTT message to a Norvi IIot that triggers a relay linked to an actuator that disposes of the defectuous item. To integrate the M5Stack UnitV2 with TinyAutomator, we had to modify the firmware of the camera to filter out the relevant value from the JSON and send the result via MQTT to a certain topic. The first thing we had to do was to integrate the **Paho MQTT Client** library: ```plaintext import paho.mqtt.client as mqttClient import time def on_connect(client, userdata, flags, rc): if rc == 0: print("Connected to broker") global Connected #Use global variable Connected = True #Signal connection else: print("Connection failed") Connected = False #global variable for the state of the connection broker_address= "192.168.0.163" port = 1883 client = mqttClient.Client("Camera Detection") #create new instance #client.username_pw_set(user, password=password) #set username and password client.on_connect= on_connect #attach function to callback client.connect(broker_address, port=port) #connect to broker client.loop_start() #start the loop ``` We've also declared some variables to be used for the detection: ```plaintext defectDetected = False noDefectDetected = False backgroundDetected = False ``` Here is a very basic detection code based on the labels we added in the previous step that will automatically send a message on state change via MQTT: ```plaintext global backgroundDetected global noGasketDetected global gasketDetected if str(doc["best_match"]) == "background" and not backgroundDetected: client.publish("test-topic",json.dumps({"value",0}) ) backgroundDetected = True defectDetected = False noDefectDetected = False if str(doc["best_match"]) == "no_defect" and not noDefectDetected: client.publish("test-topic",json.dumps({"value":1})) backgroundDetected = False defectDetected = False noDefectDetected = True if str(doc["best_match"]) == "defect" and not defectDetected: client.publish("test-topic", json.dumps({"value":2})) backgroundDetected = False defectDetected = True noDefectDetected = False ``` We also added these three lines in the **server_core.py** file at line 893 for the automatic change of the detection type: ```plaintext protocol.write("{\msg\":\"Waiting 5 seconds for the server to start.\"}\r\n". encode('utf-8')) time.sleep(5) switchFunction("online_classifier","") ``` You can find the complete **server_core.py** file in the [Github repository](https://github.com/Zalmotek/Quality_assurance_with_tiny_automator/blob/main/server_core.py). ### **Actuator control ⚙️** Using the template editor of TinyAutomator, we have created a flow in which it listens on a certain MQTT topic and if the message received corresponds to a defective object, it sends an MQTT message to an actuator. Additionally, every time a defective object is identified, a counter is incremented so we can keep track of the total number of manufacturing failures. --- Source: https://maker.wiznet.io/scarlet/projects/quality-assurance-with-tinyautomator/