Wiznet makers

Summary

This project describes a project that calls OpenAI API using WIZnet's W5500-EVB-Pico product and builds a smart home system through it. It is mentioned that cost-effective content can be created by using the W5500 product that supports S2E (Serial to Ethernet) communication at a low cost and linking it with a PC or cloud. Although a lot of AIoT content currently exists in the market, there is no movement combining LLM and boards yet. I would like to introduce how to utilize language models while lowering costs using W5500-EVB-Pico.

In this chapter 1, I will introduce the development environment. And we plan to add more and more features to complete the project.

subject

Hi! I',m Acorn 

I'm trying to use WIZnet's product, W5500-EVB-Pico, by calling the OpenAI API.

As the W5500 is a product that supports S2E at an affordable price, I think that if you implement the service by linking it with a PC or the cloud, you can create very attractive content for the price. In the case of products such as STM32 and SparkFun Edge, AIoT has already been applied in many areas, and many AIoT contents are actually available on the market, but there does not seem to be any movement to merge LLM and boards yet. So, I would like to introduce how you can use a language model really inexpensively using our product, W5500-EVB-Pico.

https://maker.wiznet.io/simons/projects/using%2Dgoogle%2Dgemini%2Dpro%2Dwith%2Dpico/ 

This is a Pico-Gemini connection project developed by Simon. You can call the gemini API through urequest communication in a micropython environment and check the output results on my Thonny interpreter screen. Because urequest is used, Https communication is possible and gemini API calls are possible.

However, as micropython is used to use requests, there is an inevitable disadvantage that it is not compatible with other C++-based external modules. As mentioned earlier, W5500 supports only Serial to Ethernet communication. In other words, https communication is not supported, so a separate SSL-TLS security protocol must be added to communicate directly with Pico.

Therefore, I designed the project to create an intermediate communication bridge by placing an http server in Flask in the middle, and to control sensors and devices by combining Flask with LangChain and processing string parsing in Pico.

This is block diagram converted to Mermaid based on the blueprint.

When you ask openai a question through Serial Monitor in the Arduino IDE, Flask connects the Lang Chain and uses Tagging to determine the response. By specifying the minimum and maximum values ​​in the schema, you set up a defense wall to prevent other values ​​from being transmitted when passing values ​​to Pico. Afterwards, data is sent only as strings and commas (,) to make it easier to parse.

led_Control_Schema = {
    "properties": {
        "brightness of lights": {
            "type": "integer",
            "minimum": 0,  # 최소값을 0으로 설정
            "maximum": 255,  # 최대값을 255로 설정
            "description": "This field specifies the brightness of lights, from 0 (completely off) to 255 (full brightness). Be sure to use only multiples of 7 to give users a variety of values. Consider the user's situation and emotions to come up with an appropriate brightness. Let's Think about step by step",
        },
    },
    "required": ["brightness of lights"]
}

Once the value has been received in Pico, parse the data to identify the device for each sensor or device, and control the device accordingly. Here, we tried to control only the LED and temperature and humidity sensors. (Other sensors will continue to be added.)

void post_to_APIServer(String device, String myQuery, Container &container) {
  String deviceData = "POST /" + device + " HTTP/1.1";
  String postData = "{\"query\": \"" + myQuery + "\"}";
  String myResponse = ""; // 서버로부터의 응답을 저장할 변수
  
  if (client.connect(server, 8080)) {
    Serial.println("Connected to server");
    // HTTP 요청 전송
    client.println(deviceData);
    client.println("Host: 192.168.0.5:8080");
    client.println("Content-Type: application/json");
    client.println("Connection: close");
    client.print("Content-Length: ");
    client.println(postData.length());
    client.println();
    client.println(postData);

    // 서버로부터의 응답 읽기
    while (client.connected() || client.available()) {
      while (client.available()) {
        char c = client.read();
        myResponse += c; // 서버로부터 읽은 문자를 응답 문자열에 추가
      }
    }
    client.stop();
    
    // HTTP 응답에서 본문만 추출
    int headerEndIndex = myResponse.indexOf("\r\n\r\n") + 4;
    String responseBody = myResponse.substring(headerEndIndex);
    responseBody.trim(); // 본문의 시작과 끝의 공백 제거
    responseBody.replace("\"", "");

    // 본문 분석 및 구조체에 값 할당
    if (responseBody.startsWith("led,")) {
      int index = responseBody.indexOf(',');
      int bright = responseBody.substring(index + 1).toInt();
      container.ledStruct.bright = bright;
    } else if (responseBody.startsWith("dht,")) {
      int firstIndex = responseBody.indexOf(',');
      int secondIndex = responseBody.indexOf(',', firstIndex + 1);
      float latitude = responseBody.substring(firstIndex + 1, secondIndex).toFloat();
      float longitude = responseBody.substring(secondIndex + 1).toFloat();
      container.dhtStruct.latitude = latitude;
      container.dhtStruct.longitude = longitude;
    } else {
      Serial.println("Unknown sensor type in response.");
    }
  } else {
    Serial.println("Connection failed");
  }
}

“Can you please turn off the light?” If you input "Turn on the lights softly," or "Turn on the lights in a moody atmosphere," the light of the pico built-in LED will turn off, and if you input things like "Turn on the lights with a moody atmosphere," the lights will turn on at a medium value, and if you say things like "Turn on the lights brightly," the lights will turn on at the maximum level. It's possible.

The current situation is at the level of a toy project, which may be somewhat disappointing from a data structure perspective and a project system flow perspective, but we will continue to make improvements, add content, and continue with the goal of implementing a system that can apply chatbots with WIZnet products. I'm planning to go out.

Conclusion

Once the server setup and tagging development using LangChain are completed, the next step involves using POST communication to receive values from Pico and display them, effectively completing the process. In this setup, the Flask server acts as an intermediary, facilitating data transmission between Pico and LangChain. User requests are sent to the Flask server, which then forwards these requests to LangChain for processing. LangChain analyzes the requested task, performs the necessary tagging, and sends the results back to the Flask server.

At this point, the server sends the data to Pico via a POST request. Pico receives this data, parses it, and based on the parsed data, performs specific actions or reads sensor values and sends them back to the server. For example, if a user requests to "turn on the light," this request is passed through the Flask server to LangChain, which analyzes it and sends an appropriate lighting control command to Pico. Pico then executes this command to control the lighting and informs the server that the action has been completed.

Consequently, this system includes the process of receiving and processing user requests and displaying the results to the user. From server setup to data processing via LangChain, and actual device control through Pico, all steps are interconnected, allowing users to control devices without complex coding. This signifies the completion of a prototype project that enables the construction of a powerful yet cost-effective  system through a No-Coding approach.