Wiznet makers

Project Purpose

To reduce food waste and promote various cooking activities by recognizing the contents of the user's refrigerator, recommending possible cooking menus and providing related cooking videos.
 

Main Features and Components

Raspberry Pi and the Pi Camera

Capture the contents of the refrigerator and use object recognition to identify ingredients.

Object recognition system 

Use YOLOv5 to accurately recognize ingredients from images.

OpenAI API 

Recommend possible dishes based on the recognized ingredients.

YouTube crawling module

Automatically search and fetch YouTube links to cooking videos related to the suggested menu.

Telegram notification system

Sends menu recommendations and YouTube video links to users in Telegram messages.

Expected Benefits

• Reduces food waste by efficiently utilizing the remaining ingredients in the refrigerator.
• Helps users easily try different dishes, adding interest and enjoyment to cooking.

Project Implementation Steps

Respberry PI & Pico Setting

Raspberry PI 4

Raspberry PI link

We utilized Raspberry Pi to power our AI model and create data.

From the Raspberry homepage, you can save the Raspbian OS to an SD card, boot it, and install the necessary modules.

Camera Module Setting

You can install the camera module by following the link above.

Hardware connection

• Enable the serial interface: Run sudo raspi-config and select 'Interfacing Options' > 'Serial' to enable hardware serial. 
• Write Python code or other code that can send and receive data through the serial port.

Raspberry PI Pico (W5100S-EVB-Pico)

Pico communicates and receives the data inferred by Raspberry Pi 4, and utilizes Ethernet communication to make menu recommendations using chatGPT.

Connect Ethernet, power and set up the UART to communicate with the Raspberry 4.

Hardware connections

Connect GPIO pin 14 (TX) of the Raspberry Pi 4 to the UART RX pin of the Pico.
Connect GPIO pin 15 (RX) of the Raspberry Pi 4 to the UART TX pin of the Pico.
Connect the GND pins of both boards to each other.

YOLOv5(Object Detection)

YOLO Github

We detect the object refrigerator. Obviously, fast reference frames in object objectification are good for real-time detection, but we'll simply be objectifying and uniquing class values to create data.

1. To create the object detection model, I first created a dataset. Through crawling, I built a dataset of images related to food ingredients or containers. I spent about 5 hours drawing bounding boxes for each object element in the photos.

2. We trained with the YOLOv5n.pt nanomodel. The reason is lightweight. Then, we switched to Onnx TFLite. However, the capacity itself is not much different and the reference speed is not different, so we used the YOLOv5n model.
YOLOv5 Train.ipynb

# pt to onnx to -> TFlite file 
!pip3 install tensorflow==2.3.1
!pip install -r requirements.txt
!python3 export.py --weights result.pt --img 418 --batch 1 --include onnx tflite

3. Object recognition using Raspberry Pi 4. Create a reference using the appropriate code. I used RaspCam, so RaspCam, which I set to '0', which is data, enters the source part and makes an inference. Then, I post-process only the class value and bounding box coordinates of the objects detected through save-txt in the parameter and send the data to Pico.

We've coded the appropriate preprocessing in the main.py file, and we can use it to send data to pico.We've also coded the UART communication.(The author is Korean, so there is code to preprocess the label values in Korean.)

from machine import UART
import time
import sys

uart = UART(0, 115200)

while True:
    if uart.any():
        data = uart.readline()
        print("Received:", data.decode().strip())
        sys.stdout.write(data.decode())
        
 #Data received by Pico
 # Received : [Kimchi,Tofu,Pork]

I've uploaded the detailed code to github.

Chat GPT Recipe

Utilizing UART communication, we communicated from the Raspberry Pi 4 to the Raspberry Pi Pico and assigned the class values from the object expression to the pico.

#GPT prompt
while True:
        if uart.any():
            data = uart.readline().decode("utf-8").strip()  # Read data and decode to string
            prompt = f"{data} Can you recommend a menu I can make with the above ingredients? "
            print(prompt)
            response = send_prompt_to_chatGPT(prompt)
            print(f"Received from UART: {data}")
            print(f"ChatGPT Response: {response}")
        utime.sleep(0.1)

Get the api key value from the Open AI homepage, assign it, and ask GPT a question by plugging in a data list before the prompt "Can you recommend a menu I can make with the above ingredients query".

Result

As shown above, GPT will make menu suggestions with simple recipes.
 

Two-Way Communication

I sent data from the Raspberry Pi to the Pico via UART communication, so I figured I could receive it as well.

Pico code add

starred_section = extract_starred_section(response)
            if starred_section:
                uart.write(starred_section.encode("utf-8")) 
                print(f"Sent to Raspberry Pi 4: {starred_section}")

Raspberry PI 4 code add

if ser.in_waiting >0:
   received_data = ser.readline(0.decode().strip()
   received_data = received_data +"Recipe" # recipe serch

YouTube Crawling 

We'll use the menu above to crawl and search YouTube to collect links.

(I'm Korean, so I'll do the preprocessing accordingly. You can do the preprocessing according to your own language.)

Regular Expressions

The menu suggested by the prompt was in an ambiguous position to crawl, so I asked it to print it out with * in front and back. The crawl will utilize a regular expression to extract data like *{data}*.

I wanted to crawl on the Raspberry Pi Pico, but the library could not be installed due to capacity issues, so I used the two-way communication back to the Raspberry Pi to crawl.😂

Crawling Code

!pip install youtube_dl

import yt_dlp as youtube_dl  

def fetch_youtube_link(query):
    ydl_opts = {
        'default_search': 'ytsearch',
        'quiet': True,
        'format': 'bestaudio/best',
        'noplaylist': True
    }

    with youtube_dl.YoutubeDL(ydl_opts) as ydl:
        result = ydl.extract_info(query, download=False)
        if 'entries' in result:
            video = result['entries'][0]
        else:
            video = result
    
    video_url = f"https://www.youtube.com/watch?v={video['id']}"
    return video_url, video['title']

if __name__ == "__main__":
    query = input("Enter a search query for YouTube: ") # user query
    url, title = fetch_youtube_link(query)
    print(f"Title: {title}")
    print(f"URL: {url}")

Result

This is what the crawl looks like to get the titles and headings I want First, I validated the code on my PC.

Telegram notification system

First, sign up for Telegram and add 'BotFather' as a friend. and /newbot to create a bot and receive a token.

If you see a token like that, your bot has been created. We send the bot a message "hello"

api.telegram.org/bot{"your_tokens"}/getUpdates

If you go to the webpage above, you'll see a JSON file of the message you sent and your information. From there, you can save your ID and check to see if the message was sent successfully.

That's it, you're ready for Telegram. Now it's time to implement it in code!

Python Telemgram Docs

If you're not sure, check out the link above!🔥

Code

import requests

def send_telegram_message(chat_id, text):
    bot_token = "YOUR_BOT_TOKEN"
    send_text = f'https://api.telegram.org/bot{bot_token}/sendMessage?chat_id={chat_id}&text={text}'
    
    response = requests.get(send_text)

    return response.json()

#examples
send_telegram_message("5795970252", "hi , im kdb")  #Put the data received from pico in text

Result

YouTube title and link sent to Telegram. We can easily cook with ingredients we have at home and get recommendations from watching YouTube videos.

Finishing the project

1. If you have GPU resources, your AI model will reference them quickly
2. Utilizing cloud communication will make your project even more complete.
3. There are many good examples of two-way communication.

For this project, I made a lot of effort to apply existing AI technologies. I tried to lighten the model, but it didn't make a big difference, but it seems to be a homework for us to study, and it was a meaningful project because I gained a lot of knowledge about the overall communication theory chip of the hardware.

For data or code questions, please contact hyun030508@icloud.com.

Github Soruce

WIZnetMaker

SImon Github