---
title: "MQTT Kafka Raspberry Pi ElasticSearch Cluster"
url: "https://maker.wiznet.io/josephsr/projects/mqtt-kafka-raspberry-pi-elasticsearch-cluster/"
markdown_url: "https://maker.wiznet.io/josephsr/projects/mqtt-kafka-raspberry-pi-elasticsearch-cluster/md"
type: "UCC: User Created Content"
author: "rdiot"
author_url: "https://rdiot.tistory.com/339"
editor: "WIZnet"
editor_url: "https://maker.wiznet.io/"
original_author: "rdiot"
original_url: "https://rdiot.tistory.com/339"
published: "2023-12-05"
language: "en"
tags: ["W5100", "Kafka", "MQTT", "Elasticsearch", "logstash"]
hardware: ["Arduino UNO", "Adafruit Standard LCD - 16x2 White on Blue", "Adafruit Photo cell (CdS photoresistor)", "Adafruit DHT22 temperature-humidity sensor", "SparkFun Raspberry Pi 3 B+ Starter Kit", "WIZnet W5100 ethernet shield"]
likes: 0
views: 2662
comments: 0
source: "WIZnet Makers (https://maker.wiznet.io/)"
---

# MQTT Kafka Raspberry Pi ElasticSearch Cluster

> Grabs sensor data via arduino uno and wt100, send the data with mqtt to artik, and send it to kafka broker and logstash, monitors via grafana

Original author: rdiot (source: https://rdiot.tistory.com/339)

## Components

- **Arduino UNO** x 1 ([docs](https://www.farnell.com/datasheets/1682209.pdf))
- **Adafruit Standard LCD - 16x2 White on Blue** x 1: LCD1602 ([docs](https://www.adafruit.com/product/181))
- **Adafruit Photo cell (CdS photoresistor)** x 1: KY-018 ([docs](https://www.adafruit.com/product/161))
- **Adafruit DHT22 temperature-humidity sensor** x 1 ([docs](https://www.adafruit.com/product/385))
- **SparkFun Raspberry Pi 3 B+ Starter Kit** x 7: 3B ([docs](https://learn.sparkfun.com/tutorials/raspberry-pi-3-starter-kit-hookup-guide?_ga=2.146334199.278798874.1657068782-1842012612.1651468433))
- **WIZnet W5100 ethernet shield** x 1 ([docs](https://www.wiznet.hk/en/open-source-hardware/102-w5100s-ethernetshield.html?search_query=W5100+ethernet+shield&results=84))
- Software: **MQTT**

## Documents and links

- [rdiot-p016](https://github.com/rdiot/rdiot-p016) (code)

## Article

MQTT + Kafka + Raspberry Pi ElasticSearch Cluster by RDIoT

![](https://t1.daumcdn.net/cfile/tistory/998D99335A27E5422D)

![](https://t1.daumcdn.net/cfile/tistory/99C7D2335A27EE9617)

![](https://t1.daumcdn.net/cfile/tistory/99767A335A27EEA90E)

![](https://t1.daumcdn.net/cfile/tistory/99E780335A27EEBC39)

[Embedded media](https://www.youtube.com/embed/coKGRMDqAJk?rel=0)

<https://www.youtube.com/watch?v=coKGRMDqAJk>

*** GitHub** : <https://github.com/rdiot/rdiot-p016>

*** Parts**

- Arduino UNO

- [Ethernet W5100 Shield (W5100) \[B004\]](http://rdiot.tistory.com/33)

- [LCD1602 I2C (LCD1602) \[D016\]](http://rdiot.tistory.com/227)

- [Photo Resistor Module (KY-018) \[S002\]](http://rdiot.tistory.com/68)

- [Temperature and humidity DHT22 (DHT22) \[S063\]](http://rdiot.tistory.com/65)

- [TPM-300 Air Quality Module (TPM-300) \[S092\]](http://rdiot.tistory.com/111)

- [Samsung ARTIK 5 (ARTIK5-V0.5) \[B023\]](http://rdiot.tistory.com/13)

- [Raspberry Pi 3 B Model (RASPBERRY-PI-3-B) \[B088\]](http://rdiot.tistory.com/42) x 3ea : for Kafka Cluster

- [Raspberry Pi 3 B Model (RASPBERRY-PI-3-B) \[B088\]](http://rdiot.tistory.com/42) x 4ea : for ElasicSearch Cluster

- USB Power Supply

- Ethernet Hub

*** Contents**

1. Raspberry Pi ElasticSearch Cluster Setup

- version : elasticsearch 2.4.6

- master config
: https://github.com/rdiot/rdiot-p016/blob/master/elasticsearch_es-pi-master-01.yml

- data node config

: https://github.com/rdiot/rdiot-p016/blob/master/elasticsearch_es-pi-data-01.yml

: https://github.com/rdiot/rdiot-p016/blob/master/elasticsearch_es-pi-data-02.yml

: https://github.com/rdiot/rdiot-p016/blob/master/elasticsearch_es-pi-data-03.yml

![](https://t1.daumcdn.net/cfile/tistory/99F4A0335A2C788406)

2. logstash configuration

- https://github.com/rdiot/rdiot-p016/blob/master/logstash-kafka-elasticsearch-airGrade.conf

- https://github.com/rdiot/rdiot-p016/blob/master/logstash-kafka-elasticsearch-cds.conf

- https://github.com/rdiot/rdiot-p016/blob/master/logstash-kafka-elasticsearch-humidity.conf

- https://github.com/rdiot/rdiot-p016/blob/master/logstash-kafka-elasticsearch-temperature.conf

3. start logstash script example

#!/bin/sh

export LS_HEAP_SIZE="500m"

/data1/logstash/logstash/bin/logstash -f /data1/logstash/logstash-kafka-elasticsearch-temperature.conf -l logstash.log &

4. Kafka logstash Monitoring

![](https://t1.daumcdn.net/cfile/tistory/99938B335A27E5AE1B)

5. ElasticSearch data node shard

![](https://t1.daumcdn.net/cfile/tistory/991375335A2C8D0906)

6. Grafana Setup

- Install in Raspberry Pi

$ sudo apt-get install libfontconfig

$ curl -L https://github.com/fg2it/grafana-on-raspberry/releases/download/v4.0.1/grafana_4.0.1-1480722482_armhf.deb -o /tmp/grafana_4.0.1-1480722482_armhf.deb

$ sudo dpkg -i /tmp/grafana_4.0.1-1480722482_armhf.deb

$ sudo vi /etc/grafana/grafana.ini

# The ip address to bind to, empty will bind to all interfaces

http_addr = 192.168.0.20

$ sudo systemctl enable grafana-server

$ sudo systemctl restart grafana-server

- Connect Grafana WebAdmin

http://92.168.0.20:3000

id/pwd : admin/admin

- Add Data Sources : http://192.168.0.20:3000/datasources/new

![](https://t1.daumcdn.net/cfile/tistory/9943E5335A27E64F10)

- edit data source examples

![](https://t1.daumcdn.net/cfile/tistory/99B4F7335A27E69A1E)

7. Grafana Dashboard

- Kafka-Sensor-Monitoring

: Temperature, Humidity, Cds, AirGrade

![](https://t1.daumcdn.net/cfile/tistory/9992C8335A27E6D629)

---

Source: https://maker.wiznet.io/josephsr/projects/mqtt-kafka-raspberry-pi-elasticsearch-cluster/