MQTT Kafka Raspberry Pi ElasticSearch Cluster

Grabs sensor data via arduino uno and wt100, send the data with mqtt to artik, and send it to kafka broker and logstash, monitors via grafana

COMPONENTS Hardware components

Arduino - Arduino UNO

x 1

adafruit - Standard LCD - 16x2 White on Blue

x 1

LCD1602

AdaFruit - Photo cell (CdS photoresistor)

x 1

KY-018

adafruit - DHT22 temperature-humidity sensor

x 1

sparkfun - Raspberry Pi 3 B+ Starter Kit

x 7

3B

WIZnet - W5100 ethernet shield

x 1

Software Apps and online services

mqtt - MQTT

x 1

PROJECT DESCRIPTION

MQTT + Kafka + Raspberry Pi ElasticSearch Cluster by RDIoT

https://www.youtube.com/watch?v=coKGRMDqAJk

* GitHub : https://github.com/rdiot/rdiot-p016

* Parts

- Arduino UNO

- Ethernet W5100 Shield (W5100) [B004]

- LCD1602 I2C (LCD1602) [D016]

- Photo Resistor Module (KY-018) [S002]

- Temperature and humidity DHT22 (DHT22) [S063]

- TPM-300 Air Quality Module (TPM-300) [S092]

- Samsung ARTIK 5 (ARTIK5-V0.5) [B023]

- Raspberry Pi 3 B Model (RASPBERRY-PI-3-B) [B088] x 3ea : for Kafka Cluster

- Raspberry Pi 3 B Model (RASPBERRY-PI-3-B) [B088] x 4ea : for ElasicSearch Cluster

- USB Power Supply

- Ethernet Hub

* Contents

1. Raspberry Pi ElasticSearch Cluster Setup

- version : elasticsearch 2.4.6

- master config
: https://github.com/rdiot/rdiot-p016/blob/master/elasticsearch_es-pi-master-01.yml

- data node config

: https://github.com/rdiot/rdiot-p016/blob/master/elasticsearch_es-pi-data-01.yml

: https://github.com/rdiot/rdiot-p016/blob/master/elasticsearch_es-pi-data-02.yml

: https://github.com/rdiot/rdiot-p016/blob/master/elasticsearch_es-pi-data-03.yml

2. logstash configuration

- https://github.com/rdiot/rdiot-p016/blob/master/logstash-kafka-elasticsearch-airGrade.conf

- https://github.com/rdiot/rdiot-p016/blob/master/logstash-kafka-elasticsearch-cds.conf

- https://github.com/rdiot/rdiot-p016/blob/master/logstash-kafka-elasticsearch-humidity.conf

- https://github.com/rdiot/rdiot-p016/blob/master/logstash-kafka-elasticsearch-temperature.conf

3. start logstash script example

#!/bin/sh

export LS_HEAP_SIZE="500m"

/data1/logstash/logstash/bin/logstash -f /data1/logstash/logstash-kafka-elasticsearch-temperature.conf -l logstash.log &

4. Kafka logstash Monitoring

5. ElasticSearch data node shard

6. Grafana Setup

- Install in Raspberry Pi

$ sudo apt-get install libfontconfig

$ curl -L https://github.com/fg2it/grafana-on-raspberry/releases/download/v4.0.1/grafana_4.0.1-1480722482_armhf.deb -o /tmp/grafana_4.0.1-1480722482_armhf.deb

$ sudo dpkg -i /tmp/grafana_4.0.1-1480722482_armhf.deb

$ sudo vi /etc/grafana/grafana.ini

# The ip address to bind to, empty will bind to all interfaces

http_addr = 192.168.0.20

$ sudo systemctl enable grafana-server

$ sudo systemctl restart grafana-server

- Connect Grafana WebAdmin

http://92.168.0.20:3000

id/pwd : admin/admin

- Add Data Sources : http://192.168.0.20:3000/datasources/new

- edit data source examples

7. Grafana Dashboard

- Kafka-Sensor-Monitoring

: Temperature, Humidity, Cds, AirGrade

Documents

rdiot-p016

Comments Write