ITD - ELK Multi Server Deployment

Procedure :

UAT Server Details

ELK_Kafka_Cluster1_UAT  dcnluelkkafcl01 10.176.16.121
ELK_Elastic_Search1_UAT dcnluelkelse01 10.176.16.122
ELK_Logtash dcnlelklog 10.176.16.123
ELK_Kibana dcnlelkkib 10.176.16.124

PDC Server Details

ELK_Kafka_Node1_PDC	10.176.25.233	dcplekafka01
ELK_Kafka_Node2_PDC	10.176.25.234	dcplekafka02
ELK_Elastic_Search_Master_Node_1	10.176.25.235		dcplelkesmas01
ELK_Elastic_Search_Master_Node_2	10.176.25.236		dcplelkesmas02
ELK_Elastic_Search_Master_Node_3	10.176.25.237		dcplelkesmas03
ELK_Elastic_Search_Slave_Node_1	10.176.25.238		dcplelkesdat01
ELK_Elastic_Search_Slave_Node_2	10.176.25.239		dcplelkesdat02
ELK_Logstash Node 1	10.176.25.240	dcplelklogst01 
ELK_Logstash Node 2	10.176.25.241	dcplelklogst02
ELK_Kibana Node	10.176.25.242	dcplelkkibana

BCP Server Details

ELK_Kafka_Node1_BCP	10.144.24.233	drplekafka01
ELK_Kafka_Node2_BCP	10.144.24.234	drplekafka02
ELK_Elastic_Search_Master_Node_1	10.144.24.235		drplelkesmas01
ELK_Elastic_Search_Master_Node_2	10.144.24.236		drplelkesmas02
ELK_Elastic_Search_Master_Node_3	10.144.24.237		drplelkesmas03
ELK_Elastic_Search_Slave_Node_1	10.144.24.238		drplelkesdat01
ELK_Elastic_Search_Slave_Node_2	10.144.24.239		drplelkesdat02
ELK_Logstash Node 1	10.144.24.240	drplelklogst01 
ELK_Logstash Node 2	10.144.24.241	drplelklogst02 
ELK_Kibana Node	10.144.25.242	drplelkkibana

1. Install/Configure ELK Multi Components Architecture - Elasticsearch, Logstash, Kibana and Kafka all on different nodes

NOTE: Installations/Configurations are same for both DC and DR.

INSTALL KAFKA

Installing pre-requisities,

yum install java-1.8.0-openjdk
yum install java-1.8.0-openjdk-devel

wget https://mirrors.estointernet.in/apache/kafka/2.4.1/kafka_2.12-2.4.1.tgz

tar -xzf kafka_2.12-2.4.1.tgz
cp -r kafka_2.12-2.4.1 /opt/kafka

cd /opt/kafka/
bin/zookeeper-server-start.sh config/zookeeper.properties &
/opt/kafka/bin/kafka-server-start.sh  /opt/kafka/config/server.properties &

yum install net-tools
netstat -ntplu 			// check ports and services.

Enable Kafka at boot time

sudo vi /etc/systemd/system/zookeeper.service

[Unit]
Requires=network.target remote-fs.target
After=network.target remote-fs.target

[Service]
Type=simple
ExecStart=/opt/kafka/bin/zookeeper-server-start.sh /opt/kafka/config/zookeeper.properties
ExecStop=/opt/kafka/bin/zookeeper-server-stop.sh
Restart=on-abnormal

[Install]
WantedBy=multi-user.target

Next, create the systemd service file for kafka:

sudo vi /etc/systemd/system/kafka.service

[Unit]
Requires=zookeeper.service
After=zookeeper.service

[Service]
Type=simple

ExecStart=/bin/sh -c '/opt/kafka/bin/kafka-server-start.sh /opt/kafka/config/server.properties > /opt/kafka/kafka.log 2>&1'
ExecStop=/opt/kafka/bin/kafka-server-stop.sh
Restart=on-abnormal

[Install]
WantedBy=multi-user.target

sudo systemctl start kafka

To ensure that the server has started successfully, check the journal logs for the kafka unit:
 journalctl -u kafka
 sudo systemctl enable kafka

firewall-cmd --permanent --add-port=9092/tcp
firewall-cmd --reload

Configure Filebeat in Kafka machine.

Configuration of Yum for electicsearch

rpm --import https://artifacts.elastic.co/GPG-KEY-elasticsearch

vi /etc/yum.repos.d/elasticsearch.repo
[elasticsearch]
name=Elasticsearch repository for 7.x packages
baseurl=https://artifacts.elastic.co/packages/7.x/yum
gpgcheck=1
gpgkey=https://artifacts.elastic.co/GPG-KEY-elasticsearch
enabled=1
autorefresh=1
type=rpm-md

yum install filebeat
systemctl enable filebeat
systemctl start filebeat

vi /etc/filebeat/filebeat.yml

change 
#output.elasticsearch:
 #hosts: ["localhost:9200"]
output.kafka:
 hosts: ["10.176.16.121:9092"]
 topic: "filebeat"
 codec.json:
   pretty: false

filebeat modules enable system filebeat setup --pipelines --modules system systemctl restart filebeat

2.