ITD - ELK Multi Server Deployment

Procedure :

UAT Server Details

ELK_Kafka_Cluster1_UAT  dcnluelkkafcl01 10.176.16.121
ELK_Elastic_Search1_UAT dcnluelkelse01 10.176.16.122
ELK_Logtash dcnlelklog 10.176.16.123
ELK_Kibana dcnlelkkib 10.176.16.124

PDC Server Details

ELK_Kafka_Node1_PDC	10.176.25.233	dcplekafka01
ELK_Kafka_Node2_PDC	10.176.25.234	dcplekafka02
ELK_Elastic_Search_Master_Node_1	10.176.25.235		dcplelkesmas01
ELK_Elastic_Search_Master_Node_2	10.176.25.236		dcplelkesmas02
ELK_Elastic_Search_Master_Node_3	10.176.25.237		dcplelkesmas03
ELK_Elastic_Search_Slave_Node_1	10.176.25.238		dcplelkesdat01
ELK_Elastic_Search_Slave_Node_2	10.176.25.239		dcplelkesdat02
ELK_Logstash Node 1	10.176.25.240	dcplelklogst01 
ELK_Logstash Node 2	10.176.25.241	dcplelklogst02
ELK_Kibana Node	10.176.25.242	dcplelkkibana

BCP Server Details

ELK_Kafka_Node1_BCP	10.144.24.233	drplekafka01
ELK_Kafka_Node2_BCP	10.144.24.234	drplekafka02
ELK_Elastic_Search_Master_Node_1	10.144.24.235		drplelkesmas01
ELK_Elastic_Search_Master_Node_2	10.144.24.236		drplelkesmas02
ELK_Elastic_Search_Master_Node_3	10.144.24.237		drplelkesmas03
ELK_Elastic_Search_Slave_Node_1	10.144.24.238		drplelkesdat01
ELK_Elastic_Search_Slave_Node_2	10.144.24.239		drplelkesdat02
ELK_Logstash Node 1	10.144.24.240	drplelklogst01 
ELK_Logstash Node 2	10.144.24.241	drplelklogst02 
ELK_Kibana Node	10.144.25.242	drplelkkibana

1. Install/Configure ELK Multi Components Architecture - Elasticsearch, Logstash, Kibana and Kafka all on different nodes

NOTE: Installations/Configurations are same for both DC and DR.

INSTALL KAFKA

Installing pre-requisities,

yum install java-1.8.0-openjdk
yum install java-1.8.0-openjdk-devel

wget https://mirrors.estointernet.in/apache/kafka/2.4.1/kafka_2.12-2.4.1.tgz

tar -xzf kafka_2.12-2.4.1.tgz
cp -r kafka_2.12-2.4.1 /opt/kafka

cd /opt/kafka/
bin/zookeeper-server-start.sh config/zookeeper.properties &
/opt/kafka/bin/kafka-server-start.sh  /opt/kafka/config/server.properties &

yum install net-tools
netstat -ntplu 			// check ports and services.

Enable Kafka at boot time

sudo vi /etc/systemd/system/zookeeper.service

[Unit]
Requires=network.target remote-fs.target
After=network.target remote-fs.target

[Service]
Type=simple
ExecStart=/opt/kafka/bin/zookeeper-server-start.sh /opt/kafka/config/zookeeper.properties
ExecStop=/opt/kafka/bin/zookeeper-server-stop.sh
Restart=on-abnormal

[Install]
WantedBy=multi-user.target

Next, create the systemd service file for kafka:

sudo vi /etc/systemd/system/kafka.service

[Unit]
Requires=zookeeper.service
After=zookeeper.service

[Service]
Type=simple

ExecStart=/bin/sh -c '/opt/kafka/bin/kafka-server-start.sh /opt/kafka/config/server.properties > /opt/kafka/kafka.log 2>&1'
ExecStop=/opt/kafka/bin/kafka-server-stop.sh
Restart=on-abnormal

[Install]
WantedBy=multi-user.target

sudo systemctl start kafka

To ensure that the server has started successfully, check the journal logs for the kafka unit:
 journalctl -u kafka
 sudo systemctl enable kafka

firewall-cmd --permanent --add-port=9092/tcp
firewall-cmd --reload

Configure Filebeat in Kafka machine.

Configuration of Yum for electicsearch

rpm --import https://artifacts.elastic.co/GPG-KEY-elasticsearch

vi /etc/yum.repos.d/elasticsearch.repo
[elasticsearch]
name=Elasticsearch repository for 7.x packages
baseurl=https://artifacts.elastic.co/packages/7.x/yum
gpgcheck=1
gpgkey=https://artifacts.elastic.co/GPG-KEY-elasticsearch
enabled=1
autorefresh=1
type=rpm-md

yum install filebeat
systemctl enable filebeat
systemctl start filebeat

vi /etc/filebeat/filebeat.yml

change 
#output.elasticsearch:
 #hosts: ["localhost:9200"]
output.kafka:
 hosts: ["10.176.25.135:9092"]
 topic: "filebeat"
 codec.json:
   pretty: false

filebeat modules enable system
filebeat setup --pipelines --modules system
systemctl restart filebeat

INSTALL ELASTICSEARCH

rpm --import https://artifacts.elastic.co/GPG-KEY-elasticsearch

vi /etc/yum.repos.d/elasticsearch.repo
 [elasticsearch]
 name=Elasticsearch repository for 7.x packages
 baseurl=https://artifacts.elastic.co/packages/7.x/yum
 gpgcheck=1
 gpgkey=https://artifacts.elastic.co/GPG-KEY-elasticsearch
 enabled=1
 autorefresh=1
 type=rpm-md

yum install java-1.8.0-openjdk
yum install java-1.8.0-openjdk-devel

yum install elasticsearch
vi /etc/elasticsearch/elasticsearch.yml
 network.host: 0.0.0.0
 discovery.seed_hosts: ["10.176.25.135", "127.0.0.1"]                ## Just one Node in UAT ###
 . . .
 Save and close elasticsearch.yml. 
 Change Value of java Heap Size in /etc/elasticsearch/jvm.options
 -Xms1g
 -Xmx1g
to 
 #-Xms1g
 #-Xmx1g
 -Xms4g
 -Xmx4g

 Then, start the Elasticsearch service with systemctl:
 systemctl start elasticsearch
 Next, run the following command to enable Elasticsearch to start up every time your server boots:
 systemctl enable elasticsearch
 You can test whether your Elasticsearch service is running by sending an HTTP request:

 curl -X GET "localhost:9200"

INSTALL KIBANA

rpm --import https://artifacts.elastic.co/GPG-KEY-elasticsearch

vi /etc/yum.repos.d/elasticsearch.repo
[elasticsearch]
name=Elasticsearch repository for 7.x packages
baseurl=https://artifacts.elastic.co/packages/7.x/yum
gpgcheck=1
gpgkey=https://artifacts.elastic.co/GPG-KEY-elasticsearch
enabled=1
autorefresh=1
type=rpm-md

yum install java-1.8.0-openjdk
yum install java-1.8.0-openjdk-devel

yum install kibana

vi /etc/kibana/kibana.yml and change 
elasticsearch.hosts: ["http://dcplelkesmas01:9200"]

sudo systemctl enable kibana
sudo systemctl start kibana

Nginx installation on Kibana

wget https://dl.fedoraproject.org/pub/epel/epel-release-latest-7.noarch.rpm
yum install epel-release-latest-7.noarch.rpm
yum install nginx
systemctl status nginx
systemctl enable  nginx
systemctl start  nginx
Now nginx as reverse proxy should be running on the Server . 

Set the Selinux and Firewall setting 
setsebool httpd_can_network_connect 1 -P
firewall-cmd --zone=public --permanent --add-service=http
firewall-cmd --reload 

Then check the configuration for syntax errors:

nginx -t
systemctl restart nginx //to restart