Observibility Cluster Setup
Auto-generated from the uploaded PDF Observibility_Cluster_Setup.pdf. This is an extracted-text rendering for searchability; see the original PDF for exact formatting, diagrams, tables, and images.
Infosys - DGARM
Observability Cluster Setup Document
Last Updated:- 5th Feb, 2026
By
Prashant Kashyap
Document Control
Document No. Doc_1
Document Version 1
Prepared By Prashant Kashyap
Release Date 5th Feb 2026
Table of Contents
Infosys - DGARM ......................................................................................................................... 1
Document Control .......................................................................................................................2
Installation of Elasticsearch ....................................................................................................... 4
Kibana Installation ....................................................................................................................12
APM Installation on ELK setup .................................................................................................. 15
Fleet Installation in Elastic Cluster ........................................................................................... 18
Installation of Elasticsearch
1. Enable subscription-manager register
subscription-manager register --username actual_user --password
actual_pass
Note :- After installation please subscription-manager unregister
Command :- subscription-manager unregister
2 . Install java on node (server), check the java version and Installation it
yum install java*
java -version
3. make Elasticsearch as user
useradd elasticsearch
groupadd elasticsearch
4. create data directory as per elasticsearch node functionality
mkdir /data // for normal node master and data
mkdir /ml-data // for ml node
5. create ELASTIC Directory for installation work
mkdir ELASTIC
cd ELASTIC
# Download Elasticsearch rpm package
wget
https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-
9.1.5-x86_64.rpm
# Install Elasticsearch
sudo rpm -ivh elasticsearch-9.1.5-x86_64.rpm
6. take backup of original elasticsearch.yml file before make any changes
cp /etc/elasticsearch/elasticsearch.yml
/etc/elasticsearch/elasticsearch_yml.backup
7. Change ownership of elk directories and owner
chown -R root:elasticsearch /etc/elasticsearch
chown -R root:elasticsearch /data
chown -R root:elasticsearch /var/log/elasticsearch
8. Put necessary node certificates in elasticsearch certs folder
cd /etc/elasticsearch/certs
Note :- .cnf, .dgarm.gov.in.crt, fullchain.crt and key will different as per node
9. Enter host entry in hosts file
Note : host entry wil be different for DR and Prod as per cluster design or setup
nano /etc/hosts
All ELK Cluster node entry
10. go to elasticserch folder and make changes in yml file
nano elasticsearch.yml
11. start 9200 and 9300 port
systemctl start firewalld
firewall-cmd --permanent --add-port=9200/tcp
firewall-cmd --permanent --add-port=9300/tcp
systemctl reload firewalld
12. start the elasticsearch service
sudo systemctl daemon-reload
sudo systemctl enable elasticsearch
sudo systemctl start elasticsearch
sudo systemctl restart elasticsearch
sudo systemctl status elasticsearch
sudo systemctl stop elasticsearch
13. Commands for check the error of Elasticsearch
sudo journalctl -u elasticsearch.service
sudo journalctl -u elasticsearch.service -f
14. Elasticsearch Instance
ps aux | grep elasticsearch
sudo netstat -plnt | grep 9200
15. command to change password for Elasticsearch user i custom form
/usr/share/bin/elasticsearch-reset-password -u elastic -i // set
custom password
Note:- The same steps are followed for all other Elasticsearch nodes.
Overview for Elasticsearch nodes:- 3 master nodes, 2 hot data nodes, 4
warm data nodes, 2 ml nodes.
Logstash Installation
1. Enable subscription-manager register
subscription-manager register --username ABC --password
abc@123
Note :- After installation please subscription-manager unregister
Command :- subscription-manager unregister
2. Install java on node (server), check the java version and Installation it
yum install java*
java -version
3. make logstash as user and group
useradd logstash
Groupadd logstash
4. create logstash Directory for installation work
mkdir /logstash
cd /logstash/
wget https://artifacts.elastic.co/downloads/logstash/logstash-
9.1.5-x86_64.rpm
rpm -ivh logstash-9.1.5-x86_64.rpm
5. make certs folder and copy all important certificate
/etc/logstash
mkdir certs
cd /etc/logstash/certs/
6. changer owner and directory permission
chown -R logstash:logstash /etc/logstash /usr/share/logstash
/var/log/logstash /var/lib/logstash
7. Go to config directory and create conf file as per requirement
/etc/logstash/conf.d
nano output.conf
8 . Start the port
firewall-cmd --permanent --add-port=5044/tcp
systemctl reload firewalld
sudo firewall-cmd --list-ports
9.Restart logstash service and check the status
sudo systemctl daemon-reload
sudo systemctl enable logstash
sudo systemctl start logstash
sudo systemctl restart logstash
sudo systemctl status logstash
Note :- Please put Certifcate and other details as per host and server.
Same process will be followed for the other logstash node also.
Kibana Installation
1. Enable subscription-manager register
subscription-manager register --username ABC --password abc@123
Note :- After installation please subscription-manager
unregister
Command :- subscription-manager unregister
2. Install java on node (server), check the java version and Install it
yum install java*
java -version
3. Make Kibana as user and group
useradd elasticsearch
groupadd elasticsearch
4. Create Kibana Directory for installation work
mkdir /kibana
cd /kibana/
wget https://artifacts.elastic.co/downloads/kibana/kibana-9.1.5-
x86_64.rpm
yum install net-tools
rpm -ivh kibana-9.1.5-x86_64.rpm
5. put all certificates in certs directory
cd /etc/kibana/certs/
6. make changes in kibana.yml file
Note : As per node, cluster and server host name will be different
7. open port and reload
firewall-cmd --permanent --add-port=5601/tcp
systemctl reload firewalld
netstat -an | grep 5601
8. Give permission to kibana respect folders
chown -R kibana:kibana /etc/kibana/certs
chown -R kibana:kibana /opt/kibana
chown -R kibana:kibana /var/log/kibana
9. restart kibana service
sudo systemctl daemon-reload
sudo systemctl enable kibana
sudo systemctl start kibana
sudo systemctl restart kibana
sudo systemctl status kibana
10. Command to reset kibana_system user password from elasticsearch master
node.
Cd /usr/share/elasticsearch
Sudo ./bin/elasticsearch-reset-password -u kibana_system
11. Now go to search engine and access kiban dashboard or url
https://kibanalocalhost:5601
Enter elastic and its password
Username :- elastic
Password :- custom or auto generated.
Note: The same process will be followed for the other Kibana node.
APM Installation on ELK setup
1. Enable subscription-manager register
subscription-manager register --username actual_user --password
actual_pass
Note :- After installation please subscription-manager unregister
Command :- subscription-manager unregister
2. Install java on node (server), check the java version and Installation it
yum install java*
java -version
3. make apm-server as user
useradd apm-server
groupadd apm-server
4. Create APM directory and install apm-server
mkdir /apm
cd /apm/
wget https://artifacts.elastic.co/downloads/apm-server/apm-server-9.1.5-
x86_64.rpm
rpm -ivh apm-server-9.1.5-x86_64.rpm
5. Take backup of yml file and make changes
cd /etc/apm-server/
cp apm-server.yml apm-server_yml.backup
nano apm-server.yml
6. Put certificate on certs location under apm directory
cd /etc/apm-server/certs/
7. Set permission for apm folders
chown -R apm-server:apm-server /etc/apm-server/
chown -R apm-server:apm-server /var/log/apm-server/
8. Make changes in apm conf yml file
cd /etc/apm-server/
nano apm-server.yml
Save yml file
9. Open port 8200 for apm
sudo firewall-cmd --zone=public --add-port=8200/tcp –permanent
sudo firewall-cmd –reload
sudo firewall-cmd --list-ports
firewall-cmd --list-all
Note:- The same process will be followed for other apm node.
10. Put ca certificate
cp ca.crt /etc/pki/ca-trust/source/anchors/
Note :- copy ca certificate from elastic master location and copy it on apm server given location
11. Start apm service and check the status
systemctl daemon-reload
systemctl status apm-server.service
systemctl enable apm-server.service --now
journalctl -xe -f -u apm-server.service --no-pager
journalctl -xe -f apm-server.service --no-pager
journalctl -xe -f -u apm-server.service --no-pager
Fleet Installation in Elastic Cluster
1. Go to Fleet Server and Enter All ELK Host entry in host file
nano /etc/hosts
2. Copy ca.crt certificate in certificate directory
Cd /etc/pki/ca-trust/source/anchors/
Copy certificate here and run update-ca-trust command.
3. Make fleet directory and go inside fleet and install elastic-agent rpm here
3. Go to Kibana and go to fleet section via 3 line in left side and add fleet.
5.select or click to add fleet server and under advanced section give Policy name as per our
requirements can give custom any name
NOTE :- Follow the step which shows after creating policy to install fleet. To install fleet we need to install
elastic-agent on server it automatically installs fleet on that particular server.
1. Select rpm or tar option for Installation of elastic-agent on our server and copy that
installation command and run in our /fleet directory of server
2. After installing elastic-agent bring or copy all respect certificate on fleet server under
elastic-agent directory
cd /etc/elastic-agent
mkdir certs
cd /etc/elastic-agent/certs
Give proper permission to certificate and certs directory to.
3. Open port 8220 for elastic-agent and fleet
4. Run installation or enroll command which is given in kibana UI for elastic-agent
5. Check status on both UI and CLI. It should be successfully enroll elastic-agent
Note :- same steps we need to follow for another 2nd node of fleet instead of default fleet we need to give
respect fleet url.
Add new fleet server host and Rest of the steps remaining same.
Note:- The same process will be followed for the other fleet node also.