Editing Training 2022 Linux team/ (section)

==Training session Tetra team on Infosys ITD ELK understanding. Dated 24 Nov 2022 - Tetra Support Staff - Biswajit Banerjee==

{{#ev:youtube|hooXpnAjPl0|640}}

'''Video summary:''' "Training session Tetra team on Infosys ITD ELK understanding" -- a detailed architecture walkthrough of the Income Tax Department (ITD)'s ELK-based logging platform, part of the same Infosys engagement covered in other sessions on this wiki. Opens with the underlying business rationale: the Income Tax environment generates enormous volumes of application logs across many machines, and reading raw text logs server-by-server has no practical value -- what's needed is a visual, aggregated view of connection counts, server/application performance, data throughput, and error rates across the whole environment, which is exactly what ELK (Elasticsearch/Logstash/Kibana) is for. Explains the cluster's real scale: 2 master nodes plus a large number of additional Elasticsearch/Kibana/Logstash nodes, totaling roughly ~99 machines across three parallel environments -- PDC (a testing/staging predecessor, referred to loosely as "Non-Prod"), a UAT-like environment, and BCP (Business Continuity Planning, i.e. the actual primary/DR production data center pairing) -- with security protocols and single sign-on experimented on first in the pre-prod environments before being promoted to production. Contrasts self-built open-source ELK (where every component -- Filebeat/Logstash shipping, Elasticsearch nodes, Kibana -- must be assembled and wired together manually) against fully-managed cloud logging platforms (e.g. AWS/GCP's built-in equivalents) that ship as a ready-made UI with nothing to build. Walks through ITD's actual deployment documentation live: PDC has 3 master nodes plus dedicated Elasticsearch data machines and a separate log-test machine; installation followed a standard pattern of setting each node's data directory, getting nodes talking to each other, then layering on Kibana with reverse-proxy access in front of the application servers; explains Logstash's per-pipeline configuration (input source, filter, output -- e.g. targeting all three master nodes as Elasticsearch output, indexing rules, multiple parallel pipelines for different log types) and how filters can be written from scratch, found pre-built, or auto-detected via plugins to parse/tag incoming log fields for later visualization. Uses the practical framing that even in a well-organized log stream, an engineer should be able to eyeball raw records and get an intuitive sense of pattern/category (e.g. "which kind of log entries are these, and what do they mean") before formalizing that into filter rules -- and that when in doubt, the right move is to just ask the person who built the original pipeline rather than guess.