Jump to content
Main menu
Main menu
move to sidebar
hide
Navigation
Main page
Recent changes
Random page
Help about MediaWiki
TetraWiki
Search
Search
Appearance
Create account
Log in
Personal tools
Create account
Log in
Pages for logged out editors
learn more
Contributions
Talk
Editing
DGARM Design Document - 20250829 v1.2
Page
Discussion
English
Read
Edit
View history
Tools
Tools
move to sidebar
hide
Actions
Read
Edit
View history
General
What links here
Related changes
Special pages
Page information
Appearance
move to sidebar
hide
Warning:
You are not logged in. Your IP address will be publicly visible if you make any edits. If you
log in
or
create an account
, your edits will be attributed to your username, along with other benefits.
Anti-spam check. Do
not
fill this in!
''Auto-generated from the uploaded PDF [[:File:DGARM_Design_Document_-_20250829_v1.2.pdf|DGARM_Design_Document_-_20250829_v1.2.pdf]]. This is an extracted-text rendering for searchability; see the original PDF for exact formatting, diagrams, tables, and images.'' <pre> <nowiki> 2 | elastic.co | © 2023 Elasticsearch B.V. All Rights Reserved. Table of Contents Table of Contents 3 Document Properties 6 Version History 6 References 6 Executive Summary 7 Overview 7 Participants 7 Business Value 8 Outcomes 9 Solution Overview 10 Elastic Stack Overview 10 Beats 10 Elastic Agent 11 Logstash 11 Elasticsearch 11 Kibana 11 APM / Fleet Server 11 Cluster Design 12 Operating System 12 Hardware Requirements 12 Network Connectivity 13 Security - SSL/TLS Certificates 13 Cluster Resiliency 15 Elastic Deployments 15 Cluster topology 16 Search Cluster 17 Observability Cluster 18 Monitoring Cluster 18 Placement of Elasticsearch Nodes 19 Shard Allocation Awareness 19 Authentication and RBAC 20 User Roles and Permissions 20 Spaces 20 Audit Logging 20 Observability Ingestion Design 22 High Level Diagram 22 3 | elastic.co | © 2023 Elasticsearch B.V. All Rights Reserved. Design Principles 23 Data Sources 23 Fleet and Elastic Agent 25 Elastic Agent 25 Fleet Server 25 Air Gapped 25 High Availability 26 Scaling 26 Logstash 26 High Availability 26 Queuing 27 Scaling 27 APM Server 28 High Availability 28 Scaling 28 Sampling Strategies 28 Definition 28 Comparison of Sampling Strategies 29 Log and Metrics Collection 30 Elastic Integrations 30 Custom Data Source Integration 30 Log Collection - HTTP endpoint 30 Log Collection - Syslog 30 Log Collection - API 31 Metrics Collection 31 Traces Collection - Elastic APM 31 Traces Collection - OpenTelemetry 32 Real User Monitoring (RUM) 32 Data Streams 33 Naming Convention 33 Sharding Strategy 33 Index Lifecycle Management (ILM) 34 Snapshots 37 Repository 38 Snapshot Lifecycle Management (SLM) 38 Alerting 39 Overview 39 Alerting Use Cases 39 Machine Learning and AI 39 4 | elastic.co | © 2023 Elasticsearch B.V. All Rights Reserved. Search Design 41 Requirement Analysis 41 Recommended Change - Filtering on Number of Arrivals / Departures 41 Ingestion 42 Index Management 42 Index Naming 42 Index Retention 43 Shard Sizing 44 Index Template 44 Autocomplete 44 Simple Autocompletion 45 Name Autocompletion - Option 1 (recommended) 45 Name Autocompletion - Option 2 45 Ingest Pipeline 47 Index Template 48 5 | elastic.co | © 2023 Elasticsearch B.V. All Rights Reserved. Document Properties Version History Version Date Author Description 1.0 2025 August 18th Salim Alaeddine Initial draft 1.1 2025 August 26th Mattias Brunnert Search Design Mattias Brunnert, Salim 1.2 2025 August 29th Review Alaeddine References Ref. No Document Name Location Description 1 6 | elastic.co | © 2023 Elasticsearch B.V. All Rights Reserved. Executive Summary Overview The Directorate General of Analytics and Risk Management (DGARM), functioning under the Central Board of Indirect Tax and Customs (CBIC), provides intelligence inputs and leverages big data analytics to assist tax officers in policy formulation, compliance monitoring, and detection of tax evasion. In alignment with these objectives, Infosys and the TeTrain team have engaged Elastic Professional Services to design, validate, a Search and Observability solution. This engagement represents the first-time deployment of an Elastic architecture within DGARM, with the goal of ensuring best practices are embedded into the platform from inception. The scope of work covers: ● Elastic Platform Design & Best Practices ● Architecture design and design review ● Pipeline and ingestion recommendations and patterns ● Elastic cluster health check and validation against the agreed Design Document ● Review of cluster configuration, sharding strategy, lifecycle management policies, and ingestion strategy Participants Org Name Role/Position mail Biswajit Banerjee Project Manager biswajit@tetrain.com Deepa Leads the software team deepa@tetrain.com Ravinder Kuma ravinder.kumar@tetrain.com Sweta sweta@tetrain.com Tetra Takshay takshay@tetrain.com Tushar tushar@tetrain.com Rakeshdubey rakeshdubey@tetrain.com 7 | elastic.co | © 2023 Elasticsearch B.V. All Rights Reserved. Mukul mukul@tetrain.com muralidharan_S12@infosys.c Muralidharan Lead Architect om bhasha.sisodia@infosys.co Bhasha Sisodia Operations Manager m karan.neelkanth@infosys.co Karan Neelkanth Platform Lead m Divya Dhyani divya_dhyani@infosys.com Infosys Bimlesh Kumar bimlesh.kumar@infosys.com ashutosh.tiwari13@infosys.c Ashutosh Tiwari om krithika_chakrapani@infosys Krithika Chakrapani .com Vikram K vikram_k@infosys.com Anant Pendse anant_pendse@infosys.com Salim Alaeddeine Consulting Architect salim.alaeddine@elastic.co Elastic Mattias Brunnert Principle Consulting Architect mattias.brunnert@elastic.co Sri Suba Selvachamy Delivery Manager sri.selvachamy@elastic.co Business Value By adopting Elastic as the foundation for its analytics and observability strategy, DGARM can realize significant value: ● Enhanced Data-Driven Decision Making: Empower timely intelligence and actionable insights. ● ● Operational Efficiency: Centralized log, metric, and trace visibility improves system monitoring, enabling faster root-cause analysis and reduced downtime. ● Improved Compliance and Enforcement: Strengthened ability to detect anomalies, identify patterns of evasion, and support investigations. 8 | elastic.co | © 2023 Elasticsearch B.V. All Rights Reserved. ● Future-Ready Architecture: A design aligned with Elastic best practices ensures scalability, resilience, and adaptability to future data growth and regulatory needs. Outcomes The engagement with Elastic Professional Services is expected to deliver: ● A validated Elastic architecture design, aligned with business requirements and technical best practices. ● Lifecycle and sharding strategies optimized for performance, cost efficiency, and data retention requirements. ● A health check and validation, identifying risks, gaps, and remediations for the current deployment. ● Increased stakeholder confidence in the Elastic platform as a secure and reliable foundation for DGARM’s big data analytics initiatives. 9 | elastic.co | © 2023 Elasticsearch B.V. All Rights Reserved. Solution Overview Elastic Stack Overview An Elastic Stack deployment consists of several core components, each designed to support specific roles in the collection, processing, storage, analysis, and visualization of observability data. These components work together to deliver a scalable, flexible, and feature-rich monitoring and analytics platform. Elastic Stack can be deployed in various forms, including fully managed Elastic Cloud Service, self-managed orchestrated deployments, or standalone on-premise clusters. In this engagement, DGARM will deploy a standalone on-poremise cluster in their local data center. The following provides a brief description of each component and its role in an Elastic Stack deployment. Beats Beats are lightweight data shippers installed on edge devices or servers to collect and forward data to Elasticsearch or Logstash. There are different types of Beats, each tailored for specific data sources: ● Filebeat: Monitors and ships log files. ● Metricbeat: Collects system and service-level metrics. ● Auditbeat: Collects audit data from Unix-based systems. ● Winlogbeat: Captures Windows event logs. ● Packetbeat: Monitors network traffic and packets. Each Beat offers a set of predefined modules for common data sources and supports custom configurations for advanced use cases. 10 | elastic.co | © 2023 Elasticsearch B.V. All Rights Reserved. Elastic Agent Elastic Agent consolidates the functionality of multiple Beats into a single, unified agent. Managed centrally through Fleet Server, it simplifies deployment, scaling, and lifecycle management across diverse environments. Elastic Agent supports a wide array of out-of-the-box integrations for logs, metrics, and security telemetry. For a full list of integrations available see Elastic Agent Integrations. Logstash Logstash is a powerful data processing pipeline tool that supports the ingestion, parsing, enrichment, transformation, and routing of data. It uses pipelines composed of input plugins, filter plugins, and output plugins to handle a wide variety of data sources and formats. This makes Logstash ideal for complex or custom data processing workflows before the data reaches Elasticsearch. Elasticsearch Elasticsearch is the heart of the Elastic Stack. It serves as a distributed, RESTful search and analytics engine designed for scalability, speed, and relevance. It provides the storage, indexing, and powerful search capabilities that drive real-time visibility and insights from ingested data. Kibana Kibana acts as the frontend interface for the Elastic Stack. It allows users to visualize, explore, and interact with data stored in Elasticsearch. Kibana supports dashboards, alerting, lens visualizations, drilldowns, and many plugins for advanced use cases such as SIEM, APM, and observability. APM / Fleet Server The APM (Application Performance Monitoring) component provides performance insights and distributed tracing for monitored applications. Fleet Server is the centralized orchestration layer that manages Elastic Agents, ensuring policy-driven configuration, agent status tracking, and seamless integration management across the stack. 11 | elastic.co | © 2023 Elasticsearch B.V. All Rights Reserved. Cluster Design Operating System The Elastic Stack can be installed on Windows or Linux operating systems. Linux is the most commonly used operating system and therefore has a more comprehensive knowledge base for configuration and troubleshooting. DGARM aims to deploy Elastic stack on Openshift VMs. The complete Elastic Stack support matrix can be found here. Hardware Requirements This section details the recommended hardware requirements for Elastic Stack components. The sizing and recommended topology is provided in the Elastic Deployments section. Ingest, Transform and Remote Cluster Client nodes have not been included as these are typically not dedicated nodes. Component RAM CPU Storage (SSD) Disk Performance Elasticsearch - Highest IOPS 64GB max 1 CPU per 2GB RAM 32GB storage total Master (10K+) Elasticsearch - 30GB storage per 1GB Highest IOPS 64GB max 1 CPU per 2-4GB RAM Data Hot RAM (10K+) 100 GB storage per Elasticsearch - Highest IOPS 64GB max 1 CPU per 4GB RAM 1GB RAM + Snapshot Data Warm (10K+) repository Elasticsearch - Machine 64GB max 1 CPU per 2GB RAM 32GB storage total Any Learning Kibana 16GB max 1 CPU per 2GB RAM 32GB storage total Any 32GB storage unless Medium IOPS Logstash 32GB max 1 CPU per 2GB RAM Persistent Queue (PQ) (7K+) if PQ is used, is required otherwise any. Elastic Agent 16GB max 1 CPU per 2GB RAM 32GB storage Any 12 | elastic.co | © 2023 Elasticsearch B.V. All Rights Reserved. Network Connectivity As DGARM is deploying a new cluster, the following connections may be required for the Elastic Stack components. Flow Source Destination Default Port Agent Management Elastic Agent Fleet Server TCP/8220 Elastic Agent Ingestion via Logstash Elastic Agent Logstash TCP/5044 Elastic Agent Ingestion to Elastic Agent Elasticsearch TCP/9200 Elasticsearch Agent Policy Retrieval Fleet Server Elasticsearch TCP/9200 Elastic APM APM Data Ingestion APM Server TCP/8200 Agent Kibana User access User Kibana TCP/5601 Elastic Artifact Elastic Artifact Elastic agents Artifacts TCP/443 registry registry Security - SSL/TLS Certificates Elastic self-managed deployments require an initial security setup to enable features such as encrypted communications, user authentication, and secure node enrollment. This includes configuring Transport Layer Security (TLS) for both the HTTP and transport layers, setting passwords for built-in users, and generating enrollment tokens to securely connect Kibana and additional Elasticsearch nodes. Elastic supports two approaches for performing this initial security bootstrap: automatic and manual setup. Securing Kibana always requires manual configuration to complete the integration. Elastic uses TLS certificates to secure communications in two primary layers: ● HTTP Layer: Encrypts traffic between clients (e.g., users, Kibana) and Elasticsearch nodes, ensuring data in transit is protected. 13 | elastic.co | © 2023 Elasticsearch B.V. All Rights Reserved. ● Transport Layer: Encrypts inter-node communications within the cluster and between clusters (if applicable), providing mutual authentication and preventing unauthorized nodes from joining. Types of TLS certificates required and their usage Certificate Type Service/Node Usage Certificate Authority (CA) All nodes Used to sign and trust all other TLS certificates across the deployment. Elasticsearch Transport All Elasticsearch nodes Mutual TLS for inter node Certificate communication Elasticsearch HTTP certificates All Elasticsearch nodes TLS for any client making API/HTTPs connectivity Kibana Certificate Kibana nodes TLS for any end user accessing Kibana using browser Fleetserver Certificate Fleetserver node TLS for any agent accessing fleet server DGARM may choose to use their own TLS certificates by adopting the manual setup approach.. It involves enabling different layers of protection in sequence, depending on your security requirements. 1. Configure transport TLS: Required for multi-node clusters running in production mode. Secures communication between nodes and prevents unauthorized nodes from joining the cluster. 2. Configure HTTP TLS: Secures all client communications over HTTPS, including traffic between Kibana and Elasticsearch, and between browsers and Kibana. Recommended for all clusters, even single-node setups. For additional TLS configuration options, refer to Manage TLS encryption in self-managed deployments. 14 | elastic.co | © 2023 Elasticsearch B.V. All Rights Reserved. Cluster Resiliency As a distributed system, Elasticsearch is capable of tolerating failure even if some of the components are not available. Elasticsearch accomplishes this through redundancy of every component, including the data stored. A resilient cluster must have the following characteristics: ● At least three master-eligible nodes ● At least two nodes of each used role ● At least one replica shard for every primary shard, unless the index is a searchable snapshot ( the data is backed up in the snapshot repository) The above configuration is the minimum requirements for resiliency i.e. the cluster will stay operational if a single node fails. This concept can be expanded for further resiliency by dividing the cluster into availability zones. Each availability zone should have independent power supply and other supporting infrastructure. In practice, this may be a different fault domain within a data centre or even another data centre entirely. By leveraging availability zones, the cluster remains functional even if an entire availability zone becomes unavailable. For a cluster to survive an availability zone failure, the following characteristics must be present: ● At least three master-eligible nodes, each situated in a different server racks ● A symmetrical deployment of each node role across at least two server racks ● At least one replica shard for every primary shard, unless the index is a searchable snapshot spread across at least two availability zones using shard allocation awareness. Elastic Deployments DGARM is planning to deploy the Elastic clusters locally within their primary datacenter, maintaining a single physical site and single availability zone architecture. This approach introduces a known risk, any failure impacting the primary datacenter, such as power outages, hardware failures, or connectivity loss, could result in a full cluster outage, affecting service 15 | elastic.co | © 2023 Elasticsearch B.V. All Rights Reserved. availability and access to data. DGARM acknowledged and accepted the inherent risk associated with a single availability zone deployment. To mitigate the risk of data loss and to support business continuity objectives, DGARM has planned an active–passive disaster recovery (DR) strategy: ● A passive cluster will be deployed in the DR site. ● Snapshots from the primary site will be replicated to the DR site. ● In the event of a disaster, a manual failover process will be initiated, restoring data from the most recent snapshots. ● Disaster recovery strategies and procedures will be enforced and periodically reviewed to validate recovery time (RTO) and recovery point objectives (RPO). While this approach does not provide instantaneous failover or cross-site replication (as in multi-AZ or cross-cluster replication designs), it aligns with DGARM’s current infrastructure, budget, and licensing constraints, providing a balance between risk management and operational feasibility. Cluster topology Initial sizing for the Elastic deployment was carried out during presales discussions between the Elastic Solution Architects and the client. Based on this exercise, a single unified production cluster was originally proposed to support both Search and Observability use cases. Following best practices and further refinement during discovery workshops, the design was adjusted to better address the criticality of the workloads, the service-level requirements (SLAs), and the need for resilience and operational visibility. As a result, the topology was updated to include: ● Dedicated Search Cluster – to handle search-intensive workloads independently. ● Dedicated Observability Cluster – to ingest and analyze logs, metrics, and traces without impacting search performance. ● A minimal Monitoring Cluster – to provide end-to-end visibility of the production clusters’ performance and health. 16 | elastic.co | © 2023 Elasticsearch B.V. All Rights Reserved. The overview topology for each cluster can be found in the diagram below Search Cluster Role CPU Memory (GB) Disk (SSD) Quantity Hot ingest and Master Nodes 16 64 2TB 3 Machine Learning 8 64 200GB 1 Kibana 8 16 200GB 2 17 | elastic.co | © 2023 Elasticsearch B.V. All Rights Reserved. Observability Cluster Role CPU Memory (GB) Disk (SSD) Quantity Master Nodes 8 16 200 GB 3 Hot ingest Nodes 16 64 2TB 2 Warm nodes 16 64 5 TB 4 Machine Learning 16 32 200 GB 2 Kibana 8 16 200 GB 2 Fleet Server 8 16 200GB 2 APM Server 8 16 200GB 2 Snapshot Repository - - - DGARM acknowledges the license limitations may cause performance impact to the cluster. DGARM will closely monitor cluster performance and scale up and out if required. Monitoring Cluster A dedicated monitoring cluster will be deployed to collect the logs and metrics generated by the Elasticsearch clusters. The monitoring cluster is critical to all production grade Elasticsearch deployments. As part of best practice, a 3-node monitoring cluster is recommended to ensure high availability, resilience, and fault tolerance. However, to align with the current procured license entitlement, the monitoring cluster has been initially deployed as a single-node cluster to provide the minimum monitoring capabilities required. The client will revisit this approach in future licensing and scaling exercises to align the monitoring cluster with high-availability standards. Below is the desired topology Role CPU Memory (GB) Disk (SSD) Quantity Master/Hot Data Nodes 4 8 1 TB 3 Kibana 2 4 200 GB 2 18 | elastic.co | © 2023 Elasticsearch B.V. All Rights Reserved. However to stay within the license, the initial cluster size could be as follow Role CPU Memory (GB) Disk (SSD) Quantity Master/Hot Data Nodes 4 8 1 TB 1 Kibana 2 4 200 GB 1 Placement of Elasticsearch Nodes To increase resiliency of the Elastic stack it is important to place virtual machines so that they don’t share common hardware resources, including server rack, power supplies, etc. It is especially important that each stack component do not share hardware resources within the same cluster, including: ● Master nodes ● Hot data nodes ● Warm data nodes ● Machine learning nodes ● Kibana ● Fleet Shard Allocation Awareness Once the cluster grows to a larger size and not sharing hardware between nodes becomes impractical, Elastic recommends DGARM implementing shard allocation awareness. It allows DGARM to use custom node attributes as awareness attributes to enable Elasticsearch to take your physical hardware configuration into account when allocating shards. If Elasticsearch knows which nodes are on the same physical server, in the same rack, or in the same zone, it can distribute the primary shard and its replica shards to minimize the risk of losing all shard copies in the event of a failure. 19 | elastic.co | © 2023 Elasticsearch B.V. All Rights Reserved. Authentication and RBAC You can manage and authenticate users natively, or integrate with external user management systems such as LDAP and Active Directory. If none of the built-in realms meet your needs, you can also build your own custom realm and plug it into the Elastic Stack. At this point there is no directory service identified, DGARM will be leveraging Elasticsearch’s Internal authentication methods to manage user authentication. User Roles and Permissions Roles in Elastic are collections of privileges that define the actions a user can perform. Instead of granting privileges directly to users, they are assigned one or more roles that collectively determine their access level. When a user is assigned multiple roles, the user receives a union of the roles’ privileges. This means that assigning additional roles cannot reduce the user’s privileges. To adjust a user’s access level, you must modify or remove one of their assigned roles. Spaces Kibana Spaces allow the separation of content and dashboards within a single Kibana instance, enabling role-based access control (RBAC) tailored to different user groups or operational personas. Each space can be restricted by role, ensuring that users only access the data and visualizations relevant to their responsibilities. Audit Logging Audit logging is a security feature that allows tracking of access to the Elastic cluster, logging of security-related events. Elasticsearch’s audit logging can be used to monitor clusters for suspicious activities, such as unauthorized data access or changes to user security configurations. By enabling and configuring audit logging, DGARM can have visibility into cluster operations and strengthen its overall security posture. 20 | elastic.co | © 2023 Elasticsearch B.V. All Rights Reserved. Audit logging is disabled by default in Elasticsearch. To enable it, the following changes need to be made to the settings for Elasticsearch and Kibana in this documentation. Audit logs can be highly verbose and noisy, especially in active clusters, Elasticsearch events ignore policies and Kibana ignore filters can be used for fine-grained control over which audit events are collected. For more information, see Auditing settings. As DGARM is starting with a minimal Monitoring cluster, enabling audit logging at this stage may not be optimal. When capacity allows, audit logging can be enabled with appropriate filtering policies to address the required use cases. 21 | elastic.co | © 2023 Elasticsearch B.V. All Rights Reserved. Observability Ingestion Design High Level Diagram 22 | elastic.co | © 2023 Elasticsearch B.V. All Rights Reserved. Design Principles The ingestion architecture for DGARM is designed according to the following ingestion best practices for Elasticsearch: ● All logs ingested will align with the Elastic Common Schema (ECS) ● Logs and metrics will be collected using Fleet-managed Elastic Agent as the default ingestion agent ○ If an Elastic Integrations exists, it will be used to collect the relevant logs and metrics ○ In cases where Elastic Agent is unusable, Beats, Logstash or other mechanisms may be proposed as the fallback option depending on the ingestion scenario. ● Data parsing at Elasticsearch Ingest Pipelines is preferred over Logstash ● All logs and metrics will be encrypted in transit Data Sources As part of the technical discovery phase, The following data sources have been identified for onboarding. Custom DataSource Type Model Count OOTB integration Integration Red Hat System Logs & RHEL Server(Virtual) Server Enterprise Linux 800/900 Metrics 9.5 (Plow) HPE ProLiant DL380 Gen11 & System Logs & Server Server 113 HPE ProLiant Metrics DL385 Gen11 Kubernetes Openshift Kubernetes securepass emudhra Application NA NA (iDAM) PostgreSQL postgres Database NA PostgreSQL 16.9 Platform/D Data Fabric HP Ezmeral 7.8.0 NA Custom B Unified Analytics HP Platform/D 1.5.x NA Custom Ezmeral B IBM mq Application 9.4.0.10 NA IBM MQ Neo 4J Application jws-6.1 NA Nginix+API Gateway Application F5 nginx/1.27.4 NA Custom Jboss Web server Server jws-6.1 NA Custom Git Hub Application NA GitHub 23 | elastic.co | © 2023 Elasticsearch B.V. All Rights Reserved. Manage Engine -ITSM Application NA Custom / Syslog Manage Engine - Patch Application NA Custom / Syslog Tenable Application NA Tenable Imperva Application NA Impreva Cisco SAN Storage Cisco 6 Cisco Nexus Syslog / SNMP Switch/Storage NetApp Metric API or Netapp NetApp Storage Storage Netapp 6 Harvester/Prom etheus Input integration Veritas Back up Backup Veritas 4 Custom / API Palo Alto Firewall - Palo Alto Security PA-3410 4 Internal Next-Gen Firewall Fortinet Firewall - Fortigate Fortinet FortiGate Security 4 External FG-401F Fortigate Fortinet FortiGate Sandbox Security 4 FSA-500G Firewall Logs Juniper SRX Juniper Router Network 14 Juniper SRX 2300, SRX380 Aruba Networking CX 8100, Core - HPE Switches Network Aruba 36 HPE Aruba CX 8325-32C, OOB - Aruba Networking CX 6200F WAF Security F5 WAF -R2600 4 F5 BIG-IP F5 -DDOS - DDOS Security 4 F5 BIG-IP R2600 Forcepoint Web Proxy Security Forcepoint 4 Security Forcepoint Web DLP Security Forcepoint Virtual Security F5 Load balancer Security F5 Virtual F5 BIG-IP Application Performance Monitoring data sources (Java and Node.js.) Application/Microservice Name Type flightScheduleService Microservice watchlistService Microservice 24 | elastic.co | © 2023 Elasticsearch B.V. All Rights Reserved. riskManagement Microservice mapService Microservice manToMachineService Microservice userManagementService Microservice masterdataservice Microservice regAuthService Microservice searchService Microservice auditLoggingService Microservice communicationService Microservice documentService Microservice adminweb Microservice cacheManagementService Microservice edifactProcessingService(MQ Consumer) Microservice Fleet and Elastic Agent Elastic Agent Elastic Agent will serve as the primary ingestion service to collect logs and metrics from the various mentioned data sources. Elastic agents will be deployed at scale, and centrally managed through Fleet Server. The Elastic Integration repository offers a large number of pre-configured and tailored options for readily integrating with common technologies and data sources to easily collect log and metrics information from applications. Fleet Server Fleet Server a special Elastic Agent which connects other Elastic Agents to Fleet for agent management. It acts as a control plane for updating agent policies, collecting status information and coordinating actions across Elastic Agents. Air Gapped In case the deployments do not have internet access. The Elastic Package Repository (EPR) and the Elastic Artifacts Repository needs to be self-hosted. The EPR hosts the elastic integrations which contain the out of the box log parsers, dashboards and other content. The Elastic Artifacts Repository 25 | elastic.co | © 2023 Elasticsearch B.V. All Rights Reserved. hosts the binaries which are required to install and upgrade the Elastic Agents. Instructions to self-host EPR can be found here. Instructions to self-host artifacts repository can be found here. High Availability There will be two Fleet Servers deployed in the observability cluster. These Fleet Servers will be spread across two VMs. Fleet Server is stateless, therefore no local configuration to be maintained. Fleet servers can be placed behind a load balancer or multiple Fleet Server URLs can be specified to provide automatic failover. Scaling Fleet servers sizing will depend on the number of agents managed. The following documentation outlines the hardware requirements per agent count. The sizing provided in the cluster topologies will be sufficient for up to 3000-5000 agents. Logstash Although not the primary ingestion tool, Logstash is included in the architecture to alleviate potential bottlenecks which may occur as the deployment scales to larger volumes. There are also some specific use cases which require Logstash. Some potential use cases for Logstash are listed below: 1. Offload parsing, enrichment and extract, transform and load (ETL) functionality off the Elasticsearch cluster for high throughput or compute heavy workloads. 2. Serve as a network and log aggregation point for Elastic Agents to: a. Reduce the number of concurrent connections into Elasticsearch b. Simplify networking configuration, including firewall rules 3. Perform further processing of APM data 4. Perform specific log actions such as splitting events and aggregating events High Availability Logstash can be deployed across multiple VMs as we have only one availability zone. A network load balancer will be deployed in front of Logstash to spread the connections across multiple instances. 26 | elastic.co | © 2023 Elasticsearch B.V. All Rights Reserved. This load balancer will be TLS passthrough to ensure that TLS verification can happen between the Logstash and Elastic Agent to ensure log traffic is encrypted. Currently, the only planned input to Logstash is from Elastic Agent on TCP/5044. Queuing By default Logstash utilises an in-memory queue to buffer events during processing. It is possible to configure a persistent queue which will write events to disk. This protects against message loss if Logstash is terminated unexpectedly and the in-flight events have not been fully transmitted. The persistent queue also allows Logstash to absorb bursts of events without an external buffering technology such as Kafka. However, as the events are committed to disk prior to sending to the configured output, this will introduce latency and additional performance overhead. Scaling Logstash should be scaled horizontally to optimise the hardware utilisation. Ideally, Logstash hosts do not need to exceed 16GB of RAM (8GB of JVM Heap). When monitoring Logstash, pay close attention to CPU since it is usually the limiting factor for maximum throughput. Heap usage, and disk performance if persistent queues are enabled, are also key metrics. Logstash metrics can be collected and sent to the monitoring cluster and analysed to make scaling decisions. The following metrics are important to consider: 1. CPU Utilisation 2. JVM usage 3. Pipeline Events metrics 4. Pipeline Flow metrics 5. Persistent Queue metrics 6. Disk metrics - collected by Elastic Agent or Metricbeat system module 27 | elastic.co | © 2023 Elasticsearch B.V. All Rights Reserved. APM Server Elastic APM server will collect logs, metrics and traces sent through APM agents. It transforms these into documents which can be sent directly to Elasticsearch or to Logstash for further processing. Elastic APM server also supports APM data coming from OpenTelemetry and will enable the use of OpenTelemetry traces to be used with Elastic Observability. High Availability APM server will be deployed across two VMs. The endpoints will be exposed via the load balancer. This load balancer will be TLS passthrough to ensure that TLS verification can happen between the APM servers and APM sources. Scaling APM Server performance can be dependent on many factors, primarily the number of transactions per second and the number of APM agents/sources sending data. The following documentation describes some of the scaling guidance which Elastic has recommended for APM Server. APM server is a stateless ingest component, therefore it can be scaled horizontally with ease. Sampling Strategies Sampling controls how much trace data is collected and stored, balancing cost and visibility. There are 3 sampling strategies: no sampling, head-based sampling and tail based sampling. Definition No Sampling ● All traces are captured and ingested. ● Provides complete visibility but may generate a high volume of data. ● Suitable for low-traffic apps or short-term troubleshooting. Head-Based Sampling ● Sampling decision is made at the start of the request. ● Simple and low-overhead. ● Limitation: rare or slow transactions may be missed. 28 | elastic.co | © 2023 Elasticsearch B.V. All Rights Reserved. Tail-Based Sampling ● Sampling decision is made after the request completes, based on content (e.g., errors, latency). ● Allows intelligent selection of important or anomalous traces. ● Requires Elastic APM Server as only Elastic APM Server currently supports tail-based sampling. Comparison of Sampling Strategies The table below provides comparisons for the 3 sampling strategies Criteria No Sampling Head-Based Tail-Based Sampling Sampling Sampling All traces are retained At the start of a After the request Decision request completes, based on outcome Setup Simplest Simple, fully More complex – requires Complexity OTEL-native Elastic APM Server Latency Impact None Minimal Slightly higher due to span buffering Trace Full visibility of all May miss rare or High accuracy in Completeness transactions low-frequency traces capturing rare or anomalous traces Data Volume & High Moderate (controlled Could be lower (selective Storage Cost by sampling ratio) trace retention) Flexibility Limited control over Flexible, tunable via Sampling policy handled data volume OTEL sampling by APM Server logic configs 29 | elastic.co | © 2023 Elasticsearch B.V. All Rights Reserved. Log and Metrics Collection Elastic Agents will be deployed on endpoints located within both on-premises infrastructure and virtual machines, and where required. These agents will be responsible for collecting logs and metrics from distributed systems and workloads, ensuring consistent and centralized data ingestion across the entire ecosystem. Elastic Integrations Elastic Integrations are Elastic prebuilt packs which reduce the time taken to onboard and derive value from log sources. Each integration comes pre-packaged with assets that support your needs and allow you to easily collect, store, and visualize any data from any source. which may be relevant to DGARM’s observability use cases. Many of the log sources DGARM wishes to onboard already have an Elastic integration available, see the table above. Log sources onboarded with an integration will be automatically parsed and mapped into the Elastic Common Schema (ECS). Log parsing is performed through Elasticsearch ingest pipelines which run on the Ingest Nodes. Custom Data Source Integration Log Collection - HTTP endpoint In scenarios where external systems can push events to a HTTP Endpoint, the custom HTTP Endpoint Log integration initializes a listening HTTP server that collects incoming HTTP POST requests containing a JSON body. The body must be either an object or an array of objects. Any other data types will result in an HTTP 400 (Bad Request) response. For arrays, one document is created for each object in the array. Log Collection - Syslog If logs are forwarded using syslog, then Elastic Agents will be set up as syslog listeners behind a network load balancer. These Elastic Agents will run either an existing integration or the custom TCP/UDP input. 30 | elastic.co | © 2023 Elasticsearch B.V. All Rights Reserved. To simplify the parsing process, each different syslog log type or format should be forwarded to a unique port. E.g. syslog from UNIX servers should go to port 5514, syslog from a network device can go to port 5515 Log Collection - API In some cases an API will need to be polled to retrieve data. In this case, if an integration exists, it will be utilised otherwise Elastic Agent Custom API integration or the Custom CEL integration will be used to collect. Note that proxy access might need to be provided for these API endpoints. Metrics Collection The collection method for metrics will be determined by the Elastic integration. If no integration exists, one of the following options can be used: 1. If the application has APM enabled, some metrics will be collected through the APM agents. 2. If an metrics API exists, the metrics can be polled using the Elastic Agent Custom API input 3. If metrics are forwarded using syslog, Elastic Agents can be set up as syslog listeners behind a network load balancer. These Elastic Agents will run either an existing integration or the custom TCP/UDP input. 4. In cases where external systems can push metrics to a HTTP Endpoint, the custom HTTP Endpoint Log integration initializes a listening HTTP server that collects incoming HTTP POST requests. Traces Collection - Elastic APM Elastic APM will be used to collect logs, metrics and traces from applications. Generating application traces and metrics generally requires the direct instrumentation of an APM Agent into the application code. The instrumentation process is different for each language and requires a separate APM Agent. Java and Node.js. are the common application languages used in DGARM applications. The APM data can be tagged with metadata and resource attributes to correlate traces with application logs and metrics. Elastic provides APM agents for a variety of languages and will tag the APM data automatically. The instrumentation process depends heavily on the language. 31 | elastic.co | © 2023 Elasticsearch B.V. All Rights Reserved. Refer to respective agent’s documentation for more details on supported instrumentation approaches: ● Java agent ● Node.js agent Traces Collection - OpenTelemetry Traces generated by OpenTelemetry can also be collected via Elastic APM Server through the OTLP protocol. There are two patterns to receive data from OpenTelemetry: 1. Traces, metrics and logs forwarded to APM Server from an OpenTelemetry Collector 2. Traces metrics and logs delivered directly to APM Server from the OpenTelemetry agent. When exporting from an OpenTelemetry Collector, it is important to choose the oltp exporter, as this sends data in the correct format to APM Server which will allow the data to be formatted correctly for use within the Kibana Observability applications. Real User Monitoring (RUM) The frontend application plays an important role in user experience. To gain visibility into client-side performance, this application is instrumented using the Elastic Real User Monitoring (RUM) JavaScript Agent. APM RUM JavaScript agent -> APM Server -> Elasticsearch In this architecture: ● The Elastic JavaScript Agent is embedded into the frontend codebase. Once loaded in the browser, it captures a variety of performance metrics, including page load times, route changes, JavaScript errors, and HTTP requests made from the client. ● The agent also collects user context such as browser metadata, geolocation (if enabled), session ID, and user agent string. This information is essential for analysing frontend performance in real-world user conditions. ● All telemetry data is sent directly from the browser to the Elastic APM Server, which processes and forwards it to Elasticsearch for storage and analysis. 32 | elastic.co | © 2023 Elasticsearch B.V. All Rights Reserved. Data Streams A data stream lets you store append-only time series data across multiple indices while giving you a single named resource for requests. Data streams are well-suited for logs, events, metrics, and other continuously generated data. There are multiple benefits to utilising data streams: ● Automatic index management using Index Lifecycle Management (ILM) ● Automatic index bootstrapping for index rollover ● Automatic index rollover to ensure optimal shard sizes ● Elastic Integrations optimise storage and mappings when using the recommended data stream naming conventions ● Leverage Elastic pre-built content by following data stream naming conventions ● Leverage storage efficiency features such as LogsDB Naming Convention All logs should be written to data streams. Data streams are designed specifically for append-only data. Data streams must follow the recommended Elastic naming conventions i.e. <type>-<dataset>-<namespace>. The type and dataset is typically static and based on the type of data being ingested. Below are samples based on some custom data sources: ● Netapp Metrics ○ metrics-netapp.status-default ● Veritas Custom Logs ○ logs-veritas.eventlog-default ○ logs-veritas.admin_jobs-default ○ logs-veritas.alerts-default Sharding Strategy Each index in Elasticsearch is divided into one or more shards. Each document in an index belongs to a shard. Each shard has some overhead and having too many shards can negatively impact 33 | elastic.co | © 2023 Elasticsearch B.V. All Rights Reserved. Elasticsearch performance and cluster stability. Therefore, enforcing a strict sharding strategy is crucial. The following guidelines should be followed for shard sizing: 1. Shards should be between 10GB and 50GB in size. 2. In cases where logsdb data stream type is used, 10-30GB is an ideal size. 3. Shards should contain no more than 200 million documents. 4. No more than 3000 indices per 1 GB of master heap. Cluster shard limits prevent creation of more than 1000 non-frozen shards per node, and 3000 frozen shards per dedicated frozen node. Make sure you have enough nodes of each type in your cluster to handle the number of shards you need. Index Lifecycle Management (ILM) ILM automatically transitions time series datasets to different data tiers over time. This allows cheaper hardware to be leveraged for data which is older. This optimises the hardware utilisation and saves infrastructure costs. Arbitrarily assigning ILM phases and retention durations which apply broadly across all data will lead to a suboptimal cluster configuration and over utilisation of storage. Retention strategies should be defined based on the usage requirements of each data source and the strengths of each tier. Consider the following for each tier: ● Hot Tier - Primarily used for Ingestion, near real-time search for alerting, live dashboards ● Warm Tier - Primarily used for low latency search and for processing large data sets and reports Initially DGARM will utilise the following data retention: ● 3 days Hot ● 27 days Warm It is recommended to define separate ILM policies for logs and metrics, as they may have different retention requirements. 34 | elastic.co | © 2023 Elasticsearch B.V. All Rights Reserved. Usage should be monitored and the ILM policy should be tuned to match the appropriate usage patterns. To ensure optimal shard sizing, the rollover phase of the ILM will be configured with the following parameters: ● Max primary shard size - 50GB ● Maximum age - 30 days or min shard size (to prevent too many small shards from being generated) ● Maximum documents per shard - 200 million The examples below illustrate how logs and metrics can be managed with different lifecycle requirements. These ensure data is retained for the appropriate number of days and that rollover is based on shard best practices. None # Sample ILM policy for logs PUT _ilm/policy/DGARM-logs-3h-27w-d { "policy": { "phases": { "hot": { "actions": { "rollover": { "max_age": "30d", "max_primary_shard_size": "50gb" }, "set_priority": { "priority": 100 } }, "min_age": "0ms" }, "warm": { "min_age": "3d", 35 | elastic.co | © 2023 Elasticsearch B.V. All Rights Reserved. "actions": { "set_priority": { "priority": 50 } } }, "delete": { "min_age": "30d", "actions": { "delete": { "delete_searchable_snapshot": false } } } } } } # Sample ILM policy for metrics #PUT _ilm/policy/DGARM-metrics-3h-27w-d { "policy": { "phases": { "hot": { "actions": { "rollover": { "max_age": "30d", "max_primary_shard_size": "50gb" }, "set_priority": { "priority": 100 } }, "min_age": "0ms" }, 36 | elastic.co | © 2023 Elasticsearch B.V. All Rights Reserved. "warm": { "min_age": "3d", "actions": { "set_priority": { "priority": 50 } } }, "delete": { "min_age": "30d", "actions": { "delete": { "delete_searchable_snapshot": false } } } } } } Snapshots A snapshot in Elasticsearch is a backup of a running cluster’s data and state. It captures a point-in-time copy of indices and optionally the cluster metadata, and stores it in a registered snapshot repository. Snapshots are stored incrementally and multiple snapshots can reference the same underlying objects in a snapshot repository. Therefore, multiple snapshots covering the same indices do not increase the storage requirements in a Blob Storage provided that the snapshots reference the same repository. Deleting a snapshot will not remove the data in the repository as long as there is at least 37 | elastic.co | © 2023 Elasticsearch B.V. All Rights Reserved. one snapshot still attached. This is important as it means that searchable snapshots can be deleted without impacting the backups and vice versa. Repository To enhance redundancy and ensure business continuity, it is recommended to configure an off-site repository (like Azure, AWS, GCP, Shared file system) for Elasticsearch snapshots. In the case of DGARM, snapshots will be stored on a shared file system repository backed by a Network-Attached Storage (NAS) solution. This provides fault isolation by storing backups outside the primary cluster storage, ensuring they remain accessible even in the event of a site-wide or hardware failure. Once the NAS is mounted and accessible by all eligible master and data nodes, the snapshot repository can be registered using the fs (file system) type. This setup aligns with Elastic’s best practices for resilient backup strategies, offering a scalable, shared, and cost-effective solution suitable for high-frequency snapshot operations and long-term retention.. Snapshot Lifecycle Management (SLM) Cluster snapshots can be managed using Snapshot Lifecycle Management (SLM). Note this is different from the searchable snapshots which are managed as part of ILM actions. When configuring long term snapshot retention, it is important to consider the frequency of the snapshots as well as the retention of a snapshot. Each snapshot has some overhead on the master nodes, so having an appropriate snapshot lifecycle management strategy is important for cluster stability. The recommended strategy is to apply multiple SLM policies with varying intervals and retention to provide flexibility in both snapshot granularity as well as retention length. The following three policies could be used in conjunction: 1. Hourly snapshots which are retained for 24 hours - 24 total snapshots 2. Daily snapshots which are retained for a month - 31 total snapshots 3. Monthly snapshots which are retained for a year - 12 total snapshots 38 | elastic.co | © 2023 Elasticsearch B.V. All Rights Reserved. Combining the above three policies will ensure that snapshots are taken every hour minimising the potential data loss and/or re-ingestion required if recent data needs to be restored. In addition, it reduces the overall number of snapshots required by taking snapshots at longer intervals. This process is described in detail in the following documentation. Alerting Overview Alerting enables you to define rules, which detect complex conditions within different Kibana apps and trigger actions when those conditions are met. Alerting is integrated with Observability, Security and Machine Learning. It can be centrally managed from Stack Management and provides a set of built-in connectors and rules for you to use. Alerting Use Cases During the engagement, potential alerting use cases were discussed to support proactive issue detection and resolution. Examples included the CPU Usage Alert, monitoring system performance by triggering notifications when thresholds are exceeded, and the Error Logs Alert, identifying error-level events in ingested data for early issue detection. Elastic’s built-in connectors, such as Email (SMTP), Slack, Microsoft Teams, ServiceNow, and custom webhooks, offer flexible options for real-time alert delivery. DGARM team will configure preferred connectors, such as email for direct alerts, Microsoft Team, or Zendesk for collaborative notifications as needed. Leveraging these capabilities will ensure timely delivery of alerts and fully utilize Elastic’s monitoring and alerting potential. Machine Learning and AI Elastic’s Machine Learning (ML) and Anomaly Detection features enable proactive monitoring by identifying unusual patterns in logs, metrics, and time-series data. These tools automatically detect anomalies such as spikes in log rate, error log surges, or unexpected network activity, helping DGARM address issues before they escalate. Prebuilt ML jobs introduced during the engagement, 39 | elastic.co | © 2023 Elasticsearch B.V. All Rights Reserved. such as log rate analysis and infrastructure monitoring, can be customized and integrated with alerting rules to ensure timely notifications for critical anomalies. The AI Assistant simplifies data exploration with natural language queries, making it easier for teams to interact with Elastic data without requiring advanced query skills. It provides context for detected anomalies, actionable insights for resolution, and proactive recommendations to optimize performance. By combining ML-powered anomaly detection with the AI Assistant, DGARM can enhance its observability and security efforts, enabling faster response times and more efficient operations. 40 | elastic.co | © 2023 Elasticsearch B.V. All Rights Reserved. Search Design Elastic has reviewed the requirements and done a brief analysis of the functional requirements, shared in SearchFRS.docx on the 26th of August. An attempt has been made to identify key areas of Elasticsearch implementation and assist in the design of those. The configurations and artefacts are provided as guidance and are not complete, final nor ready for production use. Requirement Analysis The requirements include the following identified key elements: 1. The use case is search, i.e. users may search across all records 2. The search indices must support all CRUD operations 3. Records are time based, i.e. they will receive CRUD for a limited period of time and then become static 4. Record retention polices control how long records are kept 5. Search features are provided in a structured form and include basic filtering, auto complete and fuzzy search Recommended Change - Filtering on Number of Arrivals / Departures Elastic has specifically noted the Passenger Search - Journey Details requirements: ● 2. g. Number of Arrivals >= ● 2.h. Number of Departures >= As per the functional requirements the filters should take into account the time filters applied by the user. Such filtering on aggregated data is not practical in Elasticsearch. Elastic recommends adding a metric that is easily searchable, e.g. number of trips last six months and that can be statically assigned to the passenger as part of the passenger record. 41 | elastic.co | © 2023 Elasticsearch B.V. All Rights Reserved. Ingestion Data will be ingested using custom Python code running within the Iceberg platform. Design of the application is out of scope for this document, Elastic makes the following overall recommendations: 1. Utilise the official Elasticsearch python client. This will speed up development and reduce risk 2. Use the Bulk API to write data to elasticsearch, single write requests are very expensive at scale. The Python client comes with helper classes for this purpose. 3. Size bulk requests appropriately, aiming for a 1-10MB payload is a good starting point 4. Use multiple workers to send data to Elasticsearch, however ensure that indexing load does not harm search speed 5. See also tune for disk usage, note don’t force merge indexing until after all writes have completed for an index Index Management Since Elasticsearch will need to support all CRUD operations, Elastic recommends using indices named with a date suffix. Index Naming Each index should be named according to the convention <data set name>-<date suffix>. Elastic recommends using monthly indices, as a suggestion the below example names can be used: ● flights-2025-08 ● passenger-2025-08 ● crew-2025-08 When writing data to Elasticsearch, the destination index suffix should be based on a date field that does not change for a specific record, for example, scheduled departure date. 42 | elastic.co | © 2023 Elasticsearch B.V. All Rights Reserved. Index Retention Index retention should be managed by using Elasticsearch index lifecycle management (ILM). Elastic recommends using the following policy for a 6 month policy: 1. Hot a. Rollover disabled b. Set index priority 100 2. Delete a. When 217 days old* * Note we need to keep the index for seven months, since the time is calculated from index creation and each index contains data up to creation date + one month. The ILM policy can be created using the following API call: None PUT _ilm/policy/dgarm { "policy": { "phases": { "hot": { "min_age": "0ms", "actions": { "set_priority": { "priority": 100 } } }, "delete": { "min_age": "217d", "actions": { "delete": { 43 | elastic.co | © 2023 Elasticsearch B.V. All Rights Reserved. "delete_searchable_snapshot": true } } } } } } Shard Sizing ● Elastic recommends 1 primary and 1 replica shard for each index as a starting point. ● If the monthly primary shard size grows beyond 50GB, Elastic recommends adjusting the primary shard setting to a sufficient number to bring down the shard size under 50GB. ● If the monthly index size, counting primaries only, grows beyond 200GB for any dataset, Elastic recommends transitioning to daily indices and setting the primary shard back to 1. Index Template Elastic recommends using index templates to apply mappings and settings to indices. Component templates can be used to reuse configuration between different index patterns. See further down for an example index template. Autocomplete The search application for searching fields requires autocomplete on the following fields: 1. Name 2. Airport 3. Nationality 4. Operator 44 | elastic.co | © 2023 Elasticsearch B.V. All Rights Reserved. Simple Autocompletion For simple strings, i.e., 2-4 in the list above, elasticsearch recommends mapping each field as keyword and using the terms enum API for autocompletion. The same approach can be applied to other fields, including pnr, document number, origins and destination. Name Autocompletion - Option 1 (recommended) The user may be typing names starting with either the first name or the last name. Elastic recommends introducing two fields; 1. firstname_lastname 2. lastname_firstname When autocompleting on name, the search client sends two terms enum requests in parallel,one to each field, and merges the results. This is following the same approach as the simple autocompletion. The selected option can then be used to make an exact query against the appropriate keyword field, or a text query as desired. Name Autocompletion - Option 2 If DGARM requires autocomplete to combine advanced query features such as fuzzy search, with autocomplete, the following approach can be used. 1. Copy the full name into two separate fields, for example: a. fullname_complete, mapped as search-as-you-type b. fullname, mapped as keyword 2. Generate name suggestions using a modified search-as-you-type query,, see example below a. Optional: apply any other user selected filters to discard irrelevant suggestions 3. Use aggregations to remove duplicate names Note Elastic does not recommend this option due to search latency and the impact on user experience. If DGARM requires this option, please consider: 1. Benchmark performance, for example using ESRally 45 | elastic.co | © 2023 Elasticsearch B.V. All Rights Reserved. 2. Ensure there is a delay between keypresses and sending an autocomplete request, so that not every keypress causes a request 3. Apply user selected filters to reduce the result set 4. Consider increasing the CPU count of Elasticsearch nodes if performance suffers None GET passengers/_search { "query": { "size": 0, "bool": { "must": [ { "multi_match": { "query": "mattias b", "type": "bool_prefix", "fields": [ "fullname_complete", "fullname_complete._2gram", "fullname_complete._3gram" ] } }, { "match": { "name_complete": { "query": "fullname_complete", "fuzziness": "AUTO" } } } ] } }, "aggs": { 46 | elastic.co | © 2023 Elasticsearch B.V. All Rights Reserved. "sample": { "sampler": { "shard_size": 100 }, "aggs": { "name": { "terms": { "field": "fullname" } } } } } } Note that the sampler aggregation is used to reduce the number of results analysed to the top N relevant hits in each shard, in this case 100. Ingest Pipeline To populate firstname_lastname and lastname_firstname fields, an index pipeline can be used. The pipeline below uses the set processor. The pipeline can be automatically executed by adding the default_pipeline setting to each index. None PUT _ingest/pipeline/passenger { "processors": [ { "set": { "field": "firstname_lastname", "value": "{{{firstname}}} {{{lastname}}}" 47 | elastic.co | © 2023 Elasticsearch B.V. All Rights Reserved. } }, { "set": { "field": "lastname_firstname", "value": "{{{lastname}}} {{{firstname}}}" } } ] } Index Template Please see below for an example index template, including mappings. Note that fields have generally been mapped as keyword to be used as filters, either through the use of drop downs or autocomplete, see the Autocomple section. None PUT _index_template/passenger { "index_patterns": [ "passenger-*" ], "template": { "settings": { "index.lifecycle.name": "dgarm", "number_of_shards": 1, "number_of_replicas": 1 }, "mappings": { "properties": { "pnr": { 48 | elastic.co | © 2023 Elasticsearch B.V. All Rights Reserved. "type": "keyword", "normalizer": "lowercase" }, "first_name": { "type": "keyword", "normalizer": "lowercase" }, "last_name": { "type": "keyword", "normalizer": "lowercase" }, "firstname_lastname": { "type": "keyword", "fields": { "text": { "type": "text" } } }, "lastname_firstname": { "type": "keyword" }, "date_of_birth": { "type": "date" }, "nationality": { "type": "keyword" }, "document_number": { "type": "keyword" }, "document_expiry_date": { "type": "date" }, "gender": { 49 | elastic.co | © 2023 Elasticsearch B.V. All Rights Reserved. "type": "keyword" }, "contact_number": { "type": "keyword" }, "contact_email": { "type": "keyword" }, "travel_agency": { "type": "keyword" }, "cabin_class": { "type": "keyword" }, "origin": { "type": "keyword" }, "destination": { "type": "keyword" }, "scheduled_departure": { "type": "date" }, "scheduled_arrival": { "type": "date" }, "actual_departure": { "type": "date" }, "actual_arrival": { "type": "date" } } } } 50 | elastic.co | © 2023 Elasticsearch B.V. All Rights Reserved. } 51 | elastic.co | © 2023 Elasticsearch B.V. All Rights Reserved. </nowiki> </pre>
Summary:
Please note that all contributions to TetraWiki may be edited, altered, or removed by other contributors. If you do not want your writing to be edited mercilessly, then do not submit it here.
You are also promising us that you wrote this yourself, or copied it from a public domain or similar free resource (see
TetraWiki:Copyrights
for details).
Do not submit copyrighted work without permission!
Cancel
Editing help
(opens in new window)
