The ELK stack, Elasticsearch, Logstash and Kibana, is the log analysis platform that the enterprise world standardized on, and Filebeat is the lightweight shipper that feeds it: a Go agent that tails log files, parses them and ships the entries to Elasticsearch or Logstash with minimal resource cost. The feature that makes Filebeat essential for the Java fleet is the multiline parser, which assembles the multi-line stack traces that Java applications produce into single events rather than shipping each line as a separate entry. This article builds the complete setup: the Filebeat configuration, the multiline pattern for Java stack traces, the Elasticsearch output and index management, the ILM lifecycle, and the Kibana queries that the fleet uses for analysis.
The architecture and where Filebeat sits
The ELK architecture has four layers: the applications that produce logs, the shippers that collect and forward them, the indexing and storage layer that makes them searchable, and the visualization layer that presents them. Filebeat is the shipper, installed on every server, reading the log files from the paths the configuration names, and shipping the parsed entries to Elasticsearch directly or through Logstash for additional processing. The direct-to-Elasticsearch path is the simpler architecture, and the Logstash intermediary is the path for the fleets that need enrichment, filtering or format transformation between the shipper and the index.
The contrast with the Loki stack from the previous article is the indexing model: Elasticsearch indexes the full text of every log entry, making text search fast at the cost of storage and indexing CPU, while Loki indexes only labels and searches content at query time. The fleet that needs full-text search across log content chooses Elasticsearch, and the fleet that needs label-based filtering at lower cost chooses Loki, with some fleets running both for different tiers of log analysis.
Filebeat configuration for journal and file inputs
Filebeat's configuration is a YAML file at /etc/filebeat/filebeat.yml, and the inputs section defines what to collect. The file input for application logs:
filebeat.inputs:
- type: filestream
id: applog
paths:
- /var/log/myapp/*.log
fields:
host: app01
service: myapp
fields_under_root: true
The filestream input is the modern replacement for the deprecated log input, and the fields attach metadata to every entry from this input. The journald input via the systemd module reads the journal directly:
filebeat.modules:
- module: system
syslog:
enabled: true
and the system module covers the standard syslog and auth logs, the sources that every server produces. Each input carries its own fields, and the fields are the metadata that the Elasticsearch queries filter on.
The multiline pattern for Java stack traces
The Java stack trace is the multiline log entry that breaks naive log shippers: the exception line starts a new event, but the subsequent lines, the at com.example lines and the Caused by lines, are continuations of the same event, and the shipper that treats each line as a separate entry produces hundreds of entries for one exception. The multiline pattern assembles the continuations:
filebeat.inputs:
- type: filestream
id: java-app
paths:
- /var/log/myapp/app.log
parsers:
- multiline:
type: pattern
pattern: '^\\s+at |^\\s+Caused by|^\\s+\\.\\.\\. \\d+ more'
negate: false
match: after
The pattern matches lines that start with whitespace followed by at, Caused by, or the ellipsis continuation, and the match after means these lines are appended to the previous event. The negate false means the pattern identifies continuation lines, and the assembled event is a single entry containing the exception header and the full stack trace. The multiline parser is the feature that makes Filebeat essential for Java fleets, and the pattern is the configuration that every Java service's Filebeat input carries.
The Elasticsearch output and index management
The output section sends the collected entries to Elasticsearch:
output.elasticsearch:
hosts: ["https://es.internal:9200"]
index: "filebeat-%{[agent.version]}-%{+yyyy.MM.dd}"
username: filebeat-writer
password: the-credential
ssl.certificate_authorities: ["/etc/filebeat/ca.crt"]
The index name carries the Filebeat version and the date, the daily index pattern that ILM manages and that the retention policy deletes. The username and password authenticate through Elasticsearch's security layer, and the CA certificate verifies the Elasticsearch server's TLS certificate.
The ILM lifecycle and index retention
Elasticsearch's Index Lifecycle Management (ILM) is the policy that manages each index from creation through deletion, and the ILM policy is the retention mechanism that replaces the logrotate approach for the indexed logs. The ILM policy phases: the hot phase for the active index receiving writes, the warm phase for the read-only index on slower storage, the cold phase for the archived index, and the delete phase for the index past its retention. The policy is defined in Elasticsearch, not in Filebeat, and the Filebeat output references the policy through the index template.
The ILM setup is an Elasticsearch-side configuration, and the Filebeat side only needs the index name pattern that matches the ILM policy's index template. The retention that ILM enforces is the same compliance-driven retention that the logrotate article established, translated from filesystem rotation to Elasticsearch index lifecycle.
Kibana queries and the dashboard
Kibana is the search and visualization interface, and the queries that the fleet runs are the Kibana Query Language (KQL) expressions that filter the indexed entries. The query for the Java exceptions in the last hour:
service:myapp and message:Exception
The dashboard that the team builds is the set of visualizations from the saved queries: the exception count over time, the error distribution by service, the response time histogram, and the log stream for the drill-down. The dashboard is the tool that the on-call opens during an incident, and the queries are the ones that the KQL article's diagnostic pattern follows.
The checklist for the ELK deployment
The deployment follows the article's order. Install Elasticsearch on the central server, configure security and the ILM policy. Install Kibana for the query interface. Install Filebeat on every server, configure the inputs with the multiline pattern for the Java services. Configure the Elasticsearch output with the TLS and credentials. Test the multiline parsing with a real stack trace. Test the ILM by confirming the index transitions. Build the dashboards. And monitor the Elasticsearch cluster's health and disk usage, because the indexing load that a high-volume fleet produces is the resource that the Elasticsearch cluster must be sized for.
The processor pipeline for enrichment
Filebeat's processors transform entries between the input and the output, and the processors that production uses are the add_fields, the drop_event and the decode_json_fields. The add_fields processor attaches environment metadata to every entry from a specific input. The drop_event processor filters out the entries that the index does not need. The decode_json_fields processor parses JSON-formatted log lines into structured fields, the applications that log JSON natively getting their fields indexed without a Logstash intermediary.
The resource tuning for the high-volume fleet
Filebeat's resource consumption is the concern on servers that already run production workloads, and the tuning parameters control the trade between shipping latency and resource cost. The registry.flush_interval controls how often the file position is saved. The bulk_max_size controls the Elasticsearch bulk API batch size. The high-volume fleet tunes these upward at the cost of higher memory usage.
The comparison with the Loki stack
The ELK and Loki stacks serve different query patterns. ELK's full-text indexing answers every content query instantly. Loki's label indexing answers label queries at lower cost. The fleet that queries by label chooses Loki. The fleet that searches by content chooses Elasticsearch.
The closing frame
Filebeat is the shipper that the ELK architecture needs, and the multiline parser is the feature that makes it work for the Java fleet. The architecture composes with the rest of this series: the journald articles build the local log, the rsyslog article builds the transport, and the Filebeat article builds the indexed analysis layer. The complete stack from the application's stdout to the Kibana dashboard is the log architecture that the enterprise world standardized on.
The multiline pattern deserves a second example for the Python traceback, because the pattern for Java does not match Python's continuation lines. The Python traceback starts with Traceback and continues with indented lines, and the multiline pattern that assembles it:
parsers:
- multiline:
type: pattern
pattern: '^\\s'
negate: false
match: after
matching any line that starts with whitespace, the Python convention for traceback continuation. The pattern is broader than the Java one, and the broader pattern is correct for Python because every indented line in a Python traceback belongs to the same exception.
The index template that Filebeat creates in Elasticsearch defines the field mappings, and the field mappings are the schema that the Kibana queries depend on. The dynamic mapping that Elasticsearch applies by default is usually correct for text fields, but the explicit mapping for the timestamp field, the keyword fields and the numeric fields is the schema discipline that prevents the mapping explosion that the high-cardinality fields cause. The index template is defined in Elasticsearch, not in Filebeat, and the template's field mappings are the schema that the fleet maintains alongside the ILM policy.
The Filebeat registry, the file that tracks how far each input has been read, is the state that prevents duplicate entries after a restart. The registry lives in /var/lib/filebeat/registry, and the registry's integrity is the prerequisite that the restart depends on: a corrupted registry causes Filebeat to re-read from the wrong position, producing duplicate entries or missing entries. The registry's backup is part of the server's backup scope, and the registry's state after a clean shutdown is the state that the next start resumes from.
The monitoring of Filebeat's own health is the meta-monitoring that the fleet adds to the shipper: Filebeat exposes its own metrics through the HTTP endpoint, the events processed, the events shipped, the errors, and the metrics feed the same Prometheus monitoring that the rest of the infrastructure uses. The Filebeat metrics are the health signals that confirm the shipper is alive and shipping, and the alert on the metrics absence is the alert that catches the shipper that stopped without a log entry.
The deployment checklist for the ELK stack is the article compressed into the working order. Install Elasticsearch, configure security and the ILM policy. Install Kibana. Install Filebeat on every server, configure the inputs with the multiline pattern for the Java and Python services. Configure the output with TLS and credentials. Test the multiline parsing. Test the ILM transitions. Build the Kibana dashboards. Monitor Filebeat's metrics and the Elasticsearch cluster health. The stack that passes the checklist is the log analysis platform that the fleet's diagnostic capability is measured by, and the platform is the one that the enterprise world standardized on because the search and the visualization are the capabilities that the incident response needs.