The central log server collects entries from the fleet, and the query interface that makes the collection useful is the difference between a log archive and a diagnostic tool. Loki and Promtail are the Grafana ecosystem's answer: Promtail scrapes logs from every source, attaches labels that make each entry queryable, and ships them to Loki, which indexes the labels and stores the entries for the LogQL query language to search. This article builds the complete setup: the Loki server, the Promtail agents on the fleet, the label design that makes queries fast, the LogQL vocabulary, and the integration with the other tools in this series.
The Loki architecture and the label-indexed model
Loki indexes labels, not log content, and the design is the fundamental choice that separates it from Elasticsearch: the query filters by label values first, and only the matching entries are searched for text. The label set on each stream is the query's primary dimension, and the label design is the performance decision that determines whether queries take milliseconds or minutes. The architecture has three components: Loki the server that stores and indexes, Promtail the agent that scrapes and ships, and Grafana the dashboard that queries and displays, each deployed independently and communicating over HTTP.
Loki's storage is the object store or the local filesystem, and the retention is managed by the compactor, which deletes the expired chunks on the schedule the configuration sets. The single-binary deployment mode serves the small and medium fleet, the one process that does everything, and the microservices mode splits the components for the large fleet where the read and write paths need independent scaling.
Promtail configuration for journal and file scraping
Promtail runs on every server and scrapes the log sources that the server produces. The configuration file, /etc/promtail/config.yml, defines the scrape targets and the label pipeline:
server:
http_listen_port: 9080
positions:
filename: /tmp/positions.yaml
clients:
- url: http://loki.internal:3100/loki/api/v1/push
scrape_configs:
- job_name: journal
journal:
labels:
host: app01
job: journal
relabel_configs:
- source_labels: [__journal__systemd_unit]
target_label: unit
- source_labels: [__journal_priority]
target_label: priority
- job_name: applog
static_configs:
- targets: [localhost]
labels:
host: app01
job: applog
__path__: /var/log/myapp/*.log
The journal scrape reads from the systemd journal with the same field filters that the journalctl article described, and the relabel_configs extract the journal fields into Loki labels. The file scrape watches the application's log files, and the labels on each stream are the dimensions that the LogQL queries filter on. The positions file tracks how far each source has been read, the resume point that prevents duplicate entries after a Promtail restart.
The label design and the cardinality discipline
The labels are the index, and the label design is the performance decision that determines whether queries are fast. The good labels are the low-cardinality dimensions: the hostname, the service name, the environment, the log level. The bad labels are the high-cardinality dimensions: the user id, the request id, the timestamp, the fields whose values are unique per entry, because each unique label value combination creates a new stream, and the stream explosion is the performance death of the Loki instance.
The label discipline is the design review that the fleet applies before deploying Promtail: every label is documented, every label's cardinality is estimated, and the labels whose cardinality exceeds a hundred values are moved from labels to structured metadata, the Loki feature that stores the high-cardinality data alongside the entries without indexing it. The label set is the query's interface, and the interface that the fleet designs is the interface that the fleet queries.
LogQL, the query language for label-filtered log search
LogQL is Loki's query language, and the syntax is the PromQL that the Prometheus users already know, extended with log-specific operations. The basic log query:
{host=\"app01\", unit=\"nginx.service\"} |= \"error\"
which selects all entries from app01's nginx that contain the word error, the label filters narrowing to the stream and the line filter narrowing to the content. The metric queries extract values from the log entries:
rate({job=\"applog\"} |= \"ERROR\" [5m])
which counts the ERROR lines per second over five minutes, the metric that the Grafana dashboard graphs and the alerting rule thresholds. The pipeline stages between the selector and the metric, the regexp and the json stages, extract the fields from the log lines, and the extracted fields become the metric's labels:
rate({job=\"applog\"} | regexp \level=(?P<level>\\w+)\ | level=\"error\" [5m])
which counts the errors whose level field is error, the structured extraction that the JSON logging from the previous articles feeds.
The Grafana integration and the dashboard design
Grafana is the query interface that the team uses daily, and the Loki data source is configured in Grafana with the Loki server's URL. The dashboards that the fleet builds are the queries from the runbook rendered as panels: the error rate per service, the log volume per host, the top-N error messages, and the log stream for the incident investigation. The dashboard design follows the label structure: one row per service, one panel per metric, and the log stream panel for the drill-down.
The retention and the storage management
Loki's retention is managed by the compactor's retention_enabled flag and the retention_period setting, and the retention deletes the expired chunks on the compactor's schedule. The storage backend for the small fleet is the local filesystem, and the production backend is the S3-compatible store that the restic article already established, the same object storage serving both the backup and the log archive. The retention period is the compliance policy's number, and the storage cost is the calculation that the fleet reviews on the same quarterly calendar as the backup costs.
The checklist for the log aggregation deployment
The deployment checklist follows the article's order. Deploy Loki on the central server, configure the storage backend and the retention. Deploy Promtail on every server, configure the journal and file scrapes with the label set. Design the labels with the cardinality discipline. Build the Grafana dashboards with the LogQL queries from the runbook. Wire the alerting rules to the metric queries. And test the query performance under real label cardinality, because the label design that works in development may not work at production scale, and the test that catches the cardinality explosion before production is the test that saves the Loki deployment.
The pipeline stages for structured log parsing
Promtail's pipeline stages transform each log entry between the scrape and the push, and the stages that production uses are the regex extraction, the JSON parsing and the timestamp override. The regex stage extracts fields from unstructured log lines and promotes the level to a label. The JSON stage parses structured log lines from the structured logging article. The timestamp stage overrides the entry's timestamp with a value from the log line.
The multi-tenant setup and tenant isolation
Loki supports multi-tenancy through the X-Scope-OrgID header, and the multi-tenant mode is the production pattern for teams sharing the Loki instance. Each Promtail agent is configured with its tenant ID, and the queries from Grafana carry the tenant header. The authentication in front of Loki is the security layer that the bastion article's pattern provides.
The alerting rules and the recording rules
The Loki alerting rules run the LogQL metric queries on a schedule and fire alerts when results cross thresholds. The alert that catches the error rate spike uses the rate function over five minutes with a threshold. The recording rules pre-compute the expensive queries and store the results as new metrics.
The comparison with Elasticsearch
Loki indexes labels, not log content, which makes label-filtered queries fast at lower storage cost. Elasticsearch indexes full text, which makes text search fast but costs more storage and CPU. The fleet that queries by label chooses Loki; the fleet that queries by arbitrary text needs Elasticsearch.
The closing checklist
Deploy Loki on the central server. Deploy Promtail on every server with the label set. Design labels with the cardinality discipline. Build the Grafana dashboards. Wire the alerting rules. Test query performance under real cardinality. The fleet that runs this stack has turned its log sources into a unified observability platform.
The pipeline stages for structured log parsing
Promtail's pipeline stages transform each log entry between the scrape and the push, and the stages that production uses are the regex extraction, the JSON parsing and the timestamp override. The regex stage extracts fields from unstructured log lines and promotes the level to a label. The JSON stage parses structured log lines from the structured logging article. The timestamp stage overrides the entry's timestamp with a value from the log line. Each stage transforms the entry in sequence, and the pipeline is the transformation that turns the raw log text into the labeled, structured stream that Loki indexes.
The multi-tenant setup and tenant isolation
Loki supports multi-tenancy through the X-Scope-OrgID header, and the multi-tenant mode is the production pattern for teams sharing the Loki instance. Each Promtail agent is configured with its tenant ID, and the queries from Grafana carry the tenant header, so team A's logs are invisible to team B. The authentication in front of Loki is the security layer that the bastion article's pattern provides, and the Loki instance behind the authentication is the shared infrastructure that the fleet's teams use without seeing each other's data.
The alerting rules and the recording rules
The Loki alerting rules run the LogQL metric queries on a schedule and fire alerts when results cross thresholds. The alert that catches the error rate spike uses the rate function over five minutes with a threshold, and the recording rules pre-compute the expensive queries and store the results as new metrics, the optimization that makes the dashboards fast.
The comparison with Elasticsearch
Loki indexes labels, not log content, which makes label-filtered queries fast at lower storage cost. Elasticsearch indexes full text, which makes text search fast but costs more storage and CPU. The fleet that queries by label chooses Loki, and the fleet that queries by arbitrary text needs Elasticsearch. The operational cost comparison is equally clear: Loki's single binary is simpler to deploy and operate than an Elasticsearch cluster.
The closing checklist for the log aggregation stack
Deploy Loki on the central server with the storage backend and the retention policy. Deploy Promtail on every server with the label design and the pipeline stages. Design the labels with the cardinality discipline. Build the Grafana dashboards with the LogQL queries. Wire the alerting rules to the metric queries. Test the query performance under real label cardinality. The fleet that runs this stack has turned its log sources into a unified observability platform.
The deployment that this article describes is the stack that the fleet builds once and operates continuously: Loki for the storage and indexing, Promtail for the collection and labeling, Grafana for the query and display, and the alerting rules for the proactive detection. The fleet that runs this stack has the diagnostic capability that the previous articles built the log sources for, and the query interface that makes the logs searchable across the entire fleet from a single interface is the tool that the on-call engineer opens at 3 a.m. and finds the answer in seconds rather than in hours of SSH sessions across individual servers.