The text log is the interface that humans read, and the structured log is the interface that machines parse. Journald stores every entry as a set of fields, and the JSON output exposes the structure to the tools that process logs programmatically: the monitoring scripts that extract metrics, the alerting systems that match patterns, the analysis pipelines that correlate events across services. This article builds the structured logging practice: the journalctl JSON output modes, the jq parsing vocabulary, the structured logging patterns in application code that make the JSON fields meaningful, and the integration with the analysis systems that consume the output.
The JSON output modes and the field model
Journalctl's -o json mode produces one JSON object per line, and the -o json-pretty mode produces one indented object per entry, the two modes serving the script consumer and the human reader respectively. The field model is the structure that the JSON exposes:
journalctl -u myapp -n 1 -o json-pretty
which prints one entry with every field: MESSAGE for the text, PRIORITY for the level, _PID for the process, _SYSTEMD_UNIT for the unit, __REALTIME_TIMESTAMP for the timestamp, and the custom fields that the application added through the structured logging API. The field names are documented in the systemd.journal-fields man page, and the trusted fields, the ones prefixed with underscore, are set by journald itself rather than by the application, which makes them the authoritative metadata.
The -o json-pretty output for a single entry is the readable reference, and the -o json output for a batch is the machine interface, one line per entry, each a self-contained JSON object, the NDJSON format that every streaming consumer understands. The field extraction with jq is the next section's subject, and the field model is the vocabulary that the extraction queries use.
The jq vocabulary for journal parsing
jq is the command-line JSON processor, and the journal's JSON output is the input it consumes. The basic extraction:
journalctl -u myapp -p err -o json | jq -r .MESSAGE
which prints the message text of every error from myapp, the -r flag for raw strings without JSON quoting, and the pipeline that feeds alerting scripts. The more complex extraction that combines fields:
journalctl -u myapp -o json | jq -r 'select(.PRIORITY <= 4) | .__REALTIME_TIMESTAMP + " " + .MESSAGE'
which selects errors by priority value, combines the timestamp with the message, and produces a custom format that the downstream tool expects. The select filter is jq's conditional, the arithmetic on PRIORITY works because journald stores it as a number, and the string concatenation composes the output format.
Structured logging in application code
The applications that log through journald's native interface, systemd's sd_journal_print or the Python systemd.journal module, can add custom fields to their entries, and the custom fields are what turn the JSON from a flat text dump into a structured record:
import systemd.journal
systemd.journal.send('Payment failed', PRIORITY=3, ORDER_ID='12345', AMOUNT=99.99)
and the JSON output for this entry carries ORDER_ID and AMOUNT as first-class fields alongside MESSAGE, fields that jq can filter and aggregate on:
journalctl -u payments -o json | jq -r 'select(.ORDER_ID) | .ORDER_ID' | sort -u
which lists every order that had a payment error, the query that the traditional text log cannot answer without regex parsing and that the structured log answers with a one-line jq filter. The custom fields are the difference between logging and structured logging, and the application code that adds them is the investment that every later query returns.
The integration with analysis systems
The JSON output feeds every analysis system that accepts structured input, and the three integration patterns cover the landscape. The script pattern: the shell pipeline with journalctl and jq, the simplest integration that the monitoring scripts and the alerting hooks use. The streaming pattern: journalctl -f with the JSON output, the live stream that a consumer reads continuously, the log shipper that does not need a separate tool. The batch pattern: journalctl exporting a time range to a file, the analysis that runs offline on the exported data.
The rsyslog bridge is the integration that connects the journal to the traditional log infrastructure: rsyslog reads from the journal through imjournal and forwards to the central server through the transport that the log-forwarding article established, the bridge that means the JSON fields survive the transport if the forwarding is configured for the structured format. The Loki integration from the logging article in this series reads the journal through promtail, and the labels that promtail extracts from the journal fields are the same fields that the jq queries filter on, the consistency that makes the two interfaces, jq and Loki, query the same data.
The maintenance and the closing checklist
The journal's maintenance commands from the previous article apply unchanged, and the JSON-specific maintenance is the schema management: the custom fields that applications add are the schema, and the schema documentation is the field reference that the analysis queries depend on. The field naming convention, the application's field names consistent across services, is the schema discipline that makes cross-service queries possible, and the convention is documented in the runbook alongside the service inventory.
The checklist that closes the article is short. Enable the persistent journal, the prerequisite from the previous article. Learn the JSON output mode and the field model. Master the jq vocabulary for filtering and extracting. Add structured fields in application code where the flat message is insufficient. Wire the JSON output into the monitoring and alerting. And maintain the field naming convention across services, the schema that makes the structured logging a platform rather than a collection of individual log streams.
The server whose journal is persistent, whose applications log structured fields and whose analysis queries run through jq is the server whose diagnostic capability is the structured query, and the structured query is the tool that answers the questions that text grep cannot: how many, which user, what amount, which correlation id, the questions that the structured fields answer and that the traditional text log never will.
The Python systemd journal module in practice
The Python systemd.journal module is the application-side bridge, and its usage in a real service shows the structured logging pattern in full. The import is one line, the send function carries the message and the fields, and the fields are the schema that the JSON queries depend on:
from systemd import journal
def log_order_event(order_id, status, amount, error=None):
fields = {
'ORDER_ID': order_id,
'STATUS': status,
'AMOUNT': str(amount),
'PRIORITY': journal.LOG_ERR if error else journal.LOG_INFO,
}
if error:
fields['ERROR_DETAIL'] = str(error)
journal.send(f'Order {order_id}: {status}', **fields)
and the JSON output for this entry carries ORDER_ID, STATUS, AMOUNT and ERROR_DETAIL as fields that jq and the monitoring query, the structured record that the text log line Order 12345: failed cannot provide. The function wrapper is the pattern that makes structured logging maintainable: one function per event type, the fields documented in the function signature, the PRIORITY set from the event's severity, and the applications that call the wrapper get structured logging for free.
The Python module's dependency is systemd and python3-systemd, the package that the distribution ships, and the module works in virtual environments only if the system packages are available, the dependency that the container builds handle through the system Python rather than a venv-specific install.
The rate limiting and the journal's self-protection
Journald protects itself from log floods through rate limiting, and the rate limiter is the setting that prevents a misbehaving application from filling the journal and consuming the CPU. The default configuration, RateLimitIntervalSec=30s and RateLimitBurst=10000, allows ten thousand entries per thirty seconds per service, and the application that exceeds the burst has its messages dropped until the rate falls below the limit. The dropped messages appear in the journal as a suppression notice, the line that says Suppressed N messages from service, and the suppression notice is the signal that the application needs its own rate limiting rather than relying on the journal's.
The rate limiting configuration is per-service in journald.conf, and the per-service override is the LogRateLimitIntervalSec and LogRateLimitBurst settings in the unit file:
[Service]
LogRateLimitIntervalSec=10s
LogRateLimitBurst=100
which allows only one hundred entries per ten seconds for this service, the tighter limit for the service that is known to be chatty. The rate limiting and the journal vacuum from the previous article together form the journal's self-protection, the disk guard and the CPU guard that keep the logging infrastructure from becoming the problem it is supposed to diagnose.
The systemd service and the logging best practices
The systemd service unit is the configuration that connects the application's stdout to journald, and the standard input and error streams are the simplest logging interface: the application that writes to stdout has its output captured by journald automatically, with the unit name, the priority and the timestamp all set by the service manager.
The best practices for application logging through stdout follow the journal's field model: prefix the message with the priority in the syslog format, <3> for err, <6> for info, and journald parses the prefix into the PRIORITY field automatically. The structured fields go through the sd_journal API rather than through stdout, because stdout carries the human-readable message while the structured fields carry the machine-readable data, and the two interfaces compose: the human reads the message, the machine reads the fields.
The closing argument for structured logging
Structured logging is the difference between a log that answers questions and a log that generates more questions, and the journal's JSON output is the interface that turns the structured fields into queries. The traditional text log, grep through a wall of unstructured lines, answers the question of what happened but not of how many, to whom, or at what cost, and the structured fields answer those questions with the jq vocabulary that the analysis scripts use.
The investment that structured logging requires is the application-side change, the fields added at the point where the event is known, and the return is every subsequent query, every alerting rule and every dashboard metric that the fields make possible. The journal's JSON output is the delivery mechanism, jq is the query language, and the integration points, the monitoring, the alerting, the analysis, are the consumers that the structured fields feed, the complete pipeline from the application code to the answer on the dashboard.
The structured logging practice this article describes is the foundation that the Loki article in this series builds on, and the fields that the application adds through the journal module are the same fields that promtail extracts as labels. The investment is made once in the application code.
The journal's JSON output is the interface that turns logging from a text stream into a structured data source, and the structured source is the one that the analysis tools, the alerting rules and the dashboard queries can consume. The investment in structured fields pays for itself on the first incident where the query needs to filter by order id or by user id rather than by grepping through a wall of text.