The journald binary log is the richest source of system state on a systemd host, and journalctl is the tool that turns it from a firehose into an answer. The filters that matter in daily work are the unit filter, which narrows the output to one service, the priority filter, which narrows it to errors and warnings, and the time filter, which bounds the search to the window the incident covers. This article builds the vocabulary: the filter combinations that answer the daily questions, the persistent journal setup that makes history searchable, the periodic-failure analysis that finds the pattern in the noise, and the integration points with the other diagnostic tools in this series.
The filter combinations that answer daily questions
The most common question is what went wrong with this service in the last hour, and the answer is one command:
journalctl -u nginx.service --since -1h -p err
The -u flag filters by unit, the --since bounds the time, and -p err shows only priority error and above, the combination that produces the shortest useful output. The priority values in journald follow the syslog levels: emerg, alert, crit, err, warning, notice, info, debug, and the -p flag is inclusive, so -p err includes everything from emerg through err. The variants worth knowing:
journalctl -u nginx.service --since -1h -p warning # journalctl -u postgresql --since today -p err # journalctl -u app --since -30m
The first includes warnings alongside errors, the second bounds by today rather than a relative window, and the third drops the priority filter to show everything including informational messages for a service that is being debugged. Each combination answers a different question, and the tool's power is that the filters compose: unit plus time plus priority is the minimum that turns a million lines into ten.
The persistent journal and the boot-number navigation
Journald's default storage on some distributions is volatile, in memory and lost at reboot, and the persistent journal is the setup that makes history searchable across boots. The configuration is one line in /etc/systemd/journald.conf:
Storage=persistent
after which journalctl --list-boots shows every boot with its index, and the previous boot is -1, the one before that -2:
journalctl --list-boots -2 1a2b3c4d Mon 2026-10-12 03:00:01 UTC - Mon 2026-10-12 23:59:59 UTC -1 5e6f7a8b Tue 2026-10-13 03:00:00 UTC - Wed 2026-10-14 02:59:59 UTC 0 9c8d7e6f Wed 2026-10-14 03:00:00 UTC - current
The --boot=-1 flag reads the previous boot, the boot that crashed or was rebooted, and the combination with the other filters is the diagnosis command: journalctl -u nginx --boot=-1 -p err shows the errors from the previous boot's nginx, the evidence that survived the reboot. The boot navigation is the tool's answer to the question of what happened before the crash, and the boot number is the coordinate that turns the journal from a current-state log into a historical record.
The periodic failure analysis, finding the pattern
The failures that repeat on a schedule are the ones that individual log entries hide, and the journalctl vocabulary for finding them is the time-window analysis. The command that shows errors grouped by hour:
journalctl -u myapp --since -24h -p err | grep -oP "^-- Logs begin at.|^.\d\d:\d\d:\d\d" | cut -d: -f1-3 | sort | uniq -c | sort -rn
or more practically, the timestamp extraction that counts errors per hour across the day:
journalctl -u myapp --since -24h -p err -o short-iso | cut -dT -f2 | cut -d: -f1 | sort | uniq -c
The output is a frequency table: 14 errors at 03:00, 3 at 09:00, 0 elsewhere, and the pattern that appears is the answer to the question of whether the errors are random or scheduled. The errors that appear at exactly 03:00 every day are the cron job, the ones that appear every five minutes are the monitoring probe, and the ones that appear during business hours are the user traffic, and the three patterns have completely different diagnoses.
The output formats and the integration points
Journalctl's output formats serve different consumers, and the three that matter for integration are the short-iso format for human reading, the JSON format for machine parsing, and the export format for piping. The JSON output is the interface to jq, the JSON processor that turns log entries into structured data:
journalctl -u myapp -p err --since -1h -o json | jq -r '.MESSAGE'
which prints only the message text from each error entry, the extraction that feeds alerting scripts and log analysis. The export format is the binary stream that journalctl imports back, the journal's own portable format for shipping logs between hosts. The format flags compose with the filters, so the same filter combination produces the human output, the JSON for the script, or the export for the archive, depending on the consumer.
The maintenance commands and the closing checklist
The journal maintenance commands are the vacuum and the verification: journalctl --vacuum-size=500M bounds the journal to a size limit, journalctl --vacuum-time=30d bounds it to an age, and the combination is the retention policy that the /etc/systemd/journald.conf SystemMaxUse setting implements permanently. The verification is journalctl --verify, which checks the journal files' integrity, the corruption detector for the journal files themselves.
The checklist that closes the article is the daily diagnostic routine on a systemd server. Enable the persistent journal, because without it there is no history. Find the errors with unit, time and priority filters, the three-flag combination that turns the firehose into the answer. Check the previous boot after every crash. Analyze periodic failures with the frequency table, the pattern that names the cause. Feed the JSON output to the monitoring scripts. And maintain the journal's size with the vacuum commands, the cleanup that keeps the tool fast and the disk from filling.
The server whose journal is persistent, whose filters are memorized and whose periodic analysis runs on schedule is the server whose diagnostic capability is measured in minutes rather than in hours, and the tool that provides that capability is installed on every machine this series describes, waiting for the first question to be asked.
Field-specific filtering with the JSON output
Journald stores every log entry as a set of fields, and the JSON output exposes them all: MESSAGE for the text, _PID for the process id, _SYSTEMD_UNIT for the unit, PRIORITY for the level, __REALTIME_TIMESTAMP for the timestamp in microseconds. The field-specific filtering goes beyond the -u and -p flags into the kernel-level filtering that the journal supports natively:
journalctl _PID=1234 _COMM=myapp # journalctl _UID=1000 --since today # journalctl _SYSTEMD_UNIT=nginx.service _TRANSPORT=stdout
The _PID filter narrows to one process, useful when the same unit spawns multiple processes and the log lines from the child processes need to be separated from the parent. The _UID filter narrows to one user, useful on systems where multiple users' cron jobs write to the same journal. And the _TRANSPORT filter separates stdout logs from syslog logs from kernel messages, the three transports that journald collects into the same store.
The combination of field filters with the time and priority filters is the precise query: journalctl _SYSTEMD_UNIT=app PRIORITY=3 __REALTIME_TIMESTAMP>=timestamp shows only the err-priority messages from the app unit since a specific time, the query that the monitoring scripts build programmatically and that the human diagnostic runs by hand. The field names are documented in the systemd.journal-fields man page, and the man page is the reference that every diagnostic session consults once and then remembers.
The Grep integration and the compound pipelines
The journalctl output that the field filters cannot narrow further goes through grep, the text processor that turns log lines into matched lines, and the compound pipelines are the daily diagnostic tool on a server whose journald is the only log source. The pipeline that finds the errors mentioning a specific path:
journalctl -u app --since -1h -p err | grep /etc/app/
the errors that mention the config directory, the subset that points at the configuration problem rather than the code problem. The pipeline that counts unique error messages:
journalctl -u app --since -24h -p err | sed 's/.*ERR //' | sort | uniq -c | sort -rn | head -10
which extracts the error text, deduplicates and ranks it, the top-ten errors that the 24-hour window produced, and the top-ten list is the triage document that the next diagnostic session starts from. The pipelines compose naturally because journalctl's output is text, and the shell vocabulary that the bash articles in this series taught is the same vocabulary that the journal queries use.
**The tamper-resistance and the journal's security model
The journal's integrity model is the property that makes it the audit-grade log source: the entries are sealed with Forward Secure Sealing (FSS), the cryptographic mechanism that detects any modification of the historical entries, and the sealing is set up with one command:
journalctl --setup-keys
which generates the sealing key and the verification key, and the periodic verification with journalctl --verify confirms that no entry has been altered since the sealing key was created. The sealing is the answer to the question of whether the attacker who reached the host can erase their traces from the journal: with FSS enabled, the erasure is detected, and the detection is the evidence that the investigation needs.
The closing perspective on the journal as the diagnostic center
Journald is the diagnostic center of a systemd host, the single binary log that every service writes to and that every query reads from, and the tool that makes it useful is the filter vocabulary that turns the firehose into answers. The server whose journal is persistent, whose filters are in the muscle memory and whose periodic analysis is scheduled is the server whose diagnostic capability is the diagnostic capability that this series has been building toward: fast, structured, integrated with the other tools, and always available because the journal is always running.
The journal is the first command on every diagnostic session.
The journal that ships with systemd on every current Linux distribution is the log infrastructure that the previous generation of servers assembled from rsyslog, logrotate and custom scripts. The journal is always running, always structured, always queryable, and the tool that queries it is the first command on every diagnostic session.
The field-specific filtering with the JSON output deserves its deeper example, because the JSON structure is the interface that the monitoring scripts and the alerting tools consume. The journalctl -o json output produces one JSON object per line, and the jq processor extracts the fields that the script needs:
journalctl -u app -p err -o json | jq -r '.MESSAGE'
which prints the message text of every error from the app service, the extraction that the alerting pipeline reads. The -r flag prints raw strings without the JSON quoting, and the -c flag prints compact JSON for the scripts that parse it. The field names in the JSON are the same ones the journalctl field filters use: _PID, _UID, _COMM, _SYSTEMD_UNIT, PRIORITY, __REALTIME_TIMESTAMP, and the field reference is the systemd.journal-fields man page.
The journal's disk management is the maintenance command that keeps the journal from consuming the disk, and the two vacuum commands are the runtime tools:
journalctl --vacuum-size=500M # journalctl --vacuum-time=30d
which remove the oldest journal files until the total is under 500 megabytes or until only the last thirty days remain. The SystemMaxUse=500M setting in journald.conf implements the same limit permanently, and the journald service reads the setting at every start, the configuration that prevents the journal from filling the disk the way an unbounded log does. The verification command, journalctl --verify, checks the journal files for corruption, the integrity check that the monitoring runs after any disk event, because a corrupted journal is a diagnostic source that cannot be trusted.
The persistent journal setup is the prerequisite that every other command in this article depends on, and the setup is one line plus one service restart: the Storage=persistent line in journald.conf and the systemctl restart systemd-journald that applies it. Without the persistent storage, the journal survives only in memory, and the reboot that the diagnostic needs to analyze erases the evidence, the scenario that every first encounter with a volatile journal teaches.
The diagnostic session that the journal enables is the workflow that the other articles in this series reference: the auditd articles use journalctl to read the audit stream, the systemd articles use it to read the service output, the fail2ban articles use it to read the auth log, and the backup articles use it to read the tool's own output. The journal is the single log source that the whole series reads from, and the filters that this article teaches are the ones that make each article's diagnostic section work.