Files like /etc/passwd and /etc/sudoers decide who exists on a system and who can become root, and yet on a default installation their modification leaves no trace beyond a changed timestamp that anything could have produced. The kernel's audit subsystem closes that gap: with a handful of rules, every write, attribute change or execution touching the critical files produces a structured, timestamped record naming the actor, the operation and the full command line. This article builds the setup on a real server: the rules file layout and how rules load at boot, working watches for the identity and privilege files, directory-level rules for whole configuration trees, searching the trail when a change is noticed, and pushing the information to a human before the trail goes stale.
What auditd records and why it survives tampering attempts
The audit subsystem has two parts. The kernel generates events, and the auditd service collects them into /var/log/audit/audit.log, the single local file where every rule match lands as one or more structured records. Each event carries the system call that triggered it, the process and user responsible, the audit user id that tracks the original login even through privilege changes, the file path, the operation, and the exit status. The records are terse but complete, and their density is deliberate: they are written for tools to parse and for ausearch to answer questions with, not for casual reading.
A property that matters more than any single feature: the log is append-only by design, and only the audit service writes to it. An intruder operating as root can stop the service and truncate files, which is why serious environments forward the stream to a central collector in real time, a topic with its own section below. But even the local file alone changes the economics of quiet tampering, because deleting the evidence of a change to /etc/passwd now means leaving evidence of the deletion too, and the timestamps cross-correlate.
On a fresh system the service runs with a default ruleset that captures almost nothing file-specific, which is exactly as it should be, because rules are a site's statement about what it considers critical, and no vendor can guess that. The first check on any server is therefore auditctl -s, which prints the enabled status, and a look at /etc/audit/audit.rules or the rules.d directory to see what, if anything, the baseline includes.
The rules file layout and how rules load at boot
Rules typed with auditctl at the command line take effect immediately and vanish at the next reboot, which makes the command line the right place for experiments and the wrong place for policy. Persistent rules live as files under /etc/audit/rules.d/, conventionally named with a numeric prefix that controls loading order, and the tool augenrules compiles them into /etc/audit/audit.rules, the single file the service reads at startup. The canonical workflow is: write rules in rules.d, run augenrules --load, verify with auditctl -l, and never edit audit.rules by hand.
A minimal starting file, /etc/audit/rules.d/50-critical.rules, demonstrates the two rule families this article uses. Watch rules attach to a path:
-w /etc/passwd -p wa -k identity
-w /etc/sudoers -p wa -k sudoers
-w /etc/sudoers.d/ -p wa -k sudoers
The -w flag names a file or directory to watch, -p selects the operations that trigger a record, and -k attaches a key that turns the raw event into something searchable. The permission letters are compact: w fires on writes and file creation, a on attribute changes like permissions and ownership, r on reads, and x on execution. The wa pair is the classic choice for configuration files, recording modification and metadata changes while ignoring the harmless constant reading that services do.
Syscall rules are the second family, more verbose and more precise, capable of catching operations that watch rules miss, such as a rename that replaces a file without ever writing to the old inode. A production-grade identity rule set uses both: the watch for the common cases, and a syscall rule for the corner:
-a always,exit -F arch=b64 -S rename,unlink -F path=/etc/passwd -k identity
The -a always,exit placement attaches the rule at syscall exit so the exit code is captured, arch=b64 filters the 64-bit syscall table so 32-bit noise does not duplicate, and the -S list names the operations. The same pattern guards /etc/sudoers, and because sudo reads drop-in files under /etc/sudoers.d as well, a watch on the directory itself closes the gap where someone adds a permissive file instead of editing the main one.
After loading, the state of the subsystem is inspectable three ways: auditctl -l lists the currently loaded rules, auditctl -s prints subsystem status including the rate of events, and the log itself starts filling. A test run belongs here, before anything else: change something harmless in a watched path, touch /etc/sudoers to bump its timestamp or chown it, then search the log as the next section shows. The rule that has never fired an event you have personally seen is a rule you cannot trust in an incident.
Searching the trail when a change is noticed
The companion tool ausearch turns the audit log into answers, and its filters mirror the way questions actually arrive. The key flag matches the -k from the rules:
ausearch -k identity --format text
which returns every event from the identity rules, each as a numbered group of records with the type, timestamp, actor, and operation. The time filters bound any search, -ts today or -ts 14:00 or the verbose form -ts 2026-10-14 21:00:00, and the combination of key plus time window is the daily driver: someone noticed a changed file, the search asks what touched it in the relevant hours.
The --format text option renders records in a human layout, which is worth using whenever the output is going into a ticket or a message to a colleague, while the raw format remains the parseable default. The -f flag searches by file path instead of key, ausearch -f /etc/passwd, which finds events from any rule, including the baseline ones, that mention the path. The audit user id deserves a habit of its own: the auid field in the output names the human whose session performed the change, even when the command ran as root, because the kernel remembers the login the privilege chain grew from. That single field answers the question that classic logs cannot, which human, not which account, made this change.
Summaries come from aureport, the aggregation sibling: aureport --config-files prints a summary of events tagged by the file-related rules over the whole log, and a daily cron job formatting that output into a digest is the cheapest notification mechanism that exists, covered properly two sections down.
Notifications that reach a human
A log nobody reads is a time capsule, and the distance between an audit record and a human noticing is where most setups quietly fail. The mechanisms fall into three tiers, and choosing among them is a decision about response time rather than about technology. The baseline tier is the digest: a scheduled job that runs ausearch or aureport for the last window and emails or posts the result to the team channel, giving a daily glance at everything the rules caught. It costs ten lines of shell and requires nothing new on the server:
#!/bin/bash
ausearch -k identity,sudoers -ts yesterday --format text | mail -s "audit digest $HOSTNAME"
The middle tier is forwarding the stream to a central collector, the same architecture the sudoreplay article in this series uses: rsyslog or a dedicated audit relay ships the records to a host that applies its own alerting, so a rule match becomes a dashboard event within seconds. The audit subsystem supports this natively through the audisp-syslog plugin or the audisp-remote plugin, the latter speaking the audit protocol with disk buffering and reconnection, and a production tier treats the remote collector as the primary record and the local file as the cache.
The strict tier, for the files where minutes matter, adds a tight loop: a small service or scheduled script watching the log for a specific key and paging on match. It is deliberately not the default recommendation, because paging on every sudoers touch has a false-alarm rate that trains the on-call to ignore it, but for a compliance regime where a change to /etc/passwd must provoke a human within the hour, it is the honest implementation of that requirement.
Performance, disk and the immutable finish
Every rule costs a syscall inspection, and the honest numbers are modest: a handful of file watches on a server produces a few events per day and no measurable load, while hundreds of rules, or a rule watching a hot path like a database file, produces both volume and latency. The guidelines from the audit documentation are worth keeping: watch specific files, not directories that change every minute, and prefer wa triggers over r, because reads on configuration files happen constantly and record nothing an investigation needs.
Disk is managed in /etc/audit/auditd.conf through three settings that belong in every setup: max_log_file in megabytes, num_logs the number of rotated files, and max_log_file_action, which on production is ROTATE rather than the default that can halt logging when the limit trips. The space_left and space_left_action pair gives the service a graceful reaction to a filling disk, with the email action providing the early warning that keeps the audit trail from being the thing that took the server down. A useful sizing exercise on a new fleet: run for a week, read aureport to see event volume, and set the retention numbers from evidence instead of folklore.
The last line of the boot-time rules file is the one that separates audited systems from audit-ready ones:
-e 2
The immutable flag locks the rule set until reboot, meaning even root cannot flush or replace the rules at runtime, and any attempt is itself recorded. It pairs with the requirement that some environments have for reboot-only policy changes, and it costs one line, which is why it belongs in the rules.d file from day one rather than after the first incident teaches its value.
The field checklist
The complete setup compresses into an order worth following on every new server. Install and start auditd, confirm status with auditctl -s. Write the rules.d file with watches on /etc/passwd, /etc/sudoers and /etc/sudoers.d, syscall rules for rename and unlink on the identity file, keys on everything. Load with augenrules --load, list with auditctl -l, and prove each rule with a harmless test change that you then find with ausearch. Set the disk policy in auditd.conf and the immutable flag in the rules. Stand up the digest, and on the hosts where it matters, the central forwarding. The reward for one evening of work is a property of the server that most of the industry still lacks: when someone asks who changed sudoers last Tuesday and what exactly they ran, the answer is one ausearch command away, structured, timestamped and signed by the kernel itself.
## The wider watch list beyond passwd and sudoers
Passwd and sudoers are the two canonical examples, but the same rule pattern extends to every file where a quiet change means privilege, and a mature ruleset names them explicitly. The identity family continues with /etc/shadow, where password hashes live, and /etc/group, where membership in privileged groups is decided, both deserving the identical wa watch with the identity key. The access family starts with the SSH trust store: /etc/ssh/sshd_config for the service configuration, and the per-user authorized_keys files, where an attacker with a moment of root plants the backdoor that survives a password change. The watch on home directories goes one level up:
-w /home/ -p wa -k ssh-trust
covering the keys of all users in one rule, at the price of noise from ordinary home activity, which is the tradeoff each site calibrates: tight watches per known user versus a directory watch that catches the account nobody remembered. The persistence family completes the standard list: /etc/cron.d/ and /etc/crontab, where a scheduled task is the classic persistence mechanism, /etc/systemd/system/ for unit files that start things at boot, and /etc/ld.so.conf.d/ and /etc/ld.so.preload, the shared library paths where a one-line edit compromises every dynamically linked binary on the system.
The sttylistic rule for the list is the same as for the rest of the audit plan: every entry answers a specific question about a specific persistence or privilege path, and every entry has a key that names that path in the search results. A ruleset built that way reads like a threat model in configuration form, which is exactly what an auditor with a clipboard hopes to find and exactly what an attacker with a clipboard hopes you did not write.