The rm command is instantaneous and the regret that follows it is universal, but on an ext4 filesystem the deleted file's data blocks survive for a while after the directory entry is removed. The window between deletion and permanent loss is the recovery opportunity, and extundelete is the tool that reads the journal and the inode structures to reconstruct what was removed. This article builds the procedure: why the window exists and how fast it closes, the immediate unmounting rule that preserves it, running extundelete on the unmounted filesystem, reading the output, the alternatives when the primary tool fails, and the prevention habits that make the recovery tool a rare guest rather than a regular one.
Why deleted files survive and how fast the window closes
ext4 deletes a file by marking its blocks as free in the allocator and removing the directory entry, but the data blocks themselves are not zeroed, the content stays on disk until the allocator reuses those blocks for a new file. The window is therefore proportional to the write activity on the same filesystem: a quiet server's deleted files can survive for days, a busy database server's disappear in minutes. The ext4 journal, which records metadata changes, can contain enough information to reconstruct the file's block map even after the inode is cleared, and extundelete reads the journal for exactly this purpose.
The journal-based recovery is the reason extundelete works on ext4 where older ext2 tools like debugfs lsdel fail: the journal preserves the extent information that the inode no longer carries, and the tool reconstructs the file's layout from the journal's history rather than from the inode structure alone. The window closes when the journal wraps, when the blocks are reallocated, or when the filesystem is remounted and the journal replays past the recovery point.
The immediate unmounting rule
The rule that determines whether the recovery succeeds is the first one: unmount the filesystem immediately, before anything else touches it. Every write to the same filesystem after the deletion reduces the recovery chance, because every new allocation may reuse the deleted file's blocks. The commands that implement the rule:
umount /dev/sda1
or, when the filesystem cannot be unmounted because a process holds it open, the read-only remount that stops new writes:
mount -o remount,ro /dev/sda1
The read-only remount is the fallback for the root filesystem, which cannot be unmounted while the system runs, and the remount stops the kernel from writing new data to the same blocks. The honest urgency of this step is worth stating: every second between the rm and the umount is a second during which the kernel may allocate the deleted blocks to a new write, and the filesystem that stays mounted while the recovery is researched is a filesystem whose recovery chances are actively degrading.
Running extundelete on the unmounted filesystem
extundelete runs against the block device, not the mount point, and the recovery output goes to a directory on a different filesystem, the rule that the recovered files must not land on the same device they are being recovered from:
extundelete /dev/sda1 --restore-file etc/nginx/nginx.conf # extundelete /dev/sda1 --restore-all
The --restore-file flag targets one path, the --restore-all flag attempts every recoverable file on the filesystem, and the output lands in a RECOVERED_FILES/ directory in the current working directory, which should be on a different disk or at minimum on /tmp when the target filesystem is not the root. The tool prints its progress through the journal, each replayed transaction a line of output, and the completion message reports how many files were restored and how many blocks were unrecoverable.
Reading the output and the success criteria
The RECOVERED_FILES directory after a run contains the files the tool could reconstruct, and the quality of each recovered file is the verification the operator performs: the text file that opens cleanly, the binary that runs, the image that renders. The files that open with garbage or that truncate mid-content are the blocks that were partially overwritten before the recovery ran, and the partial recovery is often still valuable, the config file's first three quarters readable, the database dump's first tables present.
The success criteria for the recovery are simple: the file is recovered when its content matches what the operator expects, and the verification is the same readback that any restore uses, the tool's output being a candidate rather than a guarantee. The files that extundelete cannot recover are the ones whose blocks were reused, and the honest answer to the question of what was lost is the list of files in the RECOVERED_FILES directory compared against the list the operator remembers deleting.
The alternatives when the primary tool fails
When extundelete cannot recover the file, the alternatives descend in order of precision. The journal examination through debugfs, the ext filesystem debugger that can dump inodes and block pointers directly, is the tool for the case where extundelete's journal interpretation misses a specific file. The file carving tools, testdisk and photorec from the recovery article in this series, scan the raw blocks for file signatures and recover by content rather than by structure, the approach that finds the photos and the documents even when the filesystem metadata is gone.
The backup is the last and most reliable alternative, and the backup tools from the previous articles in this series are the recovery path that should have been tried first if the schedule was running. The restic repository, the Borg archive, the rsnapshot rotation, the tools that were taken before the rm happened, and the recovery from them is instant because the copy exists and the tool is the one that made it.
Prevention and the habits that make recovery rare
The prevention tools that make extundelete a rare guest rather than a regular one are three. The trash command, trash-cli or its equivalent, intercepts rm and moves files to a recoverable location instead, the alias that the shell rc file carries and the habit that takes one day to build. The safe-rm wrapper checks paths against a deny list before deleting, the guard against the rm -rf that starts from the wrong directory. And the backup schedule from the earlier articles is the prevention that catches what the other two miss, the deletion that was intentional but regretted, the file that was trash-cli'd and then the trash was emptied.
The checklist for the rm emergency
The checklist for the moment after the accidental rm is short and every step matters. Unmount the filesystem immediately, umount or remount,ro. Do not write anything to the same filesystem, no logs, no temp files, no recovery attempts to the same disk. Install extundelete if it is not present, from a different filesystem or from a live environment. Run extundelete with --restore-file or --restore-all, output to a different filesystem. Verify each recovered file by opening it. If extundelete fails, try testdisk or photorec for file-carving recovery. Check the backup for the file, because the backup may have it. And after the recovery, install the prevention tools, because the second rm on the same server should not require the same recovery.
The ext4 deletion mechanics in detail
The ext4 filesystem stores file data in extents, and the inode points to the extent tree. When a file is deleted, the extent tree is zeroed and the block bitmap marks the extents as free, but the actual data blocks retain their content until overwritten. The journal records the metadata transaction atomically, and the journal entry contains enough information to reconstruct the extent tree if the journal has not yet been overwritten.
extundelete reads the journal backward and reconstructs the extent information from the before-image in the journal's descriptor block. This journal-based reconstruction is what makes extundelete work on ext4 where older tools depended on the inode structure, and it is also why the journal size matters: the deleting transaction scrolls out of the journal history within hours on a busy system.
Recovery on LVM and RAID volumes
On LVM the device is the logical volume, /dev/vg/data, which must be inactive or read-only for the recovery to be safe. On RAID the device is the md device, and the array must be degraded or stopped. The layer only changes the device naming and the activation state, and the procedure applies unchanged once the right device is identified.
The recovery on the root filesystem
The root filesystem cannot be unmounted while the system runs, so the recovery goes through the live environment: boot from SystemRescue, identify the root device, remount read-only, and run extundelete from the live environment with output to a USB drive.
The end-to-end walkthrough
An engineer deletes /srv/app/config.yaml on ext4 root at /dev/sda2. Sequence: remount read-only. Install extundelete from live environment. Run extundelete /dev/sda2 --restore-file srv/app/config.yaml, output to /tmp. Verify the recovered file. Copy back after remounting read-write. Restart the application. Install trash-cli.
The timing matters: from rm to read-only remount should be seconds, from remount to extundelete minutes, and from run to verified recovery the same session. Every hour between deletion and recovery is an hour of write activity on the same filesystem.
The journal wrap and the recovery window
The ext4 journal is a circular buffer of fixed size, typically 128 megabytes on a default installation, and the journal transactions it holds are the recovery raw material. When the journal wraps, the oldest transactions are overwritten by new ones, and the deleting transaction's before-image disappears from the journal's history. On a server writing one gigabyte of journal transactions per hour, a 128-megabyte journal wraps in under eight minutes, and the recovery window measured in journal transactions is the window between the deletion and the wrap, a number the administrator can calculate from the journal's write rate and size.
The journal size is configurable at filesystem creation time, mkfs.ext4 -J size=512, and the larger journal extends the recovery window proportionally. The tradeoff is the journal's memory and write amplification, and the larger journal is a deliberate choice on the filesystems where the recovery window matters more than the write overhead, the production servers whose data cannot be reconstructed from any other source. The journal size on an existing filesystem can be changed with tune2fs -J size=512, and the change takes effect after the filesystem is unmounted and remounted, the procedure that the fstab and initramfs articles have already established.
The block-reuse dynamics that close the window
Beyond the journal wrap, the second factor that closes the recovery window is the block allocator's reuse of the freed extents. ext4's allocator is designed to place new data near existing data on the same inode, the locality optimization that improves read performance, and the same optimization means that a new file created in the same directory as the deleted one is likely to receive the deleted file's freed blocks. The allocator's behavior is not malicious, it is the design working correctly, and the consequence for the recovery is that the files created after the deletion on the same directory tree are the most likely to overwrite the deleted content.
The operational implication is specific: after the rm, do not create files in the same directory tree as the deleted file, do not run anything that writes to the same filesystem, and do not restart services that write to the same paths. The read-only remount implements all three restrictions in one command, and the alternative, a filesystem left mounted read-write while the recovery tool is downloaded and installed, is the scenario where the recovery tool's own download overwrites the very blocks the recovery needs.
The tested and verified recovery
The recovered file's verification is the same readback discipline that the backup articles in this series established, and the verification for a recovered file is simpler because the expected content is known. The text file that opens without errors, the binary that runs, the configuration that the application accepts, the image that displays, each type of file has its own verification, and the verification is the step that turns a recovered block of data into a recovered file.
The partial recovery, the file that truncates mid-content or the binary that runs until it hits a zeroed block, is still valuable, because the readable portion is often the portion that answers the question or contains the data, and the partial file is a better starting point than nothing. The recovery's completion report names the blocks that were unrecoverable, and the honest assessment of what was recovered versus what was lost is the output that the incident report records, the scope of the loss defined by the tool's own accounting rather than by assumption.