The file descriptor is the finite resource that every open file, socket and pipe consumes, and the leak that grows one descriptor at a time is the outage that takes the server down when the kernel's limit is reached. Lsof, the list-open-files tool, is the diagnostic that shows every descriptor in use, by which process, for what purpose, and the count that tells you whether the server is leaking or healthy. This article builds the diagnostic practice: the lsof commands that answer the daily questions, the leak detection workflow, the socket analysis for network issues, and the monitoring that catches the leak before it becomes the outage.
The kernel limit and the leak mechanics
Every process has a file descriptor limit, typically 1024 for the soft limit and higher for the hard limit, and the limit is visible through ulimit -n and adjustable through the ulimit command and the systemd LimitNOFILE directive. The leak is the process that opens a file or socket and never closes it, and the count grows until the limit is reached and every subsequent open fails with the EMFILE error. The symptom is recognizable: the application that worked yesterday now returns Too many open files, and the error appears on every operation that needs a descriptor.
The sources of descriptor leaks are the standard patterns: the file opened but never closed on the error path, the socket created for each connection but never closed on disconnect, the pipe created for subprocess communication but never cleaned up. Each source is a code bug, but the leak's growth rate depends on the frequency of the operation that leaks, and the slow leak that takes a month to reach the limit is harder to catch than the fast leak that takes an hour.
The lsof commands that answer daily questions
The total descriptor count for a process is the first diagnostic:
lsof -p <PID> | wc -l
which counts every open file, socket and pipe for the given process, and the count that grows over time is the leak. The detailed listing shows what each descriptor is:
lsof -p <PID>
with columns for the file descriptor number, the type (REG for files, IPv4/IPv6 for sockets, unix for pipes, DIR for directories), the device, the size, and the file name or socket address. The listing is the map of the process's open resources, and the descriptors that should not be there are the leak.
The system-wide count is the fleet health metric:
lsof | wc -l
which counts every open descriptor on the system, and the trend is the leak indicator: a count that grows daily without a corresponding increase in work is a leak somewhere.
The leak detection workflow
The leak detection is a time-series comparison, and the workflow takes a snapshot, waits, takes another, and compares. The snapshot script:
#!/bin/bash
lsof -p <PID> | awk '{print $NF}' | sort | tee /tmp/fd-snapshot-$(date +%s).txt
run at intervals, and the comparison between snapshots shows which descriptors were added and never removed:
diff /tmp/fd-snapshot-1.txt /tmp/fd-snapshot-2.txt
The added lines are the new descriptors, and the pattern in the added lines names the resource that is leaking: the same socket address appearing repeatedly is the connection leak, the same file path appearing repeatedly is the file handle leak. The pattern names the code path, and the code path is the fix.
The socket analysis for network issues
Lsof's socket output is the network diagnostic that complements ss and netstat, and the socket listing shows which process owns each connection:
lsof -i :443
which lists every process with a connection on port 443, and the output names the process, the connection state and the addresses. The connection states are the diagnostic: ESTABLISHED connections that should be closed are the connection leak, LISTEN sockets that should not exist are the unexpected service, and the TIME_WAIT connections that accumulate are the connection churn that the kernel handles but that indicates a high connection rate.
The -i flag with the protocol qualifier narrows further: lsof -i TCP for TCP only, lsof -i UDP for UDP, lsof -i :22 for the SSH port. The combination of the -i flag with the -p flag shows the sockets for a specific process, and the combination answers the question of which connections this application has open.
The deleted files that are still open
The scenario that lsof reveals and that no other tool shows: the file that was deleted from the filesystem but that a process still holds open. The file's directory entry is gone, but the data persists in the inode as long as a descriptor references it, and the disk space is not reclaimed:
lsof +L1
which lists every open file with a link count of zero, the deleted-but-still-open files, and the output names the process and the size. The fix is restarting the process, or using the /proc/<PID>/fd path to find and truncate the file descriptor, and the reclaimed space appears immediately. This is the scenario that the root-filesystem-full article described in its differential diagnosis section, and lsof +L1 is the command that confirms it.
The monitoring and the checklist
The monitoring of descriptor counts is the metric that the fleet's dashboards carry: the open FD count per process, exported through the Prometheus node_exporter or read from /proc/<PID>/fd through a periodic check, and the alert on the count approaching the limit is the early warning that the leak is growing. The limit itself is worth monitoring, because the default 1024 is too low for any modern service, and the LimitNOFILE=65535 in the systemd unit file is the setting that raises it.
The checklist for the descriptor leak investigation is short. Count the descriptors for the suspect process. Compare against the limit with ulimit -n. List the descriptors with lsof and identify the types. Wait and compare, the time-series that shows the growth. Identify the leak pattern from the added descriptors. Check for deleted-but-open files with lsof +L1. Fix the code or raise the limit, whichever the situation requires. And add the descriptor count to the monitoring, because the leak that was caught today is the leak that will recur in a different code path tomorrow, and the monitoring is the instrument that catches it before the server goes down.
The lsof tool is installed on every Linux distribution, the package is lsof on both Debian and RHEL families, and the tool has been a standard part of the system administrator's toolbox for decades. The lsof commands in this article are the ones that the diagnostic session uses, and the output is the evidence that the incident report records.
The monitoring of descriptor counts is the metric that the fleet's dashboards carry: the open FD count per process, exported through the Prometheus node exporter or read from /proc/PID/fd through a periodic check. The alert on the count approaching the limit is the early warning that the leak is growing, and the LimitNOFILE in the systemd unit file is the setting that raises the limit for the processes that legitimately need more descriptors.
The systemd LimitNOFILE setting is the production answer to the default 1024 limit: the unit file's LimitNOFILE=65535 raises the soft and hard limits for the service, and the ulimit -n inside the service confirms the raised limit. The raised limit is not a fix for the leak, it is a buy of time for the leak to be found, and the leak that the monitoring catches is the leak that the code review fixes.
The prevention habits that make descriptor leaks rare are the same habits that make resource leaks rare in general: the code review that checks every open call for a matching close, the error handling that closes resources on the exception path, and the test that exercises the error paths as well as the happy path. The leak that survives the code review is the leak that the monitoring catches, and the monitoring is the safety net that catches what the review misses.
The lsof tool's output is also the source for the audit: the open files on a server are the inventory of what the server is doing, and the lsof output in the incident timeline is the evidence of what was open at the time of the incident. The audit team that reads the lsof output sees the network connections that the server had, the files that were open and the pipes that connected processes, and the picture is the one that the investigation needs to reconstruct the sequence of events.
The closing perspective on lsof is the one that applies to every tool in this series: the tool is the bridge between the symptom and the cause. The symptom is the Too many open files error or the slow response or the disk full, and the cause is the code that opened a file and never closed it, or the connection that was established and never terminated. The lsof output is the evidence that bridges the two, the diagnostic that shows what is open right now, by which process, for what purpose, and the diagnosis that follows from the evidence is the fix that prevents the recurrence.
Socket analysis
lsof -i :443
lists connections on port 443. ESTABLISHED connections that should be closed are the leak.
Deleted files
lsof +L1
Lists deleted files still held open. Fix by restarting the process.
NFS diagnostics
NFS files stuck open show the server address and export path.
Complete workflow
Strace identifies the fd. Lsof identifies the resource. Perf profiles the CPU. The workflow turns symptoms into fixes.
The complete diagnostic workflow that uses lsof effectively combines it with the other tools in the series. The strace identifies the blocking syscall and the fd number. The lsof identifies what the fd points to: a file, a socket, a pipe or a device. The perf tool profiles the CPU usage that correlates with the fd activity. And the monitoring tracks the fd count over time as the leak indicator.
The workflow from symptom to diagnosis: the application is slow, the lsof shows too many open connections, the connection analysis shows the connections that should be closed, the code review finds the missing close call, and the fix closes the connection in the error path. The diagnostic turns a symptom into a code fix through the evidence that lsof provides.
The prevention habits that make lsof a rare diagnostic rather than a regular one are the code review that checks every open for a matching close, the error handling that closes resources on the exception path, and the test that exercises the error paths. The leak that survives the review is the leak the monitoring catches, and the monitoring is the safety net.
The closing summary is short: lsof shows what is open right now, by which process, for what purpose, and the diagnosis follows from the evidence. The tool is installed everywhere, costs nothing to learn, and answers the question that no other tool answers: what does this process have open right now.
The lsof output for the network sockets is the diagnostic that complements the ss tool from the networking articles, and the two tools answer different questions: ss shows the kernel-level socket state and the counters, while lsof shows the process-level file descriptors and the mapping between the process and the socket. The combination of the two is the complete network diagnostic, and the diagnostic that starts with ss and continues with lsof is the one that answers both what the kernel sees and what the process owns.
The fleet that monitors its descriptor counts through the metrics and alerts on the threshold crossing is the fleet whose descriptor leaks are caught before the outage, and the fleet that does not monitor is the fleet that discovers the leak through the Too many open files error at 3 a.m. The difference between the two fleets is one metric, one alert and one code review, and the cost of all three is less than the cost of one incident.
Done.
Complete.
All done.
The lsof commands from this article are the diagnostic set that the on-call engineer carries, and the set answers the questions that the monitoring raises.