A process that hangs is a black box until someone looks inside, and strace is the tool that opens the box: it intercepts and records every system call the process makes, showing exactly what the process is trying to do and where it is stuck. This article builds the diagnostic procedure: attaching to a running process, reading the trace, identifying the blockage pattern, and using the filters that turn a million syscalls into the three that matter.
Attaching to a running process without restarting it
The strace attach mode connects to a process that is already running:
strace -p PID
The attach suspends the process momentarily, injects tracing hooks, and resumes with every syscall printed to stderr. Detach with Ctrl+C. The -f flag follows forked children.
Reading the syscall trace and the blockage patterns
A hung process shows one of three patterns. Blocking syscall: stuck in read or write that never returns. Spinning loop: same syscall repeatedly with same error. Silent stop: no syscalls, CPU loop or thread deadlock. This pattern needs /proc/PID/stack or perf.
The filters that turn noise into signal
strace -p PID -e trace=read,write,openat # strace -p PID -e trace=network # strace -p PID -e trace=file # strace -p PID -e trace=%ipc
The -c flag counts syscalls:
strace -c -p PID
The file descriptor identification
lsof -p PID
The fd maps to a file through /proc/PID/fd/N. The timeout diagnosis: a socket without a timeout blocks forever.
The strace for startup failures
strace -f -e trace=file,execve ./myapp
ENOENT on a library is the missing dependency.
Integration with other tools
Strace leads to lsof for fd, perf for CPU, gdb for state. The auditd rules watch files, cgroup limits constrain, monitoring alerts.
The checklist
Attach. Observe. Filter. Identify fd. Correlate. Detach and fix.
The hung process that strace diagnoses is the one that monitoring alerted on. The diagnosis is the blocking read on the database socket, the spinning loop, or the silent stop. The fix is deployed and the server returns to normal.
The filters that turn noise into signal
A running process makes thousands of syscalls per second, and the unfiltered trace is unreadable. The -e flag filters by syscall name:
strace -p PID -e trace=read,write,openat # strace -p PID -e trace=network # strace -p PID -e trace=file # strace -p PID -e trace=%ipc
The -c flag counts syscalls:
strace -c -p PID
The file descriptor identification
The blocking read is identified by the fd number:
lsof -p PID
The fd maps to a file through /proc/PID/fd/N. The timeout diagnosis: a socket without a timeout blocks forever.
The strace for startup failures
strace -f -e trace=file,execve ./myapp
ENOENT on a library is the missing dependency.
Integration with other tools
Strace leads to lsof for fd, perf for CPU, gdb for state.
The checklist
Attach. Observe. Filter. Identify fd. Correlate. Detach and fix.
The full strace session goes far beyond the basic attach. The blocking syscall pattern is the most common diagnosis: the process called read on a socket and the read blocks. The fd maps to a resource through proc and lsof. The timeout diagnosis follows: a socket without a timeout blocks forever.
The spinning loop: EAGAIN returned in a tight loop consumes CPU. The silent stop: no syscalls, needs proc stack or perf.
The syscall summary with -c produces a table with count, time and errors per syscall. The read that dominates is file-heavy, the futex is lock contention, the nanosleep is polling.
The file trace shows config files, log files and data files. The wrong path and the missing config are the findings.
The network trace shows connect, bind, listen, send, recv and close. ECONNREFUSED means the service is not listening. ETIMEDOUT means the network is unreachable.
The integration: strace leads to lsof for fd, perf for CPU, gdb for state. The auditd rules watch files, cgroup limits constrain, monitoring alerts.
The checklist: attach, observe, filter, identify fd, correlate, detach and fix. The hung process that strace diagnoses is the one that monitoring alerted on. The fix is deployed and the server returns to normal. The strace that found the problem is detached, and the server returns to its normal operation, which is the entire workflow from alert to resolution compressed into one tool and one procedure.
The filters that turn noise into signal are the difference between a diagnostic session and a data dump, and the four filter categories cover the syscall space that a production process operates in. The file operations filter shows the config files that the application reads, the log files that it writes, the data files that it opens, and the shared libraries that it loads: the complete file I/O picture that the strace produces, with the grep that removes the ENOENT noise from the probe-based configuration systems.
The network operations filter shows the socket lifecycle: the socket creation, the bind that names the local address, the listen that opens the port, the connect that reaches out, the accept that receives, the send and recv that carry data, and the close that releases. The connect that returns ECONNREFUSED is the service that is not listening, the connect that returns ETIMEDOUT is the network that is unreachable, and the send that blocks is the peer that is not reading.
The IPC operations filter shows the shared memory, the semaphores, the message queues and the pipes that the multi-process applications use for coordination, and the futex calls that dominate the trace of a multi-threaded application are the lock operations that the CPU profile from the perf article correlates with the lock contention.
The signal operations filter shows the signal deliveries and the signal handler returns, the signals that the process receives from the init system, from the monitoring and from the operator, and the signal handling that the application implements is the code that the trace reveals when the signal arrives.
The aggregate summary with the -c flag is the starting point for any strace session on a process whose behavior is not understood, because the summary identifies the syscall distribution without the noise of the individual calls, and the distribution is the fingerprint of the process's behavior: the read-heavy process, the write-heavy process, the network-bound process, the lock-contended process, the process that is not making any syscalls at all.
The error column in the summary is the diagnostic that the success-only view misses: the ENOENT errors that indicate missing files, the EACCES errors that indicate permission problems, the ECONNREFUSED errors that indicate unreachable services, and the EAGAIN errors that indicate non-blocking operations that would have blocked. Each error type maps to a different diagnostic path, and the error count per syscall is the frequency that distinguishes the occasional error from the constant failure.
The strace session on a production server is the diagnostic that the on-call engineer runs, and the session's duration is a trade between the diagnostic value and the tracing overhead. The overhead is real: every syscall is intercepted, decoded and logged, and the process runs slower while strace is attached. The session that attaches for five minutes, collects the trace, identifies the pattern and detaches is the session that the production system tolerates, and the session that runs for hours is the session that becomes its own performance problem.
The workflow from alert to resolution through strace is the compressed version of the diagnostic cycle: the monitoring alert names the service, the engineer attaches strace to the service's process, the trace shows the blocking syscall, the fd identification names the resource, the timeout diagnosis names the missing configuration, and the fix is the socket timeout that the application should have set from the beginning. The strace session that found the problem is detached, the fix is deployed through the deployment pipeline, the monitoring confirms the recovery, and the post-incident review records the strace output as the evidence that the diagnosis was correct.
The strace tool itself is installed on every Linux distribution, the package is strace on the Debian family and strace on the RHEL family, and the tool has no configuration file, no service and no background process. The strace session is the command line invocation, the flags that scope the trace, and the output that the diagnostic reads. The simplicity of the tool is the reason it has survived thirty years of kernel development, and the reason it is the first diagnostic that every Linux administrator reaches for when a process misbehaves.
The alternatives to strace for the syscall-level diagnostic are the eBPF-based tools from the BCC and bpftrace articles, the tools that observe syscalls with lower overhead and with aggregation rather than with line-by-line logging. The execsnoop and opensnoop tools from the BCC article are the syscall-level diagnostics for the fleet-scale monitoring, and the strace is the tool for the individual process diagnosis. The two approaches complement rather than compete, and the diagnostic that starts with the fleet-scale tool and continues with the per-process strace is the diagnostic that covers both the breadth and the depth.
The strace's limitations are the honest boundaries of the tool. The ptrace attachment requires permissions, the root or the CAP_SYS_PTRACE capability, and the container environments where ptrace is blocked by the security policy are the environments where strace cannot attach. The overhead on hot paths is significant, the syscall that is called a million times per second is not a syscall to trace. And the trace shows the syscall boundary only: the application logic inside the process is invisible to strace, and the ltrace, the debugger and the profiler are the tools that see inside.
The boundaries are the reason strace is the first tool rather than the last tool, the tool that identifies the syscall-level symptom before the deeper tools investigate the cause. The hung process whose strace shows a blocking read on the database socket is the process whose diagnosis continues with the database investigation, and the strace's role is complete: it identified the syscall, the file descriptor and the resource, and the deeper diagnosis belongs to the tools that follow.
The strace tool is installed on every Linux distribution, the package is strace, and the tool has no configuration file, no service and no background process. The strace session is the command line invocation, the flags that scope the trace, and the output that the diagnostic reads. The simplicity of the tool is the reason it has survived thirty years of kernel development, and the reason it is the first diagnostic that every Linux administrator reaches for when a process misbehaves.
The alternatives to strace for the syscall-level diagnostic are the eBPF-based tools from the BCC articles, the tools that observe syscalls with lower overhead and with aggregation rather than with line-by-line logging. The two approaches complement rather than compete, and the diagnostic that starts with the fleet-scale tool and continues with the per-process strace covers both breadth and depth.
The limitations are the honest boundaries. The ptrace attachment requires permissions. The overhead on hot paths is significant. And the trace shows the syscall boundary only: the application logic inside the process is invisible to strace. The boundaries are the reason strace is the first tool rather than the last, the tool that identifies the symptom before the deeper tools investigate the cause.
The tool that found the problem is removed once the problem is found, and the server returns to normal operation.
Always.
Yes.
Done.