A bash script that works in testing and fails in production is the script that did not handle its errors, and the error handling that production requires is not a try-catch but a set of shell options and a trap that together turn silent failures into loud ones. The set -euo pipefail line is the foundation, and the trap is the cleanup mechanism that runs when the script exits. This article builds the pattern from the ground up.
The three options and what each prevents
set -e stops the script on any command that returns nonzero, the bare minimum that every script needs. Without it, the script continues after a failed command, and the failure propagates silently until the damage is done. set -u stops the script on any reference to an unset variable, the protection against the typo that references $MYVARIABEL instead of $MYVARIABLE and that would otherwise expand to an empty string. set -o pipefail changes the exit code of a pipeline from the exit code of the last command to the exit code of the first command that fails, the protection against the pipeline where grep succeeds but the upstream command failed.
The combination is the standard line that starts every production bash script:
#!/bin/bash
set -euo pipefail
The trap for resource cleanup
The trap command installs a handler that runs when the script exits, and the handler cleans up the resources the script created: temporary files, locks, background processes.
TMPDIR=$(mktemp -d)
LOCKFILE=/var/run/myapp.lock
trap cleanup EXIT
cleanup() {
rm -rf $TMPDIR
rm -f $LOCKFILE
}
The EXIT trap fires on normal exit, on error exit and on signal termination, and the cleanup function removes everything the script created. The trap is the difference between a script that leaves its mess behind and a script that cleans up, and the cleanup is what makes the script safe to run repeatedly.
The error handling patterns that work
Beyond set -e, the explicit error handling patterns give the script control over what happens when things fail. The if-not pattern:
if ! command; then
echo "error: command failed" >&2
exit 1
fi
The || pattern for inline handling:
command || { echo "error" >&2; exit 1; }
The difference between the two is readability, and the pattern that the team standardizes is the one that every script in the codebase uses consistently.
The logging and the diagnostic output
The script that runs from cron or from systemd needs logging that the operator can read, and the logging goes to stderr and to the journal through the logger command:
log() { echo "$(date -Is) $" >> $LOGFILE; logger -t myscript "$"; }
log "starting backup"
rsync -a /data/ backup@remote:/backups/
log "backup complete"
The log function writes to both a local file and to the journal, and the journal entry is what the journalctl filters find. The log messages that the script writes are the evidence of what it did, and the evidence is the first thing the incident investigation reads.
The pipeline patterns that preserve error codes
The pipeline with pipefail preserves error codes across pipes, and the pipeline patterns that work with pipefail are the ones where each command handles its own errors. The pipeline that works:
grep ERROR /var/log/app.log | wc -l > /tmp/error-count
works because both grep and wc handle their own errors, and the pipefail catches the failure of either. The pipeline that fails silently:
command_that_fails | command_that_succeeds
is the pipeline where the first command fails but the second succeeds, and without pipefail the exit code is the second command's, the zero that hides the failure. With pipefail, the exit code is the first command's, the nonzero that set -e catches.
The signal handling for long-running scripts
The trap for signals handles SIGINT and SIGTERM, the signals that kill the script when the operator presses Ctrl+C or when the system shuts down:
trap shutdown_handler SIGINT SIGTERM
shutdown_handler() {
log "received signal, cleaning up"
kill -- -$$
cleanup
exit 1
}
The signal handler cleans up the child processes and the resources, and the cleanup is what distinguishes a graceful shutdown from a hard kill. The script that handles signals is the script that runs safely from systemd and from cron, because both send signals when they stop the script.
The complete script template
The template that every production bash script starts from:
#!/bin/bash
set -euo pipefail
readonly SCRIPT_DIR=$(cd $(dirname $0) && pwd)
readonly LOG_FILE=/var/log/$(basename $0).log
log() { echo "$(date -Is) $*" | tee -a $LOG_FILE >&2; }
cleanup() {
[ -n "${TMPDIR:-}" ] && rm -rf $TMPDIR
[ -n "${LOCK_FD:-}" ] && flock -u $LOCK_FD
}
trap cleanup EXIT
TMPDIR=$(mktemp -d)
exec 200>/var/run/$(basename $0).lock
flock -n 200 || { log "already running"; exit 1; }
log "starting"
# ... main logic ...
log "complete"
The template includes the error options, the logging function, the cleanup trap, the temporary directory and the file lock that prevents concurrent runs. The script that starts from this template inherits the error handling, the logging and the cleanup without writing them from scratch each time.
The checklist for production bash scripts
The checklist that closes the article is the review that every bash script passes before it reaches production. The shebang is #!/bin/bash. The options are set -euo pipefail. The logging writes to both a file and the journal. The cleanup trap removes temporary resources on exit. The signal handling catches SIGINT and SIGTERM. The file lock prevents concurrent runs. The error messages go to stderr. The exit codes are meaningful: zero for success, nonzero for failure. And the script is tested with the failure scenarios, not just with the happy path, because the script that has never failed in testing is the script that will fail in production without the error handling to catch it.
The command substitution patterns that preserve error handling
Command substitution, the $(...) syntax, is the source of silent failures because the exit code of the substituted command is lost unless explicitly captured. The pattern that loses the error code:
RESULT=$(command_that_might_fail)
echo $RESULT
The exit code of command_that_might_fail is the exit code of the assignment, which is zero because the assignment succeeded, and the failure is invisible. The pattern that preserves the error code:
if ! RESULT=$(command_that_might_fail); then
echo "command failed" >&2
exit 1
fi
The if wrapping captures the exit code of the substitution, and the failure is caught and handled. The alternative uses the temporary variable:
RESULT=$(command_that_might_fail) || exit 1
which is the inline version of the same pattern, and the compact form works in the simple cases while the if form works in the cases that need additional handling.
The command substitution that captures both stdout and stderr uses the 2>&1 redirection:
OUTPUT=$(command 2>&1) || exit 1
and the combined output is the variable that the script can inspect for both the result and the error messages. The separation of stdout and stderr into different variables requires a temporary file or a fifo, and the complexity is the reason the combined capture is the common pattern with the separate handling done by pattern matching on the output.
The awk and the data processing within bash scripts
The awk command is the data processing tool that bash scripts use for the structured data parsing, and the awk one-liners that the script uses are the aggregation and formatting tools that the script needs. The log analysis that counts errors by type:
awk '/ERROR/ { errors[$NF]++ } END { for (e in errors) print errors[e], e }' /var/log/app.log
which counts each unique error type, and the output is the summary that the script reports. The awk command is the tool that the bash script uses for the data transformation that a dedicated language would handle with a library, and the awk that the script uses is the awk that the operator already knows.
The ssh and the remote execution within bash scripts
The SSH command that the script uses for remote execution carries the options that make the remote execution non-interactive and reliable: the -o BatchMode=yes that disables password prompts, the -o StrictHostKeyChecking=yes that verifies the host key, and the -o ConnectTimeout=10 that bounds the connection attempt.
ssh -o BatchMode=yes -o ConnectTimeout=10 admin@remote 'systemctl status myapp'
The BatchMode option is the one that prevents the script from hanging on a password prompt, and the ConnectTimeout prevents the script from hanging on an unreachable host. Both are the options that the production script needs, and the options that the SSH config file can also set per-host.
The cron and the scheduling context for bash scripts
The cron schedule that runs the script is the context that determines the script's environment: the PATH is minimal, the working directory is the home directory, and the environment variables are absent. The script that works from the interactive shell but fails from cron is the script that depends on the interactive environment, and the fix is the script setting its own environment: the PATH that it needs, the working directory that it expects, and the variables that it requires.
The cron entry that runs the script:
0 3 * /usr/local/bin/backup.sh >> /var/log/backup.log 2>&1
redirects both stdout and stderr to the log file, and the redirection is the diagnostic record that the morning check reads. The MAILTO variable in the crontab sends the output to the email address, but the modern approach is the log file plus the monitoring, not the email that nobody reads.
The systemd timer as the modern cron alternative
The systemd timer is the scheduling mechanism that replaces cron for the scripts that run on systemd hosts, and the timer's advantages over cron are the journal integration, the dependency management and the random delay for fleet-wide distribution. The timer unit:
[Unit]
Description=Nightly backup
[Timer]
OnCalendar=--* 03:00:00
RandomizedDelaySec=300
[Install]
WantedBy=timers.target
The RandomizedDelaySec spreads the fleet's backup schedules across a five-minute window, preventing the thundering herd that the identical cron schedules cause. The journal integration means the script's output is queryable with journalctl -u backup.service, the same query interface that the journalctl article established. And the dependency management, After=network-online.target, ensures the script runs after the network is available, the ordering that cron cannot express.
The security considerations for production bash scripts
The bash script that runs as root is the script whose bugs are root-level bugs, and the security considerations that production demands are the principle of least privilege, the input validation and the credential management. The script that runs as root drops to an unprivileged user for the operations that do not need root, the setpriv or the su command that changes the effective user. The script that takes input validates the input before using it, the injection prevention that the web application world calls sanitization. And the script that needs credentials reads them from a file with restricted permissions or from the environment, never from the command line where they appear in the process listing.
The script's own permissions matter: mode 750, owner root, group the group that runs it, and the world-readable script is the script whose logic is visible to every user on the system. The credentials that the script uses, the database passwords and the API keys, belong in files with mode 600 in a directory with mode 700, and the credential file is read by the script at startup and never appears in the command line or in the script source.
The script that passes the review, the testing and the monitoring is the script that runs for years without incident, and the script that runs for years is the script that the team trusts enough to build on.