GNU Parallel runs commands in parallel across CPU cores, and the parallelism is the performance multiplier for the batch operations that process independent items: the backup jobs that archive multiple directories, the image conversions that process multiple files, and the log analysis that runs across multiple hosts. The tool reads items from stdin or from files, runs the command for each item with the configured parallelism, and reports the output in order. This article builds the practical usage: the installation, the basic invocation, the parallelism control, the output handling, and the patterns that accelerate the batch operations.
Installation and the basic invocation
GNU Parallel installs from every distribution repository: apt install parallel or dnf install parallel. The basic invocation reads items from stdin and runs a command for each:
cat urls.txt | parallel wget {}
where each line of urls.txt is a URL, and wget downloads each URL with the default parallelism, which is the number of CPU cores. The {} placeholder is replaced by the input item, and the parallelism is controlled with -j.
The parallelism control and the resource tuning
The -j flag controls the number of parallel jobs: -j 4 runs four jobs simultaneously, -j 200% uses twice the CPU cores, and -j 1 is sequential. The parallelism tuning follows the resource being parallelized: CPU-bound jobs use one job per core, network-bound jobs use more because the network wait does not consume CPU, and disk-bound jobs use the number of spindles or SSD channels that the storage has.
The output handling and the ordered results
GNU Parallel collects the output from each job and prints it in order, the ordering that prevents the interleaved output from parallel jobs from becoming unreadable. The -k flag keeps the output order matching the input order, and the --tag prefix each output line with the input item, the tag that identifies which job produced the output.
The backup acceleration with parallel rsync
The backup that processes multiple directories is the scenario that Parallel accelerates most visibly: the sequential rsync of ten directories takes ten times the slowest directory, while the parallel rsync takes the slowest directory's time plus the overhead. The parallel backup:
parallel -j 4 rsync -a {} backup@remote:/backups/ ::: /data/app1 /data/app2 /data/app3 /data/db
The ::: separator provides the input items on the command line, and each rsync runs in parallel with the configured jobs.
The comparison with xargs -P and the advantages
The xargs -P flag provides parallel execution, and the comparison with GNU Parallel is a choice between simplicity and features. Xargs -P 4 runs four parallel jobs and provides the basic parallelism, but the output is interleaved and unordered. GNU Parallel provides ordered output, job tags, retry on failure, progress reporting and the citation notice, the features that the production scripts use and that xargs does not have.
The checklist for GNU Parallel deployment
The deployment is one package and one script modification, and the checklist is short. Install the package. Add parallel to the batch scripts that process independent items. Configure the parallelism with -j based on the resource. Add the -k flag for ordered output. Test with --dry-run before the real run. The acceleration that Parallel provides is the multiplication of the CPU cores and the network links, and the acceleration is free with a tool that is installed in one command.
The input sources and the argument handling
GNU Parallel reads input from multiple sources: the stdin pipe, the ::: separator on the command line, the :::: separator for input files, and the combination of all three. The ::: separator provides items directly:
parallel echo ::: item1 item2 item3
The :::: separator reads items from a file, the file with one item per line that the script generates:
parallel rsync -a {} backup@remote:/backups/ :::: dirs.txt
The combination of the input sources pairs items from multiple sources: parallel echo ::: group1 group2 ::: server1 server2 pairs group1 with server1 and group2 with server2, the two-column input that the parallel operations need for the source and destination pairs.
The argument placeholder {1} and {2} reference the first and second input columns, and the {/.} strips the extension, the placeholders that the complex commands use for the input manipulation. The placeholder reference is the template language that the parallel commands use, and the template is the same concept that the heredoc article described for the config generation.
The job control and the monitoring during execution
GNU Parallel shows the progress during execution, and the --progress flag displays a live view of the completed and running jobs, the progress that the long-running batch needs for the operator to know it is alive. The --eta flag estimates the remaining time based on the completed jobs, the estimate that the overnight batch benefits from. The --bar flag shows a text-based progress bar, the visual that the terminal session displays.
The job control during execution: Ctrl+C sends SIGTERM to the running jobs and stops the submission of new jobs, the graceful stop that lets the running jobs finish. The Ctrl+C pressed twice kills the running jobs immediately, the hard stop for the case where the graceful stop takes too long. The --halt flag controls the failure behavior: --halt now,fail=1 stops on the first failure, --halt soon stops submitting new jobs after a failure, and the halt behavior is the policy that the production scripts configure.
The output collection and the result handling
GNU Parallel collects the output from each job and provides it in a controlled way: the default prints the output as the jobs complete, the -k flag keeps the output order matching the input order, and the --files writes the output to files and prints the file names, the pattern for the jobs whose output is too large for the terminal. The --results directory saves the output and the input of each job in a directory structure, the structured output that the post-processing reads.
The exit code of GNU Parallel reflects the jobs' exit codes: the --halt setting determines whether a single job failure causes a nonzero exit, and the exit code is the signal that the calling script uses to detect the batch failure. The job's exit code is available through the --joblog option, the log file that records each job's exit code, start time, end time and runtime, the audit trail for the batch operations.
The remote execution with SSH
GNU Parallel distributes jobs across multiple hosts through SSH, and the --sshlogin option names the remote hosts that the jobs run on:
parallel --sshlogin server1 --sshlogin server2 --sshlogin server3 hostname ::: job1 job2 job3
which distributes the three jobs across the three servers, the parallelism across the fleet rather than across the cores. The --sshloginfile option reads the host list from a file, and the --transfer option sends the input files to the remote hosts, the file transfer that the remote jobs need. The --return option retrieves the output files from the remote hosts, and the --cleanup option removes the temporary files.
The remote execution is the pattern that the distributed processing needs, and the pattern is the same one that the Ansible playbook implements with the parallel tasks, the two tools operating at different levels: the Ansible at the configuration level, the GNU Parallel at the batch processing level.
The comparison with bash background jobs
The bash background jobs with the & operator and the wait command implement basic parallelism, and the comparison with GNU Parallel is the comparison between the manual tool and the automated one. The background jobs require the script to manage the job list, to wait for the completion and to collect the exit codes, and the manual management is the source of the bugs that the parallel execution introduces. GNU Parallel manages the job list, the parallelism limit, the output collection and the failure handling automatically.
The xargs -P provides basic parallelism, the parallelism that the find and xargs article described, and the GNU Parallel provides the features that xargs lacks: ordered output, progress reporting, job logging, remote execution and the citation notice. The choice between xargs -P and GNU Parallel is the choice between the simple tool and the feature-rich tool, and the choice that the script makes is the one that matches the complexity of the batch operation.
The practical examples from production
The parallel scenarios from production log analysis are the recipes that the runbook collects. The image conversion that processes thousands of files: parallel -j 8 convert {}.png {.}.webp ::: *.png. The log analysis that processes multiple hosts: parallel -j 10 'ssh {} grep ERROR /var/log/app.log | wc -l' ::: host1 host2 host3. The database maintenance that runs on multiple databases: parallel -j 4 'psql -h {} -c VACUUM' ::: db01 db02 db03.
Each recipe is a one-liner that the operator runs on the production server, and the parallelism is the performance multiplier that the sequential execution lacks. The GNU Parallel citation notice, the --citation flag that prints the citation for the academic use, is the one-time notice that the first run displays, and the citation is the project's request for the acknowledgment that the open-source tool deserves.
The closing checklist
The deployment of GNU Parallel is one package and one script modification, and the checklist is short. Install the package. Identify the batch scripts that process independent items. Replace the sequential loops with the parallel invocations. Configure the parallelism with -j based on the resource. Add the job logging with --joblog. Add the ordered output with -k. Test with --dry-run. And monitor the resource usage during the parallel runs, because the parallelism that saturates the host is the parallelism that needs tuning.
The GNU Parallel tool is the one that turns the sequential batch scripts into the parallel ones without changing the logic: the same commands, the same arguments and the same output, with the only difference being the parallelism that the tool provides. The speedup on the multi-core host is the multiplication of the core count, and the speedup on the multi-host fleet is the multiplication of the host count.
The batch operations that the parallel execution accelerates are the ones that this series described: the backup that processes multiple directories, the log analysis that processes multiple hosts, the image conversion that processes multiple files, and the database maintenance that runs on multiple databases. Each batch operation is independent of the others, and the independence is the prerequisite for the parallel execution, because the dependent operations need the sequential execution that guarantees the ordering.
The tool that provides the parallelism is the tool that the production fleet uses, and the tool is installed on every server in this series. The investment in learning the tool is the one afternoon that the batch operations save every week, and the return on the investment is measured in the hours that the parallel execution saves over the sequential execution on every batch run.
The GNU Parallel citation that the first run displays is the project's request for acknowledgment, and the --citation flag suppresses the notice after the first run. The citation is the academic convention that the free tool asks for, and the acknowledgment is the cost of using a tool that the maintainer develops without compensation. The --will-cite flag suppresses the notice permanently, and the suppression is the acknowledgment that the user has read the citation.
The installation from the distribution repository sometimes lags behind the current release, and the installation from the source or from the vendor's repository provides the latest version with the latest features. The version that the production environment uses should be consistent across the fleet, and the version consistency is the same discipline that the other tools in this series require.
The GNU Parallel's handling of the special characters in the input is the detail that the production scripts care about: the input items with spaces, quotes and special characters are handled correctly by the default quoting, and the quoting is the feature that the xargs pipeline needs the -0 flag for. The GNU Parallel quotes the arguments automatically, which means the pipeline that passes filenames with spaces works without the null-separator workaround, and this simplicity is the advantage over the raw xargs pipeline.
The environment variables that GNU Parallel passes to the jobs are the ones from the calling shell, and the PARALLEL environment variable can set the options permanently, the environment file that the production scripts source. The options set through the environment are the same options that the command line accepts, and the environment file is the configuration that the fleet distributes.
The GNU Parallel is the tool that the batch processing needs, and the batch processing is the workload that the production server runs nightly, weekly and monthly. The parallel execution that the tool provides is the performance multiplier, and the multiplier is the reason that the batch operations complete in the maintenance window rather than overrunning it, the property that the production environment needs and that the sequential execution cannot guarantee.