Home / Programming / Bash / Bash Parallel – easily run parallel processes in bash

Bash Parallel – easily run parallel processes in bash

You don’t have to write complicated bash shell scripts to accomplish parallel processing!

Take this as an example of the hard way, It’s going to create 5 “threads” and each while loop can run with 5 at a time.

Eh… Complicated. Why make life hard or reinvent the wheel? Just use parallel
The two main flags you would probably care about are:

-j maxjobs – Use to limit the number of jobs that are run at the same time.

-l maxload – Wait as needed to avoid starting new jobs when the system’s load average is not below the specified limit.
Some examples:

– Move files to a new directory
ls | parallel mv {} destdir
– This will recompress all files in the current directory with names ending in .gz using bzip2, running one job per CPU (-j #) in parallel.
ls *.gz | parallel -j 4 "zcat {} | bzip2 >{.}.bz2 && rm {}"

About kyle

Leave a Reply

Your email address will not be published. Required fields are marked *