Quite often, in shell environment, you want to run one program in multiple processes to either fully utilize you CPU resources or other purpose. For example, you wan to do checksum or some checks to large number of files, using same program, or script. How to run one program in parallel to speed up the whole processing time?


In the begining time, I used the way like this 

 

for i in `seq 1 100`
{
   mytask &   
   sleep 1;
}

Also, tried this way

for i in `seq 1 100`
{
   mytask &  
   while  [ `ps -ef | grep mytask | grep -v grep |wc -l` -gt 3 ]   
   do
     sleep 1;
   done
}

Similar way by using function

function max2 
  {
    while [ `jobs | wc -l` -gt 2 ]
      do
        sleep 1;
        done
  }

 cat ./jobs.lst | while read name
   do
     max2;
     mytask ${name} &
   done

Here is the easist way I'd like to use now, using xargs

find ./x* | xargs -n 1 -P 5 bzip2

or

for i in `seq 1 100` | xargs -n 1 -P 5 mytask 

where,

 -n controls number of parameters to be processed by each task
-P controls how many CPU/processes to run in parallel

Similarily, there is GNU parallel tool you can use, but I haven't got chance to try it yet, however, there are some CPU intensive tools like pigz,pbzip2   etc.. they have multiple threads functions embeded.