parallel compression utilities on linux -- lbzip2, pbzip2 and pigz

In the article of Compression tools on linux, I compared 4 different compression utilities which using different compression algorithms. Here I'll compare 3 popular parallel compression utilities. lbzip2, pbzip2 and pigz. lbzip2 and pbzip2 uses the same compression algorithm as bzip's, while pigz uese zlib which gzip uses.
Here are home pages of three compression utilities.

Test condition

The test bed was on the machine that has
2 sockets, 8 Intel(R) Core(TM) i7 CPUs         870  @ 2.93GHz
16GB memory, 2 sets of mirrored local SATA disks.
Linux kernel 2.6.32-358.18.1.el6.x86_64

Even though there is no disk bottle nect, I load file into memory before each test, so that there is no io concern at all.
Each tool utility has compression levels, I test level(fast) 1, 6(default) and 9(best). The ratio is the percentage reduction for each file compressed or decompressed(compatible with gzip -v)

Performance test(De refer to decompress)

  fast 1  De  default  6  De  best  9  De
   ratio      time(s)  time(s)  ratio  time(s)  time(s)  ratio      time(s)  time(s)
lbzip2  68.08%  1m20.83  0m59.87  70.50% 1m18.41  1m1.11  70.87% 1m27.52 1m0.19
pbzip2  68.05%  2m1.91 0m59.86  70.24% 1m59.79 1m2.32  70.82% 2m10.27 1m1.62
pigz  63.48% 1m59.50 1m0.22  67.11% 0m50.53 1m0.09  67.70% 3m0.01 1m0.48

Realiability test

The second is to compare their stability. So, I used the following simple script.
The purpose is that for each compression utility, I run 100 times compress and uncompress. with mdsum and filesize check.
  echo "$tool compress start"
  for i in `seq 1 100`
         echo "test No. $i"
         cp -f ${testfile}.orig $testfile
         rm -f ${testfile}.bz2
       ./to_mem $testfile
      time -p $tool -v -9 $testfile
      ls -l ${testfile}.bz2 >>${tool}.chk
      md5sum -b ${testfile}.bz2 >>${tool}.chk
      time -p $tool -v -d ${testfile}.bz2
         md5sum -b $testfile >>${tool}.chk
      sleep 3
  echo "$tool tests end"

test results are from 100 round tests
   ratio      compress time  uncompress time
 lbzip2  70.87% 83.6321  72.5878
 pbzip2  70.82%   130.576  69.3038
 pigz  67.70%  181.315  71.1547

Compatibility test

lbzip2 and pbzip2 use the same compression algorithm, so I also cross tested format compatibilities of bzip2, pbzip2 and lbzip2, they are all good, all passed md5sum file check.
As for pigz, there is one interesting thing, every time it yields out different compressed file, same file size, but md5sum, it does can recover the compressed file back to origional file with correct md5sum. I tested it with small files, I don't see this type of 'issue'.
As for compability with gzip, I don't see any issue, all good.

Note: I don't have any parallel compression utility which use LZMA algothrim, xz looks is heading to this.

Comments powered by CComment