parallel compression utilities on linux -- lbzip2, pbzip2 and pigz
In the article of Compression tools on linux, I compared 4 different compression utilities which using different compression algorithms. Here I'll compare 3 popular parallel compression utilities. lbzip2, pbzip2 and pigz. lbzip2 and pbzip2 uses the same compression algorithm as bzip's, while pigz uese zlib which gzip uses.
Here are home pages of three compression utilities.
The test bed was on the machine that has
2 sockets, 8 Intel(R) Core(TM) i7 CPUs 870 @ 2.93GHz
16GB memory, 2 sets of mirrored local SATA disks.
Linux kernel 2.6.32-358.18.1.el6.x86_64
Even though there is no disk bottle nect, I load file into memory before each test, so that there is no io concern at all.Each tool utility has compression levels, I test level(fast) 1, 6(default) and 9(best). The ratio is the percentage reduction for each file compressed or decompressed(compatible with gzip -v)
The purpose is that for each compression utility, I run 100 times compress and uncompress. with mdsum and filesize check.
test results are from 100 round tests
As for pigz, there is one interesting thing, every time it yields out different compressed file, same file size, but md5sum, it does can recover the compressed file back to origional file with correct md5sum. I tested it with small files, I don't see this type of 'issue'.
As for compability with gzip, I don't see any issue, all good.
Note: I don't have any parallel compression utility which use LZMA algothrim, xz looks is heading to this.