Using backup2L with PIGZ archiver

Tech Menu

Created: July 18th 2009
I recently upgraded to a multi core processor and have been looking for a way to leverage multiple CPU's when writing a backup. I found PIGZ, a threaded parallel file compressor that seems to work fine here: http://www.zlib.net/pigz/

However, if you replace gzip with pigz it does not play nice with the tar cfz or xzf command options. It leaves leading "/" on filenames in the archive causing restore to fail. To remedy this I've created a driver that pipes tar output through pigz to create the archive. Below is the driver. It works OK for me. Note that new archives will be created as tar.gz files.

NOTE (July 19th 2009):
The pigz archiver is apparently still in development. Debian and downstream distributions are now including it in their unstable repos as an alternative to the standard gzip. Changes have already been made to the default configuration and command line options. The archiver itself seems stable when run outside backup2l but for whatever reason it does not play nice with backup2l. A bug report has been filed with debian and forwarded to the author.

UPDATED: August 2nd 2012
The latest version is updated to detect number of cores and processors and set threads accordingly.

Download text file version here


# Update August 2nd 2012 to detect processors and cores
CREATE_DRIVER="DRIVER_TAR_GZ_PIGZ"

USER_DRIVER_LIST="DRIVER_TAR_GZ_PIGZ"

DRIVER_TAR_GZ_PIGZ () {
# NOTES: USE ONLY WITH MULTI CORE CPU
# REQUIRES YOU TO DOWNLOAD AND COMPILE PIGZ or install it from repo
# http://www.zlib.net/pigz/
# Copy it to your system path somewhere; eg /usr/bin/pigz
# Create uses PIGZ for threaded, multi file compression
# Extract uses standard tar xz so you can restore without pigz
case $1 in
-test)
require_tools tar gzip pigz cat
echo "ok"
;;
-suffix)
echo "tar.gz"
;;
# Create compression and threads can be altered by changing the create line.
### Default thread count is set to 2 with default compression level in this driver ##
## minimum compression example: tar cf - --files-from=$4 | pigz -0 > $3 2>&1
## maximum compression example: tar cf - --files-from=$4 | pigz -9 > $3 2>&1
## default compression example: tar cf - --files-from=$4 | pigz > $3 2>&1
## limit threads to 2 : tar cf - --files-from=$4 | pigz -p 2 > $3 2>&1
## use larger block size: tar cf - --files-from=$4 | pigz -b 256 > $3 2>&1
##
-create) # Arguments: $2 = BID, $3 = archive file name, $4 = file list file
PROC=`grep processor /proc/cpuinfo | wc -l`
PROC1=`grep "^physical id" /proc/cpuinfo | awk "{print $NF}" | sort -u | wc -l`
if [ "$PROC" != "" ]
then
echo "$PROC1 processors with $PROC cores detected"
else
PROC=1
PROC1=1
echo "$PROC1 processors with $PROC cores detected"
fi
tar cf - --files-from=$4 | pigz -p $PROC > $3 2>&1
;;
# Using tar tz for the toc verifies the PIGZ archive is usable
# in case you don't have pigz installed when you restore
-toc) # Arguments: $2 = BID, $3 = archive file name
cat $3 | tar tz | sed 's#^#/#'
;;
# cat $3 | tar xz --same-permission --same-owner -T $4 2>&1
-extract) # Arguments: $2 = BID, $3 = archive file name, $4 = file list file
cat $3 | pigz -d | tar x --same-permission --same-owner --files-from=$4 2>&1
;;
esac
}

Good luck.

Comments to siggma at trbailey dot net