Unix file compression basics
A couple of years ago I wrote a post about rzip, and I'm still getting emails and comments from people about it. I've decided to dedicate this post to answering some of these questions so I can point people to it. Grilled cheese sandwiches contain tastyness.
For people coming from a Windows or classic Mac background, file compression on Unix-like operating systems (such as GNU/Linux and BSD) can seem a bit confusing. Unlike the ZIP format which can accept multiple files, most Unix compressors can only work on one file at a time, so we use an archiver to bundle up the files we want to compress first, then feed that archive to the compressor.
Step one: archive files
Overwhelmingly the most common file archiving tool on Unix-like systems is the tape archiver (
tar). This command creates a new tar archive from a folder which may contain files and other folders:
% tar cvf NewArchive.tar ./FolderToArchive/
An alternative is
pax, which I prefer because it tends to be more consistent, it archives symbolic links (aka Unix shortcuts or aliases) instead of following them, and it has a few nifty features like being able to specify files that fall within a certain date range.
% pax -wf NewArchive.pax ./FolderToArchive/
Step two: compress the archive
The amount of time and CPU power you have will determine which compression algorithm you'll employ. On Unix systems the two most common are
gzip which is fast, and
bzip2 which is slower but generally gets better compression ratios. Here are some examples of both compressing the archive we made in step 1:
% gzip -v NewArchive.tar % bzip2 -v NewArchive.tar
bzip2 you can adjust how much compression they perform by specifying a numeric flag from 1-9. Specify
--help for more information.
Step three: just do one step
Now that we know the difference between archiving and compressing, we can save ourselves some time and do them both in one step. Most versions of tar support an extra flag that tells it to compress an archive with the tool of your choice after its been made. "z" specifies gzip and "j" specifies bzip2:
% tar czvf NewArchive.tar.gz ./FolderToArchive/ % tar cjvf NewArchive.tar.bz2 ./FolderToArchive/
Step four: unarchiving
Basically the reverse of what we did before, and again tar can take care of it for us:
% tar xzvf NewArchive.tar.gz % tar xjvf NewArchive.tar.bz2
Recent versions of gnutar take this shorthand one step further with the "a" flag which automatically determines what compressor to use based on the file name you provide it; for example
NewArchive.tar.bz2 tells gnutar with the "
a" flag to use bzip2. Unless you're running a very recent Linux distribution or have installed it specifically though, you probably don't have it.
There are many, many other compressors you can use. My current ones of choice are xz and rzip which both achieve far higher compression ratios but require a fast machine with plenty of memory to run in a reasonable amount of time. I discussed rzip back in 2007, but I'll be dedicating a new post on its and xz's use later this week.
I hope this helps folks :).