When rsync compression maketh sense


Mikuru Beam!

I use and rely upon rsync so much for my backups and general file transfers, its not even a joke anymore. Well, it could still be a joke if I were the funny type. I type rsync commands, does that count?

Aside: In much the same way Nagato Yuki began gracing computer hardware posts here, in 2007 Asahina Mikuru began accompanying compression posts. I think I was implying her Mikuru Beam was more effective than regular file compression algorithms. I’ve long since switched to xz from rzip that I mentioned in that linked post, though I admit discussing rzip in a post about rsync is rather awesome.

Making rsync s/faster/slower with compression

One feature rsync has over traditional cp is temporary compression. Files are automatically compressed before transfer, then extracted at the destination. Under certain circumstances, it works a treat.

As I failed to realise last night however, rsync with compression obviously has the same limitations as compressing regular files. Check out the table below showing some transfers I ran late last night over USB 2.0:

Data type with -z without -z Diff
Anime video 13.45MB/s 29.41MB/s - 45.73%
Assignment PDFs 43.12MB/s 28.13MB/s +153.29%

Even when using gz, the impact on performance when using compression on losslessly compressed multimedia (do people still use that term?) files and textual data was rather jaw dropping. Lesson learned: if your data can be compressed well before transmission, and your bandwidth is more of a constraint than disk IO, memory or CPU, rsync with compression will yield great results. Otherwise, you may risk making your transfer run even slower!

For general usage

I’ve aliased rsync in my tcshrc twice, once to issue when dealing strictly with data that I know will compress well, and another for everything else:

  alias ry 'rsync -ahv --progress --stats'
  alias ryc 'rsync -ahv --compress --progress --stats'

I’m also looking into this --skip-compress option, which lets you define extensions that it won’t compress. This could be useful for defining losslessly compressed media we want ignored, and could negate the need for my two separate aliases. We’ll see!

Author bio and support


Ruben Schade is a technical writer and infrastructure architect in Sydney, Australia who refers to himself in the third person in bios. Hi!

The site is powered by Hugo, FreeBSD, and OpenZFS on OrionVM, everyone’s favourite bespoke cloud infrastructure provider.

If you found this post helpful or entertaining, you can shout me a coffee or send a comment. Thanks ☺️.