When rsync compression maketh sense
Software
I use and rely upon rsync so much for my backups and general file transfers, its not even a joke anymore. Well, it could still be a joke if I were the funny type. I type rsync commands, does that count?
Aside: In much the same way Nagato Yuki began gracing computer hardware posts here, in 2007 Asahina Mikuru began accompanying compression posts. I think I was implying her Mikuru Beam was more effective than regular file compression algorithms. I’ve long since switched to xz from rzip that I mentioned in that linked post, though I admit discussing rzip in a post about rsync is rather awesome.
Making rsync s/faster/slower with compression
One feature rsync has over traditional cp is temporary compression. Files are automatically compressed before transfer, then extracted at the destination. Under certain circumstances, it works a treat.
As I failed to realise last night however, rsync with compression obviously has the same limitations as compressing regular files. Check out the table below showing some transfers I ran late last night over USB 2.0:
| Data type | with -z | without -z | Diff |
|---|---|---|---|
| Anime video | 13.45MB/s | 29.41MB/s | - 45.73% |
| Assignment PDFs | 43.12MB/s | 28.13MB/s | +153.29% |
Even when using gz, the impact on performance when using compression on losslessly compressed multimedia (do people still use that term?) files and textual data was rather jaw dropping. Lesson learned: if your data can be compressed well before transmission, and your bandwidth is more of a constraint than disk IO, memory or CPU, rsync with compression will yield great results. Otherwise, you may risk making your transfer run even slower!
For general usage
I’ve aliased rsync in my tcshrc twice, once to issue when dealing strictly with data that I know will compress well, and another for everything else:
alias ry 'rsync -ahv --progress --stats' alias ryc 'rsync -ahv --compress --progress --stats'
I’m also looking into this --skip-compress option, which lets you define extensions that it won’t compress. This could be useful for defining losslessly compressed media we want ignored, and could negate the need for my two separate aliases. We’ll see!
