compression

There are a few tools we can use to compress & decompress files: compress, gzip, bzip2 and zip. The decision to use one or another is down to speed, compress ratio and compatibility. Generally compress might be the fastest and bzip2 the slowest, with gzip & zip somewhere in between. But the compress ratio tends to show the opposite order.

Let's see some examples of how they are used:

# compress ­-f /tmp/rsync.log                    → compress and overwrite if necessary
.
# uncompress ­-c rsync2nfs.hist.Z | cat          → uncompress to stdout rather than a file
.
# compress -­rf /var/applog                      → recursively compress everything underneath
.
# gzip -­fk /var/tmp/app3.log                    → compress file, overwrite if needed, but keep original
.
# gzip -­c /tmp/gark.txt > /tmp/gk.gz            → compress to stdout and then to given file
.
# gzip ­-rf /var/app7                            → compress forcefully & recursively all contents of folder
.
# find . ­-name "*.[ch]" ­-print | zip source ­-@  → compresses into source.zip all found files
.
# zip ­-r -­ . | dd of=/dev/st0 obs=16k           → compress current folder and write it to tape
.
# zip ­-d foo.zip foo/tom/junk foo/harry/* *.o   → deletes all files/folders matching the path
.                                                 “foo/tom/junk” & “foo/harry/” and “*.o” from foo.zip
.
# bzip2 rsync2nfs.log ­-c | dd of=/dev/st0       → compresses file to stdout → dd → tape
.
# bzip2 ­-kf9 alert.log                          → compresses max, overwrites if needed but keeps original

At times we might find ourselves in a situation where we have a lot of compressed files in a storage volume that doesn't have enough free space to decompress more than a few of them. What do we do if we have to look for a regexp pattern in a huge list of compressed files? We can indeed move them around or write a script to decompresses-analyses-compresses them sequentially. But that's anything but convenient! What if we could decompress and view a file in one shot without using extra storage? Well, we can indeed with zcat, zmore, zless, bzcat, bzmore & bzless:

zcat application.log.Z | wc -l 
845964
.
file boot.log.Z
boot.log.Z: compress'd data 16 bits
.
zmore boot.log.Z
[ OK ] Started Show Plymouth Boot Screen.
[ OK ] Reached target Paths.
[ OK ] Started Forward Password Requests to Plymouth Directory Watch.
[ OK ] Found device HGST_HTS541010A9E680 3.
[ OK ] Found device /dev/disk/by-uuid/c4c5492b-4a5c-4c32-bbaf-a11b9df60c60.
[...]
.
zless boot.log.Z
[ OK ] Started Show Plymouth Boot Screen.
[ OK ] Reached target Paths.
[ OK ] Started Forward Password Requests to Plymouth Directory Watch.
[ OK ] Found device HGST_HTS541010A9E680 3.
[ OK ] Found device /dev/disk/by-uuid/c4c5492b-4a5c-4c32-bbaf-a11b9df60c60.
[...]
.
file boot.log.bz2
boot.log.bz2: bzip2 compressed data, block size = 900k
.
# bzcat boot.log.bz2 | wc -l
309
.
# bzmore boot.log.bz2
------> boot.log.bz2 <------
[ OK ] Started Show Plymouth Boot Screen.
[ OK ] Reached target Paths.
[ OK ] Started Forward Password Requests to Plymouth Directory Watch.
[ OK ] Found device HGST_HTS541010A9E680 3.
[ OK ] Found device /dev/disk/by-uuid/c4c5492b-4a5c-4c32-bbaf-a11b9df60c60.
[...]
.
bzless boot.log.bz2
[ESC[0;32m OK ESC[0m] Started Show Plymouth Boot Screen.
[ESC[0;32m OK ESC[0m] Reached target Paths.
[ESC[0;32m OK ESC[0m] Started Forward Password Requests to Plymouth Directory Watch.
ESC%GESC%G[ESC[0;32m OK ESC[0m] Found device HGST_HTS541010A9E680 3.
ESC%G[ESC[0;32m OK ESC[0m] Found device /dev/disk/by-uuid/c4c5492b-4a5c-4c32-bbaf-a11b9df60c60.
[...]

 

<< archival commands          other commands >>