Visualizing progression of file operations using pv (Pipe Viewer)

Compressing large files from command line can be time consuming, and unless using a file manager like Midnight Commander, there is usually no way to know in advance how long the operation is going to last.

Enter pv (Pipe Viewer), a neat CLI tool allowing to monitor the progress of data through a pipeline, by displaying a progress bar and indicating both processing speed and estimated time until completion.

Here are a few examples showing how to use the tool :

Compressing using gzip :

pv access.log | gzip > access.log.gz
 482MB 0:00:26 [18.3MB/s] [==================================>] 100%

Compressing using bzip2 :

pv access.log | bzip2 > access.log.bz2
 482MB 0:04:45 [1.69MB/s] [==================================>] 100%

Decompressing a gzip archive :

pv access.log.gz | gunzip > access.log
48.5MB 0:00:08 [5.85MB/s] [==================================>] 100%

Decompressing a bzip2 archive :

pv access.log.bz2 | bunzip2 > access.log
25.2MB 0:00:42 [ 607kB/s] [==================================>] 100%

One-liner to compress the current directory :

tar cpf - . | pv -s $(du -sb . | cut -f1) | gzip > archive.tar.gz
1.48GB 0:01:33 [16.3MB/s] [==================================>] 100%

This one is a little more complicated and requires an explanation : as pv cannot know in advance the amount of data to be transferred, we have to specify it beforehand using the du command.

There is so much more pv can do, for example limiting the transfer rate of a pipe (using the -L output modifier). Together with the -q display switch (which turns the progress bar off), it can be used to render text animations.

Playing a VT100 animation at 9600 bytes per second (a spinning and moving globe) :

curl -s http://artscene.textfiles.com/vt100/movglobe.vt | pv -q -L 9600

dnc : a CLI tool to check domain names configuration and statistics

For a long time, I needed a simple and minimalistic tool showing a quick overview of my domain names configuration and statistics. I originally planned to implement it as a shellscript, but Node makes it so easy to write efficient CLI tools and distribute them via npm so it made more sense to go this way. The tool performs asynchronous DNS requests and as a bonus, displays a nice colored Unicode table, thanks to the great cli-tables module.

Here is the output of running dnc with two domains in the configuration file :

dnc showing a report for two domains

To install, simply invoke :

npm install -g dnc

Source code and documentation are on GitHub : https://github.com/fcambus/dnc

NXDOMAIN Hijacking : Dnsmasq to the rescue!

I've been playing a lot with OpenWrt lately (an embedded Linux distribution targeted at routers), which uses Dnsmasq to provide DNS forwarding and DHCP. After reading about Dnsmasq in detail (should you want to do the same, "Alternative DNS Servers" has an entire chapter on it), I discovered a really interesting option : bogus-nxdomain.

To illustrate how it works, we are going to configure Dnsmasq to use OpenDNS resolvers, which perform NXDOMAIN hijacking by default.

Let's add this directive to dnsmasq.conf, specifying where to look for resolvers :

resolv-file=/etc/resolv.conf.dnsmasq

We then create the resolv.conf.dnsmasq file, and add the hosts we want Dnsmasq to forward queries to :

nameserver 208.67.220.220
nameserver 208.67.222.222

Now, let's attempt to resolve a non existant domain :

dig domain.nxdomain +short
67.215.65.132

Sure enough, instead of getting a NXDOMAIN response, we are redirected to an IP address belonging to OpenDNS.

We can verifiy that it's indeed the case by performing a reverse lookup :

dig -x 67.215.65.132 +short
hit-nxdomain.opendns.com.

We now add the bogus-nxdomain directive to dnsmasq.conf :

bogus-nxdomain=67.215.65.132

And finally, after restarting Dnsmasq, querying a non existant domain returns NXDOMAIN as it should :

dig domain.nxdomain +noall +answer +comments

; <<>> DiG 9.8.4-rpz2+rl005.12-P1 <<>> domain.nxdomain +noall +answer +comments
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NXDOMAIN, id: 53036
;; flags: qr rd ra; QUERY: 1, ANSWER: 0, AUTHORITY: 1, ADDITIONAL: 0

Storing ASCII art in the DNS

From time to time, I like to do some quick DNS related experiments, and storing ASCII art in the DNS had been on my idea list for quite a while.

The naive approach to this problem would be to simply use TXT records, but there are a couple of issues with this method. First, it's not possible to specify the order of priority of TXT records, and results seem to vary depending on implementations. But moreover, there is no way to have several times the exact same line, as the records would be identical and only one would be taken into account when parsing the zone.

On the other hand, NAPTR records look like the perfect candidates for this purpose, as they can be ordered using the Order and Preference fields (see RFC 2915, for the NAPTR RR Format).

For this experiment, the first ASCII diagram from RFC 1035 will be used.

Here is the content of the zone :

ascii NAPTR 100 10 "" "              Local Host                        |  Foreign    " "" .
ascii NAPTR 110 10 "" "                                                |             " "" .
ascii NAPTR 120 10 "" " +---------+               +----------+         |  +--------+ " "" .
ascii NAPTR 130 10 "" " |         | user queries  |          |queries  |  |        | " "" .
ascii NAPTR 140 10 "" " |  User   |-------------->|          |---------|->|Foreign | " "" .
ascii NAPTR 150 10 "" " | Program |               | Resolver |         |  |  Name  | " "" .
ascii NAPTR 160 10 "" " |         |<--------------|          |<--------|--| Server | " "" .
ascii NAPTR 170 10 "" " |         | user responses|          |responses|  |        | " "" .
ascii NAPTR 180 10 "" " +---------+               +----------+         |  +--------+ " "" .
ascii NAPTR 190 10 "" "                             |     A            |             " "" .
ascii NAPTR 200 10 "" "             cache additions |     | references |             " "" .
ascii NAPTR 210 10 "" "                             V     |            |             " "" .
ascii NAPTR 220 10 "" "                           +----------+         |             " "" .
ascii NAPTR 230 10 "" "                           |  cache   |         |             " "" .
ascii NAPTR 240 10 "" "                           +----------+         |             " "" .

And the result of querying the zone for NAPTR records :

dig ascii.statdns.org naptr +short
;; Truncated, retrying in TCP mode.
100 10 "" "              Local Host                        |  Foreign    " "" .
110 10 "" "                                                |             " "" .
120 10 "" " +---------+               +----------+         |  +--------+ " "" .
130 10 "" " |         | user queries  |          |queries  |  |        | " "" .
140 10 "" " |  User   |-------------->|          |---------|->|Foreign | " "" .
150 10 "" " | Program |               | Resolver |         |  |  Name  | " "" .
160 10 "" " |         |<--------------|          |<--------|--| Server | " "" .
170 10 "" " |         | user responses|          |responses|  |        | " "" .
180 10 "" " +---------+               +----------+         |  +--------+ " "" .
190 10 "" "                             |     A            |             " "" .
200 10 "" "             cache additions |     | references |             " "" .
210 10 "" "                             V     |            |             " "" .
220 10 "" "                           +----------+         |             " "" .
230 10 "" "                           |  cache   |         |             " "" .
240 10 "" "                           +----------+         |             " "" .

Parsing JSON from command line using Python

With JSON becoming ubiquitous as an API response format, it is sometimes desirable to parse JSON data from command line. There are multiple tools available in order to do so, and I'm personally using jq, a lightweight and flexible command-line JSON processor. While jq is a fantastic tool, it introduces a dependency and might not be available in all environments.

Python, on the other end, comes preinstalled on most Linux distributions and supports JSON out of the box since version 2.6. It can be used to pretty print or parse data, here are a few examples :

Pretty printing JSON data :

curl -s http://www.telize.com/geoip/46.19.37.108 | python -mjson.tool
{
    "area_code": "0",
    "asn": "AS196752",
    "continent_code": "EU",
    "country": "Netherlands",
    "country_code": "NL",
    "country_code3": "NLD",
    "dma_code": "0",
    "ip": "46.19.37.108",
    "isp": "Tilaa V.O.F.",
    "latitude": 52.5,
    "longitude": 5.75,
    "timezone": "Europe/Amsterdam"
}

A one-liner for parsing simple JSON data :

curl -s http://www.telize.com/geoip/46.19.37.108 | python -c 'import sys, json; print json.load(sys.stdin)["country"]'
Netherlands

A one-liner for parsing JSON arrays :

curl -s http://api.statdns.com/x/46.19.37.108 | python -c 'import sys, json; print json.load(sys.stdin)["answer"][0]["rdata"]'
statdns.net.