How it works...

Overall, no truly abstract concepts were introduced in this script, except for counting numbers of occurrences and the benefit of sort. Sorting can be a time-consuming process to reduce unneeded or extra data or when the order matters, but it can also be rewarding when performing bulk operations, and pre-processing yields faster returns overall.

Onward and upward to the recipe:

1. Running these two wc commands will produce both a character and line count of the file testdata/duplicates.txt. It also begins to show another problem. The data can be padded with the filename prefixed with a space:

$ wc -l testdata/duplicates.txt
18 testdata/duplicates.txt
$ wc -c testdata/duplicates.txt
438 testdata/duplicates.txt

2. In step 2, we use awk and cut to remove the second field. The cut command is a useful command for trimming strings, which may be delimited or merely using hard-coded values such as remove X characters. Using cut, -d stands for delimiter, the space in this example (' '), and -f1 stands for field 1:

$ wc -c testdata/duplicates.txt | cut -d ' ' -f1
438
$ wc -c testdata/duplicates.txt | awk '{ print $1 }'
438

3. In the final step, we run the sort command three times. We run it once to merely sort the elements in testdata/duplicates.txt, but then we use the -u to sort and keep only unique elements, and the final command counts the number of unique elements. Of course, the returned value is 9 because we had 18 lines in the original duplicates file:

$sort testdata/duplicates.txt

127.0.0.1 localhost
127.0.0.1 localhost
127.0.1.1 moon
127.0.1.1 moon
::1 ip6-localhost ip6-loopback
::1 ip6-localhost ip6-loopback
fe00::0 ip6-localnet
fe00::0 ip6-localnet
ff00::0 ip6-mcastprefix
ff00::0 ip6-mcastprefix
ff02::1 ip6-allnodes
ff02::1 ip6-allnodes
ff02::2 ip6-allrouters
ff02::2 ip6-allrouters
# The following lines are desirable for IPv6 capable hosts
# The following lines are desirable for IPv6 capable hosts

$ sort -u testdata/duplicates.txt

127.0.0.1 localhost
127.0.1.1 moon
::1 ip6-localhost ip6-loopback
fe00::0 ip6-localnet
ff00::0 ip6-mcastprefix
ff02::1 ip6-allnodes
ff02::2 ip6-allrouters
# The following lines are desirable for IPv6 capable hosts
$ sort -u testdata/duplicates.txt | wc -l
9