Home

Linux Tips for Web Developers: Working with Text Files, Part 2

Linux Tips for Web Developers: Working with Text Files, Part 2

Previously, we worked with linux commands cat and grep to search through text files for specific strings. Today, we will talk about combining cat and grep with a new command, awk, that will enable you to do much more with text files, log files, and CSVs (or any text-based spreadsheet).

AWK is actually a programming language specifically designed for processing text. We’ll barely scratch the surface of all of awk’s capabilities, but I’ll show you some ways it can help you in a development environment.

Let’s start with a sample web server access log:

Unix


192.168.1.1 - - [29/Aug/2016:20:36:00] "GET /test.html HTTP/1.1" 200 
192.168.1.2 - - [29/Aug/2016:20:38:00] "GET /test.html HTTP/1.1" 200
192.168.1.2 - - [29/Aug/2016:20:42:00] "GET /test.html HTTP/1.1" 200
192.168.1.4 - - [29/Aug/2016:20:53:00] "GET /test.html HTTP/1.1" 200
192.168.1.4 - - [29/Aug/2016:20:54:47] "GET /test.html HTTP/1.1" 200

We learned how to pull only lines that contain certain strings (an IP address, for example) using grep in the previous blog post, but what if we only want a certain piece of information on the line, such as the date? We can do that using awk!

Unix


awk '{print $4}' access.log

Which returns:

Unix


[29/Aug/2016:20:36:00]
[29/Aug/2016:20:38:00]
[29/Aug/2016:20:42:00]
[29/Aug/2016:20:53:00]
[29/Aug/2016:20:54:47]

What we’ve done is told awk to print the 4th column of text. By default, awk separates columns by spaces. You can also print multiple columns at once. Let’s get the IP address and the page visited.

Unix


awk '{print $4 "\t" $6}' access.log

Which returns:

Unix


[29/Aug/2016:20:36:00]	/test.html
[29/Aug/2016:20:38:00]	/test.html
[29/Aug/2016:20:42:00]	/test.html
[29/Aug/2016:20:53:00]	/test.html
[29/Aug/2016:20:54:47]	/test.html

Neat! We’ve printed the 4th and 6th columns. I’ve separated the columns with a tab using the \t in double quotes, but it isn’t necessary.

Now, what if we just needed the pages visited by a specific IP address? We can combine cat, grep, and awk to customize our output.

Unix


cat access.log | grep "192.168.1.2" | awk '{print $1 " visited " $6}'

Which returns:

Unix


192.168.1.2 visited /test.html
192.168.1.2 visited /test.html

We’re displaying the log with cat, piping it to grep to search for our IP, and the piping it to awk to get the desired columns of text. Cool, huh?

You can also separate columns of text by characters other than spaces. This need commonly arises working with comma separated files like CSVs. For example: 

Unix


Peter Griffin, 31 Spooner St, Quahog
Homer Simpson, 732 Evergreen Terr, Springfield
Fred Flintstone, 301 Cobblestone Way, Bedrock

Let’s get a list of only the street addresses from this file.

Unix


awk -F ',' '{print $2}' spreadsheet.csv 

Which returns:

Unix


 31 Spooner St
 732 Evergreen Terr
 301 Cobblestone Way

Using the -F flag, we’ve told awk to treat the commas as the column separators.

Finally, let’s separate the FirstName LastName column into two separate columns and put it in a new file.

Unix


awk '{print $1 "," $2}' spreadsheet.csv > new.csv

There is much more you can do with awk, but hopefully this has shown you some of the possibilities of this powerful command.

RYAN ROBINSONAugust 31, 2016