The awk Cheatsheet: Process Text in Columns

You came here to slice a file into columns and pull out the bits you need, probably with awk because someone said it was the tool for the job. They were right. The recipes you actually reach for are below, the most-wanted ones first, so you can copy, swap the field number, and get back to the file in front of you. One thing to know before you scroll: awk reads a file line by line, splits each line into fields on whitespace by default, and numbers them $1, $2, and so on, with $0 being the whole line and $NF being the last field. Almost everything else builds on that.

You came here to slice a file into columns and pull out the bits you need, probably with awk because someone said it was the tool for the job. They were right. The recipes you actually reach for are below, the most-wanted ones first, so you can copy, swap the field number, and get back to the file in front of you.

One thing to know before you scroll. awk reads a file line by line, splits each line into fields on whitespace by default, and numbers them $1, $2, and so on. $0 is the whole line, and $NF is the last field no matter how many there are. Almost everything below builds on that single idea, so once it clicks the rest is just variations.

Print a column: $1, $3, $NF

The one everybody wants first. Wrap the program in single quotes so the shell leaves the $ signs alone, and name the field you want. $1 is the first column, $NF is the last, and $0 is the entire line.

The comma in print $1, $3 matters. With a comma you get the two values separated by a space (the output field separator, OFS). Drop the comma and write print $1 $3 and awk glues them together with nothing in between, which is almost never what you meant.

Set the field separator: -F

By default awk splits on runs of whitespace, so a CSV or /etc/passwd confuses it. The fix is -F, placed before the program, telling awk what to split on.

The separator can be a single character, several characters, or a regular expression. If you also want your output columns joined by something specific, set OFS in a BEGIN block, as in awk 'BEGIN {FS=","; OFS="\t"} {print $1, $2}', which reads a CSV and writes a TSV.

Filter rows by a pattern

This is where awk starts to feel like grep with a brain. A pattern in front of the action decides which lines the action runs on. Leave the action off and a matching line is printed whole.

The split between pattern and action is the whole model: pattern { action }. Either part is optional. A bare pattern means "print the line", and a bare action with no pattern means "run on every line". So awk '$3 > 100 {print $1}' reads as "on lines where field three beats 100, print field one".

Compute sums and averages

Numbers are where awk earns its keep over a pipe of cut and paste. Variables start at zero, persist across every line, and you print them at the end. NR is the running count of records (lines) awk has read.

If you want a per-group total instead of one grand total, use an array keyed by a column. This sums the bytes in field 7 of an access log, grouped by the status code in field 9, and prints each group once at the end:

awk '{bytes[$9] += $7} END {for (code in bytes) print code, bytes[code]}' access.log

Arrays in awk are associative, so the key can be any string. That one pattern, accumulate into array[key] then loop over it in END, covers a surprising amount of real reporting.

BEGIN and END blocks

A full awk program has three parts, and you can use any subset of them. BEGIN runs once before the first line, the middle rules run on each line, and END runs once after the last line. Use BEGIN for headers and setup, END for totals and summaries.

Here is the shape of a small report that uses all three, printing a header, the rows it cares about, and a total. Save it or paste it inline:

awk 'BEGIN { print "user  bytes" }
     $9 == 200 { print $1, $7; total += $7 }
     END { print "total", total }' access.log

The fields line up because awk does not care about your whitespace inside the program. You can spread a rule across several lines for readability, which beats cramming a real report onto one line and squinting at it later.

That last one is the kind of thing you type without thinking once the model sticks: split /etc/passwd on colons, print the usernames in field one, and pipe them to sort. No temporary file, no editor, just the column you wanted.

One-liners for logs and CSV

The working set, the lines that actually pay rent. These assume an Apache or Nginx access log in the common format (IP first, status in field 9, bytes in field 10) and a comma-separated CSV. Swap the field numbers to match your own layout.

That !seen[$1]++ trick is worth a second look, because it shows up everywhere. It uses field one as a key into an array. The first time a value appears, seen[$1] is zero, !0 is true, so the line prints, and ++ then bumps the count to one. Every later copy finds a non-zero count, !1 is false, and the line is skipped. The result is a stable dedupe that keeps the first occurrence and the original order, which sort -u cannot give you.

Where to go from here

That's the working set. Print a column, set the separator, filter rows, total a column, and the BEGIN/END scaffolding that turns those into a real report. Most of everyday awk is some mix of those, and the deep language features (functions, multidimensional arrays, getline) wait quietly until the rare day you need them.

If your text wrangling goes past columns, the same instinct carries over. Hunting for files by name, size, or age? See the find command cheatsheet, which pairs nicely with awk when you feed it a list of paths. Reaching for search and replace across a stream instead of by field? A good sed reference covers the substitutions awk is clumsy at. Different tools, same habit: stop memorizing, keep a solid cheatsheet within reach.

Frequently asked questions