Operating system

AWK Commands and Examples in Linux: How to Use AWK?

Posted on by Bình Phạm

Introduction to the Linux AWK command usage tutorial

The awk command is a Linux tool and programming language that allows users to process and manipulate data and generate formatted reports. The tool supports various operations with advanced text processing and helps to express complex data selections.

In this tutorial, you will learn about AWK commands and examples in Linux, including what the awk command does and how to use it.

prerequisite

  • A system running Linux.
  • Access the terminal window.

AWK command syntax

How to use AWK commands in Linux? The syntax of the awk command is:

awk [options] 'selection_criteria {action}' input-file > output-file

The available options are:

Optionsdescription
-F [separator]Lets you specify a file separator. The default separator is a space.
-f [filename]Lets you specify the file that contains the AWK script. awk reads the program source from the specified file, not the first command-line argument.
-vUsed to assign variables.

Note: You may also be interested in learning about the Linux curl command, which allows you to transfer data to and from the server after processing it using awk.

How does the AWK command work?

The main purpose of this awk command is to make information retrieval and text operations easy to perform in Linux. The command works by sequentially scanning a set of input lines and searching for lines that match a user-specified pattern.

For each pattern, the user can specify the action to be performed on each row that matches the specified pattern. So, with AWK, users can easily handle complex log files and output readable reports.

AWK Operations

AWK allows users to perform various actions on input files or text. Some of the available actions are:

  • Scan files line by line.
  • Split the input line/file into fields.
  • Compares the input row or field to the specified pattern.
  • Perform various actions on the matching rows.
  • Format the output line.
  • Perform arithmetic and string operations.
  • Use control flow and loops on the output.
  • Convert files and data according to the specified structure.
  • Generate formatted reports.

AWK Statement

This command provides basic control flow statements (if-else, while, for, break) and also allows the user to group statements with curly braces {}。

  • if-else

AWK Commands in Linux and Examples: This statement works by evaluating the condition specified in parentheses, and if the condition is true, executing the statement that follows the statement. This section is optional. if-elseif else

For example:

awk -F ',' '{if($2==$3){print $1","$2","$3} else {print "No Duplicates"}}' answers.txt

The output shows the duplicates present in the line and the country without duplicates, if there are no duplicates in the line for the answers.

  • while

How to use AWK commands in Linux? While as long as the specified condition is true, the statement will be executed repeatedly. This means that it works just like in the C programming language. If the condition is true, the loop body is executed. If the condition is false, AWK proceeds.

For example, the following statement instructs AWK to print one of all input fields per line:

awk '{i=0; while(i<=NF) { print i ":"$i; i++;}}' employees.txt
  • for

The for statement also works like C, allowing the user to create loops that need to be executed a specific number of times.

Linux AWK command usage example:

awk 'BEGIN{for(i=1; i<=10; i++) print "The square root of", i, "is", i*i;}'

The above statement increases the value of i all the way up to ten and calculates the square root of i each time.

Note: The expression for in the if, whileor conditional section can contain relational operators such as <, , >=, == (equals) and !=(Not equal). Expressions can also include matching ∼ and !∼ with matching operator regular expressions, logical operators ||, &&, and !. Operators are grouped in parentheses.

  • break

The break statement immediately exits for from the closed whileor. To start the next iteration, use the continue statement.

The next statement instructs awk to jump to the next record and start scanning mode from the top. The exit statement indicates that the awk input has ended.

Here’s an example of the break statement:

awk 'BEGIN{x=1; while(1) {print "Example"; if ( x==5 ) break; x++; }}'

The above command interrupts the loop after 5 iterations.

Note: This AWK tool allows users to place annotations in the AWK program. Comments begin and end with a line #.

AWK mode

AWK commands in Linux and examples: Insert a pattern awk as a selector before the operation. The selector decides whether to perform an action. The following expressions can be used as patterns:

  • Commonly used expressions.
  • Arithmetic relational expressions.
  • String value expressions.
  • Any Boolean combination of the above expressions.

The following sections explain the above expressions and how to use them.

Note: Learn how to search for strings or patterns using the grep command.

Regular expression pattern

The regular expression pattern is the simplest form of expression that contains a string enclosed by a slash. It can be a series of letters, numbers, or a combination of both.

In the following example, the program outputs all lines that start with “A”. If the specified string is part of a larger word, it will also be printed.

awk '$1 ~ /^A/ {print $0}' employees.txt

Linux AWK Command Tutorial: Relational Expression Patterns

Another type of AWK pattern is the relational expression pattern. The relational expression pattern involves using any of the following relational operators: <, <=, ==, !=, >=, and >.

How to use AWK commands in Linux? Here’s an example of an AWK relational expression:

awk 'BEGIN { a = 10; b = 10; if (a == b) print "a == b" }'

Range mode

Linux AWK command usage example: The range pattern is a pattern consisting of two patterns separated by commas. The scope mode performs the specified action on each row between the occurrence of mode one and pattern two.

For example:

awk '/clerk/, /manager/ {print $1, $2}' employees.txt

The pattern above instructs AWK to print all input lines that contain the keywords “clerk” and “manager”.

Special modes of expression

Special expression patterns include BEGIN and END, which indicate program initialization and termination. The pattern matches the beginning of the input before the first record is processed by BEGIN. END After the last record has been processed, the pattern matches the end of the input.

For example, you can instruct AWK to display a message at the beginning and end of a process:

awk 'BEGIN { print "List of debtors:" }; {print $1, $2}; END {print "End of the debtor list"}' debtors.txt

Combination mode

AWK command and example in Linux: This awk command allows the user to combine two or more patterns using logical operators. A combination pattern can be any Boolean combination of patterns. The logical operators for the combinatorial mode are:

  • || (or)
  • && (with)
  • ! (Not)

For example:

awk '$3 > 10 && $4 < 20 {print $1, $2}' employees.txt

The output prints the first and second fields for records where the third field is greater than 10 and the fourth field is less than 20.

Linux AWK Command Tutorial: AWK Variables

The awk command has built-in field variables that divide the input file into separate sections called fields. The AWK assignee has the following variables for each data field:

  • $0. Lets you specify an entire row.
  • $1. Specify the first field.
  • $2. Specify the second field.
  • Wait a minute.

Other built-in AWK variables available are:

  • NR. Count the number of input records (typically rows). The awk command executes a pattern/action statement once for each record in the file.

For example:

awk '{print NR,$0}' employees.txt

The command displays the line number in the output.

  • NF. Counts the number of fields in the current input record and displays the last field of the file.

For example:

awk '{print $NF}' employees.txt
  • FS. Contains characters that separate fields on input lines. THE DEFAULT SEPARATOR IS A SPACE, BUT YOU CAN USE FS TO REASSIGN THE DELIMITER TO ANOTHER CHARACTER (USUALLY IN BEGIN).

How to use AWK commands in Linux? For example, you can make the etc/passwd file (list of users) more readable by changing the delimiter from a colon ( ) to a dash ( ) and printing out the field separator ::/

awk -FS 'BEGIN{FS=":"; OFS="-"} {print $0}' /etc/passwd
  • RS. Stores the current record separator. The default input line is the input record, which makes the line break the default record separator. This command is useful if the input is a comma-separated file (CSV).

For example:

awk 'BEGIN {FS="-"; RS=","; OFS=" owes Rs. "} {print $1,$2}' debtors.txt

Note: We first use the cat command to display the contents of the file, and then use AWK.

  • OFS. Stores the output field separator, and prints the delimited field. The default separator is a space. Whenever a printed file has several comma-separated parameters, OFS prints values between each parameter.

For example:

awk 'OFS=" works as " {print $1,$3}' employees.txt

AWK action

Linux AWK command usage example: This awk tool follows a rule that includes pattern-action pairs. Actions consist of statements enclosed in curly braces, and {} contains expressions, control statements, compound statements, input and output statements, and delete statements. These statements are described in the section above.

awk creates scripts using the following syntax:

awk '{action}'

For example:

awk '{print "How to use the awk command"}'

This simple command instructs awk to print the specified string each time the command is run. Use Ctrl+D to terminate the program.

How to Use AWK Commands – Examples

AWK Commands and Examples in Linux: In addition to manipulating data and generating formatted output, awk has other uses because it is a scripting language, not just a text processing command. This section explains the alternative use case for AWK.

  • Calculations. The awk command allows you to perform arithmetic calculations. For example:
df | awk '/\/dev\/loop/ {print $1"\t"$2 + $3}'

In this Linux AWK command usage example, we pipe into the df command and use the information generated in the report to calculate the total memory available and used by the mounted filesystem that contains only /dev and /loop in the name.

The generated report shows the sum of memory df for the /dev and /loop filesystems in the second and third columns of the output.

  • Filtering. The awk command allows you to filter the output by limiting the length of the line. For example:
awk 'length($0) > 8' /etc/shells

How to use AWK commands in Linux? In this example, we run the /etc/shells system file awk and filter the output to include only lines that contain more than 8 characters.

  • Monitoring. Check if a process is running in Linux via a pipeline input command. For example:<strong>ps</strong>
ps -ef | awk '{ if($NF == "clipboard") print $0}'

The output will print a list of all processes running on your machine, with the last field matching the specified pattern.

  • Count. You can use awk to count the number of characters in a row and get the numbers printed in the result. For example:
awk '{ print "The number of characters in line", NR,"=" length($0) }' employees.txt

Linux AWK command usage tutorial summary

After reading this tutorial of AWK commands and examples in Linux, you will understand what the awk command is and how to use it effectively in a variety of use cases.

The awk command is also a scripting language with many uses, and it is a must-have knowledge for every Linux user. Use it for powerful text manipulation and also as a scripting language.

Bình Phạm