Why does uniq not remove all my duplicates?

The uniq command only compares each line with the one before it, so duplicates that are not next to each other survive. Sort the input first, or use sort -u when you only need unique lines and not counts.

It sends standard error, stream 2, to wherever standard output, stream 1, is currently going. Put it after the output redirection, as in command > file 2>&1, so that both streams end up in the file. If you write them in the opposite order, errors still go to the terminal.

Should I use awk, sed, or grep?

Use grep to select lines, sed for substitutions and simple edits, and awk when you need to work with columns or calculate. They combine well, and when a task needs real data structures or complex logic, a scripting language such as Python is usually clearer.

Shell Pipes, Redirection, grep, sed, and awk for Text Processing

Introduction

The real strength of the Unix shell is that small tools each do one job and can be connected together. A pipe sends the output of one command into the next, and redirection sends output to files. With a handful of text tools such as grep, sort, sed, and awk you can search logs, clean data, and build reports in a single line. This lesson explains how data flows through the shell and walks through the most useful commands, finishing with a log analysis pipeline you can reuse.

Prerequisites

Comfort running commands and writing a short script
A sample text or log file to practice on

Step-by-Step Guide

Step 1: Understand the three streams

Every command has three channels. Standard input is numbered 0 and is where it reads data, standard output is 1 and is where results go, and standard error is 2 and is where error messages go. Redirection operators let you send each one somewhere else. A single greater-than sign overwrites a file, two append to it, and the two-greater-than form captures only error messages. Keeping errors separate from results is what makes scripts and pipelines predictable.

ls > files.txt                   # overwrite files.txt with the output
ls >> files.txt                  # append to the file
ls /nope 2> errors.txt           # save only the error messages
ls /tmp /nope > out.txt 2> err.txt
ls /tmp /nope > all.txt 2>&1     # send both streams to one file
ls /tmp /nope &> all.txt         # Bash shortcut for the same thing
some_command > /dev/null 2>&1    # throw all output away
echo "problem" >&2               # print a message to standard error

Step 2: Connect commands with pipes

The vertical bar feeds the standard output of the command on its left into the standard input of the command on its right. Because each tool does one thing, you can combine them like building blocks: list, filter, sort, count. Read a pipeline from left to right, and build it up one command at a time, checking the output after each addition. That habit makes long pipelines easy to debug.

ls -l | wc -l
ps aux | grep nginx | grep -v grep
cut -d' ' -f1 access.log | sort | uniq -c | sort -rn | head -5

Step 3: Write multi-line input with here documents

A here document feeds several lines of text to a command, which is handy for creating files or printing help text from a script. The word after the double less-than sign marks where the text ends. Quoting that word stops the shell from expanding variables inside the text. A here string, with three less-than signs, passes a single value as input without creating a file.

cat > config.ini <<EOF
[server]
host=$HOSTNAME
port=8080
EOF

cat <<'EOF'
With a quoted marker, $HOME stays literal
EOF

grep -c "a" <<< "banana"     # here string: prints 1 (one matching line)

Step 4: Search text with grep

The grep command prints lines that match a pattern. Useful options include -i to ignore case, -n to show line numbers, -v to show lines that do not match, -c to count matching lines, and -r to search every file in a directory. The -E option enables extended regular expressions, so you can write alternatives with a vertical bar. With -q, grep prints nothing and only sets the exit status, which makes it ideal inside an if statement.

grep "error" app.log
grep -i "error" app.log            # ignore case
grep -n "error" app.log            # show line numbers
grep -v "debug" app.log            # lines that do not match
grep -c "error" app.log            # count matching lines
grep -r "TODO" src/                # search a whole directory
grep -E "error|warn" app.log       # extended regular expression
grep -q "error" app.log && echo "found"

Step 5: Slice, sort, and count with small tools

Several small commands do most of the shaping work in a pipeline. The cut command extracts columns, sort orders lines, uniq collapses repeated lines and with -c counts them, and wc counts lines or words. Remember that uniq only merges neighboring duplicates, so its input must be sorted first. The head and tail commands show the start or end of the input, and tail -f follows a growing log. The tr command translates characters.

cut -d, -f1,3 data.csv             # columns 1 and 3 of a CSV
sort names.txt                     # alphabetical order
sort -n -r numbers.txt             # numeric, largest first
sort -u names.txt                  # sort and remove duplicates
sort names.txt | uniq -c           # count repeated lines
head -n 5 app.log
tail -n 20 app.log
tail -f app.log                    # follow new lines, Ctrl+C to stop
wc -l app.log
tr 'a-z' 'A-Z' <<< "shell"         # SHELL

Step 6: Edit text with sed

The sed command is a stream editor, and its most common use is substitution with the s command. By default it replaces the first match on each line, and a trailing g replaces all of them. The -n option with a p command prints only selected lines, and a d command deletes lines that match. Editing a file in place needs the -i option, but its syntax differs: GNU sed on Linux accepts it alone, while the BSD sed on macOS requires an empty string after it. Keep a copy of important files before editing them in place.

sed 's/old/new/' file.txt          # replace the first match on each line
sed 's/old/new/g' file.txt         # replace every match
sed -n '5,10p' file.txt            # print only lines 5 to 10
sed '/^#/d' config.txt             # delete comment lines
sed -i 's/foo/bar/g' file.txt      # edit in place (GNU/Linux)
sed -i '' 's/foo/bar/g' file.txt   # edit in place (macOS/BSD)

Step 7: Analyze columns with awk

The awk language reads text line by line, splits each line into fields named $1, $2, and so on, and runs actions on lines that match a condition. It is the quickest way to pick columns, filter by a numeric value, and calculate totals. The BEGIN and END blocks run before the first line and after the last one, which is where you print headings and totals. Associative arrays let awk count occurrences in a single pass.

awk '{ print $1 }' access.log                 # first field of every line
awk -F, '{ print $2 }' data.csv               # comma-separated input
awk '$3 > 100 { print $1, $3 }' data.txt      # filter on a numeric field
awk '{ sum += $2 } END { print sum }' sales.txt
awk '{ count[$1]++ } END { for (k in count) print k, count[k] }' access.log

Step 8: Find files and act on them

The find command walks a directory tree and can test names, types, sizes, and modification times. You can then print the matches, delete them, or run a command on each one with -exec. To pass many file names to another command safely, use -print0 with xargs -0, which separates names with a null character so spaces and newlines in names cannot cause trouble. Run a find command without -delete first and read the list before you add it.

find . -name "*.log"
find /var/log -type f -mtime +7          # modified more than 7 days ago
find . -type f -size +100M               # larger than 100 megabytes
find . -name "*.tmp" -delete
find . -name "*.log" -exec gzip {} \;
find . -name "*.txt" -print0 | xargs -0 grep -l "error"

Step 9: Build a log analysis pipeline

Now combine the tools. In the common web server log format, the first field is the client address, the seventh is the requested path, and the ninth is the HTTP status code. Extract one column, sort it so equal values are neighbors, count them with uniq -c, then sort the counts from largest to smallest and keep the top few. This sort, count, sort pattern answers a surprising number of questions about logs and data files.

# Top 5 client IP addresses
awk '{ print $1 }' access.log | sort | uniq -c | sort -rn | head -5

# Count of each HTTP status code
awk '{ print $9 }' access.log | sort | uniq -c | sort -rn

# Most requested paths that returned an error
awk '$9 >= 500 { print $7 }' access.log | sort | uniq -c | sort -rn | head

FAQs

Why does uniq not remove all my duplicates?: The uniq command only compares each line with the one before it, so duplicates that are not next to each other survive. Sort the input first, or use sort -u when you only need unique lines and not counts.
What does 2>&1 mean?: It sends standard error, stream 2, to wherever standard output, stream 1, is currently going. Put it after the output redirection, as in command > file 2>&1, so that both streams end up in the file. If you write them in the opposite order, errors still go to the terminal.
Should I use awk, sed, or grep?: Use grep to select lines, sed for substitutions and simple edits, and awk when you need to work with columns or calculate. They combine well, and when a task needs real data structures or complex logic, a scripting language such as Python is usually clearer.

Summary

You learned how the three streams work, how to redirect them and connect commands with pipes, and how to use here documents, grep, cut, sort, uniq, sed, awk, find, and xargs. The sort, count, sort pattern gives quick answers from any log. Next you will make scripts safe and professional with strict mode, option parsing, and cleanup.