Shell Scripting Project: A Backup Script with Rotation and a Log Report

Step 1: Plan the backup script

Before writing code, decide what the script must do and how it should fail. It should take a source directory, a destination directory, and an optional number of days to keep. It should create one compressed archive with the date and time in its name, check the archive, delete only the old archives that belong to this source, and print a log line for each step. It should stop with a clear error for bad input, and exit with a non-zero status if anything goes wrong so a scheduler can report it.

Step 2: Write the script

The header turns on strict mode and defines small helpers for logging and errors. The argument checks run first, including a regular expression test that the retention value is a whole number. The archive name combines the folder name with a timestamp, so each run creates a new file and nothing is overwritten. The tar command with -C changes into the parent directory first, which stores relative paths in the archive instead of absolute ones, so it can be restored anywhere.

#!/usr/bin/env bash
# backup.sh - archive a directory and keep only recent backups
# Usage: backup.sh <source_dir> <backup_dir> [keep_days]
set -euo pipefail

usage() {
  echo "Usage: ${0##*/} <source_dir> <backup_dir> [keep_days]" >&2
  exit 2
}
log() { echo "$(date '+%F %T') $*"; }
die() { echo "$(date '+%F %T') ERROR: $*" >&2; exit 1; }

[[ $# -ge 2 && $# -le 3 ]] || usage
src=${1%/}
dest=$2
keep_days=${3:-7}

[[ -d $src ]] || die "source $src is not a directory"
[[ $keep_days =~ ^[0-9]+$ ]] || die "keep_days must be a whole number"
mkdir -p "$dest"

name=$(basename "$src")
stamp=$(date +%Y%m%d_%H%M%S)
archive="$dest/${name}_$stamp.tar.gz"

log "Backing up $src to $archive"
tar -czf "$archive" -C "$(dirname "$src")" "$name"

# Confirm the archive can be read before deleting anything old
tar -tzf "$archive" > /dev/null || die "archive $archive is corrupt"
size=$(du -h "$archive" | cut -f1)
log "Created $archive ($size)"

log "Removing backups older than $keep_days days"
find "$dest" -maxdepth 1 -name "${name}_*.tar.gz" -type f -mtime +"$keep_days" -print -delete

log "Done"

Step 3: Understand the important lines

Three details deserve attention. First, the find command filters on the folder name prefix, so a backup directory shared by several jobs never loses another job's files. Second, the age check uses -mtime with a plus sign, which means older than that many whole days, and the print option before delete lists what was removed so it appears in your log. Third, verification happens before deletion: if the new archive is unreadable the script stops, and the old backups stay untouched. Deleting the old backups first would risk leaving you with none.

Step 4: Test it safely

Test with throwaway data first. Create a small folder, run the script, and inspect the result. To check the retention logic without waiting, create an old file with a backdated modification time. The touch -d option works on GNU/Linux, and on macOS the equivalent uses -t with a timestamp. Run the script again and confirm the old file disappears while the new archives remain. Only point the script at real data after this works.

mkdir -p ~/demo/data && echo "hello" > ~/demo/data/a.txt
chmod +x backup.sh
./backup.sh ~/demo/data ~/demo/backups 7
ls -l ~/demo/backups

# simulate an old backup (GNU touch), then run again
touch -d "10 days ago" ~/demo/backups/data_old.tar.gz
./backup.sh ~/demo/data ~/demo/backups 7
ls -l ~/demo/backups

Step 5: Restore from a backup

A backup is only valuable if you know how to restore it, so practice that too. List the contents with tar -t to see what the archive holds, then extract into a different folder with -x and -C to check the files without touching the original. Document the restore command in a README beside the script so that anyone can follow it during an incident, not just the person who wrote the script.

# list what is inside
tar -tzf ~/demo/backups/data_20261110_020000.tar.gz

# extract into a separate folder to inspect it
mkdir -p ~/demo/restore
tar -xzf ~/demo/backups/data_20261110_020000.tar.gz -C ~/demo/restore
ls -R ~/demo/restore

Step 6: Write a log summary script

The second script reports on a web server access log using the pipeline pattern from the text processing lesson. It uses -eu without pipefail on purpose: when head closes a pipe after five lines, the commands before it can receive a broken pipe signal and report status 141, which pipefail would treat as a failure even though everything worked. The script validates its argument with the colon-question expansion, then prints the request count, the status code breakdown, the busiest addresses and paths, and the number of server errors.

#!/usr/bin/env bash
# log-report.sh - summarise a web server access log
set -eu

log_file=${1:?Usage: log-report.sh <access.log>}
[[ -r $log_file ]] || { echo "Cannot read $log_file" >&2; exit 1; }

echo "Requests: $(wc -l < "$log_file" | tr -d ' ')"

echo
echo "Status codes:"
awk '{ print $9 }' "$log_file" | sort | uniq -c | sort -rn

echo
echo "Top 5 IP addresses:"
awk '{ print $1 }' "$log_file" | sort | uniq -c | sort -rn | head -5

echo
echo "Top 5 requested paths:"
awk '{ print $7 }' "$log_file" | sort | uniq -c | sort -rn | head -5

echo
echo "Server errors (5xx): $(awk '$9 ~ /^5/' "$log_file" | wc -l | tr -d ' ')"

Step 7: Improve and extend the scripts

Once both scripts work, extend them as practice. Add a dry-run option that prints what would be deleted instead of deleting it. Exclude folders such as node_modules with tar's exclude option. Send a summary email or a message to a chat webhook when a run fails. Write the log lines to a file as well as the screen with tee. Each change is small, and together they cover most of what real automation scripts need. In the final lesson you will schedule them with cron and learn to debug them.

# dry-run idea: show candidates without deleting
find "$dest" -maxdepth 1 -name "${name}_*.tar.gz" -type f -mtime +"$keep_days" -print

# exclude folders from the archive
tar --exclude='node_modules' --exclude='.git' -czf "$archive" -C "$(dirname "$src")" "$name"

# write output to the screen and to a log file
./backup.sh ~/demo/data ~/demo/backups 7 2>&1 | tee -a ~/demo/backup.log