Published on

Using Cron to Send Backup Log Files from EC2 to S3

Authors

Hey there! If you're running apps on EC2 and want to make sure your logs are safely tucked away in S3, you've come to the right place. No lengthy explanations here - just a straight-to-the-point guide on setting up cron jobs to automatically ship your logs to S3. Let's get to it!

What We're Going to Do

  1. Set up the AWS CLI on your EC2 instance
  2. Create a simple backup script
  3. Schedule it with cron
  4. Test it and make sure everything works
  5. Tackle common problems if they come up

Step 1: Get Your EC2 Instance Ready

First, let's SSH into your EC2 instance:

ssh -i your-key.pem ec2-user@your-instance-ip

Now, make sure you have the AWS CLI installed:

# For Amazon Linux
sudo yum install -y aws-cli

# For Ubuntu
sudo apt update && sudo apt install -y awscli

Step 2: Set Up AWS Access

The easiest and safest way is to use an IAM role. If your EC2 doesn't have one yet:

  1. Go to the AWS console → IAM → Roles → Create role
  2. Select EC2 as the service
  3. Add the AmazonS3FullAccess permission (you can narrow this down later)
  4. Name it something like "EC2-S3-Backup-Role"
  5. Attach this role to your EC2 instance (EC2 console → Actions → Security → Modify IAM role)

Step 3: Create Your Backup Script

Let's create a simple script that will grab your logs and send them to S3:

mkdir -p ~/scripts
nano ~/scripts/send-logs-to-s3.sh

Now paste this script in:

#!/bin/bash

# Where are your logs?
LOG_DIR="/var/log"

# Which S3 bucket are we sending to?
S3_BUCKET="your-backup-bucket"

# Let's create a unique folder name using the date
BACKUP_DATE=$(date +%Y-%m-%d-%H%M)
S3_PATH="ec2-logs/$BACKUP_DATE"

# Create a temporary directory for our backup
TEMP_DIR="/tmp/log-backup-$BACKUP_DATE"
mkdir -p $TEMP_DIR

# Copy logs we care about
echo "Copying logs to temp folder..."
cp $LOG_DIR/syslog* $TEMP_DIR/ 2>/dev/null
cp $LOG_DIR/apache2/* $TEMP_DIR/ 2>/dev/null
cp $LOG_DIR/nginx/* $TEMP_DIR/ 2>/dev/null
cp $LOG_DIR/application.log* $TEMP_DIR/ 2>/dev/null
# Add any other logs you need

# Zip them up to save space
echo "Compressing logs..."
tar -czf "/tmp/logs-$BACKUP_DATE.tar.gz" -C $TEMP_DIR .

# Send to S3!
echo "Sending to S3..."
aws s3 cp "/tmp/logs-$BACKUP_DATE.tar.gz" "s3://$S3_BUCKET/$S3_PATH/"

# Clean up
echo "Cleaning up..."
rm -rf $TEMP_DIR
rm "/tmp/logs-$BACKUP_DATE.tar.gz"

echo "Done! Logs sent to s3://$S3_BUCKET/$S3_PATH/"

Make sure to replace your-backup-bucket with your actual S3 bucket name, and adjust the log paths based on what your application actually uses.

Make the script executable:

chmod +x ~/scripts/send-logs-to-s3.sh

Step 4: Test Your Script

Before scheduling, let's make sure it works:

~/scripts/send-logs-to-s3.sh

If all goes well, you should see your logs appear in your S3 bucket!

Step 5: Schedule with Cron

Now let's set up cron to run this automatically:

crontab -e

Add one of these lines (depending on how often you want backups):

# Daily backup at 2 AM
0 2 * * * ~/scripts/send-logs-to-s3.sh

# Every 6 hours
0 */6 * * * ~/scripts/send-logs-to-s3.sh

# Every Monday and Thursday at 3 AM
0 3 * * 1,4 ~/scripts/send-logs-to-s3.sh

Save and exit (usually Ctrl+X, then Y, then Enter if you're using nano).

Step 6: Add Some Logging

If you want to know what happened during your backups, add some logging:

crontab -e

Change your entry to:

0 2 * * * ~/scripts/send-logs-to-s3.sh >> ~/backup-logs.txt 2>&1

This will save both normal output and errors to a file you can check later.

Common Problems and Quick Fixes

"Command not found" errors

If cron can't find AWS CLI or other commands, it might be using a different PATH. Update your script to include:

#!/bin/bash
export PATH=$PATH:/usr/local/bin:/usr/bin

# Rest of the script...

Not enough disk space for temporary files

If your logs are huge, check disk space first:

df -h

You might need to use a different temp directory or clean up old files first.

S3 permission errors

Make sure your IAM role has these permissions at minimum:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "s3:PutObject",
                "s3:ListBucket"
            ],
            "Resource": [
                "arn:aws:s3:::your-bucket-name",
                "arn:aws:s3:::your-bucket-name/*"
            ]
        }
    ]
}

Not getting your backups when you expect them

Check if cron is running:

grep CRON /var/log/syslog

Bonus: Rotate Your Backups

If you're backing up frequently, you might want to automatically clean up old backups:

# Add this to your script

# Delete backups older than 30 days
echo "Removing old backups..."
OLD_DATE=$(date -d "30 days ago" +%Y-%m-%d)
aws s3 rm "s3://$S3_BUCKET/ec2-logs/" --recursive --exclude "*" --include "*" --include "*/logs-$OLD_DATE*"

That's It!

You've now got automated log backups from EC2 to S3! Here's a quick checklist to make sure you've got everything covered:

  • ✅ Script is executable
  • ✅ IAM permissions are correct
  • ✅ Cron is scheduled
  • ✅ You're monitoring for any failures
  • ✅ You've tested restoring from backup (you have, right?)

Now you can relax knowing your logs are safely backed up to S3 on a regular schedule. No need to manually copy files - let cron do the heavy lifting while you focus on more important stuff!

Happy automating! 🚀

Last updated: Saturday, May 3, 2025