AWS S3 on CentOS/RHEL for off-site backups

AWS S3 has been around for a long time now.  Yet, I am just now getting around to using it for an off-site backup location.

Here’s my plan.  Backup to local RAID disk on my backup server, then make an off-site copy up to S3.  In my mind, this covers the 3-2-1 backup rule.  Here’s how I see it broken down.  Reach out if you think my thought process is off.

3 copies: The first copy is your primary data, second copy is the local backup on your backup server, and the 3rd copy is what gets put into S3.

2 media types: This is where I might be off.  One is the local backup and the second is S3.  I question this a little because I’ve seen out there on the internet where some are talking about different physical media types, but I think that is overly redundant as long as you ensure that your off-site backup is secure.  IE: first is hard drive so second can’t be hard drive.   What do you think?

1 off-site copy:  The copy out to S3.

This seems like a pretty solid backup policy.

How to set it up

yum install gcc libstdc++-devel gcc-c++ curl-devel libxml2-devel openssl-devel mailcap automake fuse-devel fuse-libs git libcurl-devel libxml2-devel make
git clone https://github.com/s3fs-fuse/s3fs-fuse.git
cd s3fs-fuse/
./autogen.sh
./configure
make && make install
ln -s /usr/local/bin/s3fs /usr/bin/s3fs
  • Once you have fuse and s3fs installed, create a bucket in S3, and record credentials for user with access to bucket. s3fs will use /etc/passwd-s3fs for credential storage. Please enter your bucket credentials in /etc/passwd-s3fs as follows:
accessKeyId:secretAccessKey
  • If you have multiple buckets that will be mounted to this machine, add the credentials in /etc/passwd-s3fs as follows:
bucketName:accessKeyId:secretAccessKey
  • create a directory for mounting the s3 bucket
mkdir -p /mnt/s3fs-bucketname
  • manually mount the bucket into the mount point
s3fs -o use_cache=/tmp/cache bucketname /mnt/s3fs-bucketname

The -f switch is helpful to run the process in the foreground to troubleshoot mounting.

  • Once you confirm the mount is successful, you can enter the mount attributes in /etc/fstab so it mounts at startup.
s3fs#bucketname /mnt/s3fs-bucketname fuse allow_other,use_cache=/tmp/cache 0 0


Set up your backup client to put an extra copy in your /mnt/s3fs-bucketname directory.  If you were really paranoid about data loss, you could always age your data in S3 to send it to Glacier at a certain time.  I need to run with this for a little while and see what works best for my use case.  Let me know if this works for you.

 

 



Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s