AWS S3 on CentOS/RHEL for off-site backupsPosted: December 5, 2017
AWS S3 has been around for a long time now. Yet, I am just now getting around to using it for an off-site backup location.
Here’s my plan. Backup to local RAID disk on my backup server, then make an off-site copy up to S3. In my mind, this covers the 3-2-1 backup rule. Here’s how I see it broken down. Reach out if you think my thought process is off.
3 copies: The first copy is your primary data, second copy is the local backup on your backup server, and the 3rd copy is what gets put into S3.
2 media types: This is where I might be off. One is the local backup and the second is S3. I question this a little because I’ve seen out there on the internet where some are talking about different physical media types, but I think that is overly redundant as long as you ensure that your off-site backup is secure. IE: first is hard drive so second can’t be hard drive. What do you think?
1 off-site copy: The copy out to S3.
This seems like a pretty solid backup policy.
How to set it up
yum install gcc libstdc++-devel gcc-c++ curl-devel libxml2-devel openssl-devel mailcap automake fuse-devel fuse-libs git libcurl-devel libxml2-devel make git clone https://github.com/s3fs-fuse/s3fs-fuse.git cd s3fs-fuse/ ./autogen.sh ./configure make && make install ln -s /usr/local/bin/s3fs /usr/bin/s3fs
- Once you have fuse and s3fs installed, create a bucket in S3, and record credentials for user with access to bucket. s3fs will use /etc/passwd-s3fs for credential storage. Please enter your bucket credentials in /etc/passwd-s3fs as follows:
- If you have multiple buckets that will be mounted to this machine, add the credentials in /etc/passwd-s3fs as follows:
- create a directory for mounting the s3 bucket
mkdir -p /mnt/s3fs-bucketname
- manually mount the bucket into the mount point
s3fs -o use_cache=/tmp/cache bucketname /mnt/s3fs-bucketname
The -f switch is helpful to run the process in the foreground to troubleshoot mounting.
- Once you confirm the mount is successful, you can enter the mount attributes in /etc/fstab so it mounts at startup.
s3fs#bucketname /mnt/s3fs-bucketname fuse allow_other,use_cache=/tmp/cache 0 0
Set up your backup client to put an extra copy in your /mnt/s3fs-bucketname directory. If you were really paranoid about data loss, you could always age your data in S3 to send it to Glacier at a certain time. I need to run with this for a little while and see what works best for my use case. Let me know if this works for you.