LabKey has a number of stand-alone, single purpose servers under management. These servers are running at a variety of hosting and cloud providers (SliceHost, Rackspace and AWS to name a few). When we first got into this business, I wrote some very simple backup scripts that performed the required steps to backup the system configuration and important data. Over time, the simplified architecture that I designed for backups began to experience some problems. As with all “quick” solutions, the problems came about when the size of the data to be backed up on some of these servers began to grow and grow.
As I began re-evaluating our backup system, I reviewed a number of commercial and open source options out there. The solution that I chose is definitely not for everyone, as we have some very peculiar requirements. The results of my search are below.
- This backup system will only be used for our “hosted” servers (ie servers located at AWS or Rackspace, etc).
- This backup system will not be used to backup servers or desktops from the office
- Each server can be it’s own backup server and backup client
- The servers are not part of a cluster, or provide shared resources for all others servers.
- The backup requirements of these servers in terms of types of data, backup frequency and retention time are nearly identical
- Data transfer between servers on different cloud or hosting providers can be expensive
- Need the ability for servers at Rackspace to use CloudFiles and servers at Amazon to use S3 as the file transfers are then free.
- Cost and operational overhead of running dedicated “backup servers” seem too high.
- Server installation and configuration is performed using the automation tool Chef.
- OS or application configuration files do not need to be backed up.
- This greatly reduces the complexity of the required backup system.
- Do not require a “bare metal” restore option
- In the case of a catastrophic failure of the server, I simply need to use Chef to rebuild the server and then restore the our data.
- Use Rackspace CloudFiles or Amazon S3 for off-server storage of backups
- Servers at Rackspace should use CloudFiles and servers at Amazon should use S3 as the file transfers are then free.
I evaluated a number of options. Many of the options did not meet my requirements and were immediately dismissed. I have some of them listed here. This is by far not an exhaustive list.
- Despite the overhead of learning Amanda, this was the most robust solution out there. The code base have been around for every (I first used this back in the late 90’s).
- It can be run either in stand alone mode (ie the server is both the client and the server) or with a centralized backup server
- It is configured using text files which makes it easy to manage via Chef
- It handles all the annoyances of managing backup frequency, retention times, aging of old backups, etc
- The only problems were
- No support for CloudFiles
- Not very good documentation for installing in stand-alone mode
- It would have been great if they had this for other operating systems beside Windows.
- These required a central backup server so they were not choosen
- Backup Manager
- This tool fits all my needs except it did not support Rackspace CloudFiles
- Does not support Rackspace CloudFiles, plus looks like development ended in 2006
- Found Duplicity which can handle writing backups to local disk, CloudFiles and S3.
- Not an ideal solution as this requires me to essentially write a backup system from scratch and this has been done before by better programmers than me.
- Increased support burden over time.
- If I leave the company, there is no one else who can come in and easily manage the system
What did I choose?
In the end, The only choice that I had was to Roll-My-Own. There are no commercial or open-source solutions which support CloudFiles. This definitely shows the immaturity and limited adoption of Rackspace’s Cloud offerings. I was not really happy about this choice as it really feels like I am re-inventing the wheel. Sadly there does not seem to be any good enterprise-class backup solutions out there for backing up servers that are hosted “in the cloud”.
Below is some further information I found during my research