[OH-Dev] Current state of backups

[Asheesh Laroia]

Hi all Devel folks,

Summary: The main OpenHatch server has daily backups, and we have a 
Jenkins job (with public logs) that, weekly, makes sure the server can be 
restored from those backups.

Details:

You can see the Jenkins job log here: 
http://vm3.openhatch.org:8080/job/Test%20server%20backups%20(weekly)/lastBuild/console

What we do is:

* Spin up a new virtual machine on Rackspace Cloud

* Log into it, and download the latest backups from rsync.net

* chroot into the backup

* run a few commands to restore the server (e.g., restore MySQL from 
backups in *.sql.gz files, rather than relying on /var/lib/mysql ; start 
Apache; etc.)

* run a few commands to check that the backup actually restored things 
properly

It's driven by an ad-hoc combination of Python, shell, and GNU make. You 
can find those scripts here:

https://github.com/openhatch/oh-restore

When the job finishes, if it's successful, we rely on me to manually 
remove the virtual machine that it created.

There is some room for improvement, but I think we're on solid ground at 
this point.

-- Asheesh.