[OH-Dev] Server backup status

[Asheesh Laroia]

Hey all,

There's been an open ticket for creating a reasonable plan for restoring 
the OpenHatch site in case of a disaster on our main server.

As background: every night, we encrypt a system backup and send it to 
rsync.net, who is donating space for us for that purpose.

I spent a few hours today doing the following:

* Preparing a virtual machine (right now, on rackspace cloud) to download 
a backup from rsync.net, and making this download process automatic.

* Chrooting into that backup, and writing commands into a Makefile that 
restore that backup into working order.

The first of these, since it downloads the backup, requires secrets that I 
shouldn't make public -- the encryption and SSH keys we use for rsync.net. 
If you're on the Login Team, I can share this repo with you if you're 
quite careful with it.

The second of these is available here: 
https://github.com/openhatch/oh-restore

*Crucially*, the public oh-restore repository has a "make check" command 
that is the restore process's final self-check. If you want to know, "Can 
I believe that these backups restore what's needed to make the site work?" 
then you should make sure you're satisfied with that stanza. Right now, it 
only tests that the Django app can run; it needs to start testing the blog 
and wiki and bug tracker, too.

I'll complete the automatic restore process shortly (as per 0.12.02 
release goals). Once that's done and shipped in 0.12.02 and tested to my 
satisfaction, I aim to add a weekly task to Jenkins where anyone can see 
that it's tested periodically and works (probably won't be ready this 
month).

-- Asheesh.