[Devel] Making background tasks simpler (by ditching rabbitmq)

[Asheesh Laroia]

Hello all,

(first, let me apologize for my absence. I have only excuses, like 
spending lots of time at Debconf. Because I haven't been around to check 
in and such, we didn't reach very many of our release goals during July.)

Summary: We use the library 'celery' for some background tasks, still. On 
the deployment, we used to store notes about background tasks in a server 
called rabbitmq. Now, we store them in the main database.

Longer version
--------------

Right now, one reason the deployment gets slow is because rabbitmq is 
chewing up lots of RAM. I could figure out how to properly configure 
rabbitmq not to do that, but I'm not that enthusiastic about learning lots 
more about rabbitmq when we can just ditch it.

So, now, it's ditched! We store all background-task-related data in the 
Django database. This does mean the Django database will become massive, 
probably, but that's okay with me.

One source of complexity in the OpenHatch code world is that the 
deployment_settings.py (which we use on the real site) has different 
settings for celery than the regular settings.py. This means that there 
are issues that pop up on the deployment that don't manifest themselves in 
development.

By stopping using rabbitmq, there will be fewer of those, I hope.

To do this, we had to upgrade the version of celery we rely on. So when 
you grab the latest version of the code, you'll need to freshly 
'./bin/buildout' plus './bin/mysite syncdb --migrate', or else you might 
get errors.

The good news is that now I don't hate celery, since it doesn't force us 
to use rabbitmq, which I do hate administering. In the past I've dragged 
my heels on solving problems that are best-solved with celery. No more!

That's it for now
-----------------

If this strikes you as particularly insane, or particularly great, do 
reply.

-- Asheesh.