[OH-Dev] [issue928] Use a cron job, rather than requests.get(), to update the blog cache

[Asheesh Laroia]

New submission from Asheesh Laroia <asheesh at asheesh.org>:

Right now, if the list of recent blog posts is empty, we use requests.get() to 
load up the blog feed.

The problem here is that if the cache is empty, every web worker will try to 
fill it. This leads to massive queuing of inbound requests, which is always bad 
news.

Therefore, we should be willing to serve from a stale cache. An easy way to do 
that is to use a file on-disk as the cache, rather than putting it in memcached.

The frontend could always serve the on-disk data. A cron job, or better, a pub-
sub-hubbub hook, could update that on-disk data.

Other architecture ideas welcome.

This addresses a key scalability problem with the site.

----------
messages: 4082
nosy: paulproteus
priority: bug
status: unread
title: Use a cron job, rather than requests.get(), to update the blog cache

__________________________________________
Roundup issue tracker <bugs at openhatch.org>
<https://openhatch.org/bugs/issue928>
__________________________________________