[OH-Dev] Two different ways to refresh data on old bugs

[Asheesh Laroia]

Excerpts from Asheesh Laroia's message of Thu Nov 01 14:37:32 -0400 2012:
> Hello, all OH-Devvers,
> 
> I've been working on oh-bugimporters, and I'm working through the changes 
> necessary to support refreshing data about older bugs: 
> http://openhatch.org/bugs/issue772
> 
> I've hit an architectural issue I wanted some feedback on.
> 
> As background: What we do for Bugzilla, Trac, and Roundup is to re-scrape 
> each bug that we've ever seen, once a day. (We have to do that if, say, 
> the query we're given only shows new bugs, and a bug gets marked as 
> resolved.) For Trac and Roundup, it's O(N) requests to the remote bug 
> tracker. For Bugzilla we can group those into one big query that gets data 
> about all the bugs.
> 
> I've been looking at the Github and Google Code Issues API, and it seems 
> that both of them prefer that you make requests like this by asking for 
> "all bug data that has changed since a date".
> 
> (References:
> 
> Github: http://developer.github.com/v3/issues/
> 
> Google Code: https://code.google.com/p/support/wiki/IssueTrackerAPI )
> 
> Given that, I think it makes sense to make the following changes:
> 
> * In oh-mainline, for the backends that support it, change 
> mysite/customs/models.py GoogleTrackerModel.as_dict() and 
> GithubTrackerModel.as_dict() to also export a special bit of data called 
> get_older_bug_data. That will point to the URL of a query that, when 
> downloaded, will give all updates on that remote bug tracker since 
> whatever date you ask. Pre-configure that query URL to get all updates 
> since the minimum last_polled date of all the bugs we know about from the 
> tracker.

Implemented: https://github.com/openhatch/oh-mainline/pull/88

> * In oh-bugimporters: for Google Code and Github bug trackers, use the 
> get_older_bug_data query to download all updates since that date, and then 
> filter down the results so that we only export the data corresponding to 
> bugs we are actually tracking.

I'll get on this shortly, probably 0-2 days from now. It's a bit 
complicated. If someone wants to beat me to it, that'd be fine too!

-- Asheesh.