[OH-Dev] State of oh-bugimporters

[Asheesh Laroia]

Excerpts from Asheesh Laroia's message of Thu Nov 15 17:04:29 -0500 2012:
> Excerpts from John Morrissey's message of Thu Nov 15 15:31:32 -0500 2012:
> > On Wed, Nov 14, 2012 at 06:56:41PM -0500, Asheesh Laroia wrote:
> > > I've just fixed two small issues that were preventing Launchpad bug
> > > data from being properly downloaded from projects.
> > > 
> > > With that fixed, we "only" need to fix up the Bugzilla bug importer,
> > > and then we'll finally have this rewrite finished, and all the bugs
> > > on https://openhatch.org/search/ will be up to date.
> > > 
> > > Ticket: http://openhatch.org/bugs/issue783
> > [snip]
> > > Does someone else happen to want to take point on this? If not, I'll
> > > probably dive into it soon, like starting 24 hours from now.
> > 
> > /me touches his nose. :-/
> > 
> > I should count myself out for at least the next couple of weeks; I'm packing
> > up in the Bay area, traveling a bit, then setting up in Cambridge.
> 
> No problem. I'm on it!
> 
> (Note that we did already seemingly port bugzilla's importers to 
> oh-bugimporters once; this is the re-port to oh-bugimporters' newfound 
> reliance on scrapy, and covering with the rich, thick gravy of automated 
> testing.)

As a status update:

* http://openhatch.org/+meta/ reports 1.45% stale bugs, down from >60% a 
week ago. I fixed a whole slew of crasher bugs with different bug 
importers. Some of those fixes are committed as hacks agains the live 
code, and will make their way into the actual Github repo shortly.

* There is a slight architectural problem with the Google importer: if a 
bug is stale in our database, and we do a query against Google Code 
looking for all updates in that bug tracker since the particular stale 
bug's last_polled time, if there are no updates on Google Code for that 
bug, then we don't get back any information on the bug. Therefore, we 
don't export any data on the bug from oh-bugimporters, and therefore we 
don't end up updating last_polled in the database. The fix, in my 
opinion, is to provide a simple dict like:

{'canonical_bug_link': "http://code.google.com/.../id=333",
 '_no_updates': True,
}

and then oh-mainline's import_bugimporter_data will know when it sees a 
note like that, that it should just update last_polled.

* I believe we don't have any Launchpad projects enabled with the 
new-style bug tracker models, and really any Bug object with a NULL 
tracker_id means it won't be updated, so I deleted any Bug objects with 
a NULL tracker_id, which resulted in a lot of Launchpad.net-based Bug 
objects vanishing.

Cheers, and good night for now,

-- Asheesh.