[OH-Dev] Updates on the bug importer

[Asheesh Laroia]

Hey all,

I've been pushing some commits to the bug importer, and testing it, and 
there's a bunch more work that can be done to make it excellent. Some 
updates, and some suggestions, in case anyone's itching for some thing to 
code:

== Status ==

* There is now a web API call you can use to get a list of all the bug 
importers, as perceived by the backend:

https://openhatch.org/+api/v1/customs/tracker_model/?format=yaml&limit=500

This way, you can get the data on any particular tracker that isn't 
working, and create a local configuration file that matches the remote 
configuration for that tracker, very easily.

* The scrapy-based importer can handle that format properly. You can see a 
demo of how to import in the "run_importer.sh" script in oh-mainline.

== Small issue: github bug importer needs to generate _tracker_name ==

As per https://openhatch.org/bugs/issue773 , we can't import bugs from 
Github because the _tracker_name field is required. So the 
bugimporters/github.py file in oh-bugimporters needs to be updated to 
generate that. To see how it's supposed to work, look at 
bugimporters/trac.py. (If it's at all unclear, feel free to ask!)

This would be a good entry-level bug; you don't have to be super familiar 
with the oh-bugimporters code to do this.

== Small issue: Make sure the docs reflect the current state ==

Documentation at http://oh-bugimporters.readthedocs.org/ needs updating to 
reflect, specifically:

* The new tracker_model API call mentioned at the top of this page -- add 
a section about it to "How this integrates with the main OpenHatch site" 
section of the docs

* The _tracker_name field -- add a comment in bugimporters/items.py to 
indicate that it's mandatory

These should be pretty easy for anyone interested in writing some 
documentation. You'd also get to learn how this whole thing works, which 
is cool. To do it, you'd have to write docs in the Sphinx format, and edit 
the docs/ directory in oh-bugimporters 
<https://github.com/openhatch/oh-bugimporters>.

I'm happy to advise!

== Architectural issue: we should validate ParsedBug data we generate 
==

In oh-bugimporters, there should be some central bit of code, perhaps as 
part of the constructor for bugimporters.items.ParsedBug, that does the 
following things:

* sets last_polled to datetime.datetime.utcnow().isoformat()

* ensures _project_name is set

* ensures _tracker_name is set

That's a somewhat more complicated issue, but not that hard.

== Architectural issue: refresh old bugs ==

Right now, if a bug stops being bitesized, or gets closed, we don't 
notice.

https://openhatch.org/bugs/issue772 is an issue about that.

It's actually fairly easy to fix. It requires passing a little more data 
from the site (oh-mainline) into the bug import code (oh-bugimporters) -- 
specifically, oh-bugimporters needs a list of bug URLs to refresh.

The issue772 explains how to fix it.

== Overall ==

Overall, I'm really happy to see that a bug import run finishes in about 
30 minutes, which is absurdly fast by comparison to how things used to go. 
I think the new architecture is a huge improvement. Comments, patches, and 
the rest welcome!

-- Asheesh.