This site is an archive; learn more about 8 years of OpenHatch.

[OH-Dev] Adding simple command line interface to oh-bugimporters

Asheesh Laroia asheesh at asheesh.org
Fri Aug 17 17:51:01 UTC 2012


Howdy devel-ers,

I've added a command line interface for oh-bugimporters that should 
dramatically simplify its use.

I've written docs for it here: 
https://github.com/paulproteus/oh-bugimporters/blob/feature/command-line-interface/docs/cli.rst

And the code is here: 
https://github.com/paulproteus/oh-bugimporters/tree/feature/command-line-interface

The exciting thing is: the command line interface that the docs describe 
actually works! Although it has some flaws at the moment, as described 
there.

I ran this command:

./env/bin/python bugimporters/main.py -i examples/sample_configuration.json -o /tmp/data.json

And after a few seconds, /tmp/data.json had data from a bug in the Twisted 
tracker.

It is quite hackish at the moment. My goal with this branch is to remove 
all of our *downloading*-related code, since the value on oh-bugimporters 
is the ability to *process* (AKA parse) bug data from the web. For now, 
I'm suggesting we replace the broken async downloading with calls to 
synchronous downloading tools. This all gets run in the background anyway, 
so it's no huge deal if it's absurdly slow.

Once we've migrated to doing synchronous downloads that way, we can remove 
all of our custom, low-quality async downloading code. Man, that will be 
so great.

Then optionally we can migrate to using scrapy for doing the downloading 
portion asynchronously, correctly.

I will say: if someone wants to contribute, the path forward is now 
dramatically more clear. So please step up, and I will be very happy to 
help as needed! Just reply to this if you want to try hacking on it.

In particular, a next task:

* Modify the input data format so that one can list CSV queries in the 
input data format. Document that in the docs, and move the sample CSV 
query out of "main.py" into examples/sample_configuration.json.


-- Asheesh.


More information about the Devel mailing list