[OH-Dev] Adding simple command line interface to oh-bugimporters
Asheesh Laroia
asheesh at asheesh.org
Fri Aug 17 17:51:01 UTC 2012
Howdy devel-ers,
I've added a command line interface for oh-bugimporters that should
dramatically simplify its use.
I've written docs for it here:
https://github.com/paulproteus/oh-bugimporters/blob/feature/command-line-interface/docs/cli.rst
And the code is here:
https://github.com/paulproteus/oh-bugimporters/tree/feature/command-line-interface
The exciting thing is: the command line interface that the docs describe
actually works! Although it has some flaws at the moment, as described
there.
I ran this command:
./env/bin/python bugimporters/main.py -i examples/sample_configuration.json -o /tmp/data.json
And after a few seconds, /tmp/data.json had data from a bug in the Twisted
tracker.
It is quite hackish at the moment. My goal with this branch is to remove
all of our *downloading*-related code, since the value on oh-bugimporters
is the ability to *process* (AKA parse) bug data from the web. For now,
I'm suggesting we replace the broken async downloading with calls to
synchronous downloading tools. This all gets run in the background anyway,
so it's no huge deal if it's absurdly slow.
Once we've migrated to doing synchronous downloads that way, we can remove
all of our custom, low-quality async downloading code. Man, that will be
so great.
Then optionally we can migrate to using scrapy for doing the downloading
portion asynchronously, correctly.
I will say: if someone wants to contribute, the path forward is now
dramatically more clear. So please step up, and I will be very happy to
help as needed! Just reply to this if you want to try hacking on it.
In particular, a next task:
* Modify the input data format so that one can list CSV queries in the
input data format. Document that in the docs, and move the sample CSV
query out of "main.py" into examples/sample_configuration.json.
-- Asheesh.
More information about the Devel
mailing list