[OH-Dev] Adding simple command line interface to oh-bugimporters

[Adrian Ancona]

It sounds awesome Asheesh. Thanks a lot!

On 08/17/2012 10:51 AM, Asheesh Laroia wrote:
> Howdy devel-ers,
>
> I've added a command line interface for oh-bugimporters that should 
> dramatically simplify its use.
>
> I've written docs for it here: 
> https://github.com/paulproteus/oh-bugimporters/blob/feature/command-line-interface/docs/cli.rst
>
> And the code is here: 
> https://github.com/paulproteus/oh-bugimporters/tree/feature/command-line-interface
>
> The exciting thing is: the command line interface that the docs 
> describe actually works! Although it has some flaws at the moment, as 
> described there.
>
> I ran this command:
>
> ./env/bin/python bugimporters/main.py -i 
> examples/sample_configuration.json -o /tmp/data.json
>
> And after a few seconds, /tmp/data.json had data from a bug in the 
> Twisted tracker.
>
> It is quite hackish at the moment. My goal with this branch is to 
> remove all of our *downloading*-related code, since the value on 
> oh-bugimporters is the ability to *process* (AKA parse) bug data from 
> the web. For now, I'm suggesting we replace the broken async 
> downloading with calls to synchronous downloading tools. This all gets 
> run in the background anyway, so it's no huge deal if it's absurdly slow.
>
> Once we've migrated to doing synchronous downloads that way, we can 
> remove all of our custom, low-quality async downloading code. Man, 
> that will be so great.
>
> Then optionally we can migrate to using scrapy for doing the 
> downloading portion asynchronously, correctly.
>
> I will say: if someone wants to contribute, the path forward is now 
> dramatically more clear. So please step up, and I will be very happy 
> to help as needed! Just reply to this if you want to try hacking on it.
>
> In particular, a next task:
>
> * Modify the input data format so that one can list CSV queries in the 
> input data format. Document that in the docs, and move the sample CSV 
> query out of "main.py" into examples/sample_configuration.json.
>
>
> -- Asheesh.
> _______________________________________________
> Devel mailing list
> Devel at lists.openhatch.org
> http://lists.openhatch.org/mailman/listinfo/devel
>