Any interest in data package install / find API?

[matthew-brett]

Over on the mailing list for the nipy project, we need stuff for installing data packages locally, and providing an API to find them. I'm wondering if y'all would be interested in this going into datapackage?

Here's the draft with use cases and a proposal. We will probably end up implementing something in one of our libraries if y'all think this isn't a match for datapackage....

https://github.com/nipy/nibabel/wiki/data_packages

[trickvi]

Hi @matthew-brett

Yes this is something I have myself been thinking about how to do. OpenSpending is a project I'm using data packages for to do inflation adjustments and it would be great to be able to store an external data package locally instead of downloading it.

Have you thought about how you'd like to accomplish this. I'm a bit torn since datapackage is a module. What I've come up with is to perhaps create a class DataPackageStore or something where you can provide a local path where you'd like to store and load data packages from. That would allow people to easily build installation/caching etc. into their own applications like nipy for instance (which could then store it in ~/.nipy/datapackages/ or something while others store it somewhere else).

What are your thoughts?

[matthew-brett]

I think it would be reasonable to provide a script with with the module. I mean, there could be a datapackage script that packages some of the datapackage functionality on the command line. My own delocate package does this - you can use the code either from Python, or, substantially, from the command line - I use both myself : https://github.com/matthew-brett/delocate

That might attract developers too because the package can become a good default command line client.

Yes, I did think about how to customize the data store paths, maybe this could be in an object as you suggest. My thought had been that there should be one default, in the likely case that the user does not need to customize the paths, but that could easily be mixed with the ability to customize, sure.

[matthew-brett]

I plan to write up something like this draft:

https://github.com/nipy/nibabel/wiki/data_packages

If you are interested, I will write this for datapackage, otherwise I guess I will put it in one of our packages. I propose to make directories like ~/.datapackage be the default locations to look for configuration, but that this can be customized to - say - ~/nipy/datapackages via the command line and Python interface. The code will be in the Python modules, with thin wrappers for the command line.

[trickvi]

Hey @matthew-brett

This sounds really great. I hadn't even thought about including a script with a module but of course that could be done. It would also be good if that could be accessed via a module (just as you propose) so that software that uses the module can do what the script does programmatically (which I think is what you're suggesting).

If you create pull requests towards the development branch I'll review and merge as quickly as I can and release a version as soon as we're ready.

[trickvi]

@matthew-brett and by the way: Thanks a lot for the awesome write ups of the proposal. Not often you see things like these 😄

(feel free to use the datapackage name for the script -- I noticed that your write-up says data-tool)

[matthew-brett]

Sorry to go quiet, I just got back from a long time away from work, catching up.

I hope I will be able to get back to this next week.

Thanks for the kind words about the write-up; I find they help me think out the APIs, and I find that rather difficult :)

[vitorbaptista]

@matthew-brett We're migrating to a new library at https://github.com/frictionlessdata/datapackage-py. If you still have interest in doing this work, please open an issue there. It looks like a great addition!

Meanwhile, I suggest we close this.

[trickvi]

Yes @matthew-brett, the project will be maintained by the frictionlessdata project so any work should be done towards their version. Closing this issue.