Can we make this the official conda repo for DataFusion?

[andygrove]

Comment:

The Arrow community would like to help maintain this feedstock so that we can release DataFusion to crates.io, pypi, and conda every time we have a new release.

The ASF release process requires the PMC to vote on the release artifacts before they are published, but it looks like releases are currently automated here based on PRs being merged? There was a 0.6.0 release earlier today, for example.

@cpcloud Are you open to us making this the official conda repo for DataFusion and complying with the ASF release process?

[cpcloud]

Sounds good to me!

[jakirkham]

What would this mean for this PR ( #19 )?

More generally should the source of this recipe still be pointing at PyPI? Or should it be pointing at a different source?

Finally would it make sense to add other maintainers here based on who would be checking off on release artifacts generally?

[andygrove]

@jdye64 is helping with conda support in the repo (see apache/datafusion-python#124, for example) and would probably be best placed to answer the questions here.

I am happy to add other Arrow PMC members to the repo here if they are interested. When I offered to do this for PyPi, nobody was interested, and I expect the same will be true here.

I am hoping to get more people interested in helping with this project, and getting more frequent releases is a step towards that.

[andygrove]

I started a thread on the Arrow dev@ mailing list asking if anyone wants to be added.

https://lists.apache.org/thread/hfrobrdojohl8kg6y4d1y0jn9pqc5v8p

[jdye64]

@jakirkham the pypi source in the recipe is still fine. Thanks for checking!

[andygrove]

The artifacts published to pypi are official releases that have already been voted on. If this repo is just redistributing those artifacts then I don't think that anything additional is needed in terms of votes after all.

[jorisvandenbossche]

FWIW, for the apache/arrow monorepo packages that are on conda-forge (cpp, python, r), we don't consider those packages as "official" binaries under the ASF release process (i.e. those artifacts are not voted upon, not uploaded to an apache.org archive).

Personally I think that is a fine situation, as it is similar to other (system) package managers or distributions that can distribute packages without directly being under control of the package maintainers (eg debian (if they would have an arrow package), homebrew, ...). See the "Other installers" section at https://arrow.apache.org/install/

In practice, this repo already is kind of the official conda package for DataFusion, because on conda-forge there can only be one such repo, and that is this one.

If this repo is just redistributing those artifacts

Strictly speaking, it's not just redistributing, it's building new artifacts AFAIK (starting from the source release)

(that said, it's certainly good if there is involvement from the package authors here and get added as recipe maintainers)

[andygrove]

Thanks for the extra info @jorisvandenbossche, this is very helpful.

I will go ahead and close this issues since it seems everything is fine as is. Sorry for taking up all of your time on this. I will definitely encourage others to get involved in helping maintain this feedstock.