Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add a libgdal-arrow-parquet package #679

Merged
merged 24 commits into from
Dec 10, 2022
Merged

Conversation

rouault
Copy link
Contributor

@rouault rouault commented Nov 18, 2022

Add a libgdal-arrow-parquet package which contains the ogr_Arrow.so/dylib/dll and ogr_Parquet.so/dylib/dll plugins

The libgdal-arrow-parquet build reuses the same build directory as libgdal, hence we just to have to turn on the required dependency and drivers in the build_arrow_parquet.sh/bat scripts.

Fixes #628

Checklist

  • Used a personal fork of the feedstock to propose changes
  • Bumped the build number (if the version is unchanged)
  • Reset the build number to 0 (if the version changed)
  • Re-rendered with the latest conda-smithy (Use the phrase @conda-forge-admin, please rerender in a comment in this PR for automated rerendering)
  • Ensured the license file is being packaged.

@conda-forge-linter
Copy link

Hi! This is the friendly automated conda-forge-linting service.

I just wanted to let you know that I linted all conda-recipes in your PR (recipe) and found it was in an excellent condition.

@rouault
Copy link
Contributor Author

rouault commented Nov 18, 2022

@conda-forge-admin, please rerender

@akrherz
Copy link
Contributor

akrherz commented Nov 18, 2022

@rouault , you are amazing!

My two cents and I could be totally wrong here, but arrow-cpp is a bit of trouble in conda-forge these days. For example, I am still stuck with gdal 3.4 due to dependency hell with arrow-cpp involved and a jam with aws-sdk-cpp conda-forge/arrow-cpp-feedstock#867

So I don't know how far this PR can get until that is cleared...

@akrherz
Copy link
Contributor

akrherz commented Nov 20, 2022

I noticed the failed win64 openssl=3 build picked up arrow-cpp: 0.8.0-py36_vc14_4 yikes.

@rouault
Copy link
Contributor Author

rouault commented Nov 21, 2022

arrow-cpp involved and a jam with aws-sdk-cpp

Selfishly, for GDAL needs, we don't need arrow-cpp to be built against aws-sdk-cpp, as GDAL plugs its VSI I/O layer into libarrow, and thus can access AWS S3 through /vsis3/ . Would dropping aws-sdk-cpp dependency from arrow-cpp be acceptable ?

@ocefpaf
Copy link
Member

ocefpaf commented Nov 21, 2022

@xhochy sorry to ping you here but do you think you can help us navigate this arrow-parquet dependency for gdal?

Copy link
Member

@xhochy xhochy left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think it would be wise to wait until we have merged arrow-cpp=10 as this resolves the dependency solve issue with aws-sdk-cpp hopefully. We can also spit the arrow-cpp package a bit more up to have fewer dependencies but I think the core already links to aws-sdk-cpp, i.e. we won't be able to get rid of that.

recipe/meta.yaml Outdated Show resolved Hide resolved
@conda-forge-linter
Copy link

Hi! This is the friendly automated conda-forge-linting service.

I was trying to look for recipes to lint for you, but it appears we have a merge conflict.
Please try to merge or rebase with the base branch to resolve this conflict.

Please ping the 'conda-forge/core' team (using the @ notation in a comment) if you believe this is a bug.

rouault and others added 3 commits December 4, 2022 05:53
which contains the ogr_Arrow.so/dylib/dll and ogr_Parquet.so/dylib/dll
plugins

The libgdal-arrow-parquet build reuses the same build directory as
libgdal, hence we just to have to turn on the required dependency and
drivers in the build_arrow_parquet.sh/bat scripts.

Fixes conda-forge#628
@conda-forge-linter
Copy link

Hi! This is the friendly automated conda-forge-linting service.

I just wanted to let you know that I linted all conda-recipes in your PR (recipe) and found it was in an excellent condition.

recipe/meta.yaml Outdated Show resolved Hide resolved
@rouault
Copy link
Contributor Author

rouault commented Dec 9, 2022

@conda-forge-admin, please rerender

@github-actions
Copy link
Contributor

github-actions bot commented Dec 9, 2022

Hi! This is the friendly automated conda-forge-webservice.

I tried to rerender for you, but it looks like there was nothing to do.

This message was generated by GitHub actions workflow run https://github.com/conda-forge/gdal-feedstock/actions/runs/3653614116.

@rouault
Copy link
Contributor Author

rouault commented Dec 9, 2022

@conda-forge-admin, please rerender

@github-actions
Copy link
Contributor

github-actions bot commented Dec 9, 2022

Hi! This is the friendly automated conda-forge-webservice.

I tried to rerender for you, but it looks like there was nothing to do.

This message was generated by GitHub actions workflow run https://github.com/conda-forge/gdal-feedstock/actions/runs/3653660196.

@rouault
Copy link
Contributor Author

rouault commented Dec 9, 2022

@conda-forge-admin, please rerender

@github-actions
Copy link
Contributor

github-actions bot commented Dec 9, 2022

Hi! This is the friendly automated conda-forge-webservice.

I tried to rerender for you, but it looks like there was nothing to do.

This message was generated by GitHub actions workflow run https://github.com/conda-forge/gdal-feedstock/actions/runs/3653704985.

recipe/meta.yaml Outdated Show resolved Hide resolved
Co-authored-by: h-vetinari <h.vetinari@gmx.com>
@rouault
Copy link
Contributor Author

rouault commented Dec 9, 2022

@conda-forge-admin, please rerender

@github-actions
Copy link
Contributor

github-actions bot commented Dec 9, 2022

Hi! This is the friendly automated conda-forge-webservice.

I tried to rerender for you, but it looks like there was nothing to do.

This message was generated by GitHub actions workflow run https://github.com/conda-forge/gdal-feedstock/actions/runs/3654269582.

@jorisvandenbossche
Copy link
Member

(if the older arrow versions give problems, I think it's also perfectly fine to only have this for arrow-cpp 10 for now, since it's a new install feature for conda users anyway)

@h-vetinari
Copy link
Member

@rouault, your last few commits make no sense, as they will be immediately undone whenever the feedstock gets rerendered. Also, what's the issue with curl? If you don't want it, you'll have to remove it from the host-dependencies.

By the argument of Joris, I think it would be much easier to just skip the new output for older arrow version

recipe/meta.yaml Outdated Show resolved Hide resolved
@rouault
Copy link
Contributor Author

rouault commented Dec 9, 2022

your last few commits make no sense, as they will be immediately undone whenever the feedstock gets rerendered. Also, what's the issue with curl?

Probably. I told you I was a conda-forge noob ;-) well, I did them based on the error messages in failed builds, and looking at the diff of this PR against main. The added curl: 7 constraint was new to this PR and totally unrelated. We do want curl, but some package didn't like that constraint.

Co-authored-by: h-vetinari <h.vetinari@gmx.com>
@rouault
Copy link
Contributor Author

rouault commented Dec 9, 2022

@conda-forge-admin, please rerender

@rouault
Copy link
Contributor Author

rouault commented Dec 9, 2022

Hooray: green build!

@akrherz akrherz merged commit 2e4642e into conda-forge:main Dec 10, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Add parquet-cpp dependency
7 participants