Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Dependency TODOs: enablements & clean-ups #944

Open
18 of 26 tasks
h-vetinari opened this issue Jan 27, 2023 · 6 comments
Open
18 of 26 tasks

Dependency TODOs: enablements & clean-ups #944

h-vetinari opened this issue Jan 27, 2023 · 6 comments

Comments

@h-vetinari
Copy link
Member

h-vetinari commented Jan 27, 2023

From looking at the list of third-party deps, we could still do better with:

Switch to shared builds for (if/once available):

  • google-cloud-cpp (win; not yet available in cf)
  • gflags (win; available, but apparently symbols aren't needed; clarified as of bda7682)
  • glog (win; available, but apparently symbols aren't needed; clarified as of bda7682)
  • grpc (win; available as of 1.55, which requires protobuf >=4.23; fixed as of 025b77a)
  • thrift (needed fixes for CMake metadata; fully cleaned up as of 2af3d60)

Bindings:

  • skyhook (needs librados)
  • tensorflow (though we should use libtensorflow, if at all, because it's a very heavy dependency to pull in)
@h-vetinari h-vetinari mentioned this issue Jan 27, 2023
3 tasks
@xhochy
Copy link
Member

xhochy commented Jan 27, 2023

We cannot unvendor jemalloc as the build is with special options. Unverdoring it would have an impact on the default allocator.

@jakirkham
Copy link
Member

Would it be possible to just build a variant of the jemalloc package with options needed?

@jakirkham
Copy link
Member

jakirkham commented Jan 27, 2023

Would not count on ucx being supported on other architectures OSes. Would also like to understand why that is relevant

@h-vetinari
Copy link
Member Author

Would also like to understand why that is relevant

I have a maximalist approach to packaging the bindings offered by a given library -- if the library offers integration for it, why shouldn't we package it?1

Users rely on conda-forge because a lot of these packages are really hard to build from source, so if a binding is not supported by us, it's effectively unusuable (even moreso if the distributed wheels also turn it off).

Footnotes

  1. whether to break a package into separate outputs is a separate question

@jakirkham
Copy link
Member

Sorry I should clarify, what I mean is why are other OSes relevant? Asking since ucx likely will remain Linux only (possibly macOS at some point and basically no chance of Windows support)

@h-vetinari
Copy link
Member Author

Sorry I should clarify, what I mean is why are other OSes relevant? Asking since ucx likely will remain Linux only (possibly macOS at some point and basically no chance of Windows support)

Not relevant indeed, unless they become: a) available for the respective platform & b) are supported by arrow on that platform.

@h-vetinari h-vetinari changed the title Unvendor more libs Dependency TODOs: enablements & clean-ups May 24, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

3 participants