Skip to content
This repository has been archived by the owner on Aug 16, 2023. It is now read-only.

don't install into package directory #129

Open
stevengj opened this issue Oct 10, 2018 · 8 comments
Open

don't install into package directory #129

stevengj opened this issue Oct 10, 2018 · 8 comments

Comments

@stevengj
Copy link
Contributor

stevengj commented Oct 10, 2018

Currently, BinaryProvider downloads into MyPackage/deps/downloads and installs into MyPackage/deps/usr. With Julia 1.0, every version of MyPackage installs into its own directory, which means that every update installs a new copy of the binary dependencies even if they didn't change.

I don't think this is acceptable in the long run. Proposal:

  • BinaryProvider should install binary dependencies into a global SHA-addressed directory, e.g. ~/.julia/BinaryProvider/{MyPackage-UUID}/sha, so that they are shared between versions.

  • Some kind of hook should be added to Pkg.gc() to allow binary dependencies that are no longer needed to be deleted.

@fredrikekre
Copy link
Member

I think that the idea is that each builder will be a separate tiny package, that installs the binary dependency in its directory. Those small packages should have much fewer releases than the actual packages that uses them, so I think that will help, if not completely fix, this issue.

@stevengj
Copy link
Contributor Author

I thought "builder" referred to the repo using BinaryBuilder, which is usually not a Julia package at all?

But I get your point — one way to resolve this is to split every package using BinaryProvider into two packages, one that provides the binary and one that uses it. (Note that this doesn't seem to be documented as a best practice.)

This is likely to be a slow process since there are ~50 packages that use BinaryProvider at the moment, and hardly any of them seem to split their dependencies off like this. Is there a way to automate it?

@fredrikekre
Copy link
Member

@staticfloat is working on this new infrastructure, see e.g. https://github.com/JuliaPackaging/Yggdrasil

@stevengj
Copy link
Contributor Author

@fredrikekre, that seems totally unrelated — those are the builder repos, i.e. the repos used for BinaryBuilder. They are not Julia packages that use BinaryProvider. Am I missing something?

@staticfloat
Copy link
Member

@fredrikekre is correct, but his link doesn't show it. :)

What he's referring to is the .jll packages idea, which is outlined in the BB roadmap. Essentially, what I'm working on is to have builder repos output small, autogenerated packages that themselves have a lifecycle completely locked to builder repository changes; they are automatically generated by BB, automatically published to the General registry and do nice things like export handles to libraries, wrapper functions for calling executables, etc... It's not documented as a best practice because I hate adding more maintenance work onto people, so it's just going to become "something that happens" rather than "something you have to do."

@SimonDanisch
Copy link
Member

Where are we with this? is there a project for this? It seems like there are lots of small work items anyone could work on to make this happen! So maybe we could create a github project and make a few seizable issues to make it easier for outsiders to work on this!

@SimonDanisch
Copy link
Member

at least we could start with something like making run not mess with the LD_LIBRARY_PATH for executable binary products :P
Currently, loading a package with an executable bindeps destroys download for some unix system...
I feel like I could easily tackle this issue, but without a more precise roadmap & issues, it doesn't seem easy to make a PR that conforms to the current plans of the maintainers.

@fredrikekre
Copy link
Member

Elliot has been thinking about this recently, and we discussed it on the Pkg triage call the other week, see JuliaLang/Pkg.jl#841.

visr pushed a commit to visr/BinaryProvider.jl that referenced this issue May 10, 2019

Verified

This commit was created on GitHub.com and signed with GitHub’s verified signature. The key has expired.
Now that the sandbox is part of the rootfs, our CI currently
does not test changes to the sandbox anymore. Add a build
step that uses the sandbox from this repo.
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Projects
None yet
Development

No branches or pull requests

4 participants