Automatically detect unregistered dependencies

[00vareladavid]

• support developed packages
• support packages which are tracking repos

resolves #1005

---

Let us banish this issue to the past

Example:

(B) pkg> status
Project B v0.1.0
    Status `/tmp/B/Project.toml`
  (empty environment)

(B) pkg> dev /tmp/C
 Resolving package versions...
  Updating `/tmp/B/Project.toml`
  [3c22e19c] + C v0.1.0 [`/tmp/C`]
  Updating `/tmp/B/Manifest.toml`
  [3c22e19c] + C v0.1.0 [`/tmp/C`]

(B) pkg> activate A
[ Info: activating environment at `/tmp/A/Project.toml`.

(A) pkg> dev /tmp/B
 Resolving package versions...
  Updating `/tmp/A/Project.toml`
  [391d23c0] + B v0.1.0 [`/tmp/B`]
  Updating `/tmp/A/Manifest.toml`
  [391d23c0] + B v0.1.0 [`/tmp/B`]
  [3c22e19c] + C v0.1.0 [`/tmp/C`]

[00vareladavid]

The sandbox isolation more heavily reuses existing Pkg mechanisms. This means that you can't use unregistered packages as test dependencies because we currently don't generally support it (I think there was special code for this before?). There now exists even more motivation for a solution to be in place.

[StefanKarpinski]

Bump. What's the next step for this?

[00vareladavid]

The purpose of these recent changes are so that we don't change the format of the Project file. Pkg will just check to see if there is a Manifest file available. If so it will walk down the dependency graph and collect any unregistered packages.

I also added some basics tests

[00vareladavid]

bors try

[bors]

try

Build succeeded

• continuous-integration/appveyor/branch
• continuous-integration/travis-ci/push

[KristofferC]

For packages tracking repos, this would only allow a "dependency chain length" of one, right? So if we have

A -[url]-> B -[url]-> C

where B is currently not downloaded, we would fail to find the manifest for C and error?

[schlichtanders]

Implementing 8 julia packages in sync, this issue is one of the most common issues I encountered.

Developing nicely separated package structure seems not easily supported by now in Julia.

I checked and tried a lot and have only found complicated solutions:

• adding all dependencies in correct order, i.e. having the need of a custom solution for what this pull request tries to solve
• having a private registry fails short because it is required to fix at least one version to a concrete git commit, however the version is not yet reliable but in development, i.e. everytime something new is developed, the registry is out of date. According to this discourse question it is intended that a registry always points to fixed versions and hence is not meant for development
• having a custom script which git clones all dependencies into .julia/dev (and adding .julia/dev onto the loadpath)
• [not an option at all for me] build a big and large monolithic julia package which has all packages in one package and makes it impossible to reuse well capsulated subfunctionalitiy
• [not an option any longer because I need to switch from windows to mac and need both to work for some time] you could develop everything locally with one user and one machine so that no one needs to be able to replicate your setting
• [not yet an option] this very pull request which would make it possible for everyone to just add the package (as soon as you have a sound manifest pointing to accessable repositories)

all this is surprisingly tedious and as a new developer for julia at least I myself feel a bit unwelcome by this needed complexity to develop clean code. This is one of these moments where I wished I found julia 2 years later ;-) luckily all my other moments of these kind are already resolved by now (I am intensively developing julia packages since 1.5 years), so I hope this one gets similarly resolved quite soon.

Being pointed to this merge request and realizing it is one year old is a bit hard for me to understand. It suggests that there already is an easy way for developers to developer new interdependent packages, but I looked hard and couldn't find anything but the above.

Just to make clear, I don't have enough capacity to solve this merge request myself, because, as you can imagine, I have troubles getting my 6 or 7 interdependent julia packages to a state where I can make them public (switching from windows to mac right now brought me this issue with all its force on top of solving my own bugs and missing features)

Hence I kindly want to suggest raising the priority of this pull request to high. Definitely something higher than "enhacnement".

[00vareladavid]

closing in favor of #1628