More complete support for unregistered dependencies #1628

00vareladavid · 2020-01-23T21:40:26Z

This adds a source field in the active project for any packages we are tracking by repo or by path. On add/dev/instantiate we crawl over any unregistered dependencies and download as necessary. After we collect all unregistered dependencies, we continue with resolve normally.

close #492 close #1005 supersedes #1088

tkf · 2020-01-27T21:33:17Z

How about looking into all manifest files checked in to the repository (e.g., test/Manifest.toml and docs/Manifest.toml)? You can use them if they include the "root" project as a dependency (which then implies that this Manifest file includes the dependencies of the "root" project).

~~I implemented it in Rogue.add and it is quite useful.~~ (Edit: it was more like a comment to #1088 than this PR)

tkf · 2020-01-27T21:38:41Z

src/Operations.jl

+        end
+        if new_pkg !== nothing
+            push!(unreg, new_pkg)
+            collect_unregistered!(ctx, new_pkg, new_git, unreg)


I wonder why you have to recurse? In principle, the manifest of pkg is likely to contain all URLs for unregistered dependencies, isn't it? Otherwise, such manifest cannot be instantiated.

What about when you do an add on a url and that package has dependencies on other unregistered packages?

In this case, I guess you'd need to parse the manifest file(s) contained in the newly to-be-added package? But what I was commenting was that, if this new package has a manifest file, it is likely that you don't need to parse the manifest files of dependencies of this newly added package. Or at least you can try to minimize the I/O by stop recursing if all direct and transitive dependencies are found. It could be an unnecessary premature optimization but don't need something like this to prevent infinite recursion anyway?

@tkf Perhaps you missed it, but this PR also introduces a source field (as discussed in #492). So there could be no manifest files involved.

Thanks for the clarification. I was confused since I came to this PR via another PR #1088.

00vareladavid · 2020-02-05T22:16:57Z

Getting very close now.

Here is an example of toggling between tracking a repo and tracking a path:

(demo) pkg> add https://github.com/00vareladavid/Unregistered.jl
   Updating git-repo `https://github.com/00vareladavid/Unregistered.jl`
   Updating registry at `~/.julia/registries/General`
   Updating git-repo `https://github.com/JuliaRegistries/General.git`
  Resolving package versions...
   Updating `/tmp/demo/Project.toml`
   dcb67f36 + Unregistered v0.2.0 `https://github.com/00vareladavid/Unregistered.jl#master`
   Updating `/tmp/demo/Manifest.toml`
   7876af07 + Example v0.5.3
   dcb67f36 + Unregistered v0.2.0 `https://github.com/00vareladavid/Unregistered.jl#master`

(demo) pkg> dev --local Unregistered
    Cloning git-repo `https://github.com/00vareladavid/Unregistered.jl`
  Resolving package versions...
   Updating `/tmp/demo/Project.toml`
   dcb67f36 ~ Unregistered v0.2.0 `https://github.com/00vareladavid/Unregistered.jl#master` ⇒ v0.2.0 `dev/Unregistered`
   Updating `/tmp/demo/Manifest.toml`
   dcb67f36 ~ Unregistered v0.2.0 `https://github.com/00vareladavid/Unregistered.jl#master` ⇒ v0.2.0 `dev/Unregistered`

(demo) pkg> add Unregistered
   Updating git-repo `https://github.com/00vareladavid/Unregistered.jl`
  Resolving package versions...
   Updating `/tmp/demo/Project.toml`
   dcb67f36 ~ Unregistered v0.2.0 `dev/Unregistered` ⇒ v0.2.0 `https://github.com/00vareladavid/Unregistered.jl#master`
   Updating `/tmp/demo/Manifest.toml`
   dcb67f36 ~ Unregistered v0.2.0 `dev/Unregistered` ⇒ v0.2.0 `https://github.com/00vareladavid/Unregistered.jl#master`

And adding a package with multiple unregistered dependencies:

(demo) pkg> add https://github.com/00vareladavid/UnregisteredA
   Updating git-repo `https://github.com/00vareladavid/UnregisteredA`
   Updating git-repo `https://github.com/00vareladavid/UnregisteredD`
   Updating git-repo `https://github.com/00vareladavid/UnregisteredB`
   Updating git-repo `https://github.com/00vareladavid/UnregisteredC`
  Resolving package versions...
   Updating `/tmp/demo/Project.toml`
   b686e103 + UnregisteredA v0.1.0 `https://github.com/00vareladavid/UnregisteredA#master`
   Updating `/tmp/demo/Manifest.toml`
   b686e103 + UnregisteredA v0.1.0 `https://github.com/00vareladavid/UnregisteredA#master`
   4a15487d + UnregisteredB v0.1.0 `https://github.com/00vareladavid/UnregisteredB#master`
   576bedce + UnregisteredC v0.1.0 `https://github.com/00vareladavid/UnregisteredC#master`
   295405fa + UnregisteredD v0.1.0 `https://github.com/00vareladavid/UnregisteredD#master`

KristofferC · 2020-02-13T16:18:07Z

When you think this is ready, it would be good with a bit of a higher level description of what this implements and how it works etc. Just to aid in reviewing it :)

00vareladavid · 2020-02-20T10:14:27Z

The way that this currently works is that, whenever we track a dependency by path or by repo, we add the source url to the source table in the project file. This solves the basic problem of storing the location for unregistered packages.

Whenever Pkg tracks an package by repo/path, we read its project file and see if it declares any dependencies on unregistered packages by checking its source table. We add any unregistered packages to the dependency graph and recurse until we have collected all reachable unregistered dependencies. Once we have collected unregistered dependencies, their versions are fixed, and we can continue resolving the rest of the dependency graph as normal.

When collecting unregistered packages, if the declared source of the package is a local directory and that directory is not a git repo, we fall back to tracking the dependency by path. This fallback allows #1005 to integrate seamlessly with the rest of the scheme.

00vareladavid · 2020-03-05T19:26:40Z

I will rebase this soon. Any feedback for this, even if high level?

StefanKarpinski · 2020-03-05T21:22:12Z

My feedback is "doooo it".

fredrikekre · 2020-03-05T21:46:04Z

I think the new source field can be discussed a bit more. The original idea (IIRC) was to probe the (checked in) manifest for source. What was wrong with that idea?

00vareladavid · 2020-03-06T03:15:13Z

The goal for this PR is to make the use of unregistered dependencies as seamless as possible. Relying on the presence of manifests has two problems: (1) people have to remember to commit it and (2) we already use the presence of a manifest as a signal to Pkg.test to use that dependency graph.

fredrikekre · 2020-03-11T09:49:25Z

Alright. In which cases is Pkg "allowed" to look at the source field? E.g. I don't think it should be allowed when sticking to registered versions for example.

KristofferC · 2020-03-11T09:56:52Z

Alright. In which cases is Pkg "allowed" to look at the source field?

Whenever tracking a package by path or url maybe?

fredrikekre · 2020-03-11T10:14:21Z

Yea, just wanted to make sure that is how this is implemented. Also; why not store the source for all packages #378 ?

KristofferC · 2020-03-11T10:38:30Z

That's in the manifest though. But yeah, that's a good idea as well. Also, I notice that Cargo stores the URL to the registry where the information for the package was retrieved instead of the URL to the git repo. Something to think about perhaps.

casabre · 2020-11-10T17:52:19Z

Any progress here? I would love to see that feature backed in the next release 👍

This is a workaround [this issue](JuliaLang/Pkg.jl#1628)

TheCedarPrince · 2022-08-07T04:38:37Z

Bump - any more progress or work here? Saw the checks did not finish...

kmsherbertvt · 2024-11-25T17:01:47Z

Was this PR supposed to have been superseded by #3783?

On the one hand:
Issue #492, which seems to have been requesting the same thing this PR seeks to address, has been closed, and @KristofferC asserted there that #3783 solved it.

On the other hand:
A couple people pointed out in #492 that the [sources] documentation indicates it is only used in active environments, not when tracing dependencies:

Note that this information is only used when this environment is active, i.e. it is not used if this project is a package that is being used as a dependency.

Moreover, I just tried it and I still get the "no known versions" error for an unregistered dependency I've always gotten, when trying to add an unregistered package. (And the most recent mention of this PR seems to be due to another group encountering the same problem.)

00vareladavid force-pushed the dv_standalone_project branch 2 times, most recently from 2a180ba to 63ceeb6 Compare January 23, 2020 22:36

This comment has been minimized.

Sign in to view

tkf reviewed Jan 27, 2020

View reviewed changes

00vareladavid force-pushed the dv_standalone_project branch from 63ceeb6 to ad2c139 Compare February 5, 2020 22:12

This was referenced Feb 11, 2020

Feature Request Registry: Refer to master branch if no version is defined #1634

Open

Automatically detect unregistered dependencies #1088

Closed

00vareladavid changed the title ~~[WIP] [RFC] more complete support for unregistered dependencies~~ More complete support for unregistered dependencies Feb 12, 2020

00vareladavid force-pushed the dv_standalone_project branch from f21ee36 to 34b8d2f Compare February 13, 2020 08:22

more complete support for unregistered dependencies

db30acd

00vareladavid force-pushed the dv_standalone_project branch from d69bdf8 to db30acd Compare February 21, 2020 00:31

maltezfaria added a commit to WaveProp/WaveProp that referenced this pull request Dec 14, 2020

explicitly add untracked packages to tutorial Project.toml

7c4c950

This is a workaround [this issue](JuliaLang/Pkg.jl#1628)

pazner mentioned this pull request Jul 31, 2021

compat issue pazner/TriplotRecipes.jl#1

Closed

quinnj mentioned this pull request Nov 23, 2022

Store some location info for deps in Project.toml #492

Closed

quinnj mentioned this pull request Dec 12, 2022

Support [sources] section in Project.toml #3263

Closed

ggebbie mentioned this pull request Sep 16, 2024

Can't add TMItransient due to unsatisfiable requirements with TMI ggebbie/TMItransient.jl#28

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

More complete support for unregistered dependencies #1628

More complete support for unregistered dependencies #1628

00vareladavid commented Jan 23, 2020 •

edited

Loading

This comment has been minimized.

tkf commented Jan 27, 2020 •

edited

Loading

tkf Jan 27, 2020

KristofferC Jan 27, 2020

tkf Jan 27, 2020

00vareladavid Jan 28, 2020

tkf Jan 28, 2020

00vareladavid commented Feb 5, 2020

KristofferC commented Feb 13, 2020

00vareladavid commented Feb 20, 2020

00vareladavid commented Mar 5, 2020

StefanKarpinski commented Mar 5, 2020

fredrikekre commented Mar 5, 2020

00vareladavid commented Mar 6, 2020

fredrikekre commented Mar 11, 2020

KristofferC commented Mar 11, 2020

fredrikekre commented Mar 11, 2020

KristofferC commented Mar 11, 2020

casabre commented Nov 10, 2020

TheCedarPrince commented Aug 7, 2022

kmsherbertvt commented Nov 25, 2024

More complete support for unregistered dependencies #1628

Are you sure you want to change the base?

More complete support for unregistered dependencies #1628

Conversation

00vareladavid commented Jan 23, 2020 • edited Loading

This comment has been minimized.

tkf commented Jan 27, 2020 • edited Loading

tkf Jan 27, 2020

Choose a reason for hiding this comment

KristofferC Jan 27, 2020

Choose a reason for hiding this comment

tkf Jan 27, 2020

Choose a reason for hiding this comment

00vareladavid Jan 28, 2020

Choose a reason for hiding this comment

tkf Jan 28, 2020

Choose a reason for hiding this comment

00vareladavid commented Feb 5, 2020

KristofferC commented Feb 13, 2020

00vareladavid commented Feb 20, 2020

00vareladavid commented Mar 5, 2020

StefanKarpinski commented Mar 5, 2020

fredrikekre commented Mar 5, 2020

00vareladavid commented Mar 6, 2020

fredrikekre commented Mar 11, 2020

KristofferC commented Mar 11, 2020

fredrikekre commented Mar 11, 2020

KristofferC commented Mar 11, 2020

casabre commented Nov 10, 2020

TheCedarPrince commented Aug 7, 2022

kmsherbertvt commented Nov 25, 2024

00vareladavid commented Jan 23, 2020 •

edited

Loading

tkf commented Jan 27, 2020 •

edited

Loading