dub.selections.json woes

[p0nce]

People seem confused by dub.selections.json on IRC.

<canidae> forgive my ignorance, but what's the point of dub.selections.json? i've specified a version in dub.json, why isn't this enough?
<canidae> perhaps there's a way to "disable" dub.selections.json?

In particular it is the 3rd time people ask how to get the latest tags in dependencies (I then advise to remove dub.selections.json + dub clean-caches).

=> dub upgrade is supposed to help but I guess it doesn't work everytime, which is something unrelated.

Honestly I somehow get why dub.selections.json and dub upgrade are there. But it is a bit backwards, it is more common to want the latest available version for everything (according to version selectors of course) that to want to stay on previous versions. If you want to pin dependencies into a specific version, you can always modify dub.json.

The other case I can think of is to pin the version of a sub-dependency, but the case is pretty rare. If a program with dependency A => B => C fail, it's quite easy to reproduce it with a program B => C. And committing dub.selections.json breaks the nice encapsulation of sub-dependencies DUB provides (A => B => C then A shouldn't know C).

Opinion: the cases where we do want dub.selections.json are a small minority.
I think it should be an opt-in choice. I remember a time where it wasn't there and never missed it.
In particular if a library pins dependencies instead of using SemVer, it creates diamond problems quickly.

[canidae]

Just adding my voice to this. While I'm not exceptionally versed in dub, this is not the first time dub.selections.json has caused versioning issues for me. This time I had a small project using DSFML version ">2.1". It was incompatible with newer versions of DMD. "dub upgrade" did not update dub.selections.json, changing version in dub.json to ">2.1.1" and then running "dub upgrade" still didn't do anything. Deleting dub.selections.json did however fix the issue, but it seems quite cumbersome if I have to delete that file each time I need to upgrade some libraries.
I don't need a dub.selections.json myself, and it's a feature I've never desired in any project I've worked on (including projects outside Dlang). While it may be useful in some cases, it seems like it's solving a problem only a minority experience. It would be nice if this feature was opt-in.

[mihails-strasuns]

I agree that it is both annoying and error-prone. Just recently I was going mad trying to reason why I can't reproduce build failure from CI on my local machine (on identical commit hash) until I have finally spotted that there is a dub.selections.json left from some old build and dub keeps using old dependencies locally (and CI gets latest ones as defined by dub.sdl because it is fresh clone).

[s-ludwig]

One thing needs to be clearly separated: Bugs are bugs and not conceptual issues. 2/3 of the text in this ticket is based on #528, which is fixed in the latest beta and doesn't have anything to do with dub.selections.json.

Assuming that everything works correctly, I don't think that there is anything error prone to the approach. Any quite frankly, it's vital not only to get a reproducible build across machines, a major issue for other packaging has been that large dependency trees became virtually unmanageable. In theory this wouldn't be the case, if everyone adhered to SemVer and all dependencies were referenced using ~>1.0.0 or ~>1.0, but in practice it's often necessary to manually control which dependencies get upgraded*. The selections file gives that control not only for direct dependencies, but for all of them.

Now let's imagine that we'd do without a per-package set of version selections and we'd always use the latest available versions. There are two possibilities:

• upgrading is still manual (with the upgrade suggestion as it is now) and issues like the CI issue would be there just like now. Nothing won.
• upgrading is automatic and in the mid of a work day dub suddenly decides to use a newer version of a dependency, resulting in breakage or changed behavior. Maybe you didn't notice it, maybe you don't even know what version it was before, but in any case now you need to take manual actions to restore the previous version(s). Since there is no selections file, the only way to achieve that in general is to directly depend on all indirect dependencies from the top level package, which defeats the whole purpose of a dependency management system.

I should probably dig up the old discussions about this topic with @jacob-carlborg, or something from Ruby/Bundler. But there simply are a lot of compelling arguments for this approach, even if it may not always seem necessary.

* Especially considering that maybe 80% of the packages have 0.x.x versions, which have no backwards compatibility rules.

[s-ludwig]

BTW, @Dicebot: Do you remember if you got upgrade suggestions during the local build? It would probably help to give them a different color on the console, as currently it's rather difficult so spot such information.

However, this would have happened the same way without a set of selections, if you wouldn't have had the latest dependencies installed on the local machine (except if the automatic upgrade mode was the default). On the other hand, putting dub.selections.json under version control would have solved it no matter what the environment is (it really should be versioned at least in 90% of the cases).

[p0nce]

upgrading is automatic and in the mid of a work day dub suddenly decides to use a newer version of a dependency, resulting in breakage or changed behavior.

This is actually what I'd want, but that's because I use SemVer and ~>1.0.
If dub upgrade has that effect (and it seems it has in principle, wasn't aware of it), then it's OK for me.

As for the rest, you made a very good case for the current approach and I think this ticket can be closed.

[jacob-carlborg]

A couple of notes on what's necessary to have dub.selection.json work correctly, assuming it's supposed to work like Gemfile.lock for bundler.

• For applications the dub.selection.json files should be in version control, for libraries it should not be
• The dub.selection.json file should never be deleted to update dependencies
• The dub.selection.json file should never be edited manually. I noticed that announcement of the new Dub beta (https://forum.dlang.org/post/nfpntb$kbl$1@digitalmars.com) containing this note:

"Implements proper optional dependency semantics, where using an optional dependency can now be controlled using dub.selections.json"

In my opinion that's broken, at least according to how I think dub.selections.json should work.

[s-ludwig]

The dub.selection.json file should never be edited manually.

Why? It would of course be nice to have a CLI for selecting/deselecting optional dependencies and for upgrading specific dependencies. But I don't see any reason to disallow manual edits.

[mihails-strasuns]

Do you remember if you got upgrade suggestions during the local build? It would probably help to give them a different color on the console, as currently it's rather difficult so spot such information.

Certainly not. Though it was rather long time ago and could be fixed in later versions.

However, this would have happened the same way without a set of selections, if you wouldn't have had the latest dependencies installed on the local machine (except if the automatic upgrade mode was the default).

Intuitively I would expect some user prompt whenever upgrade is available if dependency version is open. Which in relation with next statement ..

On the other hand, putting dub.selections.json under version control would have solved it no matter what the environment is

.. is probably what makes me feel most uneasy about this solution. If I wanted to pin dependencies hard I could as well just specified it using ==version in dub.sdl itself, without introducing any new project file to keep in repo.

[jacob-carlborg]

But I don't see any reason to disallow manual edits.

Here's the deal. I want Dub to somehow look down the dependency graph for a project to ensure that the same dependencies are always installed. I though that was how dub.selection.json works but that 's currently not the case.

As soon as you start manually editing dub.selection.json you'll get into problems. The tool cannot longer given the same guarantees about the dependency graph. It will be much easier to get a dependency graph that the tool cannot satisfy. Does Dub generate a selections graph for the optional dependencies as well? If no, then you're back to square one. If yes, then why are the optional dependencies in dub.selection.json instead of in dub.json?

The only reason to manually edit dub.selection.json is to bypass the tool to generate the looked down dependency graph. But why would you want to do that? Either advertising manually editing dub.selection.json or having features to requiring to manually editing dub.selection.json will get you the problems mentioned above.

Why? It would of course be nice to have a CLI for selecting/deselecting optional dependencies and for upgrading specific dependencies.

The solution is really simple, the optional dependencies should go in dub.json, not dub.selection.json.

[jacob-carlborg]

Intuitively I would expect some user prompt whenever upgrade is available if dependency version is open

To me that is very unexpected from a tool, a command line tool. That Dub checks for new dependencies for each build. I would expect that from a GUI application, like a browser, but not from a command line tool.

.. is probably what makes me feel most uneasy about this solution. If I wanted to pin dependencies hard I could as well just specified it using ==version in dub.sdl itself, without introducing any new project file to keep in repo.

It might work with one or two dependencies but when you have 40 dependencies (including indirect dependencies) it's a PITA. If you really want to look down the dependencies in dub.sdl you would need to specify all 40 dependencies including the exact versions. Good luck with trying to find a combination that actually works. And when you upgrade a dependency, you have to do all that again.

But seeing your opinions about the dependencies of DStep (which are 3) I assume you don't have any dependencies at all in your projects 😃.

[mihails-strasuns]

That Dub checks for new dependencies for each build.

Doesn't it do so anyway already?

It might work with one or two dependencies but when you have 40 dependencies (including indirect dependencies) it's a PITA.

How is it different from doing the same in dub.selections.json? Also forcing transitive dependencies to be not the ones defined in respective projects (without any validation from test suites) is something that just plain scares me.

Though I guess it is a problem of many dub packages declaring version compatibility they don't really check.

But seeing your opinions about the dependencies of DStep (which are 3) I assume you don't have any dependencies at all in your projects

There are from 3 to 8 dependencies in our apps (if you refer to Sociomantic) which so far feels quite manageable with even something as blunt as git submodules (== always hard locking of the version) :) My opinion of dstep dependencies comes not from their count but triviality.

[jacob-carlborg]

Doesn't it do so anyway already?

Yes, that's what I think is unexpected for a CLI tool.

How is it different from doing the same in dub.selections.json?

Dub is generating dub.selections.json. You can add one single entry in dub.json and end up with 40 entries in dub.selections.json. My whole point is that you should not edit dub.selections.json.

Also forcing transitive dependencies to be not the ones defined in respective projects (without any validation from test suites) is something that just plain scares me

Not sure I understand.

[jacob-carlborg]

There are from 3 to 8 dependencies in our apps (if you refer to Sociomantic) which so far feels quite manageable with even something as blunt as git submodules

A standard Rails (3.2) project comes with six direct dependencies, resulting in 40 (the above was just an arbitrary number I picked 😃) dependencies in total. That's what I have to deal with at work every day.

We probably have around 30 Rails projects. Try dealing with that manually. Of course, many of these projects are using different versions of Rails, resulting in a different dependency graph. Now apply that across a couple of different versions of Ruby as well and you have dependency hell 😃.

[s-ludwig]

If I wanted to pin dependencies hard I could as well just specified it using ==version in dub.sdl itself, without introducing any new project file to keep in repo.

Leaving aside that it might get unmanageable for large dependency graphs, this would be fine for application packages. However, for library packages it would mean that the library is basically unusable with other libraries that have an overlapping dependency graph, because exact version dependencies are very likely to produce conflicts. dub.selections.json only applies to the root package and thus doesn't have that problem.