Improve the version solver #912

DartBot · 2015-06-05T22:17:37Z

Edit by @nex3: I've retargeted this issue to track a major version solver refactor that should allow us to fix the solver's invalid error messages, further limit the cases in which it goes exponential, and semantically determine when a package graph is too complex to solve.

<img src="https://avatars.githubusercontent.com/u/4833191?v=3" align="left" width="96" height="96"hspace="10"> Issue by tomaskulich
Originally opened as dart-lang/sdk#17429

Attachments:
pubspec.yaml (625 Bytes)
pubtrace.zip (0 Bytes)

DartBot · 2015-06-05T22:17:38Z

<img src="https://avatars.githubusercontent.com/u/4833191?v=3" align="left" width="48" height="48"hspace="10"> Comment by tomaskulich

Attachment:
pubtrace.zip (179.34 KB)

DartBot · 2015-06-05T22:17:38Z

<img src="https://avatars.githubusercontent.com/u/4833191?v=3" align="left" width="48" height="48"hspace="10"> Comment by tomaskulich

Update: I just read http://news.dartlang.org/2013/04/pubs-new-constraint-solver.html and I'm afraid I understand, what is going on.

Some naive coder produced a (in all other terms, fine quality) package on which I depend, however, he put too restricting version-constraint on some of his dependencies. Now it's hard to find a fit for all these dependencies, and pub instead of shouting this problem all over the place starts to

wait for it

BACKTRACKING THROUGH A WHOLE UNIVERSE OF ALL POSSIBLE COMBINATIONS OF PACKAGE VERSIONS in a desperate attempt to heroically fulfill THE constrains.

Dear Dart creators, most of you probably have CS degrees, so you should understand HOW VAST THIS UNIVERSE IS!

If you really want to help the users, try something like this:
Do some random-dependency-picking (or maybe even some sort of simulated annealing) for FIXED amount of time and FIXED memory usage (does pub memory leak?); collect heuristic information about possible problems (too restrictive dependencies between packages;) and then, eventually, REPORT A PROBLEM!

It's for no use, if pub keeps heroically eating crazy amounts of my time and my RAM.

DartBot · 2015-06-05T22:17:38Z

<img src="https://avatars.githubusercontent.com/u/46275?v=3" align="left" width="48" height="48"hspace="10"> Comment by munificent

error 'Connection closed before full header was received, uri = https://pub.dartlang.org/api/packages/http/versions/0.9.2%2B1' is printed, no packages are downloaded

This is an unrelated issue. Can you repro this reliably? If so, can you give me some details on your configuration?

or, the memory is exhausted. (see zipped output from pub --verbose get)

This probably happens in verbose mode, but not in non-verbose.

However, since my pubspec.yaml is not so bug, I would expect pub behaving more correctly.

The size of your pubspec means very little in the grand scheme of things. It may have only one dependency, but if that dependency in turn has a ton, then you're going to slurp in the world. What matters is the size of your transitive dependency graph.

BACKTRACKING THROUGH A WHOLE UNIVERSE OF ALL POSSIBLE COMBINATIONS OF PACKAGE VERSIONS in a desperate attempt to heroically fulfill THE constrains.

On my machine, it does that successfully and finds a solution in 1m30s. Still slower than I'd like, but some pathological package graphs are bound to be slow.

most of you probably have CS degrees, so you should understand HOW VAST THIS UNIVERSE IS!

I actually don't have a CS degree, but I am aware that constraint solving is NP-complete. In fact, you can generate a set of pub packages that model any 3SAT problem.

Do some random-dependency-picking (or maybe even some sort of simulated annealing)

We can't do that because it would then lead to non-deterministic results. Running pub upgrade shouldn't be like pulling the lever on a slot machine!

for FIXED amount of time

If a user wants to limit the time pub spends, they can always just Ctrl-C if it takes too long. If we add an artificial limit, all that means is that we'll fail for some users who would otherwise be patient enough to wait.

We discussed this, but adding a limit seems to be strictly worse than not having one.

FIXED memory usage (does pub memory leak?);

It doesn't have any leaks that we're aware of. But note that if you run it in verbose mode, it ends up using quite a bit of memory for the log transcript. There are some optimizations we could do there.

collect heuristic information about possible problems (too restrictive dependencies between packages;) and then, eventually, REPORT A PROBLEM!

This is a great idea. I'll leave this bug open for that (and update the description).

In your case, the problem is that clean_backend depends on an old version of path. Path is a widely shared dependency, so constraining that pulls down the entire package graph.

In general, to have a healthier package ecosystem where constraint solving works well, the low level packages that everyone uses need to be pretty stable. This is part of the reason we pushed path to 1.0.0: when it churns, it causes everyone pain.

In this case, I can see that most common disjoint constraint that the solver had to backtrack on was from path, so a simple heuristic to it keep a count of how often we hit a disjoint constraint for each package. We could then either report that to the user or possibly take that into account when backtracking and try to route around it.

Added Triaged label.
Changed the title to: "Use heuristics to show why solving is taking a long time".

DartBot · 2015-06-05T22:17:38Z

<img src="https://avatars.githubusercontent.com/u/4833191?v=3" align="left" width="48" height="48"hspace="10"> Comment by tomaskulich

Attachment:
pubspec.yaml (578 Bytes)

DartBot · 2015-06-05T22:17:38Z

<img src="https://avatars.githubusercontent.com/u/4833191?v=3" align="left" width="48" height="48"hspace="10"> Comment by tomaskulich

Did you succeeded in reproducing the bug? If this is problem for you, I can grant you access to one of my Amazon AWS machines, it fails very consistently there.

DartBot · 2015-06-05T22:17:38Z

<img src="https://avatars.githubusercontent.com/u/4865287?v=3" align="left" width="48" height="48"hspace="10"> Comment by lrhn

Added Area-Pub label.

DartBot · 2015-06-05T22:17:39Z

<img src="https://avatars.githubusercontent.com/u/188?v=3" align="left" width="48" height="48"hspace="10"> Comment by nex3

This is an unrelated issue. Can you repro this reliably? If so, can you give me some details on your configuration?

This is actually probably not unrelated; we've seen connection errors before on pathological graphs. It's often caused by making too many requests to the server too quickly.

WHAT SHOULD DEFAULT BEHAVIOR OF PUB BE

I like Bob's suggestion of printing this information automatically if version solving is taking a long time. That said, even if we're pretty confident something pathological is going on, it's non-trivial to figure out the specifics. If it were easy, we'd use that to just do version resolution quickly.

It's probably possible to come up with some way of saying "it's likely that this resolution is taking a long time due to constraints on package X", but doing that well will take a considerable amount of work that we're unlikely to be able to spare at the moment. If you want to take a stab at it, please feel free.

WHY THIS BOTHERS ME SO MUCH

I think you may be overstating the severity of the issue here. As far as we've seen, these pathological problems only come up when there are substantial version clashes in a dependency graph. It should be fixable without too much trouble by submitting a few patches to upgrade third-party packages, or even forking them yourself.

MY IDEA FOR IMPROVEMENT, RANDOMNESS AND EVERYTHING

Seeding the randomness from the pubspec doesn't help much. It still trains users that if their version resolution is failing, they can re-order their dependencies (or whatever) and try again. This is not something we want to encourage our users to do.

DartBot · 2015-06-05T22:17:39Z

<img src="https://avatars.githubusercontent.com/u/188?v=3" align="left" width="48" height="48"hspace="10"> Comment by nex3

Issue #971 has been merged into this issue.

cc @keertip.

DartBot · 2015-06-05T22:17:39Z

<img src="https://avatars.githubusercontent.com/u/46275?v=3" align="left" width="48" height="48"hspace="10"> Comment by munificent

Removed Type-Defect, Priority-Unassigned labels.
Added Type-Enhancement, Priority-Medium labels.

DartBot · 2015-06-05T22:17:39Z

<img src="https://avatars.githubusercontent.com/u/188?v=3" align="left" width="48" height="48"hspace="10"> Comment by nex3

Issue #1159 has been merged into this issue.

DartBot · 2015-06-05T22:17:39Z

<img src="https://avatars.githubusercontent.com/u/188?v=3" align="left" width="48" height="48"hspace="10"> Comment by nex3

Issue #1170 has been merged into this issue.

cc @munificent.
cc @alan-knight.
cc @ricowind.
cc @whesse.
cc @kasperl.

DartBot · 2015-06-05T22:17:40Z

<img src="https://avatars.githubusercontent.com/u/3476088?v=3" align="left" width="48" height="48"hspace="10"> Comment by alan-knight

I have a theory that the connection issue is that the solver goes away into exponential land without doing any network activity long enough for the network connection to time out. I do see that the progress indicator display stutters when solving a complicated problem like this, so if it stutters long enough....

DartBot · 2015-06-05T22:17:40Z

<img src="https://avatars.githubusercontent.com/u/2156198?v=3" align="left" width="48" height="48"hspace="10"> Comment by kasperl

In my book, this is a pretty serious usability issue reported by multiple people. Issue #1159 and issue #1170 contain reproductions.

Added this to the 1.8 milestone.
Removed Type-Enhancement, Priority-Medium labels.
Added Type-Defect, Priority-High labels.

DartBot · 2015-06-05T22:17:40Z

<img src="https://avatars.githubusercontent.com/u/188?v=3" align="left" width="48" height="48"hspace="10"> Comment by nex3

Issue #1171 has been merged into this issue.

DartBot · 2015-06-05T22:17:40Z

<img src="https://avatars.githubusercontent.com/u/188?v=3" align="left" width="48" height="48"hspace="10"> Comment by nex3

Issue #1171 has been merged into this issue.

DartBot · 2015-06-05T22:17:40Z

<img src="https://avatars.githubusercontent.com/u/188?v=3" align="left" width="48" height="48"hspace="10"> Comment by nex3

Issue #1187 has been merged into this issue.

DartBot · 2015-06-05T22:17:40Z

<img src="https://avatars.githubusercontent.com/u/5475521?v=3" align="left" width="48" height="48"hspace="10"> Comment by ricowind

This is marked priority high and milestone 1.8, could we get an owner?

DartBot · 2015-06-05T22:17:41Z

<img src="https://avatars.githubusercontent.com/u/46275?v=3" align="left" width="48" height="48"hspace="10"> Comment by munificent

I know this is an annoyance to users when it occurs, but we don't have immediate plans to try to solve it in pub itself. The simplest fix is and has been to address this at the ecosystem level: make sure packages have correct version ranges and try not to rev breaking changes in popular packages too frequently.

Set owner to @munificent.
Removed this from the 1.8 milestone.
Removed Priority-High label.
Added Priority-Medium label.

DartBot · 2015-06-05T22:17:41Z

<img src="https://avatars.githubusercontent.com/u/188?v=3" align="left" width="48" height="48"hspace="10"> Comment by nex3

Issue #1237 has been merged into this issue.

kevmoo · 2018-03-08T21:06:17Z

@chrisdlangton – your comment will be deleted and you will be banned from dart-lang for violating our code of conduct.

MikeMitterer · 2018-03-20T10:28:16Z

@kevmoo I don't know what chris said but I'm probably thinking the same. Even your own packages like reflectable (https://github.com/dart-lang/reflectable/blob/master/pubspec.yaml) are failing with those version problems. I just wanted to port some of my libs from mirrors to built-mirrors (via reflectable) I can't even start because it's not possible to solve the version constraints.

It's incredible frustrating! Almost 2.0 and this is still a huge problem...

kevmoo · 2018-03-20T15:55:56Z

@nex3 is cranking hard on this https://github.com/dart-lang/pub/tree/feature.solver

munificent · 2018-03-28T14:36:15Z

🎉🎉🎉🎉

thosakwe · 2018-03-28T15:17:05Z

🎈 Congrats!

dnfield · 2018-05-18T17:41:22Z

I believe I'm getting this issue now in flutter dev and master. I get errors for a few packages on flutter_tools - if I manually roll them back to versions that were published before May, they all work.

So I changed these:

async: 2.0.6 # was 2.0.7
matcher: 0.1.12 # was 0.12.2+1
pub_semver: 1.3.7 # was 1.4.1

in https://github.com/flutter/flutter/blob/master/packages/flutter_tools/pubspec.yaml
and I'm now able to install packages and run the tool.

I suspect this is a version solving issue, but let me know if this should actually be an issue in flutter/flutter.

zoechi · 2018-05-18T18:55:32Z

Usually there are just conflicting dependency constraints
Do you get any error message?

dnfield · 2018-05-18T19:40:13Z

Error message:

Building flutter tool...
Because flutter_tools depends on async 2.0.7 which doesn't match any versions, version solving failed.
Error: Unable to pub upgrade flutter tool. Retrying in five seconds...

Some more verbose error:

SLVR:   fact: no versions of async match 2.0.7
SLVR:   conflict: no versions of async match 2.0.7
SLVR:   ! async 2.0.7 is satisfied by async 2.0.7
SLVR:   ! which is caused by "flutter_tools depends on async 2.0.7"
SLVR:   ! thus: version solving failed
SLVR: Version solving took 0:00:01.110941 seconds.
    | Tried 1 solutions.
FINE: Resolving dependencies finished (1.116s).
ERR : Because flutter_tools depends on async 2.0.7 which doesn't match any versions, version solving failed.
FINE: Exception type: SolveFailure
FINE: package:pub/src/solver/version_solver.dart 308  VersionSolver._resolveConflict
    | package:pub/src/solver/version_solver.dart 131  VersionSolver._propagate
    | package:pub/src/solver/version_solver.dart 96   VersionSolver.solve
    | ===== asynchronous gap ===========================

zoechi · 2018-05-18T19:56:12Z

I guess the error message needs to be improved.
I'd suggest you create a new issue with above info.
There is no guarantee anyone will read comments on closed issues and it's different issue anyway.

dnfield · 2018-05-19T15:50:48Z

Good advice, although at this point I'm not consistently reproducing it - may have been environmental or caching issue :(

I'll open a new issue if I see it again/am able to consistently reproduce.

hpoit · 2019-07-31T22:21:52Z

I am consistently reproducing mine.

natebosch · 2019-07-31T22:49:29Z

@Hpoint - please update to a newer Dart SDK which includes the solution to this issue.

hpoit · 2019-07-31T23:38:37Z

Hi Nate. I was on 2.4.0, but angular_router does not work with it.

kevmoo · 2019-07-31T23:47:30Z

This is likely because angular_router is not stable – https://pub.dev/packages/angular_router/versions/2.0.0-alpha+24 You'll have to put a dependency of `^2.0.0-alpha` in your pubspec

…

On Wed, Jul 31, 2019 at 4:38 PM Henry Poit ***@***.***> wrote: Hi Nate. I was on 2.4.0, but angular_router does not work with it. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#912?email_source=notifications&email_token=AAAEFCVYOOG7MNZI55NYQTLQCIPADA5CNFSM4BHIHLP2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD3I3WTI#issuecomment-517061453>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAAEFCQFI2QYUUD3OWPMT5DQCIPADANCNFSM4BHIHLPQ> .

hpoit · 2019-07-31T23:48:51Z

Great, thanks Kevin

hpoit · 2019-07-31T23:52:15Z

The problem was that I was using angular_router 1.0.2. In general, is it okay to always use the alpha releases?

kevmoo · 2019-07-31T23:53:54Z

There are no plans to release a stable version of angular_router, so that's your only option.

…

On Wed, Jul 31, 2019 at 4:52 PM Henry Poit ***@***.***> wrote: The problem was that I was using angular_router 1.0.2. In general, is it okay to always use the alpha releases? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#912?email_source=notifications&email_token=AAAEFCVMXDM62CYVJAAZY63QCIQTFA5CNFSM4BHIHLP2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD3I4M3Q#issuecomment-517064302>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAAEFCSDL7WAIINXLE35HPDQCIQTFANCNFSM4BHIHLPQ> .

hpoit · 2019-07-31T23:53:59Z

Nate and Kevin, thanks for helping out so soon. I was just about to start using AngularJS, and I prefer AngularDart, and Dart.

hpoit · 2019-07-31T23:54:20Z

There are no plans to release a stable version of angular_router, so that's your only option.

Thanks

DartBot assigned munificent Jun 5, 2015

DartBot added type-bug Incorrect behavior (everything from a crash to more subtle misbehavior) Priority-Medium labels Jun 5, 2015

dart-lang deleted a comment from chrisdlangton Mar 8, 2018

nex3 mentioned this issue Mar 14, 2018

(1.24.3) pub get spins forever on a mix of dart_style, http, and test #1825

Closed

This was referenced Mar 20, 2018

Our build_{*} packages need update notifications angulardart/angular#1095

Closed

Users updating packages can't understand output of pub angulardart/angular#1096

Closed

This was referenced Mar 26, 2018

"flutter packages get" throws no versions available but actually the version exists #1849

Closed

Activate the new version solver #1851

Merged

nex3 closed this as completed in #1851 Mar 27, 2018

matanlurey mentioned this issue May 30, 2018

Ensure up-to-date dependencies dart-lang/webdev#58

Closed

hpoit mentioned this issue Jul 31, 2019

Package angular has no versions that match >=5.0.0 <6.0.0 derived from: #2179

Closed

haxscramper mentioned this issue Jul 4, 2021

Nimble improvements nim-lang/RFCs#398

Open

Improve the version solver #912

Improve the version solver #912

Comments

DartBot commented Jun 5, 2015 • edited by nex3 Loading

DartBot commented Jun 5, 2015

DartBot commented Jun 5, 2015

DartBot commented Jun 5, 2015

DartBot commented Jun 5, 2015

DartBot commented Jun 5, 2015

DartBot commented Jun 5, 2015

DartBot commented Jun 5, 2015

DartBot commented Jun 5, 2015

DartBot commented Jun 5, 2015

DartBot commented Jun 5, 2015

DartBot commented Jun 5, 2015

DartBot commented Jun 5, 2015

DartBot commented Jun 5, 2015

DartBot commented Jun 5, 2015

DartBot commented Jun 5, 2015

DartBot commented Jun 5, 2015

DartBot commented Jun 5, 2015

DartBot commented Jun 5, 2015

DartBot commented Jun 5, 2015

kevmoo commented Mar 8, 2018

MikeMitterer commented Mar 20, 2018

kevmoo commented Mar 20, 2018

munificent commented Mar 28, 2018

thosakwe commented Mar 28, 2018

dnfield commented May 18, 2018 • edited Loading

zoechi commented May 18, 2018

dnfield commented May 18, 2018

zoechi commented May 18, 2018

dnfield commented May 19, 2018 • edited Loading

hpoit commented Jul 31, 2019

natebosch commented Jul 31, 2019

hpoit commented Jul 31, 2019

kevmoo commented Jul 31, 2019 via email

hpoit commented Jul 31, 2019 • edited Loading

hpoit commented Jul 31, 2019

kevmoo commented Jul 31, 2019 via email

hpoit commented Jul 31, 2019

hpoit commented Jul 31, 2019 • edited Loading

DartBot commented Jun 5, 2015 •

edited by nex3

Loading

dnfield commented May 18, 2018 •

edited

Loading

dnfield commented May 19, 2018 •

edited

Loading

hpoit commented Jul 31, 2019 •

edited

Loading

hpoit commented Jul 31, 2019 •

edited

Loading