Implement new spec for init #470

darkowlzz · 2017-04-27T19:39:23Z

Adds assuming project revision to be master when VersionInWorkspace
fails for not on disk projects. Failed projects were being skipped from
manifest before this change.
Adds running solver once at init. This ensures creating an initial
lock with memo hash in lock file.

darkowlzz · 2017-04-27T19:55:25Z

I would like to discuss if this is a good solution.

After debugging init, found out that the manifest variable, m, created in init refers to constraints of ProjectData here. And getProjectData() was omitting the projects from manifest when it fails to get version of projects not on disk here.

Instead of omitting the projects, I thought assuming the remote project's version to be master would be better. This ensured that all the direct, not on disk, dependencies are added to manifest.

Also, enabled running the solver to ensure that a proper lock is generated and memo is not left empty after init is run.

Need feedback.

darkowlzz · 2017-04-27T20:10:05Z

Integration tests are failing due to change in the expected behavior/results. Will fix them once we decide to go ahead with this solution.

sdboyer · 2017-04-27T20:51:22Z

Instead of omitting the projects, I thought assuming the remote project's version to be master would be better. This ensured that all the direct, not on disk, dependencies are added to manifest.

There'd be a lot of pitfalls from this. Much better is to run a solve without constraints applied, then pick the versions to put back into the manifest. Please have a look at the WIP spec doc - it describes how such an algorithm should work for picking a constraint.

Because it's out of scope for this issue to change init's flags, it'd be best to just use that algorithm for constraint selection - but (as it is now) only for projects for which we haven't gleaned a constraint from searching GOPATH.

Also, enabled running the solver to ensure that a proper lock is generated and memo is not left empty after init is run.

Excellent 😄

darkowlzz · 2017-04-28T19:11:11Z

There'd be a lot of pitfalls from this.

oops! yeah, right. Not every project has the default branch named "master" 😅

darkowlzz · 2017-04-28T19:13:22Z

cmd/dep/init.go

+			if k == x.Ident().ProjectRoot {
+				m.Dependencies[k] = getProjectPropertiesFromVersion(x.Version())
+				break
+			}


Is there a better way to pick the constraints and add to manifests?

Not one that's wrapped up in a nice function, but there is something sorta analogous - I'll make more detailed comments down on the impl. But really, the algorithm you'll want to follow for this is bulleted in the spec doc. 😄

darkowlzz · 2017-04-28T19:37:01Z

Another attempt and a little bit closer, I guess :)

Removed false version assumption for not on disk projects. Instead, running solver to get the proper version and then picking the missing constraints of not on disk projects from the solution.

Since the lock from the previous solver run had missing constraints, running solver again with appropriate project version(manifest) to obtain a final lock. Sounds good?

Also, with the above changes, manifest, lock and vendor are generated properly, but the lock Memo has a little issue. Here's an example scenario.

My project has 2 direct dependencies:

github.com/darkowlzz/openurl (not on disk)
github.com/fsnotify/fsnotify (on disk)

When init is run, fsnotify gets added to manifest first because it's on disk, hence becomes the first element of manifest map. After running solver and obtaining version of openurl, it is added to manifest map.

{map[github.com/fsnotify/fsnotify:{ master} github.com/darkowlzz/openurl:{ master}] map[] [] []}

This manifest map is then passed to the 2nd solver run to obtain the final lock, which is then written to manifest and lock files.

Now, if I run status, it fails to show expected result (list of packages) because status runs solver and compares it's solution's lock Memo with the memo in the lock file, and they don't match. On checking the inputs to the solver in status, I found that manifest map value is

&{map[github.com/darkowlzz/openurl:{ master} github.com/fsnotify/fsnotify:{ master}] map[] [] []}

openurl first and fsnotify second. Due to this the generated memo hash differs.

Any idea how to handle this?

sdboyer

Great progress!

sdboyer · 2017-04-29T00:16:59Z

cmd/dep/init.go

+			if k == x.Ident().ProjectRoot {
+				m.Dependencies[k] = getProjectPropertiesFromVersion(x.Version())
+				break
+			}


Not one that's wrapped up in a nice function, but there is something sorta analogous - I'll make more detailed comments down on the impl. But really, the algorithm you'll want to follow for this is bulleted in the spec doc. 😄

sdboyer · 2017-04-29T00:23:44Z

cmd/dep/init.go

-			return err
-		}
-		l = dep.LockFromInterface(soln)
+	// Run solver again with appropriate constraint solutions from previous run


I don't think it's necessary to solve twice. It should be sufficient to gps.Prepare() a new solver with the returned solution and the updated manifest, then call solver.HashInputs() and use that as the Memo value instead of what was returned from the initial solve.

More details about how this hash works, and what it means, are here.

sdboyer · 2017-04-29T00:28:18Z

cmd/dep/init.go

+	case gps.IsBranch, gps.IsVersion, gps.IsRevision:
+		pp.Constraint = v
+	case gps.IsSemver:
+		c, _ := gps.NewSemverConstraint("^" + v.String())


Let's add a TODO here to note that this will need to change when #225 is ready.

sdboyer · 2017-04-29T00:38:01Z

cmd/dep/init.go

+// gps.ProjectProperties with Constraint value based on the version type.
+func getProjectPropertiesFromVersion(v gps.Version) gps.ProjectProperties {
+	pp := gps.ProjectProperties{}
+	switch v.Type() {


So, we actually need to do a little more work here up front. It's not a guarantee of the interface (yet - this may change), but gps has a concept of "version pairing" - basically, combining a revision with a branch or tag. This information is not encoded in the gps.Version.Type() method, so we need to tease it out first, because - per the spec - we never want to put revisions in Gopkg.toml, if we can avoid it.

There's a rough skeleton to follow here in status.go.

sdboyer · 2017-04-29T00:39:49Z

cmd/dep/init.go

+}
+
+// getSolverSolution runs gps solver and returns a solution.
+func getSolverSolution(root string, pkgT pkgtree.PackageTree, m *dep.Manifest, l *dep.Lock, sm *gps.SourceMgr) (gps.Solution, error) {


I don't object to having this abstraction in principle, but let's leave it for a later PR where it can be a dedicated refactor that affects the other commands as well.

sdboyer · 2017-04-29T00:40:43Z

cmd/dep/init.go

+func getProjectPropertiesFromVersion(v gps.Version) gps.ProjectProperties {
+	pp := gps.ProjectProperties{}
+	switch v.Type() {
+	case gps.IsBranch, gps.IsVersion, gps.IsRevision:


Per the spec & the above note about revisions, if the actual type is gps.IsRevision, then we just want to ignore it.

sdboyer · 2017-04-29T00:49:22Z

(for those things that weren't addressed inline...)

Another attempt and a little bit closer, I guess :)

Yep! 🥅

Removed false version assumption for not on disk projects. Instead, running solver to get the proper version and then picking the missing constraints of not on disk projects from the solution.

👍

Also, with the above changes, manifest, lock and vendor are generated properly, but the lock Memo has a little issue. Here's an example scenario.
...
openurl first and fsnotify second. Due to this the generated memo hash differs.

Ordering should not be an issue here. The input hasher is very, very careful about minimizing superfluous conflicts. It even removes irrelevant output (e.g. if a package is both imported and required, then the require value is not included in the hash, because it would be redundant and cause unnecessary hash conflicts). If gps' code is correct, it should be impossible for the caller to order inputs in a way that generate a superfluous conflict.

To get a better picture of why this conflict is happening, rather than printing the manifest itself, run gps.HashingInputsAsString() at various stages. That'll give a precise picture of what the problems are.

darkowlzz · 2017-04-29T21:23:55Z

Another attempt with all my understandings 😅

Stopped running solver twice, using gps.Prepare() to obtain memo.
Removed solver solution abstraction created in the previous commit, because we are running solver just once now.
Modified getProjectPropertiesFromVersion() to comply more with the manifest constraints population algo as per the spec.

Also, the issue with different memo got resolved. I used gps.HashingInputsAsString() to check the hashing inputs at init:

-CONSTRAINTS-
github.com/darkowlzz/openurl
b-master-r-673a04d14da017f965f0b97f87ab369ec4a1fd08
github.com/fsnotify/fsnotify
b-master-r-fd9ec7deca8bf46ecd2a795baaacf2b3a9be1197
-IMPORTS/REQS-
github.com/darkowlzz/openurl
github.com/fsnotify/fsnotify
-IGNORES-
-OVERRIDES-
-ANALYZER-
dep
1

While on running status, hashing inputs were:

-CONSTRAINTS-
github.com/darkowlzz/openurl
b-master
github.com/fsnotify/fsnotify
b-master
-IMPORTS/REQS-
github.com/darkowlzz/openurl
github.com/fsnotify/fsnotify
-IGNORES-
-OVERRIDES-
-ANALYZER-
dep
1

Revision value was causing the issue. With the modified constraint selections algo implementation, it got fixed.

sdboyer · 2017-04-30T00:12:39Z

(travis should be failing too, but we introduced a bug - i just opened #486 for that)

sdboyer

idk about limited understanding, this is looking pretty good to me 😄 next step is probably to move on to adapting the tests.

sdboyer · 2017-04-30T00:24:30Z

cmd/dep/init.go

@@ -209,6 +227,45 @@ func hasImportPathPrefix(s, prefix string) bool {
 	return strings.HasPrefix(s, prefix+"/")
 }

+// getVersionConstituents extracts version constituents
+func getVersionConstituents(v gps.Version) (version, revision string) {


This is actually really close to the VersionComponentStrings() func in gps. If we were to go this route, I think I'd rather see that function reused. But I think another route might be better, as I'll describe below...

sdboyer · 2017-04-30T00:28:38Z

cmd/dep/init.go

+	pp := gps.ProjectProperties{}
+
+	// constituent version and revision. Ignoring revison for manifest.
+	cv, _ := getVersionConstituents(v)


Rather than converting to and from strings, we can do it all with the types - in getProjectPropertiesFromVersion, we can do something like this:

switch tv := v.(type) { case gps.PairedVersion: v = tv.Unpair() case gps.Revision: return pp }

Now the v is exactly what we'll want to assign into pp.Constraint, unless we're in the semver case.

sdboyer · 2017-04-30T00:29:46Z

cmd/dep/init.go

+		pp.Constraint = gps.NewVersion(cv)
+	case gps.IsSemver:
+		// TODO: remove "^" when https://github.com/golang/dep/issues/225 is ready.
+		c, _ := gps.NewSemverConstraint("^" + cv)


Let's check the error here and panic if we get one. That shouldn't be possible - any valid plain semver version should be able to have the caret operand prepended to it - but if we somehow missed something, then program behavior is undefined; panicking is the safe thing to do.

sdboyer · 2017-04-30T00:31:31Z

Revision value was causing the issue. With the modified constraint selections algo implementation, it got fixed.

Ahh, of course! Also, awesome, really glad we're going to be able to knock that one out.

darkowlzz · 2017-04-30T15:38:32Z

cmd/dep/init_test.go

+	if outsemver.Constraint.String() != expectedSemver.String() {
+		t.Fatalf("Expected %q to be equal to %q", outsemver, expectedSemver)
+	}
+}


Can we do something about these tests? Especially the semver part. And if there's a way to improve the way they are being tested?

Yeah, the issue here is that semver.rangeConstraint contains a slice, which means it's not directly comparable, even though that slice is almost always empty, including in this case.

The simple solution here is to use reflect.DeepEqual() to compare them instead.

Apart from that, I think the tests are actually fine as-is; they focus in on the key issue, which is stripping out the underlying revision, and making sure that we default to the caret. The only ones I might add, just for completeness, is a duplicate set where there isn't a revision attached (the Version is actually a gps.UnpairedVersion). This case shouldn't occur in the wild, but it's to test for and we should know if we ever regress in handling it, for some reason.

darkowlzz · 2017-04-30T15:48:13Z

And we are almost there ☘️ 😁

Removed getVersionConstituents() and instead using a simplified inline implementation of the same.
Added error checking with panic for gps.NewSemverConstraint().
Added test for getProjectPropertiesFromVersion().
Fixed the failing integration tests due to new init behavior.

sdboyer

good progress!

sdboyer · 2017-05-01T02:56:36Z

cmd/dep/init_test.go

+func TestGetProjectPropertiesFromVersion(t *testing.T) {
+	cases := []struct {
+		version  gps.Version
+		expected gps.Version


supernit: prefer want to expected

sdboyer · 2017-05-01T02:57:36Z

cmd/dep/init_test.go

+	if outsemver.Constraint.String() != expectedSemver.String() {
+		t.Fatalf("Expected %q to be equal to %q", outsemver, expectedSemver)
+	}
+}


Yeah, the issue here is that semver.rangeConstraint contains a slice, which means it's not directly comparable, even though that slice is almost always empty, including in this case.

The simple solution here is to use reflect.DeepEqual() to compare them instead.

Apart from that, I think the tests are actually fine as-is; they focus in on the key issue, which is stripping out the underlying revision, and making sure that we default to the caret. The only ones I might add, just for completeness, is a duplicate set where there isn't a revision attached (the Version is actually a gps.UnpairedVersion). This case shouldn't occur in the wild, but it's to test for and we should know if we ever regress in handling it, for some reason.

- Adds assuming project revision to be master when VersionInWorkspace fails for not on disk projects. Failed projects were being skipped from manifest before this change. - Adds running solver once at init. This ensures creating an initial lock with memo hash in lock file.

- Adds running gps solver with project versions found on disk. - Picks constraints from the solver's solution and adds them to manifest projects with missing constraints. - Runs solver again to obtain final lock with appropriate project constraints.

- Avoid running solver twice and use gps.Prepare to get the final lock. - Improve manifest constraints population based on the algo in the new spec doc.

- Removes getVersionConstituents(), inline its simplified implementation. - Adds test for getProjectPropertiesFromVersion() function.

- Added some more test cases: version without any revision attached. - Fixed breaking ensure/pkg-errors/case1 integration test.

darkowlzz · 2017-05-01T12:09:51Z

Renamed expected to want in test cases.
Added test cases for versions without revision attached.
Fixed breaking ensure/pkg-errors/case1 integration test. init in an empty project (empty manifest) should also have a memo hash.

reflect.DeepEqual for comparing semvar constraints didn't work for some reason :(
For reflect.DeepEqual(outsemver.Constraint, wantSemver), got:

Expected semver to be ^1.0.0, got ^1.0.0

Ended up comparing their strings.

sdboyer · 2017-05-01T13:31:05Z

cmd/dep/init_test.go

+
+	// Test to have caret in semver version
+	outsemver := getProjectPropertiesFromVersion(gps.NewVersion("v1.0.0"))
+	wantSemver, _ := gps.NewSemverConstraint("^1.0.0")


The actual problem here is an annoying implementation detail - it should work without needing to fall back on string comparison if you use ^v1.0.0 instead of ^1.0.0. I believe this kind of issue is improved in newer versions of the semver lib, but we're stuck with what we've got for now.

sdboyer · 2017-05-01T13:33:28Z

Now all we need is to update the CLI helptext 😄

sdboyer · 2017-05-01T13:52:07Z

LGTM! I'll tweak the helptext a bit on my own, directly - no sense holding up the functionality for that 😄

…versioninworkspace Implement new spec for init

googlebot added the cla: yes label Apr 27, 2017

darkowlzz changed the title ~~Add constraints for not on disk projects.~~ [WIP] Add constraints for not on disk projects. Apr 27, 2017

darkowlzz commented Apr 28, 2017

View reviewed changes

sdboyer suggested changes Apr 29, 2017

View reviewed changes

darkowlzz force-pushed the 356-fix-init-solver-and-versioninworkspace branch from 92740cb to a4fd873 Compare April 29, 2017 21:05

sdboyer suggested changes Apr 30, 2017

View reviewed changes

darkowlzz force-pushed the 356-fix-init-solver-and-versioninworkspace branch from a4fd873 to 14c8cc6 Compare April 30, 2017 15:31

darkowlzz commented Apr 30, 2017

View reviewed changes

darkowlzz mentioned this pull request Apr 30, 2017

dep status is not useful if memo is out of date, not even a warning #161

Closed

sdboyer reviewed May 1, 2017

View reviewed changes

darkowlzz added 6 commits May 1, 2017 16:17

Fix issues with the new init algo implementation

b444466

- Avoid running solver twice and use gps.Prepare to get the final lock. - Improve manifest constraints population based on the algo in the new spec doc.

Nit fixes to init algo and add test

f467151

- Removes getVersionConstituents(), inline its simplified implementation. - Adds test for getProjectPropertiesFromVersion() function.

Fix failed integration tests due to new init algo

1e56bdd

Nit fixes to tests

0a0f777

- Added some more test cases: version without any revision attached. - Fixed breaking ensure/pkg-errors/case1 integration test.

darkowlzz force-pushed the 356-fix-init-solver-and-versioninworkspace branch from 14c8cc6 to 0a0f777 Compare May 1, 2017 11:51

darkowlzz changed the title ~~[WIP] Add constraints for not on disk projects.~~ Implement new spec for init May 1, 2017

darkowlzz mentioned this pull request May 1, 2017

Add -gopath flag to init #496

Closed

sdboyer reviewed May 1, 2017

View reviewed changes

Update init help text.

8ab8cf5

sdboyer approved these changes May 1, 2017

View reviewed changes

sdboyer merged commit aef755e into golang:master May 1, 2017

ibrasho pushed a commit to ibrasho-forks/dep that referenced this pull request May 10, 2017

Merge pull request golang#470 from darkowlzz/356-fix-init-solver-and-…

869a852

…versioninworkspace Implement new spec for init

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Implement new spec for init #470

Implement new spec for init #470

darkowlzz commented Apr 27, 2017

darkowlzz commented Apr 27, 2017 •

edited

Loading

darkowlzz commented Apr 27, 2017

sdboyer commented Apr 27, 2017

darkowlzz commented Apr 28, 2017 •

edited

Loading

darkowlzz Apr 28, 2017

sdboyer Apr 29, 2017

darkowlzz commented Apr 28, 2017

sdboyer left a comment

sdboyer Apr 29, 2017

sdboyer Apr 29, 2017

sdboyer Apr 29, 2017

sdboyer Apr 29, 2017

sdboyer Apr 29, 2017

sdboyer Apr 29, 2017

sdboyer commented Apr 29, 2017

darkowlzz commented Apr 29, 2017

sdboyer commented Apr 30, 2017

sdboyer left a comment

sdboyer Apr 30, 2017

sdboyer Apr 30, 2017

sdboyer Apr 30, 2017

sdboyer commented Apr 30, 2017

darkowlzz Apr 30, 2017 •

edited

Loading

sdboyer May 1, 2017

darkowlzz commented Apr 30, 2017 •

edited

Loading

sdboyer left a comment

sdboyer May 1, 2017

sdboyer May 1, 2017

darkowlzz commented May 1, 2017

sdboyer May 1, 2017

sdboyer commented May 1, 2017

sdboyer commented May 1, 2017

Implement new spec for init #470

Implement new spec for init #470

Conversation

darkowlzz commented Apr 27, 2017

darkowlzz commented Apr 27, 2017 • edited Loading

darkowlzz commented Apr 27, 2017

sdboyer commented Apr 27, 2017

darkowlzz commented Apr 28, 2017 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

darkowlzz commented Apr 28, 2017

sdboyer left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

sdboyer commented Apr 29, 2017

darkowlzz commented Apr 29, 2017

sdboyer commented Apr 30, 2017

sdboyer left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

sdboyer commented Apr 30, 2017

darkowlzz Apr 30, 2017 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

darkowlzz commented Apr 30, 2017 • edited Loading

sdboyer left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

darkowlzz commented May 1, 2017

Choose a reason for hiding this comment

sdboyer commented May 1, 2017

sdboyer commented May 1, 2017

darkowlzz commented Apr 27, 2017 •

edited

Loading

darkowlzz commented Apr 28, 2017 •

edited

Loading

darkowlzz Apr 30, 2017 •

edited

Loading

darkowlzz commented Apr 30, 2017 •

edited

Loading