fix(status): concurrent BasicStatus creation #1135

darkowlzz · 2017-09-07T11:02:48Z

This change adds concurrent BasicStatus creation, maintaining the order
of status output.

What does this do / why do we need it?

It reduces the time taken by status.
Before:

real    0m16.092s
user    0m0.408s
sys    0m0.394s

After:

real    0m1.777s
user    0m0.411s
sys    0m0.339s

What should your reviewer look out for in this PR?

Correctness of concurrency code.

Do you need help or clarification on anything?

~~Some help with the detected race condition.~~

Which issue(s) does this PR fix?

Related to #1008 , making status more verbose on failures.

jmank88 · 2017-09-07T12:04:17Z

Looks like you sorted out the concurrency.
Regarding error handling, would it still be worthwhile to output as much as possible in the event of an error, rather than logging errors and suppressing results? The work is still being done either way, and partial success can still be informative.

darkowlzz · 2017-09-07T14:34:10Z

Regarding error handling, would it still be worthwhile to output as much as possible in the event of an error, rather than logging errors and suppressing results? The work is still being done either way, and partial success can still be informative.

Yeah, that sounds reasonable.

jmank88 · 2017-09-07T15:10:28Z

cmd/dep/status.go

+			for err := range errorCh {
+				ctx.Err.Println(err.Error())
+			}
+			ctx.Err.Println()


I didn't really consider if it should still return an error. Could this look like a false positive if a user overlooks the logged errors? Should we return an error, and adjust the caller's error handling to print the buffer followed by a clear error message?

Even that would work. Like conditionally print the out buffer (with incomplete info) depending on the error type. Although, I'm still not sure if that's the best behavior.

Ah right, some errors are more severe than others.

What if errored items were still displayed inline, but indicated error somehow? (probably in addition to separate logs, since we don't want to cram messages into the table). Then there would be no mistaking the tail of the output for a false positive, with the errors interleaved.

Or perhaps simplest is best, what if the error output just came last?

darkowlzz · 2017-09-09T14:50:13Z

What if errored items were still displayed inline, but indicated error somehow? (probably in addition to separate logs, since we don't want to cram messages into the table). Then there would be no mistaking the tail of the output for a false positive, with the errors interleaved.
Or perhaps simplest is best, what if the error output just came last?

@jmank88 So I thought about it and even prototyped the idea.

Right now, if status is run offline, none of the network errors are logged. Or none of the errors are logged at all. I think it would be good to log the errors in verbose mode and let the user know that something failed in non-verbose mode.

How about running status offline and getting this?

$ dep status
PROJECT                             CONSTRAINT                                 VERSION                                    REVISION  LATEST   PKGS USED
github.com/Masterminds/semver       branch parse-constraints-with-dash-in-pre  branch parse-constraints-with-dash-in-pre  a93e51b   unknown  1
github.com/Masterminds/vcs          ^1.11.0                                    v1.11.1                                    3084677   unknown  1
github.com/armon/go-radix           *                                          branch master                              4239b77   unknown  1
github.com/boltdb/bolt              ^1.0.0                                     v1.3.1                                     2f1ce7a   unknown  1
github.com/go-yaml/yaml             branch v2                                  branch v2                                  cd8b52f   unknown  1
github.com/nightlyone/lockfile      *                                          branch master                              e83dc5e   unknown  1
github.com/pelletier/go-buffruneio  *                                          v0.2.0                                     c37440a   unknown  1
github.com/pelletier/go-toml        branch master                              branch master                              fe206ef   unknown  1
github.com/pkg/errors               ^0.8.0                                     v0.8.0                                     645ef00   unknown  1
github.com/sdboyer/constext         *                                          branch master                              836a144   unknown  1
golang.org/x/sys                    *                                          branch master                              bb24a47   unknown  1

Failed to get status of some projects. Run `dep status -v` to see the error messages.
failed to fetch updates

NOTE: the LATEST field is marked unknown, which is like an inline indication that something wrong happened. And a message at the bottom about how to view the errors.

I'll push the commit with these changes. Have a look, or even try it 🙂

e70cafa

jmank88 · 2017-09-09T15:17:36Z

I really like this in principle.

Could we link the final message with the unknown tag little more directly? Something like:

The statuses of %d projects are unknown due to errors. Rerun with `-v` to see details

darkowlzz · 2017-09-10T16:45:07Z

So I added the unknown count too.

The status of 2 projects are unknown due to errors. Rerun with `-v` flag to see details.
failed to fetch updates

And created 2 different error channels for different error types. Error due to failure in listing packages (errListPkgCh) and error due to failure in listing versions(errListVerCh).

Added 3 errors errFailedUpdate, errFailedListPkg and errMultipleFailures. Only errFailedUpdate would print the result with unknown data.

jmank88 · 2017-09-10T20:57:10Z

cmd/dep/status.go

+				statusError = errFailedUpdate
+			}
+
+			errorCount = len(errListVerCh)


Should this include len(errListPkgCh) as well?
Can probably just set it at initilization to len(errListVerCh) + len(errListPkgCh), or even just inline that expression down below.

ListPackages errors are less likely to happen(no network). And even if they do, it only affects Children of a BasicStatus only for dotOutput. And for such errors, we don't show the result with unknowns. We just fail.

Error count is only for the unknown status. This should be documented properly, which errors are counted.

jmank88 · 2017-09-10T20:59:53Z

cmd/dep/status.go

+				statusError = errMultipleFailures
+			} else {
+				statusError = errFailedUpdate
+			}


What about this instead:

if statusError == nil { statusError = errFailedUpdate } else { statuError = errMultipleFailures }

jmank88 · 2017-09-10T21:09:39Z

cmd/dep/status.go

+	count int
+}
+
+func runStatusAll(ctx *dep.Ctx, out outputter, p *dep.Project, sm gps.SourceManager) (bool, bool, errorStatus) {


I wonder if the errorStatus type is worth it. The return lines are sort of complex. Did you consider adding a return value instead? (bool, bool, int, error)

In either case, this function could probably use some documentation regarding what is returned. Actually, a case could be made for naming them. They are all declared in highly visible scope already anyways (bools wouldn't change), plus returns would be simpler.
Maybe (digestMismatch bool, hasMissingPkgs bool, errCnt int, err error) or something?

I thought of returning the error count, but then thought it would look weird. Returning error count when there's no error. If you think it's fine, we can return the count 😊

Yes, the return needs documentation. I'll add those variable names for the existing variables, will add error count later if we agree that we need it.

I thought of returning the error count, but then thought it would look weird. Returning error count when there's no error. If you think it's fine, we can return the count 😊

I don't think there is a problem with this in principle, as long as the documentation makes the purpose clear. The default assumption tends to be that a non-nil err means any other return values are irrelevant, but io.Writer, for example, always returns the bytes written.

jmank88 · 2017-09-11T12:14:26Z

cmd/dep/status.go

 	}

 	// While the network churns on ListVersions() requests, statically analyze
 	// code from the current project.
 	ptree, err := pkgtree.ListPackages(p.ResolvedAbsRoot, string(p.ImportRoot))
 	if err != nil {
-		return digestMismatch, hasMissingPkgs, errors.Wrapf(err, "analysis of local packages failed")
+		errStatus.err = errors.Wrapf(err, "analysis of local packages failed")
+		return digestMismatch, hasMissingPkgs, errStatus


With the named return vars, you can just do a bare return when the values are already set.

yep, but I think it's better to do explicit returns when the function is big, for clarity.

In that case, can we change most of these to return constant true or false for both vars? It looks like only the final hasMissingPkgs is actually dynamic.

changed and pushed!

jmank88

Looks good, just a few optional nits

jmank88 · 2017-09-11T21:45:26Z

cmd/dep/status.go

@@ -482,7 +596,7 @@ func runStatusAll(ctx *dep.Ctx, out outputter, p *dep.Project, sm gps.SourceMana
 			ctx.Err.Printf("\t%s: %s\n", fail.ex, fail.err.Error())
 		}

-		return digestMismatch, hasMissingPkgs, errors.New("address issues with undeducible import paths to get more status information")
+		return digestMismatch, hasMissingPkgs, 0, errors.New("address issues with undeducible import paths to get more status information")


This one could be true, false, 0...

jmank88 · 2017-09-11T21:45:54Z

cmd/dep/status.go

@@ -502,7 +616,7 @@ outer:
 	}
 	out.MissingFooter()

-	return digestMismatch, hasMissingPkgs, nil
+	return digestMismatch, hasMissingPkgs, 0, nil


This one could be true, hasMissingPkgs...

And then line 566 can go away too.

jmank88 · 2017-09-11T21:50:17Z

cmd/dep/status.go

-	digestMismatch, hasMissingPkgs, err := runStatusAll(ctx, out, p, sm)
-	if err != nil {
-		return err
+	digestMismatch, hasMissingPkgs, errCount, errStatus := runStatusAll(ctx, out, p, sm)


Maybe s/errStatus/err?

jmank88 · 2017-09-11T21:50:36Z

cmd/dep/status.go

-func runStatusAll(ctx *dep.Ctx, out outputter, p *dep.Project, sm gps.SourceManager) (bool, bool, error) {
-	var digestMismatch, hasMissingPkgs bool
-
+func runStatusAll(ctx *dep.Ctx, out outputter, p *dep.Project, sm gps.SourceManager) (digestMismatch bool, hasMissingPkgs bool, errCount int, errStatus error) {


s/errStatus/err here too?

darkowlzz · 2017-09-12T11:45:16Z

@jmank88 thanks for spending so much time on this 😊

jmank88 · 2017-09-12T12:03:09Z

cmd/dep/status.go

 		out.BasicFooter()

-		return digestMismatch, hasMissingPkgs, nil
+		return false, false, errCount, err


Can this be false, false, 0, nil?

Everything else looks good!

err... no, I don't think so. That's the only place where we are counting and returning error, for partial results.

Whoops thought I was looking at the final return.

sdboyer

approving, though there are some nits - please do fix those before merging 😄

sdboyer · 2017-09-21T13:50:10Z

cmd/dep/status.go

@@ -42,6 +43,17 @@ print an extended status output for each dependency of the project.
 Status returns exit code zero if all dependencies are in a "good state".
 `

+const (
+	shortRev = iota
+	longRev  = iota


nit: don't need second iota

sdboyer · 2017-09-21T13:52:30Z

cmd/dep/status.go

@@ -42,6 +43,17 @@ print an extended status output for each dependency of the project.
 Status returns exit code zero if all dependencies are in a "good state".
 `

+const (
+	shortRev = iota


prefer to include a type on iota assignments, for clarity/less magic

sdboyer · 2017-09-21T14:28:13Z

cmd/dep/status.go

-			case *dotOutput:
-				ptr, err := sm.ListPackages(proj.Ident(), proj.Version())
+			go func(proj gps.LockedProject) {
+				defer wg.Done()


i don't see multiple returns in this func, so no real need for the defer - just put it at the bottom.

sdboyer · 2017-09-21T14:30:17Z

cmd/dep/status.go

+					bs.Children = prm.FlattenFn(paths.IsStandardImportPath)
+				}
+
+				// Split apart the version from the lock into its constituent parts


nit: complete sentences (that have periods) are preferred, so add a period - yes, i realize this is my original comment 😄

This change adds concurrent BasicStatus creation, maintaining the order of status output.

Adds options to specify shortRev or longRev to getConsolidatedLatest().

Return explicit values instead of variables in runStatusAll().

googlebot added the cla: yes label Sep 7, 2017

darkowlzz changed the title ~~fix(status): concurrent BasicStatus creation~~ [WIP] fix(status): concurrent BasicStatus creation Sep 7, 2017

darkowlzz changed the title ~~[WIP] fix(status): concurrent BasicStatus creation~~ fix(status): concurrent BasicStatus creation Sep 7, 2017

darkowlzz force-pushed the concurrent-status branch from 5595c1a to 60e5c84 Compare September 7, 2017 14:37

jmank88 reviewed Sep 7, 2017

View reviewed changes

ibrasho added the area: status label Sep 9, 2017

darkowlzz force-pushed the concurrent-status branch from e70cafa to bdc7710 Compare September 10, 2017 16:34

jmank88 reviewed Sep 10, 2017

View reviewed changes

jmank88 reviewed Sep 11, 2017

View reviewed changes

jmank88 approved these changes Sep 11, 2017

View reviewed changes

darkowlzz force-pushed the concurrent-status branch from 48be3da to a69c951 Compare September 12, 2017 11:43

jmank88 reviewed Sep 12, 2017

View reviewed changes

sdboyer approved these changes Sep 21, 2017

View reviewed changes

darkowlzz added 7 commits September 21, 2017 23:27

fix(status): concurrent BasicStatus creation

6cec5a8

This change adds concurrent BasicStatus creation, maintaining the order of status output.

status: log errors and failure feedback

9de9063

status: add failure count for update failures

152a82a

fix(status): add shortRev and longRev for latest

cf9a971

Adds options to specify shortRev or longRev to getConsolidatedLatest().

status: add test for getConsolidatedLatest()

d1f2cdc

status: document runStatusAll() result

bce09f4

status: refactor return statements

88230aa

Return explicit values instead of variables in runStatusAll().

darkowlzz force-pushed the concurrent-status branch from a69c951 to 8d45be7 Compare September 21, 2017 18:17

status: cleanup

17bb4d6

darkowlzz force-pushed the concurrent-status branch from 8d45be7 to 17bb4d6 Compare September 21, 2017 18:21

darkowlzz merged commit 88b7ad9 into golang:master Sep 21, 2017

darkowlzz mentioned this pull request Nov 15, 2017

Make dep status -v verbose? #1008

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

fix(status): concurrent BasicStatus creation #1135

fix(status): concurrent BasicStatus creation #1135

darkowlzz commented Sep 7, 2017 •

edited

Loading

jmank88 commented Sep 7, 2017 •

edited

Loading

darkowlzz commented Sep 7, 2017

jmank88 Sep 7, 2017

darkowlzz Sep 8, 2017

jmank88 Sep 8, 2017

darkowlzz commented Sep 9, 2017 •

edited

Loading

jmank88 commented Sep 9, 2017

darkowlzz commented Sep 10, 2017

jmank88 Sep 10, 2017

darkowlzz Sep 11, 2017

jmank88 Sep 10, 2017

jmank88 Sep 10, 2017

darkowlzz Sep 11, 2017

jmank88 Sep 11, 2017

jmank88 Sep 11, 2017

darkowlzz Sep 11, 2017

jmank88 Sep 11, 2017

darkowlzz Sep 11, 2017

jmank88 left a comment

jmank88 Sep 11, 2017

jmank88 Sep 11, 2017

jmank88 Sep 11, 2017

jmank88 Sep 11, 2017

darkowlzz commented Sep 12, 2017

jmank88 Sep 12, 2017

darkowlzz Sep 12, 2017

jmank88 Sep 12, 2017

sdboyer left a comment

sdboyer Sep 21, 2017

sdboyer Sep 21, 2017

sdboyer Sep 21, 2017

sdboyer Sep 21, 2017

fix(status): concurrent BasicStatus creation #1135

fix(status): concurrent BasicStatus creation #1135

Conversation

darkowlzz commented Sep 7, 2017 • edited Loading

What does this do / why do we need it?

What should your reviewer look out for in this PR?

Do you need help or clarification on anything?

Which issue(s) does this PR fix?

jmank88 commented Sep 7, 2017 • edited Loading

darkowlzz commented Sep 7, 2017

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

darkowlzz commented Sep 9, 2017 • edited Loading

jmank88 commented Sep 9, 2017

darkowlzz commented Sep 10, 2017

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jmank88 left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

darkowlzz commented Sep 12, 2017

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

sdboyer left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

darkowlzz commented Sep 7, 2017 •

edited

Loading

jmank88 commented Sep 7, 2017 •

edited

Loading

darkowlzz commented Sep 9, 2017 •

edited

Loading