Dep status tree visualisation dot output #271

Rhymond · 2017-02-26T20:09:10Z

Couple of real life examples:

github.com/asciimoo/wuzz

github.com/wallix/awless

gopkg.in/kataras/iris.v6

BasicStatus was extended and Children field was added in order to get project child packages
-o was added to specify output file name (default: output.svg)
Default format is SVG

REF

…ph statuses

…oject_children

sdboyer

Great first steps, thanks for doing this! A few things to work through before we can merge.

sdboyer · 2017-03-03T19:45:13Z

cmd/dep/status.go

@@ -49,6 +55,7 @@ func (cmd *statusCommand) Register(fs *flag.FlagSet) {
 	fs.BoolVar(&cmd.detailed, "detailed", false, "report more detailed status")
 	fs.BoolVar(&cmd.json, "json", false, "output in JSON format")
 	fs.StringVar(&cmd.template, "f", "", "output in text/template format")
+	fs.StringVar(&cmd.output, "o", "output.svg", "output file")


Instead of -o, let's have -dot, then simply compute dot output as a string and print that to stdout. That'll be much more flexible for end-users.

If we are only printing dot output to stdout this parameter is not necessary

It is necessary - we're not printing dot output in the default case. The parameter is needed to indicate that dot output is desired. (So, a bool instead of a string)

-dot flag was in the code before: https://github.com/golang/dep/blob/master/cmd/dep/status.go#L52

This is a newly created flag for saving generated dot string into the file which is not necessary if we're stdout string now.

Hah, so it was. That was in our spec, but we never actually implemented it, so I figured it wasn't there. Didn't bother to check!

So then, yes, it's simply a matter of dropping the -o param you added.

sdboyer · 2017-03-03T19:46:22Z

cmd/dep/status.go

@@ -150,6 +158,123 @@ func (out *jsonOutput) MissingFooter() {
 	json.NewEncoder(out.w).Encode(out.missing)
 }

+type dotProject struct {


I think everything related to generating graphviz output could be safely put into its own, separate file - graphviz.go

sdboyer · 2017-03-03T19:49:18Z

cmd/dep/status.go

+		out.bsh[dp.project] = dp.hash()
+
+		// Create name boxes, and name them using hashes
+		// to avoid encoding name conflicts


nit: either use just a couple clarifying words, or a complete sentence (including a period) in comments, please

sdboyer · 2017-03-03T19:51:16Z

cmd/dep/status.go

+		fmt.Printf("%v", err)
+	}
+
+	defer syscall.Unlink(tf.Name())


This should all go away when switching to just printing to stdout instead of a tmp file, but for future - os.Remove is the call to make here.

sdboyer · 2017-03-03T19:54:09Z

cmd/dep/status.go

+			switch out.(type) {
+			case *dotOutput:
+				r := filepath.Join(p.AbsRoot, "vendor", string(proj.Ident().ProjectRoot))
+				ptr, err := gps.ListPackages(r, string(proj.Ident().ProjectRoot))


Instead of reading from vendor here, use a SourceManager's version of ListPackages(). That'll take care of ensuring the files are present, unmodified, and at the right version (which you'll have to pass in). At some point, it'll also get caching for free, too.

Also, side note - by placing this within the branch of the if where the memo matches, we create a potentially confusing situation for users where they ask for dot output, but just don't get if the memos don't match. I'm undecided if it's out of scope to handle in this PR, but if we don't, then we need a follow-up issue.

sdboyer · 2017-03-03T20:00:53Z

cmd/dep/status.go

+				}
+
+				prm, _ := ptr.ToReachMap(true, false, false, nil)
+				bs.Children = prm.Flatten(false)


There's some filtering that needs to be done here. Simply calling Flatten will get you the imports of ALL the packages in a ptree. We need only the subset of imports that come from the packages we're actually importing from a project.

The lock should contain the imported package list for each project, though, so this shouldn't be that bad to do.

@sdboyer we are doing some sort of "filtering".
https://github.com/golang/dep/pull/271/files#diff-0d223b461ecb49bf384a2c75a3893558R51
When we are generating Graphviz nodes we are only working with packages which are presented in the lock in this way we're excluding other packages.

sdboyer · 2017-03-09T01:44:39Z

bump

Rhymond · 2017-03-10T09:39:08Z

Hi @sdboyer,

Sorry for late response and Thanks for review.
I'm going to make changes and update PR this weekend.

googlebot · 2017-03-10T15:08:04Z

We found a Contributor License Agreement for you (the sender of this pull request), but were unable to find agreements for the commit author(s). If you authored these, maybe you used a different email address in the git commits than was used to sign the CLA (login here to double check)? If these were authored by someone else, then they will need to sign a CLA as well, and confirm that they're okay with these being contributed to Google.

sdboyer · 2017-03-10T15:12:47Z

Please rebase instead of merging when catching up with the mainline.

googlebot · 2017-03-10T16:30:23Z

CLAs look good, thanks!

sdboyer · 2017-03-10T16:35:00Z

(And please rewrite the history to remove that merge commit, 4275dd0, which should be unnecessary post-rebase)

freeformz · 2017-03-11T20:23:03Z

@Rhymond let me know if I can help.

Rhymond · 2017-03-14T10:48:56Z

@sdboyer merge commit was removed

sdboyer

Sorry for slow review, very busy week. Almost there, I think - just one bug to fix!

sdboyer · 2017-03-17T13:35:07Z

cmd/dep/graphviz.go

+	for _, dp := range g.ps {
+		for _, bsc := range dp.children {
+			for pr, hsh := range g.h {
+				if strings.HasPrefix(bsc, pr) {


There's a small bug here. We need to check not only that there's a prefix match, but that the next character in the bsc is a slash. This comes up a lot in this problem space; there's an (unexported) helper for it in gps. It'd probably be fine to just copy the func over to here - though if you do, please adjust comments accordingly.

Thanks!
Is status.go correct place for isPathPrefixOrEqual or it should be in graphviz.go?

Let's keep it as a helper just in graphviz.go for now. If/when we see wider use for it, we can move it into a more general location.

sdboyer · 2017-03-17T13:37:45Z

Oh, also looks like Travis is complaining about some gofmt issues. Please make sure you gomft before committing!

…ng path tree

Rhymond · 2017-03-21T10:19:51Z

Fixed gofmt issues and helper were moved to dep.
Could you please confirm that comment is correct. Otherwise, please help with corrections. Thanks!

sdboyer

Couple nits, rearrangements.

Also, I think we need some tests for this. It'd be fine to limit the tests just to the graphviz type - no need to mock up a real project for it to analyze.

sdboyer · 2017-03-21T13:37:46Z

cmd/dep/graphviz.go

@@ -91,3 +91,24 @@ func (dp gvnode) label() string {

 	return strings.Join(label, "\n")
 }
+
+// Ensure that the literal string prefix is a path tree match and


Even though this isn't exported, it should follow the Go standard of having the comment block begin with the func name.

sdboyer · 2017-03-21T13:38:03Z

cmd/dep/graphviz.go

+// Verify that either the input is the same length as the match (in which
+// case we know they're equal), or that the next character is a "/". (Import
+// paths are defined to always use "/", not the OS-specific path separator.)
+func isPathPrefixOrEqual(pre, path string) bool {


Now that I'm looking at this here, I realize the change in parameter order is terrible. Let's flip them so that it's the same as strings.HasPrefix() - string first, prefix second.

sdboyer · 2017-03-21T13:39:03Z

cmd/dep/graphviz.go

@@ -48,7 +48,7 @@ func (g graphviz) output() bytes.Buffer {
 	for _, dp := range g.ps {
 		for _, bsc := range dp.children {
 			for pr, hsh := range g.h {
-				if strings.HasPrefix(bsc, pr) {
+				if strings.HasPrefix(bsc, pr) && isPathPrefixOrEqual(pr, bsc) {


I was imagining this as a standalone implementation - move the strings.HasPrefix() call inside isPathPrefixOrEqual, rather than requiring two calls out here.

sdboyer · 2017-03-21T13:39:30Z

cmd/dep/graphviz.go

+		return false
+	}
+
+	// we assume something else (a trie) has done equality check up to the point


This comment should go away when strings.HasPrefix() moves into the func.

Rhymond · 2017-03-27T19:44:50Z

Hi, sorry for late response.
Will do changes this week

Removed strings.HasPrefix() usage

Rhymond · 2017-04-05T20:45:34Z

Hi @sdboyer,
I made tests and changed .HasPrefix, could you please check if it's alright?

sdboyer

just two tiny things 😄

sdboyer · 2017-04-07T03:38:42Z

cmd/dep/graphviz.go

+// path separator.)
+func isPathPrefix(path, pre string) bool {
+	pathlen, prflen := len(path), len(pre)
+	if pathlen < prflen || path[0:prflen] != pre || pathlen == prflen+1 {


That last check, pathlen == prflen+1, was useful because the use pattern in gps guaranteed that trailing slashes were not a possibility on the path input. Does the same apply here? If not, we should drop that check.

Yes you're right, it's unnecessary check, because there is no possibilities to have trailing slashes in manifest either in imports.

sdboyer · 2017-04-07T03:39:10Z

cmd/dep/graphviz_test.go

+	defer h.Cleanup()
+
+	b := g.output()
+	expected := h.GetTestFileString("graphviz/empty.dot")


nit: use want instead of expected

Remove unnecessary check IsPathPrefix()

freeformz · 2017-04-10T19:12:13Z

looks like it needs a rebase.

sdboyer · 2017-04-13T05:22:55Z

OK, I think we're finally ready to merge this in. Thanks so much!

Dep status tree visualisation dot output

Rhymond added 5 commits February 22, 2017 21:13

Children field in BasicStatus struct to store child data for tree/gra…

a7c6fa8

…ph statuses

Merge branch 'master' of https://github.com/golang/dep into status_pr…

1aa5000

…oject_children

Create graphiv (dot) package

a70afb5

Whitespace

80d8fc2

Merge branch 'master' of https://github.com/golang/dep into dot_output

08e23ec

googlebot added the cla: yes label Feb 26, 2017

Rhymond added 2 commits February 26, 2017 20:24

remove not used errorf

408c3aa

go imports

3ff1f50

sdboyer suggested changes Mar 3, 2017

View reviewed changes

sdboyer mentioned this pull request Mar 7, 2017

Parameterization: OS/Arch/build tags #291

Closed

googlebot added cla: no and removed cla: yes labels Mar 10, 2017

Rhymond force-pushed the dot_output branch from 272b812 to a8d5000 Compare March 10, 2017 16:30

googlebot added cla: yes and removed cla: no labels Mar 10, 2017

Rhymond added 2 commits March 12, 2017 12:38

Graphviz code moved from status.go to graphviz.go

5d3174f

Remove incorect comment

20fcfef

Rhymond force-pushed the dot_output branch from a8d5000 to 20fcfef Compare March 12, 2017 12:40

sdboyer suggested changes Mar 17, 2017

View reviewed changes

Transfer and use isPathPrefixOrEqual for checking if string is matchi…

5427779

…ng path tree

Rhymond force-pushed the dot_output branch from a9b959a to 5427779 Compare March 20, 2017 20:43

sdboyer suggested changes Mar 21, 2017

View reviewed changes

sdboyer mentioned this pull request Apr 1, 2017

Deprecated filepath.HasPrefix #296

Closed

Rhymond added 2 commits April 1, 2017 16:44

Merge branch 'master' of https://github.com/golang/dep into dot_output

70d8108

Test cases for graphviz

1a64b30

Removed strings.HasPrefix() usage

sdboyer reviewed Apr 7, 2017

View reviewed changes

Rename expected to want

b519dfa

Remove unnecessary check IsPathPrefix()

Change gps -> pkgtree

7fa0203

Rhymond force-pushed the dot_output branch from d74d1cb to 7fa0203 Compare April 10, 2017 20:28

sdboyer approved these changes Apr 13, 2017

View reviewed changes

sdboyer merged commit e4f1f3e into golang:master Apr 13, 2017

ibrasho pushed a commit to ibrasho-forks/dep that referenced this pull request May 10, 2017

Merge pull request golang#271 from Rhymond/dot_output

e3d1c4b

Dep status tree visualisation dot output

Dep status tree visualisation dot output #271

Dep status tree visualisation dot output #271

Conversation

Rhymond commented Feb 26, 2017

sdboyer left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Rhymond Mar 12, 2017 • edited Loading

Choose a reason for hiding this comment

sdboyer commented Mar 9, 2017

Rhymond commented Mar 10, 2017

googlebot commented Mar 10, 2017

sdboyer commented Mar 10, 2017

googlebot commented Mar 10, 2017

sdboyer commented Mar 10, 2017

freeformz commented Mar 11, 2017

Rhymond commented Mar 14, 2017

sdboyer left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

sdboyer commented Mar 17, 2017

Rhymond commented Mar 21, 2017

sdboyer left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Rhymond commented Mar 27, 2017

Rhymond commented Apr 5, 2017

sdboyer left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

freeformz commented Apr 10, 2017

sdboyer commented Apr 13, 2017

Rhymond Mar 12, 2017 •

edited

Loading