go/doc: add fields to Example struct to indicate type/method/suffix

[dsnet]

The doc package currently provides:

func Examples(files ...*ast.File) []*Example

Given a set of source files, this only parses out the examples as a flat list.

However, the testing package documents a syntax of how to associate examples with:

• A package (e.g., Example or Example_suffix)
• A function (e.g., ExampleF or ExampleF_suffix)
• A type (e.g., ExampleT or ExampleT_suffix)
• A method (e.g., ExampleT_M or ExampleT_M_suffix)

As the number of tools that provide godoc-like functionality increases, the manual logic to parse these example names and associate it with the appropriate top-level declaration is somewhat tedious and easy to get wrong.

Given that this syntax is documented, I propose that the doc package provide first-class support for doing this association. The additional functionality is as follows:

• New uses the ast.Package.Files input to produce an internal []Example.
• An Examples []Example field is added to the Package, Type, and Func types. They are populated as appropriate according to what the relevant examples are associated with.
• Examples that do not follow the example syntax or reference an non-existent declaration could be either ignored or just associated with the entire package.

\cc @griesemer

[dsnet]

Some evidence that this is an issue: the two most common godoc implementations don't handle this the same way:

• golang/tools/godoc: The stripExampleSuffix function assumes that the suffix may not contain another '_' character.
• golang/gddo/doc: The builder.getExamples method assumes that so long as there is a prefix match and the suffix starts with an uppercase, it doesn't care what the rest of the suffix looks like.

Arguably the gddo approach is more correct. The point is that the divergence in behavior is a why the standard library should do this association for the user.

[gopherbot]

Change https://golang.org/cl/94855 mentions this issue: go/doc: add support for classifying Examples

[rsc]

Why do you want to modify go/doc? Is it just so godoc and gddo can share something?
Also we need not change the accessor. We could provide a helper that just takes the string name and returns info about the parsed form of that string name.

[dsnet]

Why do you want to modify go/doc?

To have a canonical implementation of the syntax laid on in the testing package. As it currently stands, every godoc-like tool does something slightly different.

Is it just so godoc and gddo can share something?

and the additional godoc-like tools that are springing up.

Also we need not change the accessor. We could provide a helper that just takes the string name and returns info about the parsed form of that string name.

I'm not sure what you mean by that. Are you saying we shouldn't alter the doc.Package struct? Modifying the struct makes it easier to pass doc.Package to html/template or text/template. IMO, the helper functions are harder to work with.

[rsc]

I don't think we expect go/doc structs in their current form to be guaranteed to be helpful for any use of template.

I do think that something like

func ParseExampleName(name string) ExampleInfo

// For "ExampleFoo_M_suffix"
type ExampleInfo struct {
    Name string // "Foo"
    Method string // "M"
    Suffix string // "suffix"
}

could be useful for godoc and cmd/doc and other things to share. That would still give you a canonical implementation of the syntax. Or add a few fields to the doc.Example struct:

Name string // "ExampleFoo_M_suffix" (already in struct)
Base string // "Foo"
Method string // "M"
Suffix string // "suffix"

?

[dsnet]

That is not sufficient. You need a list of top-level type, function, and method identifiers to disambiguate whether Foo_Bar_baz refers to a method named Foo.Bar or a type named Foo_Bar. It is only a stylistic recommendation that most top-level Go identifiers do not have _ in their name. Thus, classifying by _ delimiter alone is insufficient. The gddo tool does this more correctly except for the worst-case situation where both Foo.Bar and Foo_Bar exists; however that case is rare and impossible to reconcile.

Another ambiguity is whether Foo_bar refers to Foo with a suffix of bar or an actual type named Foo_bar. If we were relying on purely lexical parsing, I would not recommend us adding any logic to go/doc at all since I would still perform this on my own (as it is more correct).

[rsc]

The ambiguity that @dsnet mentions strikes out the ParseExample func, but it seems like the doc.Example field additions are still fine, since they're returned as part of something that looks at the whole package and knows its types and methods.

[rsc]

OK, it sounds like we should add the fields to the Example struct (the second half of my message from March 5).

And maybe also we should provide the Example slice in the overall Package struct instead of having a second call, although there may be some concerns around opening files during doc.New that were previously not opened (*_test.go).

@dsnet and @griesemer, if you both agree about adding the fields to the Example struct, please mark this proposal-accepted.

[rsc]

Ping @dsnet and @griesemer, waiting on your acceptance.

[dsnet]

Spoke to @griesemer, adding fields to the doc.Example struct and extracting top-level declarations from the set of input *ast.File SGTM.

This will require some adjustments to how doc.Examples is called as it seems most usages just pass into only the test files (e.g., here).

And maybe also we should provide the Example slice in the overall Package struct instead of having a second call, although there may be some concerns around opening files during doc.New that were previously not opened (*_test.go).

I support this, but can be a future addition.