proposal: x/tools/cmd/godoc: add support for sections

[dsnet]

Problem Statement

As packages grow organically over time, their API surface often expands to the point where it is hard to understand how to use the package when all functions and types are shown as a vast alphabetically ordered list. Aside from reading examples (and other 3rd party tutorials), it becomes nearly impossible figuring out how to use a package by the godoc alone.

For example, it is not clear from the godoc of the proto package that the core set of functions are Marshal, Unmarshal, Equal, Clone, and Merge apart from reading the example. The vast majority of other functions are for more advanced (and legacy) functionality that should be categorized separately from the core functionality.

I propose that godoc provides support for user-defined sections.

Proposed Solution

(Approach presented here is Solution A)

It is observed that Go code is often written in such a way that declarations that would be grouped together under a section are already located in proximity of each other within the source code. Assuming that most sections follow this pattern, I propose that a special Section: marker in a top-level comment be used to signify the start of a section. All const, var, func, and type declarations below that marker will be considered part of that section. A section will not cover methods since those are already implicitly grouped under the receiver type.

The coverage of the section group extends either until:

• The end of the source file.
• The next occurrence of a section marker.
• The next occurrence of a special End section. marker, which must be a top-level comment with a blank line above and below it. As a matter of style, it will be discouraged practice to use the end marker unless necessary.

The section marker syntax allows for a required title and an optional description:

// Section: My wonderful title
//
// Let me tell you more details about my wonderful section
// in multiple lines of exciting text.

The title is a string that follows the Section: magic string and must be one line. The title may be any arbitrary string, but white space will be collapsed together. The description is optional and comprises of zero or more paragraphs (similar to package documentation) and must be preceded by a blank line.

For example, the proto package contains a set of helper functions that are only used when dealing with proto2 messages. In the source code, they are already co-located near each other with a comment that looks very similar to a section header. This portion could be modified to be more presentable in the public documentation:

// Section: Basic type helpers
//
// The proto2 format uses pointers for optional basic types (ints, uints, floats, and strings)
// It uses the the nil value to signal that they were are not present in the marshaled message.
// The proto package provides some helpers to aid in allocating values for these types.
//
// These helpers are not necessary for proto3 messages.

All of the helper functions (Float32, Float64, Int, Int32, Int64, String, Uint32, Uint64) will be grouped together under this section until the next top-level end marker:

// End section.

Sections without source locality

The observation that related declarations are often close together in the source does not always hold true. There are certain rare use cases of sections where the related declarations are not necessarily close by (or even in the same source file).

To solve this use case, we permit multiple section headers that have the same title. All declarations under sections with the same title will be considered to be under the same section. Only one of those section headers may have a description. This feature, combined with the end marker, can be used to indicate which declarations in separate files belong in the same section.

For example, the proto package has several types and functions that are only used by generated code and should never be called by user code. These types and functions live in different files. A section can be used to clearly identify these types and functions that are of no interest to the user by add a section in lib.go:

// Section: Internal implementation
//
// These types and functions are only used by generated .pb.go files and
// should never be called directly by user code.
// The compatibility agreement does not cover any of these declarations.

Elsewhere in message_set.go:

+ // Section: Internal implementation

  // RegisterMessageSetType is called from the generated code.
  func RegisterMessageSetType(m Message, fieldNum int32, name string) { ... }

+ // End section.

Elsewhere in properties.go:

+ // Section: Internal implementation

  // RegisterFile is called from generated code and maps from the
  // full file name of a .proto file to its compressed FileDescriptorProto.
  func RegisterFile(filename string, fileDescriptor []byte) { ... }

+ // End section.

How is godoc rendered?

All const, var, func, and type declarations that do not have a section will be shown at the top of the index as they are currently shown. Methods will always be grouped under their receiver type. The "core functionality" of a package should be a small set of declarations that do not have a parent section so that they appear first in the index.

For each section, the title will be printed and then the section's declarations will be shown. The list of sections are ordered alphabetically and the list of declarations are also listed alphabetically in the same way that they are shown today.

See the attached images for an example of what the proto package could look like.

• GoDoc before
• GoDoc after

Questions:

Q: Will there be support for arbitrary ordering of sections?

No. Since sections are ordered alphabetically, it is trivial for the ordering to be defined by prepending the section title with some prefix (e.g., the section number). It is unlikely that the number of sections will exceed 10, but if it does, a leading 0 can be prepended to the prior 9 sections. People are welcome to design their own ordering system (or carefully choose titles such that they appear in the desired order).

Q: Will there be support for sub-sections?

No. The number of indentations needed to visually show nested sections would make godoc too unreadable. Again, since the sections are ordered alphabetically, a prefix (e.g., 3.1.) can be used to create the concept of a "sub-section".

Q: What about other approaches to describe a section?

I looked other approaches of describing what was in a section and evaluated possible designs. They are described below.

Alternative Solution B (File-based locality)

(Idea by @josharian)

A variation on the source locality idea is to restrict it such that the section header can only be applied at most once at the top of the source file. The same section title may be present on multiple files. This approach avoids the need for a End section. marker since sections effectively span the entire file.

Alternative Solution C (Forward References)

An alternative to defining sections by source locality is to use forward references. In this approach, each section header contains a list of all the consts, vars, funcs, and types that belong in that section.

  // Section: Basic type helpers
  //
  // The proto2 format uses pointers for optional basic types (ints, uints, floats, and strings)
  // It uses the the nil value to signal that they were are not present in the marshaled message.
  // The proto package provides some helpers to aid in allocating values for these types.
  //
  // These helpers are not necessary for proto3 messages.
+ //
+ // References:
+ //  Float32
+ //  Float64
+ //  Int
+ //  Int32
+ //  Int64
+ //  String
+ //  Uint32
+ //  Uint64

Alternative Solution D (Reverse References)

An alternative to defining sections by source locality is to use reverse references. In this approach, the comment for each const, var, func, and type declaration contains a reference to the parent section it belongs in.

  // Int64 is a helper routine that allocates a new int64 value to store v and returns a pointer to it.
+ //
+ // Section: Basic type helpers
  func Int64(v int64) *int64 { ... }

Summary

A summary of the possible solutions:

	Source Locality (A)	File locality (B)	Forward references (C)	Reverse references (D)
Provides sections to godoc	✅	✅	✅	✅
Similar to how people naturally document Go code	✅ (1)	✅ (1)	❌	❌
New syntax does not negatively affect older doc-like tools	✅	✅	✅	❌ (2)
Using sections requires no movement of actual code	✅	❌ (3)	✅	✅
Renaming a section is trivial	⚠️ (4)	⚠️ (4)	✅	❌
Renaming or deleting a declaration is trivial	✅	✅	❌ (5)	✅
Easy to associate declarations with a section	✅	✅	❌ (6)	⚠️ (6)
Avoids End section. marker	❌ (7)	✅	✅	✅

1. (A, B) In looking at examples of what could use sections, it was common to find a comment block that was very similar to a section header. Unfortunately, the beautiful prose written is not visible on the godoc.

2. (D) The Section: Foo title string will be visible for doc-like tools with no understanding of sections.

3. (B) Since sections are defined on a per-file basis, it may be necessary to move code around, which muddle the code history.

4. (A, B) In the common case, there is only one section title and renaming is trivial. In rarer cases the same title may be used in multiple places and will need to be updated together.

5. (C) It is easy for the forward references to get out of sync when declarations are renamed or deleted. This is trivially checked by the lint tool.

6. (C) Any approach using references needs to provide that reference information somewhere and specifying these references can be cumbersome.

   For forward references, in the event that two section headers references the same declaration, which section should that declaration be in? Also, if there are many identifiers (e.g., lots of constants), do all identifiers have to be listed in the section header? The problem gets worse when different section headers refer to different identifiers within the same declaration block. Should the block be split apart to be in different sections?

   For backward references, a reference is still needed, but it avoids the problem with block declarations since the parent section is applied on the entire block.

7. (A) The end section marker is not always needed since the end of a file or start of a section ends the current section's scope. Forgetting the end section marker can accidentally add more declarations to a section than intended. Getting this wrong is obvious when viewing the godoc.

I propose that godoc add support for user-defined sections by source locality (Solution A) as opposed to other approaches (using forward references or reverse references). I believe that Go documentation is often clean and palatable when the package is small, but quickly becomes overwhelming as the number of exported declarations in the package grows. User defined documentation sections solves this problem and encourages clean documentations at scale.

Related proposals:

• #18130: support gradual code repair while moving a type between packages

It may be the case that when a package accrues many types over time that they should to be broken off into their own seperate package. However, support for movement of types does not change the fact that these types still exist in the original package for compatibility reasons. If support for #18130 happens, that is more reason that we should support sections so that documentation can clearly identify that a group of types have moved and provide details about the move.

• #17056: godoc: hide deprecated things by default

As a possible alternative to #17056, a section could be used to contain all deprecated items under it. However, sections would not be able to group deprecated methods or fields. The existence of both features in godoc do not hamper the other, but could be used powerfully augment each other. You can imagine allowing godoc to hide an entire section with the Deprecated: magic string in the section description.

[dsnet]

\cc @neild @light @bcmills @bradfitz @LMMilewski

[minux]

Should we just split huge packages or have better introduction in package docs and package level example instead?

[dsnet]

In retrospect, some of these packages should be split apart and doing so will definitely help the situation. The reality is, many packages are currently bloated and are unlikely to shed API surface anytime soon due to compatibility reasons.

1. Even if we start gradually moving types using aliases (or whatever the solution ends up being), there is still a reference in the package. Sections help categorize and document the move.
2. Even after some cruft is remove from a package, there are usually still a number of distinct sections that still belong together in a package, and it makes sense to document them as such.
3. Better introductions help, but I believe are still insufficient. Some internal packages at Google have ridiculously long introductions, and they don't really help make the package docs be any more readable. IMHO, most of that introduction should be split into the relevant sections that they describe.
4. Examples help, but I believe are still insufficient.

[neild]

+1 for some form of sections in godoc.

Splitting large packages can make sense, but it's very difficult to do after the fact and a maze of highly interdependent small packages is not necessarily any clearer than a single large one. Even small packages can benefit from sections, too; for example, database/sql is not particularly large, but grouping the Null* types into a single section might provide a small boost in readability.

[cznic]

Please keep godoc as simple as it is. It's a great value, though some would realize that only after it's gone.

Projects needing more structured documentation can easily link to any of the uncountable formats available. For example guru or go/types do that.

[bcmills]

It appears that the godoc on golang.org already supports sections to some extent. For example, there are now heading links to the Examples section of the testing package.

Another option might be to expand godoc to accept package comments from multiple source files. The package comment(s) in source files without any other declarations would become the top-level "Overview", and the package comment in each other source file would be the section header for the declarations in that file. Declarations in files without their own package comment would be grouped into the main section with the overview.

[bcmills]

For the example of the proto package in particular: I maintain that it would be cleaner to split that package into two (or more) distinct packages: one with only the user-facing API, and another with the pseudo-internal hooks for generated proto packages.

(I could envision even-finer-grained splits, too: one for the proto2 pointer-to-primitive helper functions, perhaps one for the extension API, etc.)

[dsnet]

It appears that the godoc on golang.org already supports sections to some extent.

I contend that the most useful feature of sections that I want to see is the ability to group a set of const, var, func, and type declarations together. Right now, all we have is the ability to pretty print some header title.

I maintain that it would be cleaner to split that package into two (or more) distinct packages.

Absolutely agree the pseudo-internal hooks should have been a different package. I believe the exported stuff to help with custom implementations of Message should also be split out. But those pointer-to-primitive functions are so pervasively used (over 200k usages!) that it's pretty annoying needing to import both a proto package and proto2 package. The point of sections is to group those functions together without creating a tiny package just for them.

---

To organize the pointer-to-primitive helpers, you could do something similar to binary.LittleEndian, but I find that method a hack and doesn't always work. I also find it dirty altering the implementation to fit the godoc, rather than the other way around.

Even then, binary.ByteOrder could still benefit from sections. The source code has ByteOrder, LittleEndian, BigEndian all located together in the source, but show up on opposite sides of the godoc.

[josharian]

I contend that the most useful feature of sections that I want to see is the ability to group a set of const, var, func, and type declarations together.

Thinking out loud...

What if that grouping was determined, perhaps optionally, by source file? Code is frequently already organized this way, with some exceptions, like autogenerated files. And if feels less fragile than relying on any ordering within a file. Perhaps if you still want section headers and more control, there could be zero or one section header per file, which applied to all code in that file, with identical section headers being coalesced.

[bcmills]

But those pointer-to-primitive functions are so pervasively used (over 200k usages!) that it's probably pretty annoying needing to import both a proto package and proto2 package. The point of sections is to group those functions together without creating a tiny package just for them.

1. The pointer-to-primitive functions aren't needed for proto3 messages, so only a subset of users of the proto package will actually use them.
2. goimports makes having lots of tiny packages fairly benign, as long as there is some reasonable heuristic to figure out which item is in which package. (A counterexample: io vs. ioutil.)

The net and net/http packages in the standard library might be other good examples to consider. If you had support for sections, how would you want to split those out? (And does that split match the division of files today?)

[neild]

I disagree that lots of tiny packages are benign, even with goimports. The user still needs to remember which package contains which function; easy when you have one or two packages, less so when you have a dozen.

Obviously, stuffing unrelated functionality into a single package isn't right either. It's a balance. (For the proto package, I'd say the pointer-to-primitive functions should stay--they're small, cheap, and anyone who uses proto2 will want them--but obviously all the internal hooks should get pulled out. Something to work on once type aliases or the moral equivalent become available.)

[dsnet]

What if that grouping was determined, perhaps optionally, by source file?

In the example I posted, when I was organizing the proto package, 5 of the sections were pretty much by file. The last section, which was about "pseudo-internal stuff" was spanning several files. Arguably, those functions and types could be all moved to a single internal.go file and be made one section. Thus, grouping by file-alone can lead to some unfortunate of code-churn.

That said, if we end up going with grouping by file alone, I'd be fine with that approach.

[neild]

As a specific example from the net package, it contains eight error types:

DNSConfigError
DNSError
Error
InvalidAddrError
OpError
ParseError
UnknownNetworkError

Those are currently spread throughout the package index; collecting them into a single section would improve clarity.

These type names could be rewritten to use a common prefix (e.g., ErrParse) so that they sort together, but using less-natural names to work around godoc doesn't seem like a good situation.