go/ast, go/token: additions to support type parameters

[findleyr]

This proposal formally introduces the changes we’ve made to support parameterized functions and types in the go/ast and go/token packages. See the full write-up here:
https://go.googlesource.com/proposal/+/master/design/47781-parameterized-go-ast.md

There will be a separate issue for go/types, where type parameters add a much larger API surface.

We’ve gotten some experience with these new APIs over the last few months by using them in go/types. The additional MultiIndexExprIndexListExpr node type is the most significant change, but felt cleanest of the alternatives we considered.

Any feedback is appreciated.

CC @griesemer

Changelog:

• TParams fields were renamed to TypeParams
• MultiIndexExpr was renamed to IndexListExpr

[scott-cotton]

Thanks for the proposal.

Typo: "invalid or invalid" -> "valid or invalid"?

I think the general representation of optional arguments/parameter is ok.

Perhaps a better name can be found for MultiIndexExpr: BracketedExpr? (I do not think of type parameters or instantiations as "indices")

But, I don't quite understand why suppositions about the completeness of a set of node types is a concern in the stdlib but the same is not a concern about token: The dev.typeparams commit 7c5d7a4 introduced token.TILDE. Why does the same concern not hold there? I think that one way to handle these issues is documentation: we could state that for go/* stdlib libraries, all exported sets of constants and types can be extended to facilitate language changes: do not assume switches handle all (possibly future) cases.

Could you post a link to the issue for the type proposal here when you are ready?

[findleyr]

@scott-cotton thank you for the feedback.

Typo: "invalid or invalid" -> "valid or invalid"?

Fixed, thanks.

Perhaps a better name can be found for MultiIndexExpr: BracketedExpr? (I do not think of type parameters or instantiations as "indices")

I think an advantage of MultiIndexExpr is similarity with IndexExpr. If we could start over these would be the same node, but since we can't it's important to keep them associated.

But, I don't quite understand why suppositions about the completeness of a set of node types is a concern in the stdlib but the same is not a concern about token

You're right that we should extend the same concern to TILDE. I was more concerned about node completeness because I was already aware of places in the x/tools codebase that would panic when encountering an unknown node.

One additional argument for adding a new node is that the panic exists to be sure your type switch covers all syntax. In this context, it would actually be unhelpful to hide the change by packing the new syntax into existing nodes. Perhaps that's overly optimistic.

Could you post a link to the issue for the type proposal here when you are ready?

Yes, will do.

[findleyr]

@scott-cotton the go/types proposal is filed as #47916.

[mdempsky]

What's the rationale for adding TParams to FuncType instead of FuncDecl? I can see it allows interface methods to have type parameters (in case we ever decide to support those). Are there any other reasons?

[findleyr]

@mdempsky that is the rationale. If we put TParams on the FuncDecl and then type parameters are allowed on interface methods, we'd likely end up with TParams on both FuncDecl and FuncType (or Field?!). Doing it this way hopefully keeps things clean, at the cost of a bit of memory now.

It wasn't an obvious choice though.

[rsc]

This proposal has been added to the active column of the proposals project
and will now be reviewed at the weekly proposal review meetings.
— rsc for the proposal review group

[mvdan]

I'm slightly surprised by the choice of TParams as a name. TypeParams is just slightly longer, and much clearer. I also might mistake TParams for Params or vice versa when quickly reading or reviewing code.

There were several alternatives considered for representing this (MultiIndexExpr) syntax.

I agree with the decision to go with MultiIndexExpr. One major disadvantage of overloading existing types by adding fields is that existing tools might silently break in confusing ways, as they used to inspect or modify node types that can now represent new features. MultiIndexExpr is a new node, so it makes that transition much clearer.

There is code in the wild that assumes the completeness of node sets, i.e. panicking if an unknown node is encountered.

Continuing on the point above: this is how I write my type switches in general, including those using go/ast types. And I think this is good, even in the face of new node types. If a tool of mine needs updating to support type parameters, the tool will start panicking when the new syntax is used, and it will be abundantly clear (and probably rather easy) for me to add the extra switch case.

If the addition was made to an existing type, the strong and clear signal via a default case panic would not be there, and I might not notice that my tool actually needs changes to support new or changed node struct fields in existing types.

[findleyr]

@mvdan

I'm slightly surprised by the choice of TParams as a name. TypeParams is just slightly longer, and much clearer.

Thanks for the feedback. I don't feel the same way, but I've been staring at this for a while. With TParams (and TArgs), I've grown accustomed to seeing 'T' as a modifier. With that said, we already have the TypeParam type in go/types, and you are right that TypeParams is not significantly more verbose. We can consider changing the naming convention, but of course if we decide to switch we should apply this change to go/types as well.

I agree with the decision to go with MultiIndexExpr.

Yes, I was concerned at first but have come around to adding a new node as the clear best option. With something like #47783 I don't think it will be a large burden on tool authors to update their code to avoid the panic, especially since many tools will want to have explicit support for type parameters anyway.

[rsc]

It might be nice to have MultiIndexExpr next to IndexExpr in the docs. Maybe IndexListExpr?

[rsc]

It does seem a bit odd to have the type params in ast.FuncType and not ast.FuncDecl.
The receiver, for example, is only in ast.FuncDecl.

[findleyr]

@griesemer and I just discussed, and here is our current thinking:

• As suggested by @mvdan here, and @rsc regarding go/types: additions to support type parameters #47916, it's clearer to just spell out Type rather than use T as a modifier. I'll go ahead and change TParam to TypeParam and TArg to TypeArg throughout both proposals.
• IndexListExpr sounds good. I mildly prefer how MultiIndexExpr reads, but it does seem worthwhile being adjacent to IndexExpr in the docs.
• Leaving the type params on ast.FuncType has the one distinct advantage of being forward compatible with type parameters on interface methods (and function literals, though it's less likely that matters). I think it's a valid critique to say that we shouldn't design our APIs around future language features that may never materialize, but I also don't think it's that odd to have type parameters associated with the func type. We're inclined to keep it as-is, but can reconsider if there are strong objections.

[gopherbot]

Change https://golang.org/cl/348375 mentions this issue: go/ast: rename TParams fields to TypeParams

[rsc]

Based on the discussion above, this proposal seems like a likely accept.
— rsc for the proposal review group

[gopherbot]

Change https://golang.org/cl/348609 mentions this issue: go/ast: rename MultiIndexExpr to IndexListExpr

[gopherbot]

Change https://golang.org/cl/348382 mentions this issue: 47781-parameterized-go-ast.md: rename MultiIndexExpr to IndexListExpr

[rsc]

No change in consensus, so accepted. 🎉
This issue now tracks the work of implementing the proposal.
— rsc for the proposal review group

[findleyr]

Thanks for the feedback, all!

Closing, as the accepted proposal matches the current implementation at tip.

[gopherbot]

Change https://golang.org/cl/353882 mentions this issue: cmd/cgo: update to handle ast.IndexListExpr