go/parser: poor error recovery after missing function call argument

[adonovan]

This program contains a syntax error in Abs(,) that causes the parser's recovery to generate a BadExpr for math.Lde.

func g() {
        math.Abs(,)

        //

        math.Lde()
}

This causes go/types not to recognize math as a reference to a package, and causes completions in gopls to misbehave.

I think the parser could do a better job of recovery. @griesemer gives reasons why this is difficult in #24327 (comment):

go/parser is likely falling behind the compiler due to the compiler's parser (package syntax) being actively improved whenever we run into unsatisfying error messages. It's fairly time-consuming to back-port those fixes because the parsers don't have the same structure, and small local changes sometimes have unexpected consequences with respect to error handling (hence those changes, even if small, are time-consuming to get right).

Ideally, in the long run we should migrate to package syntax. Failing that, one option might be to replace go/parser with the syntax package parser while keeping the same API. That might be not too hard since the syntax trees are reasonably close.

Related issue of gopls completion after a syntax error:

• go/parser: redundant error message with unexpected comma #24327
• x/tools/gopls: bad completion with syntax errors #39721
• go/parser: inconsistent error messages when missing braces after struct/interface in return type #36595
• x/tools/gopls: vendor go/parser #45408

[griesemer]

This specific error is perhaps not too hard to fix individually.

Re-using the compiler's syntax package's parser code, but producing go/ast may be a good solution in the long run. Perhaps parts of it can be copied in pieces. E.g., the parts the parse parameter lists could be copied and adjusted to produce go/ast, and as a "side-effect" this bug might get fixed.

[adonovan]

Another version of the same recovery cascade:

package main

import "fmt"

func nop(...any){}

func main() {
        nop(1, 2, 3 fmt.Println) // notice missing comma after 3

        nop()
        nop()
        nop()

        var a, b, c int
        println(a, b, c)
}

Errors:
play.go:8:21: missing ',' in argument list
play.go:8:24: expected operand, found '.'
play.go:14:9: missing ',' in argument list
play.go:14:21: missing ',' in argument list
play.go:14:24: expected operand, found newline
play.go:16:3: expected ')', found 'EOF'
play.go:16:3: expected ';', found 'EOF'
play.go:16:3: expected ';', found 'EOF'
play.go:16:3: expected '}', found 'EOF'
play.go:16:3: missing ',' in argument list
play.go:14:13: undeclared name: a
play.go:14:16: undeclared name: b
play.go:14:19: undeclared name: c
play.go:3:8: "fmt" imported but not used

FWIW, pretty printing the damaged tree produces only a small part of it:

func main() {
    nop(1, 2, 3, BadExpr,

        a, b, c, BadExpr,
    )
}

[griesemer]

For reference, for the above program, the compiler produces simply

foo.go:8:21: syntax error: unexpected fmt in argument list; possibly missing comma or )

[gopherbot]

Change https://go.dev/cl/493616 mentions this issue: gopls/internal/lsp/cache: skip type errors after parse errors

[adonovan]

Here's a variant distilled from #68918 in which the parser swallowed (originally) thousands of lines during recovery:

package p

func f() {
	h(a b, 

	case c:
		if e.Len != nil {
		} else {
		}
	}
}

func longfunc() {
   //...
}

func g() {
	if stmt.Tok == token.DEFINE {
	}
}

Using the types playground, this parse tree is pretty printed as:

package p

func f() {
h(a, BadExpr,

e.Len != nil{}, BadExpr,

//...

stmt.Tok == token.DEFINE{},
BadExpr)
}

Observe that recovery consumes the if of various statements of the form if x == y {, changing the parser's disposition towards the { so that it becomes a composite literal y{...}. There are two examples above: nil{...} and token.DEFINE{...}. It also sucks in and discards a potentially long function in the middle.