panic: resolveTable item not yet handled: N (with )

[rozcietrzewiacz]

When?

Converting a large input comprising multiple documents divided with ---. No issues with the same input when using yq for example.

Source of the original file (kubernetes crds):
https://doc.crds.dev/raw/github.com/crossplane/provider-aws@v0.23.0

What happened?

$ curl https://doc.crds.dev/raw/github.com/crossplane/provider-aws@v0.23.0 > in.yaml
$ wc -l in.yaml 
46157 in.yaml
$ yaml2json < in.yaml > out.json
panic: resolveTable item not yet handled: N (with ) [recovered]
	panic: resolveTable item not yet handled: N (with )

goroutine 1 [running]:
github.com/bronze1man/yaml2json/vendor/gopkg.in/yaml%2ev2.handleErr(0xc000113cc8)
	/Users/a/work/yaml2json/src/github.com/bronze1man/yaml2json/vendor/gopkg.in/yaml.v2/yaml.go:249 +0x9a
panic(0x509cc0, 0xc000126eb0)
	/usr/local/go/src/runtime/panic.go:513 +0x1b9
github.com/bronze1man/yaml2json/vendor/gopkg.in/yaml%2ev2.resolve(0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0)
	/Users/a/work/yaml2json/src/github.com/bronze1man/yaml2json/vendor/gopkg.in/yaml.v2/resolve.go:198 +0xe40
github.com/bronze1man/yaml2json/vendor/gopkg.in/yaml%2ev2.(*decoder).scalar(0xc0003a2040, 0xc00010e070, 0x511a60, 0xc000126ea0, 0x194, 0x511a60)
	/Users/a/work/yaml2json/src/github.com/bronze1man/yaml2json/vendor/gopkg.in/yaml.v2/decode.go:376 +0x92
github.com/bronze1man/yaml2json/vendor/gopkg.in/yaml%2ev2.(*decoder).unmarshal(0xc0003a2040, 0xc00010e070, 0x511a60, 0xc000126ea0, 0x194, 0x1)
	/Users/a/work/yaml2json/src/github.com/bronze1man/yaml2json/vendor/gopkg.in/yaml.v2/decode.go:330 +0xd6
github.com/bronze1man/yaml2json/vendor/gopkg.in/yaml%2ev2.(*decoder).document(0xc0003a2040, 0xc00010e000, 0x511a60, 0xc000126ea0, 0x194, 0xc000113c30)
	/Users/a/work/yaml2json/src/github.com/bronze1man/yaml2json/vendor/gopkg.in/yaml.v2/decode.go:344 +0x7a
github.com/bronze1man/yaml2json/vendor/gopkg.in/yaml%2ev2.(*decoder).unmarshal(0xc0003a2040, 0xc00010e000, 0x511a60, 0xc000126ea0, 0x194, 0x194)
	/Users/a/work/yaml2json/src/github.com/bronze1man/yaml2json/vendor/gopkg.in/yaml.v2/decode.go:320 +0x1d3
github.com/bronze1man/yaml2json/vendor/gopkg.in/yaml%2ev2.(*Decoder).Decode(0xc000113d20, 0x503820, 0xc000126ea0, 0x0, 0x0)
	/Users/a/work/yaml2json/src/github.com/bronze1man/yaml2json/vendor/gopkg.in/yaml.v2/yaml.go:130 +0x1da
github.com/bronze1man/yaml2json/y2jLib.TranslateStream(0x554fa0, 0xc00008c000, 0x7f27482db000, 0xc00008c008, 0xc00008c008, 0x0)
	/Users/a/work/yaml2json/src/github.com/bronze1man/yaml2json/y2jLib/lib.go:16 +0x1fb
main.main()
	src/github.com/bronze1man/yaml2json/main.go:19 +0xa9

[rozcietrzewiacz]

I have narrowed it down: this has nothing to do with the size of the file. The error occurs when the last yaml document in file is empty. Here's a minimal example:

simple.yaml:

first:
  file: without --- at top
---
second:
  file: which is actually the last one
  next: there's nothing after the last delimiter
---

Result:

$ yaml2json < simple.yaml
{"first":{"file":"without --- at top"}}
{"second":{"file":"which is actually the last one","next":"there's nothing after the last delimiter"}}
panic: resolveTable item not yet handled: N (with ) [recovered]
	panic: resolveTable item not yet handled: N (with )

goroutine 1 [running]:
github.com/bronze1man/yaml2json/vendor/gopkg.in/yaml%2ev2.handleErr(0xc000079cc8)
	/Users/a/work/yaml2json/src/github.com/bronze1man/yaml2json/vendor/gopkg.in/yaml.v2/yaml.go:249 +0x9a
panic(0x509cc0, 0xc00007c5f0)
	/usr/local/go/src/runtime/panic.go:513 +0x1b9
github.com/bronze1man/yaml2json/vendor/gopkg.in/yaml%2ev2.resolve(0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0)
	/Users/a/work/yaml2json/src/github.com/bronze1man/yaml2json/vendor/gopkg.in/yaml.v2/resolve.go:198 +0xe40
github.com/bronze1man/yaml2json/vendor/gopkg.in/yaml%2ev2.(*decoder).scalar(0xc0000b6240, 0xc0000b5880, 0x511a60, 0xc00007c5e0, 0x194, 0x511a60)
	/Users/a/work/yaml2json/src/github.com/bronze1man/yaml2json/vendor/gopkg.in/yaml.v2/decode.go:376 +0x92
github.com/bronze1man/yaml2json/vendor/gopkg.in/yaml%2ev2.(*decoder).unmarshal(0xc0000b6240, 0xc0000b5880, 0x511a60, 0xc00007c5e0, 0x194, 0x1)
	/Users/a/work/yaml2json/src/github.com/bronze1man/yaml2json/vendor/gopkg.in/yaml.v2/decode.go:330 +0xd6
github.com/bronze1man/yaml2json/vendor/gopkg.in/yaml%2ev2.(*decoder).document(0xc0000b6240, 0xc0000b5810, 0x511a60, 0xc00007c5e0, 0x194, 0xc000079c30)
	/Users/a/work/yaml2json/src/github.com/bronze1man/yaml2json/vendor/gopkg.in/yaml.v2/decode.go:344 +0x7a
github.com/bronze1man/yaml2json/vendor/gopkg.in/yaml%2ev2.(*decoder).unmarshal(0xc0000b6240, 0xc0000b5810, 0x511a60, 0xc00007c5e0, 0x194, 0x194)
	/Users/a/work/yaml2json/src/github.com/bronze1man/yaml2json/vendor/gopkg.in/yaml.v2/decode.go:320 +0x1d3
github.com/bronze1man/yaml2json/vendor/gopkg.in/yaml%2ev2.(*Decoder).Decode(0xc000079d20, 0x503820, 0xc00007c5e0, 0x0, 0x0)
	/Users/a/work/yaml2json/src/github.com/bronze1man/yaml2json/vendor/gopkg.in/yaml.v2/yaml.go:130 +0x1da
github.com/bronze1man/yaml2json/y2jLib.TranslateStream(0x554fa0, 0xc00008c000, 0x7f0acb96b000, 0xc00008c008, 0xc00008c008, 0x0)
	/Users/a/work/yaml2json/src/github.com/bronze1man/yaml2json/y2jLib/lib.go:16 +0x1fb
main.main()
	src/github.com/bronze1man/yaml2json/main.go:19 +0xa9

[DRuggeri]

I've been able to confirm this issue with a similar reproduction recipe - but in this case, the following file is enough to cause the same panic on the latest release:

properties:
  property1: FOO
  property2:

This can even be further simplified - any object that has a null value seems to cause the panic, such as a file containing only key:

$ echo 'key:' | ./latest-yaml2json
panic: resolveTable item not yet handled: N (with ) [recovered]
        panic: resolveTable item not yet handled: N (with )
...

I was able to track this down to a regression between tags 1.2 and 1.3. I don't have a build environment to see if the latest un-released commits have the same issue, though.

[bronze1man]

This bug has already been fixed in v1.3.2 .
should be fixed by gopkg.in/yaml.v3 v3.0.1
will add test case in this issues in later version:

{
			`first:
  file: without --- at top
---
second:
  file: which is actually the last one
  next: there's nothing after the last delimiter
---`,
`{"first":{"file":"without --- at top"}}
{"second":{"file":"which is actually the last one","next":"there's nothing after the last delimiter"}}
null
`,
},
{
			`properties:
  property1: FOO
  property2:`,
	`{"properties":{"property1":"FOO","property2":null}}
`,
}, // https://github.com/bronze1man/yaml2json/issues/23

[martinrangelov96]

Hi @bronze1man

We use this library and face the same issue. We see there is a fix in v1.3.2 but there is no official release with a binary. Are you planning to release this version?

Thank you!

[bronze1man]

@martinrangelov96 ok, I release with a binary

[martinrangelov96]

@bronze1man thanks for the fast response and release!