v3 panics in some cases when marshalling unicode char in comments

[mjambert]

With yaml.v3, try this in go playground => get panic.

import (
	"gopkg.in/yaml.v3"
)

func main() {
	var rootNode yaml.Node
	data := []byte(
		`## Some nasty character like this crashes yaml marshalling -> š
dummyStuff:`)
	_ = yaml.Unmarshal(data, &rootNode)

	yaml.Marshal(&rootNode)
}

[mikefarah]

Another example:

value: newValue #проверить

[niemeyer]

It wasn't the writing that was broken, but the reading.

This should be fixed with 5308cda. Please drop me a note if you see related issues.

[mikefarah]

Updated, unfortunately I still get the same error when reading

value: newValue #проверить

$ ./yq r examples/sample.yaml
panic: runtime error: index out of range [10] with length 10 [recovered]
        panic: runtime error: index out of range [10] with length 10

goroutine 1 [running]:
gopkg.in/yaml%2ev3.handleErr(0xc0001078e0)
        C:/Users/mikefarah/go/pkg/mod/gopkg.in/yaml.v3@v3.0.0-20200504163728-5308cda29e3d/yaml.go:276 +0xa1
panic(0x670160, 0xc000012e40)
        c:/go/src/runtime/panic.go:967 +0x174
gopkg.in/yaml%2ev3.write(0xc000128700, 0xc00000cb30, 0xa, 0x10, 0xc000106250, 0x53b401)
        C:/Users/mikefarah/go/pkg/mod/gopkg.in/yaml.v3@v3.0.0-20200504163728-5308cda29e3d/emitterc.go:93 +0x210
gopkg.in/yaml%2ev3.yaml_emitter_write_comment(0xc000128700, 0xc00000cb30, 0xa, 0x10, 0x1)

[niemeyer]

Works fine here and in play.golang.org:

https://play.golang.org/p/-Frq5ryF1Lb

[mikefarah]

Ah but it doesn't work with Decoder/Encoder:

https://play.golang.org/p/NYU0kEpK71M

[mikefarah]

@niemeyer

[lingsamuel]

It doesn't work with marshal/unmarshal if you specify a type:
https://play.golang.org/p/xFD8Q0W7GHb
Please reopen this issue while it's not fixed yet.

[lingsamuel]

Okay I got the point:
In scannerc.go function yaml_parser_scan_line_comment (line 2830) :

// ...
if parser.mark.index >= seen {
	if len(text) == 0 {
		start_mark = parser.mark
	}
	text = append(text, parser.buffer[parser.buffer_pos]) // ERROR!
}
skip(parser)
// ...

In function skip, it add buffer_pos according to character width:

	parser.buffer_pos += width(parser.buffer[parser.buffer_pos])

width:

// Determine the width of the character.
func width(b byte) int {
	// Don't replace these by a switch without first
	// confirming that it is being inlined.
	if b&0x80 == 0x00 {
		return 1
	}
	if b&0xE0 == 0xC0 {
		return 2
	}
	if b&0xF0 == 0xE0 {
		return 3
	}
	if b&0xF8 == 0xF0 {
		return 4
	}
	return 0
}

The solution:

width := width(parser.buffer[parser.buffer_pos])
text = append(text, parser.buffer[parser.buffer_pos:parser.buffer_pos+width]...)

[niemeyer]

Thanks for investigating it. I've fixed a similar issue in 5308cda, and will use the same approach here.

[niemeyer]

Okay, that same logic was applied to inline comments on 913338d.