[dev.fuzz] testing: crash due to incompatible type

[dsnet]

Using 529e5d0.

Consider this snippet:

func FuzzFoo(f *testing.F) {
	for _, td := range myTestdata {
		f.Add(0, []byte(td))
	}

	f.Fuzz(func(t *testing.T, seed int64, b []byte) {
		...
	})
}

Currently, this crashes with:

panic: reflect: Call using int as type int64 [recovered]
	panic: reflect: Call using int as type int64

goroutine 54 [running]:
testing.tRunner.func1.2(0x5fd600, 0xc00058e220)
	testing/testing.go:1153 +0x34b
testing.tRunner.func1(0xc0000f8300)
	testing/testing.go:1156 +0x4b6
panic(0x5fd600, 0xc00058e220)
	runtime/panic.go:972 +0x1d4
reflect.Value.call(0x603a20, 0x653690, 0x13, 0x64222a, 0x4, 0xc001c9c240, 0x3, 0x4, 0x4, 0x30, ...)
	reflect/value.go:409 +0x1917
reflect.Value.Call(0x603a20, 0x653690, 0x13, 0xc001c9c240, 0x3, 0x4, 0x2, 0x4, 0x786c00)
	reflect/value.go:338 +0xb9
testing.(*F).Fuzz.func1.1(0xc0000f8300)
	testing/fuzz.go:343 +0x225
testing.tRunner(0xc0000f8300, 0xc001c9c1e0)
	testing/testing.go:1203 +0xef
created by testing.(*F).Fuzz.func1
	testing/fuzz.go:338 +0x370

The cause of the crash is because testing.F.Add is called with an int, while the Fuzz function itself is expecting an int64.

Either, the testing framework should:

• report an with a more clearer message about what's wrong, or
• automatically convert assignable values.

Also, with generics on the horizon, is there an API design that would guarantee type safety between testing.F.Add and testing.F.Fuzz? It's probably better to statically prevent type errors like than to have it occur at runtime.

\cc @katiehockman @jayconrod

[katiehockman]

Ah thanks for testing and reporting this! There's probably something better we can do here, like you said. So I'll think about that more. At the very least we can improve the error messaging to say something more direct.

It would be great to be able to use generics at some point soon. It would likely help for cases like this. If we can't, then at the very least we can add go vet checks that can verify this at build time so it isn't caught at runtime.

[dsnet]

It would be great to be able to use generics at some point soon. It would likely help for cases like this.

Although, I'm not sure if it's going to be possible with the currently accepted generics proposal since it doesn't support variadic type parameters. A go vet check might be necessary.

@ianlancetaylor @griesemer

[mknyszek]

I put this in backlog for now, but is it a release blocker?

[katiehockman]

One tricky thing to consider is that we have to decide what our source of truth is if the arguments differ.

For example, consider these two fuzz targets:
a)

func FuzzFoo(f *testing.F) {
	f.Add(int32(0))
	f.Fuzz(func(t *testing.T, i int64) { ... })
}

b)

func FuzzFoo(f *testing.F) {
	f.Add(int64(0))
	f.Fuzz(func(t *testing.T, i int32) { ... })
}

Should we pick int64 values to fuzz and write to the corpus, since that can hold the most bytes, and is thus the most likely to be able to assign to? Or should we assume that the arguments in the f.Fuzz function are the source of truth, and if whatever is passed via f.Add or in testdata can't be cast to that type, then we fail the test? I think I'd prefer the latter.

Do you have an opinion on this?

[dsnet]

Or should we assume that the arguments in the f.Fuzz function are the source of truth, and if whatever is passed via f.Add or in testdata can't be cast to that type, then we fail the test? I think I'd prefer the latter.

I believe f.Fuzz should be authoritative with the rationale that f.Fuzz is given explicit types in the arguments list. On the other hand, the types given to f.Add are inferred by the values given.

[gopherbot]

Change https://golang.org/cl/325702 mentions this issue: [dev.fuzz] testing: convert seed corpus values where possible