Make capnpc-go output deterministic

[kormat]

Currently the output of capnpc-go is not deterministic: the var schema_XXX line can change even if the input (or tool version) does not.. This is a problem when you want to store the generated files in git (and want to be sure that you catch drift between the source .capnp files and the output .go files).

This is caused by how the schema is encoded
(from https://github.com/capnproto/go-capnproto2/blob/master/capnpc-go/capnpc-go.go#L159):

	var buf bytes.Buffer
	z, _ := zlib.NewWriterLevel(&buf, zlib.BestCompression)
	if err := capnp.NewPackedEncoder(z).Encode(msg); err != nil {

Neither zlib nor capnp are guaranteed to produce deterministic/canonical output (AIUI).

Would it be possible to change to outputting a canonical representation of the schema instead?

[zombiezen]

Hm. Are you claiming that the output is not deterministic given the same version of Go and capnpc-go? That shouldn't be the case. I agree that the zlib output will not be deterministic over Go versions, but that shouldn't matter for the purposes of the generator.

[kormat]

Based on your comment, i did some more digging, and eventually discovered something Odd.

Using the .capnp files in https://gist.github.com/kormat/89d9a07b9d83f4ebebe14666b43bc326:

# Just compile a.capnp
$ rm *capnp.go; capnp compile -I$GOPATH/src/zombiezen.com/go/capnproto2/std -ogo a.capnp; md5sum a.capnp.go
08b0802596a305c23f5ffa3696bac1c1  a.capnp.go

# Compile a.capnp and b.capnp at the same time
$ rm *capnp.go; capnp compile -I$GOPATH/src/zombiezen.com/go/capnproto2/std -ogo a.capnp b.capnp; md5sum a.capnp.go
ff0ea0a407a03c09109978547a83bf6e  a.capnp.go

I.e. the output for a given .capnp file changes if other .capnp files are processed at the same time.

Environment:

• ubuntu 16.04, amd64
• $ capnp --version: Cap'n Proto version 0.5.3
• go-capnproto2 version: def4a3e i.e. current HEAD (but the same issue also exists in at least 659aba4)

[kormat]

Here's a diff of the generated files:

$ diff -urN a.capnp.go.solo a.capnp.go.group
--- a.capnp.go.solo     2019-07-31 13:24:18.331702979 +0200
+++ a.capnp.go.group    2019-07-31 13:24:42.259623661 +0200
@@ -67,14 +67,14 @@
        return A{s}, err
 }
 
-const schema_866c06cc12d9b24d = "x\xda\x12\xf0s`2d\xdd\xcf\xc8\xc0\x10(\xc2\xca" +
-       "\xf6\x7fu\xf9N\xa3#\x0f\xbd\xde1\x04r02\xfe" +
-       "\xf7\xddtS\xe8\x0c[N\x1b\x03\x0b;\x03\x83\xe0\xd1" +
-       " \xc1\x93`\xda\x9eA\xf7\x7f\xa2^rbA^\x81" +
-       "\x15\x03\xa3c\x00#c \x0b3\x0b\x03\x03\x0b#\x03" +
-       "\x83 \xaf\x10\x03C \x073c\xa0\x08\x13#c\"" +
-       "#\x1b\x03\x13#\x1b\x03# \x00\x00\xff\xff\xc4\x08\x16" +
-       "\x85"
+const schema_866c06cc12d9b24d = "x\xda\x12Hq`2d\xcdgb`\x08\x94ae" +
+       "\xfb\xbf\xba|\xa7\xd1\x91\x87^\xef\x18\x029\x18\x19\xff" +
+       "\xfbn\xba)t\x86-\xa7\x8d\x81\x85\x9d\x81AX\x96" +
+       "1HX\x91\x11\xc2\xb2gp\xfe\x9f\xa8\x97\x9cX\x90" +
+       "W`\xc5\xc0\xe8\x18\xc0\xc8\x18\xc8\xc2\xcc\xc2\xc0\xc0\xc2" +
+       "\xc8\xc0 \xc8+\xc4\xc0\x10\xc8\xc1\xcc\x18(\xc2\xc4\xc8" +
+       "\x98\xc8\xc8\xc6\xc0\xc4\xc8\xc6\xc0\x08\x08\x00\x00\xff\xffq" +
+       "W\x14{"
 
 func init() {
        schemas.Register(schema_866c06cc12d9b24d,

[kormat]

Looking at the decoded CodeGeneratorRequest, it doesn't look obviously wrong (implying the issue is in capnpc-go, and not in capnp).

$ capnp compile -I$GOPATH/src/zombiezen.com/go/capnproto2/std -o- a.capnp b.capnp | capnp decode ~/go/src/zombiezen.com/go/capnproto2/std/capnp/schema.capnp CodeGeneratorRequest

I've updated the gist above (https://gist.github.com/kormat/89d9a07b9d83f4ebebe14666b43bc326) with the full generated capnp.go files, as well as the decoded code gen requests.

[kormat]

I found the issue, and sent a PR to fix it (#146)