Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

reflect: StructOf panics when using unexported fields #432

Closed
jonathangold opened this issue Dec 31, 2018 · 11 comments
Closed

reflect: StructOf panics when using unexported fields #432

jonathangold opened this issue Dec 31, 2018 · 11 comments

Comments

@jonathangold
Copy link
Contributor

Versions:

  • mgmt version (eg: mgmt --version):
    mgmt version 0.0.15-127-g96dccca why isn't this 0.0.16...?

  • operating system/distribution (eg: uname -a):
    Linux skynet 4.19.10-300.fc29.x86_64 #1 SMP Mon Dec 17 15:34:44 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux

  • golang version (eg: go version):
    go version go1.11.2 linux/amd64

Description:

Panic Reproducer: https://gist.github.com/jonathangold/58cc2f2fb2bcc460a546e1b09c512e95

When I run the .mcl file included with the above panic reproducer, it panics:

jon@skynet ~/mgmt 🗙 (res/dhcpd) $ ./mgmt run --tmp-prefix lang --lang examples/lang/dhcpd0.mcl 
2018-12-31 17:28:26.583481 I | cli: lang: lexing/parsing...
2018-12-31 17:28:26.604908 I | cli: lang: init...
2018-12-31 17:28:26.604931 I | cli: lang: interpolating...
2018-12-31 17:28:26.605011 I | cli: lang: building scope...
2018-12-31 17:28:26.605027 I | cli: lang: running type unification...
2018-12-31 17:28:26.605266 I | cli: lang: input: examples/lang/dhcpd0.mcl
2018-12-31 17:28:26.605290 I | cli: lang: tree:
.
├── dhcpd0.mcl
└── metadata.yaml
17:28:26 hello.go:49: this is: mgmt, version: 0.0.15-127-g96dccca
17:28:26 hello.go:50: main: start: 1546295306605323199
17:28:26 main.go:191: main: warning: working prefix directory is temporary!
17:28:26 main.go:191: main: working prefix is: /tmp/mgmt-skynet-571455772
17:28:26 pgp.go:88: PGP: Created key: 18B2A691
17:28:26 main.go:191: main: etcd: seeds: no seeds specified!
17:28:26 etcd.go:435: Etcd: Bootstrapping...
17:28:26 etcd.go:1463: Etcd: Nominated: skynet=http://localhost:2380
17:28:26 etcd.go:1478: Etcd: StartServer(newCluster: true): skynet=http://localhost:2380
17:28:26 etcd.go:350: Etcd: Connect: Endpoints: []
17:28:26 etcd.go:364: Etcd: Connect: CtxError...
17:28:26 etcd.go:644: Etcd: CtxError: Reason: CtxDelayErr(1s): No endpoints available yet!
17:28:26 etcd.go:1749: Etcd: StartServer: Starting server...
17:28:27 etcd.go:1756: Etcd: StartServer: Done starting server!
17:28:27 etcd.go:1774: Etcd: StartServer: Server running...
17:28:27 etcd.go:1501: Etcd: Addresses are: http://localhost:2379
17:28:27 etcd.go:348: Etcd: Connect: Endpoints: skynet=http://localhost:2379
17:28:27 etcd.go:455: Etcd: Startup: Volunteering...
17:28:27 etcd.go:1669: Etcd: Ideal cluster size is now: 5
17:28:27 etcd.go:1022: Etcd: Set(/_mgmt/idealClusterSize): &{cluster_id:14841639068965178418 member_id:10276657743932975437 revision:4 raft_term:2  <nil>}
17:28:27 etcd.go:1263: Etcd: Members: List: [skynet]
17:28:27 etcd.go:1286: Etcd: Leader: skynet
17:28:27 etcd.go:1305: Etcd: Volunteers: [skynet]
17:28:27 etcd.go:1309: Etcd: Quitters: []
17:28:27 etcd.go:1321: Etcd: Candidates: []
17:28:27 etcd.go:1263: Etcd: Members: List: [skynet]
17:28:27 etcd.go:1286: Etcd: Leader: skynet
17:28:27 etcd.go:1305: Etcd: Volunteers: [skynet]
17:28:27 etcd.go:1309: Etcd: Quitters: []
17:28:27 etcd.go:1321: Etcd: Candidates: []
17:28:28 main.go:191: main: waiting...
17:28:28 main.go:191: main: running...
17:28:28 main.go:191: main: waiting...
17:28:28 main.go:492: gapi: generating new graph...
17:28:28 main.go:492: gapi: swap!
17:28:28 main.go:492: gapi: lang: lexing/parsing...
17:28:28 main.go:492: gapi: lang: init...
17:28:28 main.go:492: gapi: lang: interpolating...
17:28:28 main.go:492: gapi: lang: building scope...
17:28:28 main.go:492: gapi: lang: running type unification...
17:28:28 main.go:492: gapi: lang: building function graph...
17:28:28 main.go:492: gapi: lang: function engine initializing...
17:28:28 main.go:492: gapi: lang: function engine validating...
17:28:28 main.go:492: gapi: lang: function engine starting...
17:28:28 main.go:492: gapi: lang: stream...
17:28:28 main.go:492: gapi: lang: loop...
17:28:28 main.go:492: gapi: lang: funcs: func `str(192.168.42.3)` started
17:28:28 main.go:492: gapi: lang: funcs: func `str(192.168.42.3)` changed
17:28:28 main.go:492: gapi: lang: funcs: func `str(foo)` started
17:28:28 main.go:492: gapi: lang: funcs: func `str(foo)` changed
17:28:28 main.go:492: gapi: lang: funcs: func `str(BA:98:76:54:32:10)` started
17:28:28 main.go:492: gapi: lang: funcs: func `str(192.168.42.1)` started
17:28:28 main.go:492: gapi: lang: funcs: func `str(192.168.42.3)` stopped
17:28:28 main.go:492: gapi: lang: funcs: func `str(foo.com)` started
17:28:28 main.go:492: gapi: lang: funcs: func `str(192.168.42.2)` started
17:28:28 main.go:492: gapi: lang: funcs: func `str(BA:98:76:54:32:10)` changed
17:28:28 main.go:492: gapi: lang: funcs: func `str(BA:98:76:54:32:10)` stopped
17:28:28 main.go:492: gapi: lang: funcs: func `str(192.168.42.100/24)` started
17:28:28 main.go:492: gapi: lang: funcs: func `str(01:23:45:67:89:AB)` started
17:28:28 main.go:492: gapi: lang: funcs: func `str(192.168.42.2)` changed
17:28:28 main.go:492: gapi: lang: funcs: func `str(192.168.42.200/24)` started
17:28:28 main.go:492: gapi: lang: funcs: func `str(192.168.42.200/24)` changed
17:28:28 main.go:492: gapi: lang: funcs: func `str(192.168.42.2)` stopped
17:28:28 main.go:492: gapi: lang: funcs: func `str(192.168.42.1)` changed
17:28:28 main.go:492: gapi: lang: funcs: func `str(192.168.42.101/24)` started
17:28:28 main.go:492: gapi: lang: funcs: func `str(bar.net)` started
17:28:28 main.go:492: gapi: lang: funcs: func `str(192.168.42.100/24)` started
17:28:28 main.go:492: gapi: lang: funcs: func `str(192.168.42.200/24)` stopped
17:28:28 main.go:492: gapi: lang: funcs: func `str(192.168.42.101/24)` changed
17:28:28 main.go:492: gapi: lang: funcs: func `str(192.168.42.1)` stopped
17:28:28 main.go:492: gapi: lang: funcs: func `str(192.168.42.101/24)` stopped
17:28:28 main.go:492: gapi: lang: funcs: func `struct(mac: str(BA:98:76:54:32:10); ip: str(192.168.42.101/24))` started
17:28:28 main.go:492: gapi: lang: funcs: func `list(str(192.168.42.2), str(192.168.42.3))` started
17:28:28 main.go:492: gapi: lang: funcs: func `list(str(192.168.42.1))` started
17:28:28 main.go:492: gapi: lang: funcs: func `struct(mac: str(BA:98:76:54:32:10); ip: str(192.168.42.101/24))` changed
17:28:28 main.go:492: gapi: lang: funcs: func `list(str(192.168.42.1))` changed
17:28:28 main.go:492: gapi: lang: funcs: func `list(str(192.168.42.1))` stopped
17:28:28 main.go:492: gapi: lang: funcs: func `struct(mac: str(BA:98:76:54:32:10); ip: str(192.168.42.101/24))` stopped
17:28:28 main.go:492: gapi: lang: funcs: func `str(bar.net)` changed
17:28:28 main.go:492: gapi: lang: funcs: func `str(bar.net)` stopped
17:28:28 main.go:492: gapi: lang: funcs: func `str(192.168.42.100/24)` changed
17:28:28 main.go:492: gapi: lang: funcs: func `str(192.168.42.100/24)` stopped
17:28:28 main.go:492: gapi: lang: funcs: func `str(01:23:45:67:89:AB)` changed
17:28:28 main.go:492: gapi: lang: funcs: func `str(01:23:45:67:89:AB)` stopped
17:28:28 main.go:492: gapi: lang: funcs: func `str(192.168.42.100/24)` changed
17:28:28 main.go:492: gapi: lang: funcs: func `str(192.168.42.100/24)` stopped
17:28:28 main.go:492: gapi: lang: funcs: func `struct(mac: str(01:23:45:67:89:AB); ip: str(192.168.42.100/24))` started
17:28:28 main.go:492: gapi: lang: funcs: func `struct(from: str(192.168.42.100/24); to: str(192.168.42.200/24))` started
17:28:28 main.go:492: gapi: lang: funcs: func `struct(mac: str(01:23:45:67:89:AB); ip: str(192.168.42.100/24))` changed
17:28:28 main.go:492: gapi: lang: funcs: func `struct(mac: str(01:23:45:67:89:AB); ip: str(192.168.42.100/24))` stopped
17:28:28 main.go:492: gapi: lang: funcs: func `str(foo.com)` changed
17:28:28 main.go:492: gapi: lang: funcs: func `str(foo)` stopped
17:28:28 main.go:492: gapi: lang: funcs: func `list(str(192.168.42.2), str(192.168.42.3))` changed
17:28:28 main.go:492: gapi: lang: funcs: func `list(str(192.168.42.2), str(192.168.42.3))` stopped
17:28:28 main.go:492: gapi: lang: funcs: func `str(foo.com)` stopped
17:28:28 main.go:492: gapi: lang: funcs: func `list(str(foo.com), str(bar.net))` started
17:28:28 main.go:492: gapi: lang: funcs: func `list(struct(mac: str(01:23:45:67:89:AB); ip: str(192.168.42.100/24)), struct(mac: str(BA:98:76:54:32:10); ip: str(192.168.42.101/24)))` started
17:28:28 main.go:492: gapi: lang: funcs: func `list(str(foo.com), str(bar.net))` changed
17:28:28 main.go:492: gapi: lang: funcs: func `list(str(foo.com), str(bar.net))` stopped
17:28:28 main.go:492: gapi: lang: funcs: func `struct(from: str(192.168.42.100/24); to: str(192.168.42.200/24))` changed
17:28:28 main.go:492: gapi: lang: funcs: func `struct(from: str(192.168.42.100/24); to: str(192.168.42.200/24))` stopped
17:28:28 main.go:492: gapi: lang: funcs: func `list(struct(mac: str(01:23:45:67:89:AB); ip: str(192.168.42.100/24)), struct(mac: str(BA:98:76:54:32:10); ip: str(192.168.42.101/24)))` changed
17:28:28 main.go:492: gapi: lang: funcs: func `list(struct(mac: str(01:23:45:67:89:AB); ip: str(192.168.42.100/24)), struct(mac: str(BA:98:76:54:32:10); ip: str(192.168.42.101/24)))` stopped
17:28:28 main.go:492: gapi: generating new graph...
17:28:28 main.go:492: gapi: lang: running interpret...
17:28:28 main.go:191: main: loop: exited
panic: reflect.StructOf: field "from" is unexported but missing PkgPath

goroutine 235 [running]:
reflect.runtimeStructField(0xc000266a68, 0x4, 0x0, 0x0, 0x1c5e980, 0x15fd880, 0x0, 0x0, 0x0, 0x0, ...)
	/usr/lib/golang/src/reflect/type.go:2786 +0x1b9
reflect.StructOf(0xc00036a750, 0x2, 0x2, 0x0, 0x0)
	/usr/lib/golang/src/reflect/type.go:2378 +0x1fa8
github.com/purpleidea/mgmt/lang/types.(*Type).Reflect(0xc000889310, 0x2, 0x2)
	/home/jon/go/src/github.com/purpleidea/mgmt/lang/types/type.go:766 +0x233
github.com/purpleidea/mgmt/lang/types.(*StructValue).Value(0xc000452380, 0x1c55d20, 0xc000452380)
	/home/jon/go/src/github.com/purpleidea/mgmt/lang/types/value.go:845 +0x4f
github.com/purpleidea/mgmt/lang.(*StmtRes).Output(0xc0008724c0, 0x2ab0430, 0x0, 0x0)
	/home/jon/go/src/github.com/purpleidea/mgmt/lang/structs.go:357 +0x648
github.com/purpleidea/mgmt/lang.(*StmtProg).Output(0xc000248de0, 0x46527c, 0x29e6a90, 0x33)
	/home/jon/go/src/github.com/purpleidea/mgmt/lang/structs.go:2092 +0x1d1
github.com/purpleidea/mgmt/lang.interpret(0x1c4d480, 0xc000248de0, 0x0, 0x0, 0x0)
	/home/jon/go/src/github.com/purpleidea/mgmt/lang/interpret.go:35 +0x4c
github.com/purpleidea/mgmt/lang.(*Lang).Interpret(0xc0002183c0, 0x148ed14, 0xc00082ed30, 0x10)
	/home/jon/go/src/github.com/purpleidea/mgmt/lang/lang.go:316 +0xa5
github.com/purpleidea/mgmt/lang.(*GAPI).Graph(0xc0004269c0, 0xc00082ede0, 0x2, 0x1)
	/home/jon/go/src/github.com/purpleidea/mgmt/lang/gapi.go:438 +0x42
github.com/purpleidea/mgmt/lib.(*Main).Run.func10(0x1a040e0, 0xc000295174, 0xc0002383c8, 0xc000824480, 0xc0002383d0, 0xc0000d5200, 0xc000267328, 0x6, 0xc000451200, 0xc00026a560, ...)
	/home/jon/go/src/github.com/purpleidea/mgmt/lib/main.go:547 +0x3a3
created by github.com/purpleidea/mgmt/lib.(*Main).Run
	/home/jon/go/src/github.com/purpleidea/mgmt/lib/main.go:428 +0xd96

related issue: golang/go#25401

@purpleidea
Copy link
Owner

type Reservation struct {
	mac string `lang:"mac" yaml:"mac"`
	ip  string `lang:"ip" yaml:"ip"`
}

Seems the fields are private. We shouldn't panic, but that's one unrelated issue.

@jonathangold
Copy link
Contributor Author

Yeah, they should be public, as discussed in IRC. Making them private was a last-ditch attempt at bypassing the type unification error when they were public. I thought reflect might do the heavy lifting, but it was a slim chance. :P

@purpleidea
Copy link
Owner

Yeah, I'll recap the issue here in case anyone is interested:

When a golang resource has a struct that needs fulfilling, we need to feed it with a matching struct in mcl. The two are not exactly identical, because a golang struct can have empty fields (that default to the zero value) where as a struct in mcl has an exact type which contains each and every field!

So how do we solve the situation where we might only want to specify only one or two of the possible fields in a struct? It's tricky because in mcl:

struct{ foo int; bar str; baz bool}

is a DIFFERENT TYPE ENTIRELY from:

struct{ foo int; bar str}

Here's where the magic of the unification comes in. At mcl run time, all the types must be exact, but at mcl compile time we work out what everything should be. Printf being dynamic here is the canonical example. So all we need to do is unify any of the possible struct input permutations which could be valid against the expected golang type, and if we find a single match, then we're golden!

Eg: if we're expected in golang a struct{ foo int; bar str; baz bool} then we could unify from a struct{ foo int; bar str} but never from a struct{ foobad int; bar str; baz bool} and never from a struct{ foo float; bar str; baz bool}, etc... (Note the incompatible foobad and foo float fields.)

Interestingly this is the same issue that we need to solve for the meta param field in the struct. We should probably add it as property named Meta (similar to:

Property string // TODO: iota constant instead?
)

Questions welcome!

@jonathangold
Copy link
Contributor Author

So how do we solve the situation where we might only want to specify only one or two of the possible fields in a struct?

When I make the fields public, and specify all of them in my mcl file, I still get a type unification error.

Here's the updated code:
https://gist.github.com/jonathangold/401f04f7f6805e39ff14c20feeebcb6b

And here's the result of running it:

jon@skynet ~/mgmt 🗙 (res/dhcpd) $ ./mgmt run --tmp-prefix lang --lang examples/lang/dhcpd0.mcl 
2018-12-31 19:09:09.331444 I | cli: lang: lexing/parsing...
2018-12-31 19:09:09.346465 I | cli: lang: init...
2018-12-31 19:09:09.346496 I | cli: lang: interpolating...
2018-12-31 19:09:09.346554 I | cli: lang: building scope...
2018-12-31 19:09:09.346567 I | cli: lang: running type unification...
2018-12-31 19:09:09.346709 I | run: error: cli parse error: could not unify types: can't unify, invariant illogicality with equals: struct fields differ

Is this still the same issue?

@purpleidea
Copy link
Owner

Is this still the same issue?

Exactly. (Not the panic.) This is as expected until I patch the problem in #432 (comment)

Sorry that this blocks you. I'll try and work on it next.

@jonathangold
Copy link
Contributor Author

No worries. I can work around it for the time-being (and shouldn't need to actually run any code for a while.)

@purpleidea
Copy link
Owner

Okay, so you shouldn't actually be blocked on this anymore. I fixed a small bug, and added some fixes and some tests. Eg:

d38592e

The one catch is that your struct must be complete, not partial. Eg if it's defined as:

type Reservation struct {
	Mac string `lang:"mac" yaml:"mac"`
	IP  string `lang:"ip" yaml:"ip"`
}

You must specify both those fields. I'll eventual consider adding the code that let's you have a struct like:

type Reservation struct {
	Mac string `lang:"mac" yaml:"mac"`
	IP  string `lang:"ip" yaml:"ip"`
	SomethingOptional  string `lang:"somethingoptional" yaml:"somethingoptional"`
}

That will not fail during unification if you leave one or more of the fields out.

Please lmk if you have any more issues. I'll investigate the panic too! (Coming soon.)

@purpleidea
Copy link
Owner

Fix for the panic is in f30842e (will merge when tests pass.)

Thanks for the report! Virtually all panics are bad, and even if it's the user's "fault" we shouldn't really panic, it just meant they found a clever way to poke us that we should have prevented!

I'm going to close this, please ping or reopen if you have any more issues.

@purpleidea
Copy link
Owner

Actually, I think there is another bug or maybe there is a heisenbug happening on my machine. I thought I fixed it, but now... Hmmm.

@purpleidea purpleidea reopened this Jan 3, 2019
@purpleidea
Copy link
Owner

Oh. I might have "fixed" something just now, which actually unintentionally triggers this :/

@purpleidea
Copy link
Owner

@jonathangold I think this should be fixed now, but I didn't retest the exact code. We have a newer dhcp resource already merged that uses automatic grouping instead of complex internal structs, but this was still a valid bug. If you can still repro, please ping or open a new issue! Thanks =D

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants