Discussion: code generation

[mappu]

Hi,

Very cool project, certainly there is a need for high level object functions on top of Bolt.

I saw this conversation on GitHub https://www.reddit.com/r/golang/comments/5dfzm7/bolthold_an_embeddable_nosql_store_for_go_types/da5uc53/ .

For me codegeneration is a way to avoid reflection and interface{}. E.g. the Where("field") concept seems weak compared to a codegen'd Where_MyType_MyField() replacement. It helps catch errors at compile-time.

Possibly this is too great a departure from the package's current behaviour?

[timshannon]

Yeah, having strongly typed queries is something I hadn't considered, and I can see their usefulness.

[ghost]

You can do code gen very simply by running a scan on the dB to gen a schema of sorts.
Then gen go code from the schema.

Even just generating named strings for the various places where strings are using in the current API is a developer productivity win.

Gets you nothing at run time, but it gets the ball rolling towards opportunities to use less reflection moving forward.

[rmg]

For my own uses, the main advantage of generated code with strong typing would be in the removal of most of the reflection overhead.

Considering how much of the code in bolthold is dedicated to type inspection, I think a codegen approach would be a completely different project.

I imagine you would want to split bolthold in to a lower layer that doesn't do all those checks and then the upper layer would either be the current interface{} based API or the generated code that goes in to a project. It seems to me like maybe that lower layer might actually just be boltdb itself. 🤔

[timshannon]

In my personal tests, the reflection overhead didn't come anywhere near the cost of disk access, even with SSDs or running against /dev/shm.

If you remove the need to remove reflection from the generated code, where you only care about type safety, then this becomes much more doable.

[rmg]

In my personal tests, the reflection overhead didn't come anywhere near the cost of disk access, even with SSDs or running against /dev/shm.

In my own code I've had to write wrapper types that normalize/denormalize shared nested objects. Without this the I/O overhead was unbearable. I hadn't verified but suspected that the reflection might have been the next bottleneck.

If you remove the need to remove reflection from the generated code, where you only care about type safety, then this becomes much more doable.

Very true. Based on my above description, you can probably imagine why generated code wouldn't be as high priority to me since I need to write typed wrappers myself anyway.

[timshannon]

In my own code I've had to write wrapper types that normalize/denormalize shared nested objects.

I find that interesting, as Bolthold only ever cares about the highest level fields on structs. My tests were with very simple objects, so if the cost of reflecting a highly nested object is greater than a simple one (which very likely is the case), then I can see how that could get expensive.

[ghost]

I see what you both mean.

Ok here is another approach. You dont get rid of the reflection but you can at least get strong typing by making a simple code gen that scans the db for "Tables" and "field" names.
It generates them as constants, so with your golang queries you can stop using strings everywhere.
Its very much a convenience thing, and i guess useful when the number of types you store gets big.