Is there any way to generate avro schema file using go struct?

[iyashjayesh]

I'm currently working on a project where I need to generate an Avro schema file from a Go struct. I want to know if there is any existing solution that can assist me in achieving this.

Ideally, I would like to find a way to automatically generate the Avro schema based on the defined struct in Go. This would greatly simplify the process of defining and maintaining the schema separately.

If anyone has any insights, recommendations, or example code snippets on how to accomplish this task, I would greatly appreciate your help!

Example:
For instance, consider the following Go struct:

type User struct {
    ID        int    `avro:"id" json:"id"`
    Name      string `avro:"name" json:"name"`
    Age       int    `avro:"age" json:"age"`
    Email     string `avro:"email" json:"email"`
    Timestamp int64  `avro:"timestamp" json:"timestamp"`
}

I would like to generate an Avro schema file that reflects the structure and field types of this Go struct automatically. This would enable me to share the schema with other systems.

Any guidance or suggestions on how to achieve this would be greatly appreciated. Thank you in advance for your help!

[nrwiersma]

Nothing like this existing in this library.

A method would be to use reflect to parse the struct an make the required Schema. From there you could JSON marshal it into a schema string.

[TuSKan]

I have a code for that. From Go Struct to avro.Schema and vice versa.
@nrwiersma Do you have any interest to include this feature in the library ?

By the way, in my implementation I'm loosing the properties.
As a avro.Field is not a avro.Schema, the properties belong to the field and not to the type of the field which is a avro.Schema
Any idea on that?

[nrwiersma]

@TuSKan The library already has Schema -> Struct in cmd/avrogen. I would gladly accept Struct -> Schema, perhaps in cmd/schemagen?

By the way, in my implementation I'm loosing the properties.

The fields should maintain the properties as it inherits the properties struct, like most Schemas (not all Schemas can have properties), but like most Schemas, we dont marshal it back to JSON. It should be possible to do though. Check MarshalJSON.

[iyashjayesh]

can anyone share any example?

[TuSKan]

My problem is different from avrogen, I need to generate a Go Struct as a reflect.Type dynamically to Encode or Decode in runtime, rather than have a code gen to use further.

I looked to MarshalJSON, as you mention there is no properties there.

Why a avro.Field is not a avro.Schema ? Just missings a Fingerprint...

[nrwiersma]

These are the properties on the Field: https://github.com/hamba/avro/blob/main/schema.go#L537

Why a avro.Field is not a avro.Schema ? Just missings a Fingerprint...

The Field is not a Schema by spec.

There is no need to create a struct on the fly, and you can just decode/encoder from a map[string]any.

[TuSKan]

I build a recursive func for avro.Schema, so for avro.RecordSchema I lose the properties of the Field. A Fingerprint method for Field doesn't make sense?

There is no need to create a struct on the fly, and you can just decode/encoder from a map[string]any.

mp[string]any is not a option to me. I work with huge files and databases, I need to have a real schema.

[nrwiersma]

Fields are not Schemas, regardless of the fingerprint. Assuming you parsed a non-canonical schema, any properties of the Field still exist and are accessible in the same way they are on a Schema.

[TuSKan]

@iyashjayesh let me know if this code helps you
https://gist.github.com/TuSKan/35ae72fac0ec6aa9e71a0ffbedd4790f

[nrwiersma]

Closing as this is not a planned feature of the library.

[rayokota]

@iyashjayesh let me know if this code helps you https://gist.github.com/TuSKan/35ae72fac0ec6aa9e71a0ffbedd4790f

@TuSKan , do you mind providing the definition of avroDefaultField in that method? Thanks in advance!

[TuSKan]

I just update the gist with avroDefaultField method