Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Custom deserializer for avro raw format #843

Merged
merged 4 commits into from
Oct 11, 2021

Conversation

bgranvea
Copy link
Contributor

@bgranvea bgranvea commented Oct 5, 2021

This PR implements #833.

It allows the decoding of Avro raw binary format, with a mapping configuration similar to what already exists for Protobuf, for example:

        akhq:
          connections:
            my-cluster:
              deserialization:
                avro-raw:
                  schemas-folder: "/app/avro_schemas"
                  topics-mapping:
                    - topic-regex: "data.*"
                      value-schema-file: "Schema.avsc"

A couple of remarks regarding this PR:

  • it only addresses Avro "raw" binary format, that is objects encoded with low-level DatumWriter (no header), which is my use case. It could be extended to support single-object encoding (https://avro.apache.org/docs/current/spec.html#single_object_encoding) and it could also be interesting to decode object container files (https://avro.apache.org/docs/current/spec.html#Object+Container+Files) which embed the schema
  • there is no support for schema referencing another schemas
  • I'm not a Gradle specialist but I didn't manage to have Avro schemas used for unit tests that are compiled by Gradle Avro plugin as well as also packaged as resources, so I ended up duplicating the files...
  • I modified AvroDeserializer so that displayed JSON keeps the same field order than Avro schema. This is convenient for unit tests and I think it is also for the user

@tchiotludo
Copy link
Owner

@bgranvea for that !
There is some falling test, can you look at please ? thanks 👍

@bgranvea
Copy link
Contributor Author

bgranvea commented Oct 9, 2021

Sorry for that, it was difficult to setup my dev environment, unit tests don't seem to work at all on Windows (some weird Kafka errors), I finally managed to get it work with WSL.

I fixed most of the unit tests, but there is still one failing test. I don't understand because I have the same error when I switch to the dev branch so this doesn't seem to be related to my PR. Any idea?

  org.akhq.controllers.SchemaControllerTest

    ✔ deleteNotExistApi()
    ✔ crud()
    ✘ listApi()

      org.opentest4j.AssertionFailedError: expected: <5> but was: <3>
          at org.junit.jupiter.api.AssertionUtils.fail(AssertionUtils.java:55)
          at org.junit.jupiter.api.AssertionUtils.failNotEqual(AssertionUtils.java:62)
          at org.junit.jupiter.api.AssertEquals.assertEquals(AssertEquals.java:150)
          at org.junit.jupiter.api.AssertEquals.assertEquals(AssertEquals.java:145)
          at org.junit.jupiter.api.Assertions.assertEquals(Assertions.java:510)
          at org.akhq.controllers.SchemaControllerTest.listApi(SchemaControllerTest.java:35)

I also noticed this in Record:

    @JsonIgnore
    private SchemaRegistryClient client;

    private ProtobufToJsonDeserializer protobufToJsonDeserializer;

Shouldn't this field also be annotated with JsonIgnore?

@tchiotludo
Copy link
Owner

Test are now working, I have some flakky test I never found a way to fix.
That for pointing the missing @JsonIgnore 👍 it's fixed on dev branch.

All seems to be good, just please add some documentation on the README.md about this feature and I will merged that one

Thanks for all 👍

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants