Apicurio/confluent compatibility issues

[janvyhnanek]

I have a question about compatibility between apicurio and confluent ser/des clients.

If an application is written in C# it has to use confluent client since apicurio does not provide such feature. It is possible to make it working with producer on apicurio if producer sets properties apicurio.registry.as-confluent=true, apicurio.registry.use-id=contentId and apicurio.registry.headers.enabled=false. But as a consequence all other consumers with apicurio clients must set parameters apicurio.registry.use-id=contentId, apicurio.registry.as-confluent=true which will ensure usage of Legacy4ByteIdHandler. Also, serialization with confluent client requires schema registered with default groupId as mentioned in #1593.

In my comprehension, if there is even one application which cannot use apicurio clients, all other applications have to conform to confluent standards marked as legacy in apicurio to make it working. Am I right or is there any option to use non legacy ids in this case? It seems that because of one application writen in C# we will have to use only default groupId and legacy id handling everywhere.

Thank you

[EricWittmann]

I think you are exactly right unless you are willing to write your own custom SerDes in C#. Certainly it would be great if we had the bandwidth to implement C# and Python SerDes. That's the situation right now at least. Some issues could be resolved perhaps with a proxy between the confluent client and the registry, where the proxy could utilize the registry API instead of the ccompat API and e.g. add group info.

[EricWittmann]

Other ideas @Apicurio/developers ?

[janvyhnanek]

Thanks for information, it seems that indeed we will have to switch to confluent mode of communication, it is not in our capacity either to create clients for other languages in apicurio mode now.

We have found another problem related to missing groupId in confluent - if we switch to confluent mode, groupId must be always default. However, we still want the possibility to differ between schemas by namespaces. Confluent client works fine, if groupId is registered as null and namespace is specified directly in artifactId (e.g. artifactId=io.test.SchemaName), but apicurio strategy RecordIdStrategy does not work on such schema, because the unqualified name of record is used for schema lookup. GroupId can be switched to default by configuration apicurio.registry.artifact.group-id="default", but I have found no option to make artifactId qualified. Is the right approach to implement our own strategy "QualifiedRecordIdStrategy" where the only difference would getting artifactId by schema.getFullName() instead of schema.getName()? Maybe it would be useful, if you add such configuration for the current strategies?

Thank you

[famarting]

Hi, yes it's a perfectly fine approach to implement your own strategy "QualifiedRecordIdStrategy" we had the strategy interface for that purpose.

JFYI Also there is another interface SchemaResolver that it's also acceptable to re-implement, that may help you in the future if you find a very specific usecase, just have in mind doing so is a very advanced usecase :)

[EricWittmann]

Note to @janvyhnanek - we would welcome a contribution of the QualifiedRecordIdStrategy if you ended up creating it!

[janvyhnanek]

@EricWittmann I have created and tested the class but I have troubles creating a feature branch for pull request, I get an authorization error. Could you grant me the necessary rights or how else should I proceed? Thank you

[famarting]

Hi @janvyhnanek you have to create your pull request from your fork of the repository.

[EricWittmann]

Yes @famartinrh is correct - please create a fork of the repository, then push the feature branch to your fork and submit the PR from that. And thank you in advance for the contribution!