Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Ccompat api - cannot use protobuf from dotnet or Python #2440

Closed
tomikiss opened this issue Apr 13, 2022 · 2 comments · Fixed by #2496
Closed

Ccompat api - cannot use protobuf from dotnet or Python #2440

tomikiss opened this issue Apr 13, 2022 · 2 comments · Fixed by #2496
Assignees
Labels
type/bug Something isn't working

Comments

@tomikiss
Copy link

tomikiss commented Apr 13, 2022

Hello,

I'm testing out Apicurio using apicurio/apicurio-registry-mem:2.2.2.Final docker image and testing confluent python and dotnet packages using protobuf. When I try to use proto schema I'm receiving status code 422.

It seems like there is an inconsistency how proto schemas represented between the confluent schema-registry and apicurio. After debugging the issue a bit more confluent packages use Protobuf base64 encoded format while apicurio expect the input being string and return "Failed to discover artifact type from content. (HTTP status code 422, SR code 42201)" error.

Python package:

  • confluent-kafka 1.8.2

Endpoint used :
http://localhost:8080/apis/ccompat/v6
Failing request:
POST
http://localhost:8080/apis/ccompat/v6/subjects/test-value/versions
Request content:

'{"schema": "Cgp1c2VyLnByb3RvIiYKBFVzZXISEAoIdXNlck5hbWUYASABKAkSDAoEdGltZRgCIAEoCWIGcHJvdG8z", "schemaType": "PROTOBUF", "references": []}'

Example reproducer following confluent python example - I have a similar one for dotnet as well if that helps.

import proto.user_pb2
from confluent_kafka import SerializingProducer
from confluent_kafka.serialization import StringSerializer
from confluent_kafka.schema_registry import SchemaRegistryClient
from confluent_kafka.schema_registry.protobuf import ProtobufSerializer


def delivery_report(err, msg):
    if err is not None:
        print("Delivery failed for User record {}: {}".format(msg.key(), err))
        return
    print('User record {} successfully produced to {} [{}] at offset {}'.format(
        msg.key(), msg.topic(), msg.partition(), msg.offset()))


def send_message():
    topic = "test"
    schema_registry_conf = {'url': "http://localhost:8080/apis/ccompat/v6"}
    schema_registry_client = SchemaRegistryClient(schema_registry_conf)

    protobuf_serializer = ProtobufSerializer(proto.user_pb2.User,
                                             schema_registry_client,
                                             {'use.deprecated.format': False})

    producer_conf = {'bootstrap.servers': "localhost:9092",
                     'key.serializer': StringSerializer('utf_8'),
                     'value.serializer': protobuf_serializer}

    producer = SerializingProducer(producer_conf)

    print("Producing user records to topic {}. ^C to exit.".format(topic))

    # Serve on_delivery callbacks from previous calls to produce()
    producer.poll(0.0)
    try:
        user_name = "test"
        user = proto.user_pb2.User(userName=user_name)
        producer.produce(topic=topic, partition=0, key="1", value=user,
                             on_delivery=delivery_report)
    except ValueError:
        print("Invalid input, discarding record...")

    print("Flushing records...")
    producer.flush()
    print("Done")


if __name__ == '__main__':
    send_message()

user.proto:

syntax = "proto3";

message User {
	string userName = 1;
}

On python side this is where the conversion happening before sending the schema to the registry:

https://github.com/confluentinc/confluent-kafka-python/blob/e7e066120f05ca6f9c37b18d77f974f1a6cad222/src/confluent_kafka/schema_registry/protobuf.py#L290

The equivalent happening on dotnet side as well when sending protobuf schemas to the registry.

@carlesarnal
Copy link
Member

carlesarnal commented May 11, 2022

First of all, thanks for raising this.

I have created a PR that will solve this issue. We were not handling the situation of encoded protobuf properly in our automated type discovery tool.

I have used your example request to test the fix and now everything should work as expected.

@carlesarnal carlesarnal self-assigned this May 11, 2022
@carlesarnal carlesarnal added the type/bug Something isn't working label May 11, 2022
@carlesarnal carlesarnal linked a pull request May 11, 2022 that will close this issue
@tomikiss
Copy link
Author

Thank you very much for this. Will test it out after merge with .net and python side as well.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
type/bug Something isn't working
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants