gRPC/Python high-level Client

[dgzlopes]

I've been building a high-level Python client [0] for Liftbridge.

First time building something more than a basic POC with gRPC, so the code might be a little rough. It's still early stage, but I'm trying to follow the features from the Go client [1] and making them more Pythonic.

Any thoughts, reviews and/or contributions are highly welcome.

[0] https://github.com/dgzlopes/python-liftbridge
[1] https://github.com/liftbridge-io/go-liftbridge

[tylertreat]

Awesome! I was planning to get to a Python client, glad you got a head start. I will definitely take a look. Let me know if you need any help.

[tylertreat]

Also FYI there is a rather large refactor going on that adds in support for stream partitioning. This is a breaking change that needs to be done prior to a stable 1.0 release. It's still WIP and I'm working through the changes needed for the Go client. I can make the changes to the Python client once it's ready.

[dgzlopes]

@tylertreat Sure! Feel free to make any changes. Any PR is welcome. Also, the status of the project is a little dirty... So any improvements in design/organization/best-practices would be great.

Today I added a couple of basic improvements for making development easier. Scaffolding for logging, version file, tooling, sub/pub debug flag... I hope now the project is a little bit cleaner.

Actually, I'm a little lost with the ACK functionality and with finding a good way to test all the gRPC stuff. Would love any pointers/contributions on both!

[tylertreat]

Yeah, testing is tricky. The Go client actually just starts the server in the tests rather than mocking. Could do similar for other clients. For example, the NATS Python client does this: https://github.com/nats-io/nats.py/blob/master/tests/utils.py

What's your question on the acking?

[dgzlopes]

Interesting. I'm going to research more the server option then. Thanks!

Sorry if It looks obvious but... The ack related question is that, if I use AckPolicyLeader() or AckPolicyAll() as a client... Should I expect a confirmation response? Or wait? Or is all handled by the server?

I think it's closely related to ackInbox usage, and what happens if none is set.

[tylertreat]

AckPolicyLeader means the server will send an ack once the stream leader has stored the message. AckPolicyAll means the server will send an ack once the in-sync replica set have all stored the message.

From the client perspective, Publish should wait for an ack if AckPolicyLeader or AckPolicyAll is set (i.e. an ack is expected from the server) and a timeout is specified. This timeout is passed through the gRPC context. Here's the server-side code for handling publishes/acking.

I also plan to introduce a "streaming" publish API that would be for higher throughput publishing rather than just synchronous publishes for performance reasons.

[tylertreat]

To clarify, with the Publish API the server handles the ack and sends it in the response back to the client: https://github.com/liftbridge-io/liftbridge-grpc/blob/f254d5f49c85dfa7600b075f2919ff17ef66e571/api.proto#L52-L55

AckInbox is if the client wants to take on the responsibility of handling acks, but if they use the Publish API that's not necessary.

[dgzlopes]

Sorry for late response @tylertreat, I was on vacations! Thank you for the insights on ACK usage, really helpful. This week I'm going to work on both issues discussed here as I have some bandwidth to work on the client.

[LaPetiteSouris]

Hi guys.

I just pick up the subject of Python client. @dgzlopes thanks for the work.

It seems that it is outdated with version 1.0.0 of the api, so I did make several rough update to get basic publish subscribe mechanism back to working order.

I think that given the huge popularity of Python, it's kinda important to have a good python client for Liftbridge. I intend to dedicate a bit of my time to the subject, as I also need a python client for my projects.

Do you guys have any plan to move forward to an operational client compatible with api version 1.0.01 ? Any road-map or planning , or at least any "official" repo so I can start my contribution ?

[tylertreat]

@LaPetiteSouris Thanks for picking up the Python client. It was going to be next on my list after the Java one, but I haven't been able to get to that yet.

There isn't an official repo yet for the Python client. My thought is to move clients to the liftbridge-io org once they are mature/stable enough. I think you can either use @dgzlopes repo or fork it if you want to use that as a foundation. Then when we're ready we can move it to an "official" repo.

[dgzlopes]

@LaPetiteSouris If you want to use my repo as a foundation, I'm happy to give you write access (as a collaborator) :)

[LaPetiteSouris]

@LaPetiteSouris If you want to use my repo as a foundation, I'm happy to give you write access (as a collaborator) :)

thanks. That will be my pleasure :)

I will try my best to make it compatible with api version 1.0.0