singleton client causes operation to fail

[walt-liuzw]

Describe the bug
I use the netcore framework to build microservices and inject ZeebeClient into StartUp.

Then, I call the process in the Service layer

When I deploy on k8s, the previous calls return the correct result, but after a period of time (about 30min), it will cause a timeout.

Expected behavior
I hope that the singleton client always guarantees success when called

Enviroment (please complete the following information):

• OS: aliyun k8s

[ChrisKujawa]

Hey thanks,

for using the c#-client and opening this issue. Is it correct that you use an await without an async method? You should probably mark your deploy method async and return the corresponding task.

I hope that the singleton client always guarantees success when called

How? In a distributed system requests can always fail. You need to handle these kind of errors, with a strategy which fits to your needs (e.g. retry etc.).
If you have/see errors it is helpful if you post them, at least the stacktrace.

[walt-liuzw]

thanks for your response,
using ZeebeClient like this:
var client = ZeebeClient.NewZeebeClient(request.ServerUrl);
Task.Run(async () => {
var deployResponse = await client.NewDeployCommand().AddResourceStringUtf8(contentStr, fileName).Send();
workflow = deployResponse.Workflows[0];
}).GetAwaiter().GetResult();

After the microservice is deployed, I initiate an HTTP request on the front end, and it can be executed successfully. But after running the microservice for a period of time,
when I execute the deployment command, I can’t wait for the return response, it timeout

[walt-liuzw]

try{ var deployResponse = await client.NewDeployCommand().AddResourceStringUtf8(contentStr, fileName).Send(); } catch Grpc.Core.RpcException Message: Stream Moved

[ChrisKujawa]

But after running the microservice for a period of time,
when I execute the deployment command, I can’t wait for the return response, it timeout

What do you mean with that?
Lets do an step back.

What version of the client and broker do you use?
How is your Broker configured and your client? Does request topology work?

Greets
Chris

[walt-liuzw]

1. zb-client: 0.11.0
2. zeebe cluster: camunda/zeebe:0.22.0-alpha1
   environment:
   • ZEEBE_LOG_LEVEL=debug
   • ZEEBE_STANDALONE_GATEWAY=true
   • ZEEBE_GATEWAY_CONTACT_POINT=node0:26502
   • ZEEBE_GATEWAY_CLUSTER_PORT=26502
   • ZEEBE_GATEWAY_CLUSTER_HOST=zeebe
   • ZEEBE_GATEWAY_MANAGEMENT_THREADS=3
3. topology:

Brokers:

Address: 172.18.0.4:26501,
Host: 172.18.0.4,
NodeId: 0,
Port: 26501,
Partitions:
PartitionId: 1,
IsLeader: True,
Role: LEADER, PartitionId: 7,
IsLeader: True,
Role: LEADER, PartitionId: 4,
IsLeader: True,
Role: LEADER,

Address: 172.18.0.6:26501,
Host: 172.18.0.6,
NodeId: 1,
Port: 26501,
Partitions:
PartitionId: 2,
IsLeader: True,
Role: LEADER, PartitionId: 8,
IsLeader: True,
Role: LEADER, PartitionId: 5,
IsLeader: True,
Role: LEADER,

Address: 172.18.0.7:26501,
Host: 172.18.0.7,
NodeId: 2,
Port: 26501,
Partitions:
PartitionId: 3,
IsLeader: True,
Role: LEADER, PartitionId: 6,
IsLeader: True,
Role: LEADER

[ChrisKujawa]

Hey thanks for the quick response.

Could you please try to use the latest client version, which is 0.13.

Is that correct that you have replication factor one?

[walt-liuzw]

my project use netcore2.1 (it depends on the company plan)

[ChrisKujawa]

Not sure what you mean with that. Why is this a problem to not use 0.13.0?

[walt-liuzw]

1.my cluster include 3 nodes, 8 partitions, 1 replication factor. is that correct?
2.this my problem:
client 0.13.0 references
<PackageReference Include="Microsoft.Extensions.DependencyInjection.Abstractions" Version="3.1.0" />
<PackageReference Include="Microsoft.Extensions.Logging.Abstractions" Version="3.1.0" />

now my project use netcore2.1. it references
<PackageReference Include="Microsoft.Extensions.DependencyInjection.Abstractions" Version="2.1.1" />
<PackageReference Include="Microsoft.Extensions.Logging.Abstractions" Version="2.1.1" />

so after upgrading the zb-client version 0.13.0, the dll version conflicts

[ChrisKujawa]

Hey @walt-liuzw thanks for pointing that out, wasn't aware of that. I will investigate that. Might be that I need to bump the version than.

Could you try to build the project with lower Microsoft.Extensions.DependencyInjection.Abstractions version and verify with your project whether this helps you or not?

[walt-liuzw]

thanks @Zelldon after upgrading the client 0.13.0. the request is not be blocked. but sometimes it throws exception:ERROR Status(StatusCode=DeadlineExceeded, Detail="Deadline Exceeded")

[ChrisKujawa]

For which request do you see that? Still deployment?

Regarding to you replication factor. Is it correct that you do not want not replicated data yes?

[walt-liuzw]

sorry. maybe it my code bug:
client.NewDeployCommand().AddResourceString(contentStr, Encoding.UTF8, fileName).Send(TimeSpan.FromSeconds(5))

i set the timeout too small. i updated 5s to 15s and tried

[ChrisKujawa]

Yes might be the case. 👍

[ChrisKujawa]

So do you think it is necessary to bump the abstraction dependency version again ? Or is this fine for you now?

[ChrisKujawa]

I think you could also add this to you project.

<PackageReference Include="Microsoft.Extensions.DependencyInjection.Abstractions" Version="3.1.0" />
<PackageReference Include="Microsoft.Extensions.Logging.Abstractions" Version="3.1.0" />

Then it should probably be fine right?

[walt-liuzw]

that is a good suggestion. thanks

[ChrisKujawa]

Ok I will close this issue then. Feel free to reopen it.

[ChrisKujawa]

Hey @walt-liuzw I created a new release where I bumped that version
https://github.com/zeebe-io/zeebe-client-csharp/releases/tag/0.14.0