CosmosDiagnostics: Add active client count

[j82w]

The diagnostics currently only has the number of created clients. This issue is to track adding the number of active clients. This way we can see if a user is properly disposing of clients for certain cases.

[ealsur]

We need to think about scenarios where the customer is doing:

using (CosmosClient client = new CosmosClient(...))

If this is executed concurrently and in high numbers it is still a bad scenario, the number of clients created would be high and the fact that the active clients might be low doesn't mean everything is correct.
Creating and disposing continuously will also lead to Connection limiting or issues, because connections are not closed right away (https://www.aspnetmonsters.com/2016/08/2016-08-27-httpclientwrong/).

[j82w]

There will be a TotalNumberOfClientsCreated and a ActiveNumberOfClientsCreated. If there is some crazy number of total clients it will still be obvious. The active number will tell if the user is disposing of the client or has N clients active. It may also help in scenarios where users think they are disposing of the clients but are not.

[ezgambac]

In our scenario we are having the clients in cache where they are removed after a certain time being unused, so we would be creating and deleting them often.

[kirankumarkolli]

There are genuine scenarios to create multiple clients (ex: multiple accounts). The data needs to dis-ambiguate or at-least have right semantics of interpretation. One option is to have normalized counter like per account #clients (MAX)

[j82w]

For troubleshooting I think having a total active count is sufficient. It doesn't matter if they have 100 SDK instances with 100 different accounts vs 100 SDK instance for the same account. Both will run into the same issue of causing to many connections.

[ezgambac]

We are currently tracking total number of active clients globally, not per account, if that helps as a data point.
We also make an effort to have 1 client per account.

[sourabh1007]

Recommendation is: Customer should have 1 Client Instance per Account. if User have multiple accounts, then there is no other way except creating multiple client instances.

So We should collect how many clients is being created per account. E.g. Customer have 10 accounts and then TotalNumberOfClientsCreated and ActiveNumberOfClientsCreated will say 10 instances are being created and active (we will consider it as a bad practice), but in actual, nothing can be done to reduce the number of client instance.
So These 2 metrics may mislead us. But, if we collect it for each account then, we will have clear picture if we should even look into this direction or not.

[kirankumarkolli]

UserAgent do already cover some aspects of it.
What extra context do we get through this telemetry?

Or if the # createdClients is not relevant why not change to activeClients in userAgent?

[j82w]

The number of created clients is more important than number of active clients. The number of created clients tells if the application is creating multiple instances or recreating instances. If just the number of active instances is recorded it will not be possible to tell if the customer is recreating instance for each call. This is the most common issue observed.

The problem with just capturing total number of clients created is there are scenarios where customers by design recreates instances. This usually occurs is middle tier services where the customer is reselling Cosmos DB. In those scenarios it's not possible to tell how many active clients instance exist. The process having 2 active clients vs 200 is a big difference, and can be helpful in root causing certain issues.

[ezgambac]

In our scenario, we keep the clients in a timed cache, so if they are not accessed within X minutes, they will be disposed of and recreated on the next access. This is done to prevent port/connection exhaustion on our side. From: j82w ***@***.***> Sent: Tuesday, February 8, 2022 1:28 PM To: Azure/azure-cosmos-dotnet-v3 ***@***.***> Cc: Ezequiel Gambaccini ***@***.***>; Comment ***@***.***> Subject: Re: [Azure/azure-cosmos-dotnet-v3] CosmosDiagnostics: Add active client count (#2589) The number of created clients is more important than number of active clients. The number of created clients tells if the application is creating multiple instances or recreating instances. If just the number of active instances is recorded it will not be possible to tell if the customer is recreating instance for each call. This is the most common issue observed. The problem with just capturing total number of clients created is there are scenarios where customers by design recreates instances. This usually occurs is middle tier services where the customer is reselling Cosmos DB. In those scenarios it's not possible to tell how many active clients instance exist. The process having 2 active clients vs 200 is a big difference, and can be helpful in root causing certain issues. - Reply to this email directly, view it on GitHub<https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2FAzure%2Fazure-cosmos-dotnet-v3%2Fissues%2F2589%23issuecomment-1033078871&data=04%7C01%7CEzequiel.Gambaccini%40microsoft.com%7Cc89dc447e9fe4f1a5bbb08d9eb49e7b4%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637799524930745977%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=nDq5twiuGGMGPwHvH4BT9HhRT2OwbOtBT4apWeg%2FImY%3D&reserved=0>, or unsubscribe<https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fnotifications%2Funsubscribe-auth%2FAMTEQJGWTRU3FEIHQRNSSTTU2GDGTANCNFSM474MRYAA&data=04%7C01%7CEzequiel.Gambaccini%40microsoft.com%7Cc89dc447e9fe4f1a5bbb08d9eb49e7b4%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637799524930745977%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=U%2FvVpXdZbD6mQyDLh90fU8jPBiGtmYnwHd7bBY17JpU%3D&reserved=0>. Triage notifications on the go with GitHub Mobile for iOS<https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fapps.apple.com%2Fapp%2Fapple-store%2Fid1477376905%3Fct%3Dnotification-email%26mt%3D8%26pt%3D524675&data=04%7C01%7CEzequiel.Gambaccini%40microsoft.com%7Cc89dc447e9fe4f1a5bbb08d9eb49e7b4%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637799524930745977%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=5Y1L3AR%2Bal1%2FYzoM7brInWX0mS6zSGpsQCOAxlMl92s%3D&reserved=0> or Android<https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fplay.google.com%2Fstore%2Fapps%2Fdetails%3Fid%3Dcom.github.android%26referrer%3Dutm_campaign%253Dnotification-email%2526utm_medium%253Demail%2526utm_source%253Dgithub&data=04%7C01%7CEzequiel.Gambaccini%40microsoft.com%7Cc89dc447e9fe4f1a5bbb08d9eb49e7b4%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637799524930745977%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=j1C3koVA6EavkcgygCMxgrWPBDTVxuXNNiQk3A1MduM%3D&reserved=0>. You are receiving this because you commented.Message ID: ***@***.***>