Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

ClientTelemetryTests flaky #2728

Closed
j82w opened this issue Sep 13, 2021 · 3 comments · Fixed by #2777
Closed

ClientTelemetryTests flaky #2728

j82w opened this issue Sep 13, 2021 · 3 comments · Fixed by #2777
Assignees
Labels
Engineering engineering improvements (CI, tests, etc.)

Comments

@j82w
Copy link
Contributor

j82w commented Sep 13, 2021

The trace even shows the CPU as NaN: DocDBTrace Information: 0 : System Usage recorded by telemetry is : {"systemHistory":[{"dateUtc":"2021-09-13T15:25:24.0897374Z","cpu":NaN,"memory":11566524.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.6365,"availableThreads":32764,"minThreads":8,"maxThreads":32767}}]}

This causes the following assert to fail.
Assert.AreEqual failed. Expected:<2>. Actual:<1>. System Information Count doesn't Match; [{"metricInfo":{"metricsName":"MemoryRemaining","unitName":"MB","mean":10821.59375,"count":1,"min":10822.0,"max":10822.0,"percentiles":{"50":10821.6240234375,"90":10821.6240234375,"95":10821.6240234375,"99":10821.6240234375,"99.9":10821.6240234375}}}]

@j82w j82w added the Engineering engineering improvements (CI, tests, etc.) label Sep 13, 2021
@sourabh1007
Copy link
Contributor

It will return CPU usage as NaN only in following cases:

  • If Native Function (Windows : kernel32.dll, Linux: /proc/stat) is not returning any value.
  • If totalElapsedTime is zero means difference between 2 consecutive reading are zero.

Related PR is out to fix this isue

@sourabh1007
Copy link
Contributor

Fix will be available in the latest Direct package.

@j82w
Copy link
Contributor Author

j82w commented Sep 15, 2021

Please keep this issue open until the direct package is updated and the assert is re-enabled in the test. Without an issue to track it the assert will not get enabled again.

@j82w j82w reopened this Sep 15, 2021
@j82w j82w closed this as completed in #2777 Oct 6, 2021
j82w added a commit that referenced this issue Oct 6, 2021
* Adds improved 449 retry logic to be more aggressive on retries
* Fixes a bug where a InvalidOperationException can occur if a metadata operation fails while multiple item operations are waiting for the result
* Diagnostics: Fix issue causing CPU usage to be NaN intermittently. Related Issue on Github : #2728
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Engineering engineering improvements (CI, tests, etc.)
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants