-
Notifications
You must be signed in to change notification settings - Fork 72
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[API-537] Add support for the binary metrics #441
Conversation
Here is the benchmark I used to compare the old vs new approach import timeit
import psutil
def f1():
psutil.virtual_memory().total
psutil.virtual_memory().free
psutil.virtual_memory().used
psutil.swap_memory().total
psutil.swap_memory().free
psutil.cpu_percent()
psutil.cpu_count()
p = psutil.Process()
p.memory_info().rss
p.num_fds()
p.rlimit(psutil.RLIMIT_NOFILE)[1]
p.cpu_times()
p.create_time()
def f2():
v = psutil.virtual_memory()
v.total
v.free
v.used
s = psutil.swap_memory()
s.total
s.free
psutil.cpu_percent()
psutil.cpu_count()
p = psutil.Process()
with p.oneshot():
p.memory_info().rss
p.num_fds()
p.rlimit(psutil.RLIMIT_NOFILE)[1]
p.cpu_times()
p.create_time()
print(timeit.timeit(f1, number=10000))
print(timeit.timeit(f2, number=10000)) |
Implemented the mechanism to compress the metrics we collect in a way that is compatible with the server-side. With this change, the latest Management Center versions are able to show statistics regarding the Python client. For backward compatibility, we also continue sending metrics in a comma separated key-value format (client attributes). In this PR, I also significantly cleaned up the statistics service. In the old code, we were trying to reuse the results of some calls. In the benchmarks I did, I saw that it didn't really improve much. (New approach approx. 0.5 ms, old approach approx. 0.4 ms per statistics collection) This really small difference does not justify the need for complexity so I went with the simpler approach. Also, aligned the log levels with the Java client for the various warnings. Also, bumped the version of psutil to 5.8.0.
c2f301b
to
3e990f1
Compare
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Looking good left some minor comments
Before merging, can you test that if metrics can be sent to management center on windows(python client runs on windows)? some of them like number of file descriptors may not work. |
Yes, I have verified this, on Windows, this feature works with a subset of the stats we sent from Linux. For example, file descriptor count is not sent on Windows, as there is no such thing in Windows, or the load average will not be sent, as it is not implemented there in the Python standard library. |
7ea0d8c
to
ddfc21d
Compare
Codecov Report
@@ Coverage Diff @@
## master #441 +/- ##
==========================================
- Coverage 86.62% 85.95% -0.67%
==========================================
Files 336 340 +4
Lines 17398 17850 +452
==========================================
+ Hits 15071 15343 +272
- Misses 2327 2507 +180
Continue to review full report at Codecov.
|
Implemented the mechanism to compress the metrics we collect in a
way that is compatible with the server-side.
With this change, the latest Management Center versions are able to
show statistics regarding the Python client.
For backward compatibility, we also continue sending metrics in
a comma separated key-value format (client attributes).
In this PR, I also significantly cleaned up the statistics service.
In the old code, we were trying to reuse the results of some
calls. In the benchmarks I did, I saw that it didn't really improve
much. (New approach approx. 0.5 ms, old approach approx. 0.4 ms per
statistics collection) This really small difference does not justify
the need for complexity so I went with the simpler approach.
Also, aligned the log levels with the Java client for the various
warnings.
Also, bumped the version of psutil to 5.8.0.