expose transfer bytes total (count) rather than just gauge of current value

[ntabris]

Currently the worker Prometheus endpoint exposes transfer_incoming_bytes and transfer_outgoing_bytes as gauges—i.e., the current value at a single point in time.

A better way to expose this sort of data is as a monotonically increasing count metric type (this should be exposed as transfer_incoming_bytes_total and transfer_outgoing_bytes_total).

It's easy to get rate from an accumulated count, but you can't get accurate count from a sampled rate.

[mrocklin]

I'm in favor of this generally

…

On Thu, Oct 6, 2022, 9:36 AM Nat Tabris ***@***.***> wrote: Currently the worker Prometheus endpoint exposes transfer_incoming_bytes and transfer_outgoing_bytes as gauges—i.e., the current value at a single point in time. A better way to expose this sort of data is as a monotonically increasing count metric type (this should be exposed as transfer_outgoing_bytes_total and transfer_outgoing_bytes_total). It's easy to get rate from an accumulated count, but you can't get accurate count from a sampled rate. — Reply to this email directly, view it on GitHub <#7123>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AACKZTGM2ZFQI42QDXJL2F3WB35XXANCNFSM6AAAAAAQ6ZYKTI> . You are receiving this because you are subscribed to this thread.Message ID: ***@***.***>

[hendrikmakait]

We expose transfer_incoming_bytes and transfer_outgoing_bytes as gauges as these point-in-time metrics inform data transfer throttling and should therefore be visible. If you find exposing transfer_{incoming|outgoing}_bytes_total useful, we can add those. We should just make sure we have a good definition for those metrics (see #6936 (comment)) for a more in-depth comment).

[fjetter]

@ntabris is there anything left to do or are your concerns addressed?

[ntabris]

I think we don't expose actually bytes transferred over time, i.e., what @crusaderky calls "cumulative" values in #6936 (comment)

That's what I was asking for. If it's not high-value, feel free to ignore for now though.

For context, host metrics can tell us how much data moves in/out of each worker. What it can't exactly tell us (at least not easily) is how much of that is transfer vs data moving into/out of cluster (e.g., S3). I think it would be nice if Dask could tell us how many bytes of host network traffic is for transfer.

[crusaderky]

See also #7357

[gjoseph92]

I would also find this useful for benchmarking. Total amount of data transferred is a useful metric to compare when working on changes to scheduling.