CPU regression with NonBlockingStatsDClient

[elenatomlinson]

Hi there,

We recently migrated from NonBlockingStatsDClient.java to DataDog's NonBlockingStatsDClient.java. We see significant increase of cpu usage, proportional to number of instances of the client. e.g. single application pod cpu usage in millicores increased from 2 to 10 after migration, then from 10 to 29 after instantiating 3x the number of clients during experimentation.

[conderls]

it seems that the sample rate is out of use when aggregation is enabled. the usage of cpu and memory may increase when monitoring massive messages.

https://github.com/DataDog/java-dogstatsd-client/blob/master/src/main/java/com/timgroup/statsd/NonBlockingStatsDClient.java#L1191-L1209

    private void send(String aspect, final long value, Message.Type type, double sampleRate, String[] tags) {
        if (statsDProcessor.getAggregator().getFlushInterval() != 0 && !Double.isNaN(sampleRate)) {
            switch (type) {
                case COUNT:
                    sampleRate = Double.NaN; // <-- reset sampleRate to NaN
                    break;
                default:
                    break;
            }
        }

        if (Double.isNaN(sampleRate) || !isInvalidSample(sampleRate)) {

            sendMetric(new StatsDMessage<Long>(aspect, type, value, sampleRate, tags) {
                @Override protected void writeValue(StringBuilder builder) {
                    builder.append(this.value);
                }
            });
        }
    }