Constant CPU usage with axum/tower & ConnectOptions

[luizfonseca]

Make sure that these boxes are checked before submitting your issue -- thank you!

• Included below version and environment information
• Included a [Minimal, Complete, and Verifiable example] (https://stackoverflow.com/help/mcve)

NATS version (grep 'name = "nats"' Cargo.lock -A 1)

0.29.0

NATS server version

v2.9.15

rustc version (rustc --version - we support Rust 1.41 and up)

1.68.2

OS/Container environment:

MacOS Ventura 13.1, M1 mac

Steps or code to reproduce the issue:

Hello there. I have been learning Rust and in the process of learning NATS as well.
I have a mock deployment that I manage locally mimicking a production environment using dokku
and noticed something particularly odd about CPU usage.

Important: code runs with --release.

1. Have an axum and tower server with a shared state using Arc:

use std::{net::SocketAddr, sync::Arc};
use axum::{routing::get, Router};
use async_nats::ConnectOptions;

struct SharedState {
   nats: async_nats::Client,
}

#[tokio::main]
async fn main () -> Result<(), anyhow::Error> {
   let nats_client = ConnectOptions::with_user_and_password("user", "pw").connect("url").await?;
   
   // Creates shared state for all handlers
   let shared_state = Arc::new(SharedState {
      nats: nats_client
   });
   
   let app = Router::new()
                   .route("/", get { || async { "Test" } })
                   .with_state(shared_state);
   
    let addr = SocketAddr::from(([0, 0, 0, 0], 3030));
     axum::Server::bind(&addr)
        .serve(app.into_make_service_with_connect_info::<SocketAddr>())
        .await?;
}

2. Observe as CPU usage hovers within 2-3% during the server execution, even without subscribers/publishers

increase in CPU usage after installing async-nats. Blue in the graph showcases the issue.

Expected result:

• CPU behaves normally when idle (see screenshot above for periods without the issue)

Actual result:

• While on idle (no messages or subscribers), CPU sits at 2-3% all the time.
• Occurs when using the async-nats crate
• When reverting to nats 0.24.0, the issue goes away and CPU usage is back to < 1%.

[paolobarbolini]

Could you try increasing the flush_interval?
It's something I noticed too when looking at the code (see point 2 here #905)

[caspervonb]

Aye it's being woken up by the flush interval.
As a work-around, bumping the flush interval can reduce this for the time being.

I'll take a stab at doing a proper waker for this.

[caspervonb]

Not quite resolved yet @Jarema, flush interval still keeps the CPU usage at around 7% on idle.
We need to conditionally kick off a delay after seeing commands.