feat(BOUN-1168): add dynamic route provider (Discovery Library)

[nikolay-komarevskiy]

Description

API boundary nodes (BNs) have already been introduced into the Internet Computer (IC) landscape. Currently, the agent does not support dynamic discovery of existing API BNs or the dynamic routing of requests to these nodes based on specific strategies, such as latency. This PR enhances the agent by adding functionality for the dynamic routing to API BNs. In a nutshell the newly introduced DynamicRouteProvider is the implementation of the RouteProvider trait.

Example usage 1 - ergonomic:

#[tokio::main]
async fn main() {
    let client = Client::builder().build().expect("failed to build a client");

    // Initialize the agent with a custom dynamic transport layer
    // NOTE: this requires an async context
    let agent = Agent::builder()
        .with_discovery_transport(client)
        .await
        .build()
        .expect("failed to create an agent");
    //
    // ... use agent
    //

}

Example usage 2 - fully customized:

#[tokio::main]
async fn main() {
    let client = Client::builder().build().expect("failed to build a client");
    let seeds = vec![Node::new("ic0.app").expect("domain is invalid")];
    let snapshot = LatencyRoutingSnapshot::new();
    let route_provider = DynamicRouteProviderBuilder::new(snapshot, seeds, client.clone()).build().await;
    let transport = ReqwestTransport::create_with_client_route(Arc::new(route_provider), client.clone()).unwrap();

    // Initialize the agent with a custom dynamic transport layer
    let agent = Agent::builder()
        .with_transport(transport)
        .build()
        .expect("failed to create an agent");
    //
    // ... use agent
    //

}

How Has This Been Tested?

1. Added a mainnet test
2. Created a draft PR and tested on the api_boundary_node_decentralization_test, which is a system-test in the ic-repo.
   NOTE: @adamspofford-dfinity and @lwshang the suggestion to fully remove the fetch_root_key() was probably wrong. As system-test fails without this call. Could You commit a conditional fetch_root_key() invocation, which would work for both mainnet and testnets.

Checklist:

• The title of this PR complies with Conventional Commits.
• I have edited the CHANGELOG accordingly.
• I have made corresponding changes to the documentation.

[r-birkner]

I would just use the same as here: https://github.com/dfinity/ic/blob/master/rs/registry/canister/src/mutations/common.rs#L131-L155

[nikolay-komarevskiy]

i think reusing the Url library is more concise:

pub fn is_valid_domain<S: AsRef<str>>(domain: S) -> bool {
    // Prepend scheme to make it a valid URL
    let url_string = format!("http://{}", domain.as_ref());
    Url::parse(&url_string).is_ok()
}

[r-birkner]

This means that to create an agent with the discovery library, you need to do something like:

let snapshot = LatencyBasedSnapshot::new();
let http_client = Client::builder().build().unwrap();
let route_provider = HealthCheckRouteProvider::new(
            snapshot,
            vec![Node::new(domain_seed)],
            http_client
);
let transport = ReqwestTransport::create_with_client_route(
        Arc::clone(&route_provider) as Arc<dyn RouteProvider>,
        http_client,
    )
    .expect("failed to create transport");
    
let agent = Agent::builder()
        .with_transport(transport)
        .build()?;

Can't we do something like and just provide defaults:

let agent = Agent::builder()
        .with_latency_based_routing()
        .build()?;

or

let agent = Agent::builder()
        .with_health_based_routing()
        .build()?;

[r-birkner]

If someone wants not the default, they can then go through the process above and create their own snapshot/route provider, etc.

[nikolay-komarevskiy]

Eventually this is the direction to go, yes! But i don't want to change the agent during the library migration phase., better in the follow up.