Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

prioritized client updates #17354

Merged
merged 3 commits into from
May 31, 2023
Merged

prioritized client updates #17354

merged 3 commits into from
May 31, 2023

Commits on May 30, 2023

  1. prioritized client updates

    The allocrunner sends several updates to the server during the early lifecycle
    of an allocation and its tasks. Clients batch-up allocation updates every 200ms,
    but experiments like the C2M challenge has shown that even with this batching,
    servers can be overwhelmed with client updates during high volume
    deployments. Benchmarking done in #9451 has shown that client updates can easily
    represent ~70% of all Nomad Raft traffic.
    
    Each allocation sends many updates during its lifetime, but only those that
    change the `ClientStatus` field are critical for progressing a deployment or
    kicking off a reschedule to recover from failures.
    
    Add a priority to the client allocation sync and update the `syncTicker`
    receiver so that we only send an update if there's a high priority update
    waiting, or on every 5th tick. This means when there are no high priority
    updates, the client will send updates at most every 1s instead of
    200ms. Benchmarks have shown this can reduce overall Raft traffic by 10%, as
    well as reduce client-to-server RPC traffic.
    
    This changeset also switches from a channel-based collection of updates to a
    shared buffer, so as to split batching from sending and prevent backpressure
    onto the allocrunner when the RPC is slow. This doesn't have a major performance
    benefit in the benchmarks but makes the implementation of the prioritized update
    simpler.
    
    Fixes: #9451
    tgross committed May 30, 2023
    Configuration menu
    Copy the full SHA
    116ad2a View commit details
    Browse the repository at this point in the history

Commits on May 31, 2023

  1. Configuration menu
    Copy the full SHA
    eae6854 View commit details
    Browse the repository at this point in the history
  2. add note to method about lock

    tgross committed May 31, 2023
    Configuration menu
    Copy the full SHA
    88527e5 View commit details
    Browse the repository at this point in the history