Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Make function pinning asynchronous #21

Open
vsreekanti opened this issue May 26, 2020 · 0 comments
Open

Make function pinning asynchronous #21

vsreekanti opened this issue May 26, 2020 · 0 comments

Comments

@vsreekanti
Copy link
Member

Right now, the management server synchronously waits for functions to pin, which can cause huge backlogs of status updates. This is bad because -- when the function takes a long time to pin -- on the next pin round, it will think there was a huge influx of thousands of requests, and it will try to add some absurd number of replicas.

Two changes need to be made:

  1. Status updates should be eagerly pulled of the queue (i.e., dequeue everything that's there, not one per loop) because otherwise they get starved after long pin operations.
  2. Pins should be made asynchronous. In the interim, there should be a set of sequestered executors that are waiting for pin responses. Those executors should not be used for other pin operations. If the pin succeeds, put them into function_locations. If it fails, put it back into the pool of available executors.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant