-
Notifications
You must be signed in to change notification settings - Fork 15
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
Periodically Check For Migration Opportunities (#214)
* proto: add field to set migration check period * scheduler: add function migration thread that sleeps for a fixed amount of time, and a simple test. * scheduler: add logic for check for migration opportunities method * scheduler: start/stop migration thread and fix race condition when removing from in-flight map * tests: add test for function migration thread and fix bug in thread shutdown * util: add method to deep copy a faabric message + test * function call server: add call to add a pending migration to remote hosts * tests: add further testing * mpi: add migration points, link with executor, and tests * tests: fix data race * pr: re-factor methods as suggested in the pr comments * executor: throw exception to properly shutdown executing task * executor: close the loop to alllow for function migration * scheduler: add UNDERFULL scheduling topology hint * Add migration exception, catch in executor * Remove manual message copying * Remove unnecessary migration APIs * Make getMemoryView public * scheduler: get functions to migrate properly * cleanup * mpi: use initLocalQueues() instead of repeating the logic, remove assertion from the method * mpi: remove unused getMpiMockedPendingMigrations * scheduler: remove unused function declarations * scheduler: factor out method to start the function migration thread if necessary * mpi: use a boolean flag to indicate that app has been migrated, and check for flag in barriers to remove used pendingMigrations in the scheduler * proto: make topologyHint a message field, add test for json serialisation, remove unnecessary field to callFunctions, and re-factor necessary calls to callFunctions Co-authored-by: Simon Shillaker <mail@simonshillaker.com>
- Loading branch information
1 parent
5b67a98
commit 4f4628a
Showing
24 changed files
with
1,213 additions
and
53 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,26 @@ | ||
#pragma once | ||
|
||
#include <condition_variable> | ||
#include <mutex> | ||
#include <thread> | ||
|
||
namespace faabric::scheduler { | ||
// Start a background thread that, every wake up period, will check if there | ||
// are migration opportunities for in-flight apps that have opted in to | ||
// being checked for migrations. | ||
class FunctionMigrationThread | ||
{ | ||
public: | ||
void start(int wakeUpPeriodSecondsIn); | ||
|
||
void stop(); | ||
|
||
int wakeUpPeriodSeconds; | ||
|
||
private: | ||
std::unique_ptr<std::thread> workThread = nullptr; | ||
std::mutex mx; | ||
std::condition_variable mustStopCv; | ||
std::atomic<bool> isShutdown; | ||
}; | ||
} |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.