Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Implement shared-memory communication #7

Open
3 tasks
mcopik opened this issue Feb 16, 2023 · 0 comments
Open
3 tasks

Implement shared-memory communication #7

mcopik opened this issue Feb 16, 2023 · 0 comments
Labels
enhancement New feature or request help wanted Extra attention is needed

Comments

@mcopik
Copy link
Contributor

mcopik commented Feb 16, 2023

Serverless functions are becoming more powerful, and AWS Lambda now supports using up to 6 vCPUs per function. We could benefit by running multiple "workers" as threads on a single function and letting them communicate locally. We cannot use processes within a single function as that would require shmem, which is not supported on AWS Lambda.

  • Introduce a memory-based communication channel.
  • Adjust the function launcher to support a hierarchy of functions - local and external ones.
  • Implement in collectives and P2P communication the selection of external and local channels based on the recipient.

This might require making collectives to be natively multi-protocol first #6.

@mcopik mcopik added enhancement New feature or request help wanted Extra attention is needed labels Feb 16, 2023
@mcopik mcopik mentioned this issue Feb 16, 2023
5 tasks
@mcopik mcopik moved this to Todo in Full MPI support Feb 16, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request help wanted Extra attention is needed
Projects
Status: Todo
Development

No branches or pull requests

1 participant