Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

chore: Gracefully shutdown the embedder #9693

Merged
merged 6 commits into from
May 6, 2024
Merged

Conversation

Dschoordsch
Copy link
Contributor

@Dschoordsch Dschoordsch commented Apr 30, 2024

Description

Fixes #9693
End all streams on receiving a kill signal to properly shutdown the embedder. Ending just the merged stream does not work in the current situation as it will wait indefinitely if all the workers are idle.

Demo

Nothing to see here.

Testing scenarios

  • run the embedder outside of pm2
    sh -c 'export SERVER_ID=7 && exec node dev/embedder.js'
    
  • send SIGINT via Ctrl+C
  • see the process ending gracefully

Final checklist

  • I checked the code review guidelines
  • I have added Metrics Representative as reviewer(s) if my PR invovles metrics/data/analytics related changes
  • I have performed a self-review of my code, the same way I'd do it for any other team member
  • I have tested all cases I listed in the testing scenarios and I haven't found any issues or regressions
  • Whenever I took a non-obvious choice I added a comment explaining why I did it this way
  • I added the label Skip Maintainer Review Indicating the PR only requires reviewer review and can be merged right after it's approved if the PR introduces only minor changes, does not contain any architectural changes or does not introduce any new patterns and I think one review is sufficient'
  • PR title is human readable and could be used in changelog

@github-actions github-actions bot added size/m and removed size/s labels Apr 30, 2024
throw err
},
return: async () => {
await Promise.allSettled(iterators.map((iterator) => iterator.return?.()))
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This refactoring was done so we can define this return function. This is necessary so all streams are notified of the return. The async generator function we had previously would hang in await capability.promise and handle the return only afterwards. If all embedders are idle, the promise would never resolve.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

interesting! really good find having to define our own return here

@@ -6,5 +6,5 @@ export const logMemoryUse = () => {
const {rss} = memoryUsage
const usedMB = Math.floor(rss / MB)
console.log('Memory use:', usedMB, 'MB')
}, 10000)
}, 10000).unref()
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This was an attempt to regularly end the event loop. It will continue if there are still timers running unless these are unrefed. This did not allow proper shutdown as there are many things in dependencies still running, but it's good practice nonetheless if we're not keeping a reference to the timer to allow cleanup.

@Dschoordsch Dschoordsch merged commit 695ccad into master May 6, 2024
5 checks passed
@Dschoordsch Dschoordsch deleted the chore/9677/gracefullyDie branch May 6, 2024 15:28
@github-actions github-actions bot mentioned this pull request May 7, 2024
24 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants