Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

How to set a memory limit for a sandboxed processor? #1555

Open
ppedziwiatr opened this issue Nov 29, 2022 · 10 comments
Open

How to set a memory limit for a sandboxed processor? #1555

ppedziwiatr opened this issue Nov 29, 2022 · 10 comments
Labels
enhancement New feature or request

Comments

@ppedziwiatr
Copy link

ppedziwiatr commented Nov 29, 2022

I assume that Sandboxed Processors work underneath as a separate node.js Workers.

How can I pass the memory configuration to the Sandboxed processor?

E.g. the resourceLimits. maxOldGenerationSizeMb from the original Worker api
https://nodejs.org/api/worker_threads.html#worker_threads_new_worker_filename_options

@manast
Copy link
Contributor

manast commented Nov 29, 2022

Currently they are implemented as spawn processes, so there is no such thing as memory configuration AFAIK.

@ppedziwiatr
Copy link
Author

ppedziwiatr commented Nov 30, 2022

Thanks for the answer. What is the advantage of using a spawned process (in comparison to worker threads)?

Also - when running a node.js app, there's an option to set the --max-old-space-size param.
E.g. node index.js --max-old-space-size=8000 - which would set the heap size to 8GB.

I believe setting this param for the spawned process (it is spawned via spawn from node:child_process, right?)
could effectively limit the memory used by sandboxed processor...

@ppedziwiatr
Copy link
Author

ppedziwiatr commented Nov 30, 2022

Ok, if I understand properly, they are implemented as forks, (not as spawned processes) -https://github.com/taskforcesh/bullmq/blob/master/src/classes/child-pool.ts#L114

Which is even better, because I believe that each fork has a new V8 instance created = better isolation from the "host" envrionment.

And if that's really the case - it should be possible to set the memory limit per forked process - with sth like:

require('child_process').fork("newProcess.js", { 
        execArgv: ['--max-old-space-size=128']
    });)

https://stackoverflow.com/a/37596117

Would you consider adding the max available memory setting in the Worker configuration options?

@manast
Copy link
Contributor

manast commented Nov 30, 2022

Yeah well, a fork is also a spawn, the only difference is that it creates a new V8 for you, but it is still a new process so there is no practical difference (with spawn you will need to call the node cmd line tool instead).
But sure, we can add support for setting custom flags to child processes, the tricky part is how would such an API look like so that it is consistent with the current design.

@ppedziwiatr
Copy link
Author

ppedziwiatr commented Nov 30, 2022

For me personally - the simplest solution (assuming, that this is more of an "advanced" feature) - with sth like this:

new Worker(updateQueueName, updateProcessor, {
    concurrency: workersConfig.update,
    metrics: {
      maxDataPoints: MetricsTime.ONE_WEEK,
    },
    sandboxExecArgv: ['--max-old-space-size=128', '--foo=bar']
  });

would be awesome!

The sandboxExecArgv would be obviously effectively used only for the sandboxed processors.

@ppedziwiatr
Copy link
Author

Hey, any updates? :-)

@ppedziwiatr
Copy link
Author

Hey - just asking while reviewing our internal issues ;-) - is there any timeline for implementing this feature?

@ppedziwiatr
Copy link
Author

ppedziwiatr commented May 29, 2023

Hey, do you plan to implement this feature?

EDIT: I see, that support for worker_threads has been recently added (0820985) - does this new implementation support https://nodejs.org/api/worker_threads.html#workerresourcelimits ?

@manast
Copy link
Contributor

manast commented May 29, 2023

It does not seem to be a lot of interest in this feature. However if someone creates a PR with a clean and simple implementation I do not have anything against merging it.

@opexu
Copy link

opexu commented Sep 23, 2024

i need this feature too

github-actions bot pushed a commit that referenced this issue Sep 30, 2024
# [5.14.0](v5.13.2...v5.14.0) (2024-09-30)

### Features

* **worker-thread:** allow passing Worker options ([#2791](#2791)) ref [#1555](#1555) ([6a1f7a9](6a1f7a9))
alexandresoro pushed a commit to alexandresoro/ouca that referenced this issue Oct 2, 2024
This PR contains the following updates:

| Package | Type | Update | Change |
|---|---|---|---|
| [bullmq](https://bullmq.io/) ([source](https://github.com/taskforcesh/bullmq)) | dependencies | minor | [`5.13.2` -> `5.15.0`](https://renovatebot.com/diffs/npm/bullmq/5.13.2/5.15.0) |

---

### Release Notes

<details>
<summary>taskforcesh/bullmq (bullmq)</summary>

### [`v5.15.0`](https://github.com/taskforcesh/bullmq/releases/tag/v5.15.0)

[Compare Source](taskforcesh/bullmq@v5.14.0...v5.15.0)

##### Features

-   **worker-fork:** allow passing fork options ([#&#8203;2795](taskforcesh/bullmq#2795)) ([f7a4292](taskforcesh/bullmq@f7a4292))

### [`v5.14.0`](https://github.com/taskforcesh/bullmq/releases/tag/v5.14.0)

[Compare Source](taskforcesh/bullmq@v5.13.2...v5.14.0)

##### Features

-   **worker-thread:** allow passing Worker options ([#&#8203;2791](taskforcesh/bullmq#2791)) ref [#&#8203;1555](taskforcesh/bullmq#1555) ([6a1f7a9](taskforcesh/bullmq@6a1f7a9))

</details>

---

### Configuration

📅 **Schedule**: Branch creation - At any time (no schedule defined), Automerge - At any time (no schedule defined).

🚦 **Automerge**: Disabled by config. Please merge this manually once you are satisfied.

♻ **Rebasing**: Whenever PR becomes conflicted, or you tick the rebase/retry checkbox.

🔕 **Ignore**: Close this PR and you won't be reminded about this update again.

---

 - [ ] <!-- rebase-check -->If you want to rebase/retry this PR, check this box

---

This PR has been generated by [Renovate Bot](https://github.com/renovatebot/renovate).
<!--renovate-debug:eyJjcmVhdGVkSW5WZXIiOiIzOC4xMDEuMSIsInVwZGF0ZWRJblZlciI6IjM4LjEwNi4yIiwidGFyZ2V0QnJhbmNoIjoibWFpbiIsImxhYmVscyI6WyJkZXBlbmRlbmNpZXMiXX0=-->

Reviewed-on: https://git.tristess.app/alexandresoro/ouca/pulls/178
Reviewed-by: Alexandre Soro <code@soro.dev>
Co-authored-by: renovate <renovate@git.tristess.app>
Co-committed-by: renovate <renovate@git.tristess.app>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

No branches or pull requests

3 participants