Initial Refactor for speed control

[Mzack9999]

Proposed changes

Due to the impact of internal changes (remove callbacks, move components from static to component-field and pass via dep-injection) the original PR is put On Hold, and a lighter approach for speed control is implemented here. The speed control will be performed with the following points:

• Deprecating RateLimitPerMinute and adding RateLimitDuration
• Speed control via SDK through types.Option options:
  • BulkSize
  • Threads
  • PayloadConcurrency
  • JsConcurrency
  • RateLimit
  • RateLimitDuration
• Any variation to these settings will be reflected internally to existing and new worker pools at next internal useful iteration (for example changing the bulk size will reflect the next time the internal engine iterate over a new template)

In order to allow autotuning it's first necessary to resolve internal design issues to allow smooth communication between the growing amount of components. As a follow up we can expose the speed control capability via local http endpoint (maybe the same metrics/stats one)

Checklist

• Pull request is created against the dev branch
• All checks passed (lint, unit/integration/regression tests etc.) with my changes
• I have added tests that prove my fix is effective or that my feature works
• I have added necessary documentation (if appropriate)

[tarunKoyalwar]

lgtm !

• suggested 2 possible changes
• if i am not wrong this lays a foundation for handler / controller logic which will be implemented in follow up PRs ( but as of now this is a No-OP behaviour wise )

More thoughts

• Looking at implementation i wonder if we could have created all these sizedpools from a single type/method like types.Options or maybe a option to register a optional handler when intializing a sizedpool created a new sizedpool ex:

swg := NewSizedPool(concurrency).RegisterController("js-pool", autoModeController)

and we would have a autoModeController running in goroutine continiously monitering memory and follow a pre-defined algorithm to rebalance different sizedpools at different locations ( ex: decrease payloadconcurrency first and observe change etc )

that way this could be later on extended to implement a global nuclei level smart rate limit controller

[Mzack9999]

Actually it's a NO-OP only for usages from CLI, while via SDK a change within the input settings, causes the internal pools to follow the new value within types.Option. I think that via cli we can expose the control via the existing http server for stats/metrics or create a new one ad-hoc.

The centralizing of speed control was the central thought at #4919, so that the various resizable pools would have referred to a central unique controller, with the advantage that references were cleaned up automatically, while with the approach you suggested, while easier to implement, requires a very careful removal of handlers, since they are tied to a global go routine. As we have seen this is the source of endless memory issues. The biggest problem with this approach is the logic scattered into ultra-wrapped callbacks with unclear intended closures and circular references.