Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Server: Worker refactor #704

Merged
merged 1 commit into from
Jan 17, 2023
Merged

Server: Worker refactor #704

merged 1 commit into from
Jan 17, 2023

Conversation

jaymell
Copy link
Contributor

@jaymell jaymell commented Nov 15, 2022

  • Add webhook-http-client, which is based on hyper, enabling several
    critical improvements:
    • Default filtering of endpoints that resolve to private
      address space, with ability to add CIDR exceptions as needed
    • Case-sensitive HTTP1 headers -- although HTTP spec states that
      headers are not case-sensitive, in practice many widely
      deployed web-servers expect to support case-sensitive headers.
      This requires a fork of hyper, which currently has but does
      not expose support for case-sensitive headers.
    • Assurance that the host header appears first in outgoing HTTP requests.
      Although HTTP spec does not prescribe an order for headers, in
      our experience, many web-servers will fail if the host header does
      not appear first in request headers. We ensure this by manually
      adding the header ourselves rather than relying on hyper to do so.
  • Use openssl instead of rusttls for outbound webhooks. In our experience,
    many widely deployed webservers use "weak" ciphers that rusttls refuses to
    support, leaving us with no choice but to rely on openssl libraries.
  • Support for soft-limiting of concurrent worker tasks. Unbounded concurrent worker
    tasks can easily overwhelm a system. This sets a default of 500 tasks with support
    for configuring other limits or setting unlimited (0) concurrent tasks
  • Refactor message dispatch code for clarity. Dispatch code is now broken into
    separate methods: prepare_dispatch, make_http_call,
    handle_successful_dispatch, and handle_failed_dispatch with the dispatch itself
    having different types depending on its stage in the dispatch process.
    Other function names have also been changed for improved clarity.
  • Prevent worker from shutting down when there are active tasks.
  • Avoid multiple DB calls for message-destination insert/retrieval

@jaymell jaymell changed the title Worker refactor 2 Refactor worker Nov 15, 2022
@jaymell jaymell marked this pull request as ready for review November 15, 2022 23:31
@jaymell jaymell force-pushed the worker-refactor-2 branch 2 times, most recently from f56534d to dfa027c Compare November 18, 2022 00:10
@jaymell jaymell requested a review from tasn November 18, 2022 00:30
@jaymell jaymell force-pushed the worker-refactor-2 branch 5 times, most recently from b9c597b to b7c7f4e Compare January 13, 2023 23:21
* Add webhook-http-client, which is based on hyper, enabling several
  critical improvements:
  * Default filtering of endpoints that resolve to private
    address space, with ability to add CIDR exceptions as needed
  * Case-sensitive HTTP1 headers -- although HTTP spec states that
    headers are not case-sensitive, in practice many widely
    deployed web-servers expect to support case-sensitive headers.
    This requires a fork of hyper, which currently has but does
    not expose support for case-sensitive headers.
  * Assurance that the `host` header appears first in outgoing HTTP requests.
    Although HTTP spec does not prescribe an order for headers, in
    our experience, many web-servers will fail if the `host` header does
    not appear first in request headers. We ensure this by manually
    adding the header ourselves rather than relying on hyper to do so.
* Use `openssl` instead of `rusttls` for outbound webhooks. In our experience,
  many widely deployed webservers use "weak" ciphers that rusttls refuses to
  support, leaving us with no choice but to rely on openssl libraries.
* Support for soft-limiting of concurrent worker tasks. Unbounded concurrent worker
  tasks can easily overwhelm a system. This sets a default of 500 tasks with support
  for configuring other limits or setting unlimited (`0`) concurrent tasks
* Refactor message dispatch code for clarity. Dispatch code is now broken into
  separate methods: `prepare_dispatch`, `make_http_call`,
  `handle_successful_dispatch`, and `handle_failed_dispatch` with the dispatch itself
  having different types depending on its stage in the dispatch process.
  Other function names have also been changed for improved clarity.
* Prevent worker from shutting down when there are active tasks.
* Avoid multiple DB calls for message-destination insert/retrieval
@jaymell jaymell requested a review from tasn January 17, 2023 18:02
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants