Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

CheckIn.cron drains hackney pool connections #970

Closed
ftes opened this issue Nov 6, 2024 · 2 comments · Fixed by #971
Closed

CheckIn.cron drains hackney pool connections #970

ftes opened this issue Nov 6, 2024 · 2 comments · Fixed by #971
Labels
bug

Comments

@ftes
Copy link

ftes commented Nov 6, 2024 •
edited
Loading

Describe the bug

Calling cron/1 and cron/2 checks out a hackney connection from the pool and doesn't release it again.
After a while (roughly 4 hours for a 5-minute-interval cron job) this causes all 50 connections in the default pool to be drained.

To Reproduce

Steps to reproduce the behavior:

  • Using AppSignal for Elixir package version 2.13.0
  • elixir version 1.17.3
  • hackney version 1.20.1
> :hackney_pool.get_stats(:default)
[name: :default, max: 50, in_use_count: 0, free_count: 0, queue_count: 0]

> Appsignal.CheckIn.cron("my_check_in")

> Process.sleep(10_000)
> :hackney_pool.get_stats(:default)
[name: :default, max: 50, in_use_count: 1, free_count: 0, queue_count: 0]

> Appsignal.CheckIn.cron("my_check_in", fn -> :ok end)

> Process.sleep(10_000)
> :hackney_pool.get_stats(:default)
[name: :default, max: 50, in_use_count: 2, free_count: 0, queue_count: 0]
@ftes ftes added the bug label Nov 6, 2024
@unflxw
Copy link
Contributor

unflxw commented Nov 6, 2024

Hi @ftes, thank you for reporting this issue! We will look into it and get back to you.

unflxw added a commit that referenced this issue Nov 6, 2024
@unflxw
Use separate Hackney pool for transmitter
37ae062 
This relates to the issue reported in #970, where the default Hackney
pool is drained of connections by the check-ins sent by the
transmitter. This does not fix the issue, but ensures that it is
a separate pool, used only by the AppSignal transmitter, that is
drained of connections, rather than the default one, which may be
used by the customer's code.
unflxw added a commit that referenced this issue Nov 6, 2024
@unflxw
Close Hackney connections after use
80827bf 
Make sure that Hackney connections are closed after use. This fixes
an issue where the Hackney pool does not correctly claim back
connections, which fixes #970.

Implement a `transmit_and_close/3` convenience method in the
transmitter for use cases where the body is not of interest, meaning
that the connection can be immediately closed.
@unflxw unflxw mentioned this issue Nov 6, 2024
Merged
unflxw added a commit that referenced this issue Nov 6, 2024
@unflxw
Use separate Hackney pool for transmitter
6ff1268 
This relates to the issue reported in #970, where the default Hackney
pool is drained of connections by the check-ins sent by the
transmitter. This does not fix the issue, but ensures that it is
a separate pool, used only by the AppSignal transmitter, that is
drained of connections, rather than the default one, which may be
used by the customer's code.
unflxw added a commit that referenced this issue Nov 6, 2024
@unflxw
Close Hackney connections after use
d500693 
Make sure that Hackney connections are closed after use. This fixes
an issue where the Hackney pool does not correctly claim back
connections, which fixes #970.

Implement a `transmit_and_close/3` convenience method in the
transmitter for use cases where the body is not of interest, meaning
that the connection can be immediately closed.
unflxw added a commit that referenced this issue Nov 6, 2024
@unflxw
Close Hackney connections after use
7233adf 
Make sure that Hackney connections are closed after use. This fixes
an issue where the Hackney pool does not correctly claim back
connections, which fixes #970.

Implement a `transmit_and_close/3` convenience method in the
transmitter for use cases where the body is not of interest, meaning
that the connection can be immediately closed.
@unflxw
Copy link
Contributor

unflxw commented Nov 6, 2024

Hi @ftes,

Thank you for reporting this issue, and especially for hinting at the Hackney connection pool! We have now released AppSignal for Elixir 2.13.1, which fixes this issue.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Fix Hackney pool saturation appsignal/appsignal-elixir
2 participants
@ftes @unflxw