Infinite Stack Recursion with HTTP

[saulshanabrook]

Issue Description

Hello!

I am coming across an error trying to use the sentry ruby bindings on with a rails app, during a rake task. The task uploads the precompiled assets to an S3 server. It exits cleanly without the sentry integration, but when I add it, it enters into an infinitely recursive loop with the patched HTTP module. I am not sure how to debug this further or what to do.

Thank you!

Reproduction Steps

edited with simpler steps:

$ git clone https://github.com/saulshanabrook/discourse-hosting
$ cd discourse-hosting
$ echo 'HOSTNAME=any-domain.com' > .env
$ docker compose run --rm upload_assets

Following the steps of https://github.com/saulshanabrook/discourse-hosting#local-development-with-docker-compose the error will be raised when running docker compose run --rm upload_assets:

Expected Behavior

The command exits cleanly without an error

Actual Behavior

It raises a SystemStackError:

$ docker compose run --rm upload_assets
[+] Running 4/0
 ⠿ Container discourse-hosting-db-1             Running 0.0s
 ⠿ Container discourse-hosting-minio-1          Running 0.0s
 ⠿ Container discourse-hosting-redis-1          Running 0.0s
 ⠿ Container discourse-hosting-glitchtip-web-1  Running 0.0s
** Invoke s3:upload_assets (first_time)
** Invoke environment (first_time)
** Execute environment
** Invoke s3:ensure_cors_rules (first_time)
** Invoke environment 
** Execute s3:ensure_cors_rules
** Execute s3:upload_assets
rake aborted!
SystemStackError: stack level too deep
...
/usr/local/bundle/gems/sentry-ruby-5.8.0/lib/sentry/net/http.rb:35:in `block in request'
/usr/local/bundle/gems/sentry-ruby-5.8.0/lib/sentry/hub.rb:100:in `with_child_span'
/usr/local/bundle/gems/sentry-ruby-5.8.0/lib/sentry-ruby.rb:446:in `with_child_span'
/usr/local/bundle/gems/sentry-ruby-5.8.0/lib/sentry/net/http.rb:32:in `request'
(eval):5:in `request'
/usr/local/bundle/gems/sentry-ruby-5.8.0/lib/sentry/net/http.rb:35:in `block in request'
/usr/local/bundle/gems/sentry-ruby-5.8.0/lib/sentry/hub.rb:100:in `with_child_span'
/usr/local/bundle/gems/sentry-ruby-5.8.0/lib/sentry-ruby.rb:446:in `with_child_span'
/usr/local/bundle/gems/sentry-ruby-5.8.0/lib/sentry/net/http.rb:32:in `request'
(eval):5:in `request'
/usr/local/bundle/gems/sentry-ruby-5.8.0/lib/sentry/net/http.rb:35:in `block in request'
/usr/local/bundle/gems/sentry-ruby-5.8.0/lib/sentry/hub.rb:100:in `with_child_span'
/usr/local/bundle/gems/sentry-ruby-5.8.0/lib/sentry-ruby.rb:446:in `with_child_span'
/usr/local/bundle/gems/sentry-ruby-5.8.0/lib/sentry/net/http.rb:32:in `request'
/usr/local/bundle/gems/aws-sdk-core-3.130.2/lib/seahorse/client/net_http/connection_pool.rb:348:in `request'
/usr/local/bundle/gems/aws-sdk-core-3.130.2/lib/seahorse/client/net_http/handler.rb:79:in `block in transmit'
/usr/local/bundle/gems/aws-sdk-core-3.130.2/lib/seahorse/client/net_http/handler.rb:133:in `block in session'
/usr/local/bundle/gems/aws-sdk-core-3.130.2/lib/seahorse/client/net_http/connection_pool.rb:104:in `session_for'
/usr/local/bundle/gems/aws-sdk-core-3.130.2/lib/seahorse/client/net_http/handler.rb:128:in `session'
/usr/local/bundle/gems/aws-sdk-core-3.130.2/lib/seahorse/client/net_http/handler.rb:76:in `transmit'
/usr/local/bundle/gems/aws-sdk-core-3.130.2/lib/seahorse/client/net_http/handler.rb:50:in `call'
/usr/local/bundle/gems/aws-sdk-core-3.130.2/lib/seahorse/client/plugins/content_length.rb:24:in `call'
/usr/local/bundle/gems/aws-sdk-core-3.130.2/lib/seahorse/client/plugins/request_callback.rb:85:in `call'
/usr/local/bundle/gems/aws-sdk-s3-1.114.0/lib/aws-sdk-s3/plugins/s3_signer.rb:132:in `call'
/usr/local/bundle/gems/aws-sdk-s3-1.114.0/lib/aws-sdk-s3/plugins/s3_signer.rb:63:in `call'
/usr/local/bundle/gems/aws-sdk-s3-1.114.0/lib/aws-sdk-s3/plugins/s3_host_id.rb:17:in `call'
/usr/local/bundle/gems/aws-sdk-core-3.130.2/lib/aws-sdk-core/xml/error_handler.rb:10:in `call'
/usr/local/bundle/gems/aws-sdk-core-3.130.2/lib/aws-sdk-core/plugins/transfer_encoding.rb:26:in `call'
/usr/local/bundle/gems/aws-sdk-core-3.130.2/lib/aws-sdk-core/plugins/helpful_socket_errors.rb:12:in `call'
/usr/local/bundle/gems/aws-sdk-s3-1.114.0/lib/aws-sdk-s3/plugins/s3_signer.rb:110:in `call'
/usr/local/bundle/gems/aws-sdk-s3-1.114.0/lib/aws-sdk-s3/plugins/redirects.rb:20:in `call'
/usr/local/bundle/gems/aws-sdk-core-3.130.2/lib/aws-sdk-core/plugins/retry_errors.rb:360:in `call'
/usr/local/bundle/gems/aws-sdk-core-3.130.2/lib/aws-sdk-core/plugins/http_checksum.rb:19:in `call'
/usr/local/bundle/gems/aws-sdk-core-3.130.2/lib/aws-sdk-core/plugins/endpoint_pattern.rb:30:in `call'
/usr/local/bundle/gems/aws-sdk-s3-1.114.0/lib/aws-sdk-s3/plugins/accelerate.rb:67:in `call'
/usr/local/bundle/gems/aws-sdk-core-3.130.2/lib/aws-sdk-core/plugins/checksum_algorithm.rb:136:in `call'
/usr/local/bundle/gems/aws-sdk-s3-1.114.0/lib/aws-sdk-s3/plugins/bucket_dns.rb:35:in `call'
/usr/local/bundle/gems/aws-sdk-s3-1.114.0/lib/aws-sdk-s3/plugins/dualstack.rb:41:in `call'
/usr/local/bundle/gems/aws-sdk-s3-1.114.0/lib/aws-sdk-s3/plugins/expect_100_continue.rb:22:in `call'
/usr/local/bundle/gems/aws-sdk-s3-1.114.0/lib/aws-sdk-s3/plugins/bucket_name_restrictions.rb:26:in `call'
/usr/local/bundle/gems/aws-sdk-s3-1.114.0/lib/aws-sdk-s3/plugins/arn.rb:62:in `call'
/usr/local/bundle/gems/aws-sdk-core-3.130.2/lib/aws-sdk-core/rest/handler.rb:10:in `call'
/usr/local/bundle/gems/aws-sdk-core-3.130.2/lib/aws-sdk-core/plugins/recursion_detection.rb:18:in `call'
/usr/local/bundle/gems/aws-sdk-core-3.130.2/lib/aws-sdk-core/plugins/user_agent.rb:13:in `call'
/usr/local/bundle/gems/aws-sdk-core-3.130.2/lib/seahorse/client/plugins/endpoint.rb:47:in `call'
/usr/local/bundle/gems/aws-sdk-core-3.130.2/lib/aws-sdk-core/plugins/param_validator.rb:26:in `call'
/usr/local/bundle/gems/aws-sdk-s3-1.114.0/lib/aws-sdk-s3/plugins/arn.rb:88:in `call'
/usr/local/bundle/gems/aws-sdk-s3-1.114.0/lib/aws-sdk-s3/plugins/sse_cpk.rb:24:in `call'
/usr/local/bundle/gems/aws-sdk-s3-1.114.0/lib/aws-sdk-s3/plugins/dualstack.rb:27:in `call'
/usr/local/bundle/gems/aws-sdk-s3-1.114.0/lib/aws-sdk-s3/plugins/accelerate.rb:56:in `call'
/usr/local/bundle/gems/aws-sdk-core-3.130.2/lib/aws-sdk-core/plugins/checksum_algorithm.rb:111:in `call'
/usr/local/bundle/gems/aws-sdk-core-3.130.2/lib/aws-sdk-core/plugins/jsonvalue_converter.rb:22:in `call'
/usr/local/bundle/gems/aws-sdk-core-3.130.2/lib/aws-sdk-core/plugins/idempotency_token.rb:19:in `call'
/usr/local/bundle/gems/aws-sdk-core-3.130.2/lib/aws-sdk-core/plugins/param_converter.rb:26:in `call'
/usr/local/bundle/gems/aws-sdk-core-3.130.2/lib/seahorse/client/plugins/request_callback.rb:71:in `call'
/usr/local/bundle/gems/aws-sdk-core-3.130.2/lib/aws-sdk-core/plugins/response_paging.rb:12:in `call'
/usr/local/bundle/gems/aws-sdk-core-3.130.2/lib/seahorse/client/plugins/response_target.rb:24:in `call'
/usr/local/bundle/gems/aws-sdk-core-3.130.2/lib/seahorse/client/request.rb:72:in `send_request'
/usr/local/bundle/gems/aws-sdk-core-3.130.2/lib/aws-sdk-core/waiters/poller.rb:65:in `send_request'
/usr/local/bundle/gems/aws-sdk-core-3.130.2/lib/aws-sdk-core/waiters/poller.rb:51:in `call'
/usr/local/bundle/gems/aws-sdk-core-3.130.2/lib/aws-sdk-core/waiters/waiter.rb:107:in `block in poll'
/usr/local/bundle/gems/aws-sdk-core-3.130.2/lib/aws-sdk-core/waiters/waiter.rb:104:in `loop'
/usr/local/bundle/gems/aws-sdk-core-3.130.2/lib/aws-sdk-core/waiters/waiter.rb:104:in `poll'
/usr/local/bundle/gems/aws-sdk-core-3.130.2/lib/aws-sdk-core/waiters/waiter.rb:94:in `block (2 levels) in wait'
/usr/local/bundle/gems/aws-sdk-core-3.130.2/lib/aws-sdk-core/waiters/waiter.rb:93:in `catch'
/usr/local/bundle/gems/aws-sdk-core-3.130.2/lib/aws-sdk-core/waiters/waiter.rb:93:in `block in wait'
/usr/local/bundle/gems/aws-sdk-core-3.130.2/lib/aws-sdk-core/waiters/waiter.rb:92:in `catch'
/usr/local/bundle/gems/aws-sdk-core-3.130.2/lib/aws-sdk-core/waiters/waiter.rb:92:in `wait'
/usr/local/bundle/gems/aws-sdk-s3-1.114.0/lib/aws-sdk-s3/waiters.rb:123:in `wait'
/usr/local/bundle/gems/aws-sdk-s3-1.114.0/lib/aws-sdk-s3/bucket.rb:98:in `wait_until_exists'
/usr/local/bundle/gems/aws-sdk-s3-1.114.0/lib/aws-sdk-s3/bucket.rb:79:in `exists?'
/var/www/discourse/lib/s3_helper.rb:404:in `s3_bucket'
/var/www/discourse/lib/s3_helper.rb:240:in `list'
/var/www/discourse/lib/tasks/s3.rake:14:in `existing_assets'
/var/www/discourse/lib/tasks/s3.rake:24:in `should_skip?'
/var/www/discourse/lib/tasks/s3.rake:36:in `upload'
/var/www/discourse/lib/tasks/s3.rake:192:in `block (2 levels) in <main>'
/var/www/discourse/lib/tasks/s3.rake:192:in `each'
/var/www/discourse/lib/tasks/s3.rake:192:in `block in <main>'
/usr/local/lib/ruby/gems/3.1.0/gems/rake-13.0.6/lib/rake/task.rb:281:in `block in execute'
/usr/local/lib/ruby/gems/3.1.0/gems/rake-13.0.6/lib/rake/task.rb:281:in `each'
/usr/local/lib/ruby/gems/3.1.0/gems/rake-13.0.6/lib/rake/task.rb:281:in `execute'
/usr/local/bundle/gems/sentry-ruby-5.8.0/lib/sentry/rake.rb:26:in `execute'
/usr/local/lib/ruby/gems/3.1.0/gems/rake-13.0.6/lib/rake/task.rb:219:in `block in invoke_with_call_chain'
/usr/local/lib/ruby/gems/3.1.0/gems/rake-13.0.6/lib/rake/task.rb:199:in `synchronize'
/usr/local/lib/ruby/gems/3.1.0/gems/rake-13.0.6/lib/rake/task.rb:199:in `invoke_with_call_chain'
/usr/local/lib/ruby/gems/3.1.0/gems/rake-13.0.6/lib/rake/task.rb:188:in `invoke'
/usr/local/lib/ruby/gems/3.1.0/gems/rake-13.0.6/lib/rake/application.rb:160:in `invoke_task'
/usr/local/lib/ruby/gems/3.1.0/gems/rake-13.0.6/lib/rake/application.rb:116:in `block (2 levels) in top_level'
/usr/local/lib/ruby/gems/3.1.0/gems/rake-13.0.6/lib/rake/application.rb:116:in `each'
/usr/local/lib/ruby/gems/3.1.0/gems/rake-13.0.6/lib/rake/application.rb:116:in `block in top_level'
/usr/local/lib/ruby/gems/3.1.0/gems/rake-13.0.6/lib/rake/application.rb:125:in `run_with_threads'
/usr/local/lib/ruby/gems/3.1.0/gems/rake-13.0.6/lib/rake/application.rb:110:in `top_level'
/usr/local/lib/ruby/gems/3.1.0/gems/rake-13.0.6/lib/rake/application.rb:83:in `block in run'
/usr/local/lib/ruby/gems/3.1.0/gems/rake-13.0.6/lib/rake/application.rb:186:in `standard_exception_handling'
/usr/local/lib/ruby/gems/3.1.0/gems/rake-13.0.6/lib/rake/application.rb:80:in `run'
/usr/local/lib/ruby/gems/3.1.0/gems/rake-13.0.6/exe/rake:27:in `<top (required)>'
/usr/local/bin/rake:25:in `load'
/usr/local/bin/rake:25:in `<top (required)>'
/usr/local/bundle/gems/bundler-2.3.22/lib/bundler/cli/exec.rb:58:in `load'
/usr/local/bundle/gems/bundler-2.3.22/lib/bundler/cli/exec.rb:58:in `kernel_load'
/usr/local/bundle/gems/bundler-2.3.22/lib/bundler/cli/exec.rb:23:in `run'
/usr/local/bundle/gems/bundler-2.3.22/lib/bundler/cli.rb:486:in `exec'
/usr/local/bundle/gems/bundler-2.3.22/lib/bundler/vendor/thor/lib/thor/command.rb:27:in `run'
/usr/local/bundle/gems/bundler-2.3.22/lib/bundler/vendor/thor/lib/thor/invocation.rb:127:in `invoke_command'
/usr/local/bundle/gems/bundler-2.3.22/lib/bundler/vendor/thor/lib/thor.rb:392:in `dispatch'
/usr/local/bundle/gems/bundler-2.3.22/lib/bundler/cli.rb:31:in `dispatch'
/usr/local/bundle/gems/bundler-2.3.22/lib/bundler/vendor/thor/lib/thor/base.rb:485:in `start'
/usr/local/bundle/gems/bundler-2.3.22/lib/bundler/cli.rb:25:in `start'
/usr/local/bundle/gems/bundler-2.3.22/exe/bundle:48:in `block in <top (required)>'
/usr/local/bundle/gems/bundler-2.3.22/lib/bundler/friendly_errors.rb:120:in `with_friendly_errors'
/usr/local/bundle/gems/bundler-2.3.22/exe/bundle:36:in `<top (required)>'
/usr/local/bundle/bin/bundle:25:in `load'
/usr/local/bundle/bin/bundle:25:in `<main>'
Tasks: TOP => s3:upload_assets

Ruby Version

3.1

SDK Version

sentry-ruby-5.8.0

Integration and Its Version

No response

Sentry Config

Sentry.init do |config|

  # this gem also provides a new breadcrumb logger that accepts instrumentations from ActiveSupport
  # it's not activated by default, but you can enable it with
  # config.breadcrumbs_logger = [:active_support_logger]

  config.dsn =  ENV['SENTRY_DSN']
end

[st0012]

@saulshanabrook Thanks for reporting this, but can you simplify the reproduction steps down to just a Ruby/Rails application? The amount of dependencies and manual operations required to run it seems to be quite high:

1. Install docker
2. Install docker compose
3. Tweaking my /etc/hosts
4. Find a place to configure Sentry integration

I gave up on step 3 because I don't feel comfortable changing system configs (even harmless) for reproducing an issue that doesn't require it.

[sl0thentr0py]

@saulshanabrook also is this a regression introduced by 5.8.0 (since that's the version in your issue) or does it also happen on older versions?

[sl0thentr0py]

ok so what is happening is this super is going back to the same patched request for some reason, still don't know why.

sentry-ruby/sentry-ruby/lib/sentry/net/http.rb

Line 35 in a7bc13e

super.tap do |res|

@saulshanabrook can you print Net::HTTP.ancestors after sticking a binding.pry/debugger in your rake task?

[saulshanabrook]

@st0012 Yeah, that totally makes sense about the complicated reproduction steps.

I have tested this to reproduce the bug, which doesn't require messing with hosts, getting a sentry config, or having a domain:

$ git clone https://github.com/saulshanabrook/discourse-hosting
$ cd discourse-hosting
$ echo 'HOSTNAME=any-domain.com' > .env
$ docker compose run --rm upload_assets

Let me know if that doesn't work to reproduce it.

[saulshanabrook]

@sl0thentr0py I was just starting to use sentry with this project. I tried it with 5.7.0 a few weeks ago when that was the latest, was hitting the same error, upgraded to 5.8.0 to see if it would fix it but to no avail. Do you recommend I try downgrading to an earlier version?

[saulshanabrook]

@sl0thentr0py Sure, is this what you mean? I believe it's [Sentry::Net::HTTP]

$ docker compose run --rm upload_assets rdbg -c --  bundle exec rake --trace s3:upload_assets
WARN[0000] The "SEND_GRID_API_KEY" variable is not set. Defaulting to a blank string.
WARN[0000] The "SECRET_KEY" variable is not set. Defaulting to a blank string.
WARN[0000] The "DSN" variable is not set. Defaulting to a blank string.
WARN[0000] The "DSN" variable is not set. Defaulting to a blank string.
WARN[0000] The "EMAIL" variable is not set. Defaulting to a blank string.
WARN[0000] The "SECRET_KEY" variable is not set. Defaulting to a blank string.
WARN[0000] The "SEND_GRID_API_KEY" variable is not set. Defaulting to a blank string.
WARN[0000] The "DSN" variable is not set. Defaulting to a blank string.
WARN[0000] The "DSN" variable is not set. Defaulting to a blank string.
WARN[0000] The "SECRET_KEY" variable is not set. Defaulting to a blank string.
WARN[0000] The "SEND_GRID_API_KEY" variable is not set. Defaulting to a blank string.
WARN[0000] The "DSN" variable is not set. Defaulting to a blank string.
WARN[0000] The "SEND_GRID_API_KEY" variable is not set. Defaulting to a blank string.
WARN[0000] The "SECRET_KEY" variable is not set. Defaulting to a blank string.
WARN[0000] The "EMAIL" variable is not set. Defaulting to a blank string.
WARN[0000] The "SEND_GRID_API_KEY" variable is not set. Defaulting to a blank string.
WARN[0000] The "EMAIL" variable is not set. Defaulting to a blank string.
WARN[0000] The "SEND_GRID_API_KEY" variable is not set. Defaulting to a blank string.
WARN[0000] The "EMAIL" variable is not set. Defaulting to a blank string.
WARN[0000] The "SEND_GRID_API_KEY" variable is not set. Defaulting to a blank string.
WARN[0000] The "EMAIL" variable is not set. Defaulting to a blank string.
WARN[0000] The "SEND_GRID_API_KEY" variable is not set. Defaulting to a blank string.
[+] Running 4/0
 ⠿ Container discourse-hosting-minio-1          Running                                                                                                                                  0.0s
 ⠿ Container discourse-hosting-redis-1          Running                                                                                                                                  0.0s
 ⠿ Container discourse-hosting-db-1             Running                                                                                                                                  0.0s
 ⠿ Container discourse-hosting-glitchtip-web-1  Running                                                                                                                                  0.0s
[4, 13] in /usr/local/bin/rake
     4| #
     5| # The application 'rake' is installed as part of a gem, and
     6| # this file is here to facilitate running it.
     7| #
     8|
=>   9| require 'rubygems'
    10|
    11| Gem.use_gemdeps
    12|
    13| version = ">= 0.a"
=>#0	<top (required)> at /usr/local/bin/rake:9
  #1	[C] Kernel.load at /usr/local/bundle/gems/bundler-2.3.22/lib/bundler/cli/exec.rb:58
  # and 14 frames (use `bt' command for all frames)
(rdbg) b /usr/local/bundle/gems/sentry-ruby-5.8.0/lib/sentry/net/http.rb:35    # break command
#0  BP - Line (pending)  /usr/local/bundle/gems/sentry-ruby-5.8.0/lib/sentry/net/http.rb:35
(rdbg) c    # continue command
** Invoke s3:upload_assets (first_time)
** Invoke environment (first_time)
** Execute environment
** Invoke s3:ensure_cors_rules (first_time)
** Invoke environment
** Execute s3:ensure_cors_rules
Installing CORS rules...
skipping
** Execute s3:upload_assets
[30, 39] in /usr/local/bundle/gems/sentry-ruby-5.8.0/lib/sentry/net/http.rb
    30|         return super if from_sentry_sdk?
    31|
    32|         Sentry.with_child_span(op: OP_NAME, start_timestamp: Sentry.utc_now.to_f) do |sentry_span|
    33|           set_sentry_trace_header(req, sentry_span)
    34|
=>  35|           super.tap do |res|
    36|             record_sentry_breadcrumb(req, res)
    37|
    38|             if sentry_span
    39|               request_info = extract_request_info(req)
=>#0	block {|sentry_span=nil|} in request at /usr/local/bundle/gems/sentry-ruby-5.8.0/lib/sentry/net/http.rb:35
  #1	Sentry::Hub#with_child_span(instrumenter=:sentry, attributes={:op=>"http.client", :start_timestamp=>16..., block=#<Proc:0x00007f9058c78a40 /usr/local/bund...) at /usr/local/bundle/gems/sentry-ruby-5.8.0/lib/sentry/hub.rb:100
  # and 102 frames (use `bt' command for all frames)

Stop by #0  BP - Line  /usr/local/bundle/gems/sentry-ruby-5.8.0/lib/sentry/net/http.rb:35 (line)
(ruby) Net::HTTP.ancestors
[Sentry::Net::HTTP]

[sl0thentr0py]

no i just wanted to know if this was a regression or not. In that case downgrading won't help, we'll need to find the problem in this setup and fix it.