Make response body accessible via hint in `beforSend` callback for SentryHttpClient #2293

martinhaintz · 2024-09-16T15:10:16Z

📜 Description

The request body is now also available via the hint in beforeSend so the user can do further steps.
This is available for failed requests and if tracing is enabled.

options.beforeSend = (event, hint) async {
  final response = hint.get(TypeCheckHint.httpResponse);
  if (response is StreamedResponse) {
    final body = await response.stream.bytesToString();
    // user can now use it
  }
  return event;
};

💡 Motivation and Context

close #2220

💚 How did you test it?

📝 Checklist

I reviewed submitted code
I added tests to verify changes
No new PII added or SDK only sends newly added PII if sendDefaultPii is enabled
I updated the docs if needed
All tests passing
No breaking changes

🔮 Next steps

github-actions · 2024-09-16T15:11:00Z

	Messages
📖	Do not forget to update Sentry-docs with your feature once the pull request gets approved.

Generated by 🚫 dangerJS against a188061

codecov · 2024-09-16T15:11:42Z

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 85.10%. Comparing base (404f5a9) to head (a188061).

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #2293      +/-   ##
==========================================
+ Coverage   85.04%   85.10%   +0.06%     
==========================================
  Files         257      258       +1     
  Lines        9194     9234      +40     
==========================================
+ Hits         7819     7859      +40     
  Misses       1375     1375

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚨 Try these New Features:

Flaky Tests Detection - Detect and resolve failed and flaky tests

github-actions · 2024-09-16T15:19:50Z

iOS Performance metrics 🚀

	Plain	With Sentry	Diff
Startup time	1242.08 ms	1264.28 ms	22.20 ms
Size	8.38 MiB	9.77 MiB	1.39 MiB

Baseline results on branch: main

Startup times

Revision	Plain	With Sentry	Diff
`824df58`	1235.72 ms	1259.02 ms	23.30 ms
`3f3ef0b`	1223.73 ms	1237.67 ms	13.94 ms
`4c78360`	1230.35 ms	1252.37 ms	22.01 ms
`ca9f398`	1227.29 ms	1236.20 ms	8.92 ms
`dd933d4`	1238.73 ms	1252.43 ms	13.70 ms
`afa6e2a`	1251.04 ms	1266.65 ms	15.61 ms
`a49594a`	1284.83 ms	1313.29 ms	28.45 ms
`a094100`	1260.27 ms	1276.75 ms	16.48 ms
`8fa3934`	1252.79 ms	1272.41 ms	19.62 ms
`613760b`	1263.10 ms	1277.27 ms	14.16 ms

App size

Revision	Plain	With Sentry	Diff
`824df58`	8.33 MiB	9.64 MiB	1.31 MiB
`3f3ef0b`	8.32 MiB	9.38 MiB	1.05 MiB
`4c78360`	8.32 MiB	9.38 MiB	1.06 MiB
`ca9f398`	8.38 MiB	9.74 MiB	1.36 MiB
`dd933d4`	8.33 MiB	9.64 MiB	1.31 MiB
`afa6e2a`	8.28 MiB	9.33 MiB	1.05 MiB
`a49594a`	8.16 MiB	9.16 MiB	1.00 MiB
`a094100`	8.16 MiB	9.17 MiB	1.01 MiB
`8fa3934`	8.09 MiB	9.07 MiB	1000.86 KiB
`613760b`	8.15 MiB	9.13 MiB	1000.46 KiB

Previous results on branch: feat/capture-http-response-body-for-sentry-http-client

Startup times

Revision	Plain	With Sentry	Diff
`b0bca7d`	1251.67 ms	1272.80 ms	21.13 ms
`9e2ff12`	1251.00 ms	1274.70 ms	23.70 ms
`ac2c668`	1243.00 ms	1263.92 ms	20.92 ms
`0282579`	1255.33 ms	1276.70 ms	21.36 ms
`8896a1e`	1246.29 ms	1269.15 ms	22.85 ms
`bc3d263`	1240.98 ms	1259.69 ms	18.71 ms
`4190467`	1243.45 ms	1267.19 ms	23.74 ms
`2308122`	1231.69 ms	1256.85 ms	25.16 ms
`b38f597`	1245.52 ms	1266.73 ms	21.21 ms
`f9abe16`	1245.40 ms	1274.94 ms	29.54 ms

App size

Revision	Plain	With Sentry	Diff
`b0bca7d`	8.38 MiB	9.77 MiB	1.39 MiB
`9e2ff12`	8.38 MiB	9.77 MiB	1.39 MiB
`ac2c668`	8.38 MiB	9.77 MiB	1.39 MiB
`0282579`	8.38 MiB	9.76 MiB	1.39 MiB
`8896a1e`	8.38 MiB	9.77 MiB	1.39 MiB
`bc3d263`	8.38 MiB	9.75 MiB	1.37 MiB
`4190467`	8.38 MiB	9.76 MiB	1.39 MiB
`2308122`	8.38 MiB	9.73 MiB	1.36 MiB
`b38f597`	8.38 MiB	9.77 MiB	1.39 MiB
`f9abe16`	8.38 MiB	9.75 MiB	1.37 MiB

dart/lib/src/sentry_options.dart

buenaflor · 2024-10-15T11:25:37Z

@kahest JS Replay uses captureNetworkBodies for both request and response

getsentry/sentry-javascript#7589

wdyt if we also adopt this flag?

kahest · 2024-10-15T17:54:09Z

@kahest JS Replay uses captureNetworkBodies for both request and response

getsentry/sentry-javascript#7589

wdyt if we also adopt this flag?

I think this has since been dropped in favor of networkDetailAllowUrls - see docs. I also don't see references to the option in code anymore.

I like the new option - it makes it more granular and explicit, though of course it's more work. It's still only available for JS SR I think, so not widely adopted. If there's no competing option for other HTTP integrations on other SDKs, I'm fine with adopting this

buenaflor · 2024-10-15T18:15:37Z

Ah I missed that, thx. I'll have a look at it

buenaflor · 2024-10-16T11:08:15Z

@kahest I haven't found anything specific in other SDKs that allow sending http response bodies other than replay.

However it's possible a user can do this:

beforeSend: (event, hint) async {
    final response = hint.get(TypeCheckHint.httpResponse);
    if (response is StreamedResponse) {
        final body = getResponseBody(response)
        // user can now use it
    }
}

this also aligns with what the js http integration would like to add: getsentry/sentry-javascript#12544

kahest · 2024-10-16T17:00:58Z

@buenaflor IIUC this would work for http+dio? I like the idea, it's very flexible and allows users to decide based on endpoint etc. if they want to add, but of course it's more complex to use than a switch

buenaflor · 2024-10-17T10:54:43Z

this would work for http+dio

yes but only errors so this won't work for transactions (like the user in the referenced issue wants) because we don't have a system in place to pair hints with transactions

I'll take a look for alternatives

…ttp-client

github-actions · 2024-11-04T09:36:00Z

Android Performance metrics 🚀

	Plain	With Sentry	Diff
Startup time	714.20 ms	766.96 ms	52.76 ms
Size	6.49 MiB	7.56 MiB	1.07 MiB

Baseline results on branch: main

Startup times

Revision	Plain	With Sentry	Diff
`453e1bc`	320.41 ms	372.73 ms	52.32 ms
`061fed2`	434.11 ms	506.49 ms	72.38 ms
`9928a74`	375.26 ms	456.30 ms	81.04 ms
`11fb408`	320.10 ms	380.24 ms	60.14 ms
`dd76eef`	461.37 ms	540.55 ms	79.18 ms
`2261c15`	370.00 ms	455.88 ms	85.88 ms
`d089990`	361.67 ms	442.50 ms	80.83 ms
`ca7f531`	395.69 ms	497.82 ms	102.12 ms
`e5b744f`	302.70 ms	342.17 ms	39.47 ms
`1e094d3`	316.22 ms	378.69 ms	62.47 ms

App size

Revision	Plain	With Sentry	Diff
`453e1bc`	5.94 MiB	6.95 MiB	1.01 MiB
`061fed2`	6.52 MiB	7.59 MiB	1.06 MiB
`9928a74`	5.94 MiB	6.96 MiB	1.02 MiB
`11fb408`	6.06 MiB	7.10 MiB	1.04 MiB
`dd76eef`	6.35 MiB	7.40 MiB	1.05 MiB
`2261c15`	6.27 MiB	7.20 MiB	957.75 KiB
`d089990`	6.34 MiB	7.28 MiB	967.79 KiB
`ca7f531`	6.33 MiB	7.26 MiB	949.75 KiB
`e5b744f`	6.06 MiB	7.09 MiB	1.03 MiB
`1e094d3`	6.16 MiB	7.14 MiB	1004.21 KiB

Previous results on branch: feat/capture-http-response-body-for-sentry-http-client

Startup times

Revision	Plain	With Sentry	Diff
`ac2c668`	469.81 ms	490.12 ms	20.31 ms
`0282579`	503.49 ms	528.57 ms	25.08 ms
`9e2ff12`	514.25 ms	517.24 ms	2.99 ms
`4190467`	458.57 ms	480.06 ms	21.49 ms
`bc3d263`	454.46 ms	498.00 ms	43.54 ms
`f9abe16`	456.26 ms	480.49 ms	24.23 ms
`b38f597`	488.83 ms	566.78 ms	77.95 ms
`2308122`	422.54 ms	491.65 ms	69.11 ms
`8896a1e`	454.63 ms	491.96 ms	37.33 ms
`b0bca7d`	461.92 ms	523.16 ms	61.24 ms

App size

Revision	Plain	With Sentry	Diff
`ac2c668`	6.49 MiB	7.56 MiB	1.07 MiB
`0282579`	6.49 MiB	7.56 MiB	1.07 MiB
`9e2ff12`	6.49 MiB	7.56 MiB	1.07 MiB
`4190467`	6.49 MiB	7.56 MiB	1.07 MiB
`bc3d263`	6.49 MiB	7.57 MiB	1.08 MiB
`f9abe16`	6.49 MiB	7.57 MiB	1.08 MiB
`b38f597`	6.49 MiB	7.56 MiB	1.07 MiB
`2308122`	6.49 MiB	7.55 MiB	1.07 MiB
`8896a1e`	6.49 MiB	7.56 MiB	1.07 MiB
`b0bca7d`	6.49 MiB	7.56 MiB	1.07 MiB

…ttp-client

buenaflor · 2024-11-11T12:16:45Z

before I look at the PR, does this also work for transactions? because afaik beforeSend is not triggered for transactions @martinhaintz

buenaflor · 2024-11-12T12:14:02Z

In order to make this fully compatible with tracing we should add the hint to the beforeSendTransaction API

It is available in our sdks so imo it's safe to implement it:

martinhaintz · 2024-11-13T16:19:58Z

before I look at the PR, does this also work for transactions? because afaik beforeSend is not triggered for transactions @martinhaintz

it wasn't done previously, but I implemented it.

martinhaintz · 2024-11-13T16:20:33Z

In order to make this fully compatible with tracing we should add the hint to the beforeSendTransaction API

It is available in our sdks so imo it's safe to implement it:

.net

javascript

this is by far, the better solution. I will change it tomorrow.

…ttp-client

vaind

This can break existing applications and cause OOM errors in some edge cases. Are there other implementation options that wouldn't have this issue?

For example, only do the capture of the response body with limits on response size (read the response header first)

dart/lib/src/utils/http_deep_copy_streamed_response.dart

…ttp-client

martinhaintz · 2024-11-18T12:45:18Z

This can break existing applications and cause OOM errors in some edge cases. Are there other implementation options that wouldn't have this issue?

For example, only do the capture of the response body with limits on response size (read the response header first)

@vaind You are right. Thanks for the hint. I will take a different approach.

buenaflor · 2024-11-18T13:06:34Z

This can break existing applications

true, but I'm not sure if adding a hint to beforeSendTransaction would warrant a new major, although I'm not opposed to marking this as a v9 feature

only do the capture of the response body with limits on response size

we generally don't want to support this with response bodies (as far as I can tell after some searching other sdks don't allow users to capture response bodies automatically with a max size flag), so we'd like to make this dependent on the user as much as possible

imo I'd prefer using hints for this so the user can decide themselves whether to add it to their events

…ttp-client

martinhaintz · 2024-11-19T16:21:40Z

@vaind I implemented a cache for the streamedResponse see streamed_response_copier.dart

@buenaflor @vaind A copied stream is now available via the hint with TypeCheckHint.httpResponse. I am not sure if it is the best way to hold the hint in the SentryTracer, but as the response is from/inside a children this was the only way I could return it as a hint.
Let me know about your feedback.

UPDATE:
This is not a good idea. For every transaction we have only one hint element which is a key/value pair with TypeCheckHint.httpResponse for the response. However, a transaction can contain multiple spans, with every span containing their own response element. Therefore I think the better solution would be to access the hint via transaction.spans[0].hint in the beforeSendTransaction.

I also updated the analyzer in some plugins to ignore the mocks.mocks.dart files, because it threw some warnings after rebuilding.

…ttp-client

martinhaintz · 2024-11-20T10:17:29Z

This is the current approach for accessing the hints for two spans inside a transaction:

options.beforeSendTransaction = (transaction, hint) {
        final firstHint = transaction!.spans[0].hint
        final secondHint = transaction!.spans[1].hint
        return transaction;
      };

@vaind @buenaflor should I revert the changes regarding the parameter options.beforeSendTransaction = (transaction, hint) {} and remove the hint, or leave it there to have the hint already there for the future?

buenaflor · 2024-11-20T10:47:34Z

We can add the hint for beforeSendTransaction there but probably will mark this as a v9 feature unless we think this is okay to do in v8

capture response body for failed requests and if tracing is enabled.

2308122

unified event capturing for request and response

da865b2

buenaflor reviewed Sep 18, 2024

View reviewed changes

dart/lib/src/sentry_options.dart Outdated Show resolved Hide resolved

Merge branch 'main' into feat/capture-http-response-body-for-sentry-h…

bc3d263

…ttp-client

martinhaintz added 3 commits November 5, 2024 16:40

make response body accessible via hint

710ab1a

fix warning

f9abe16

add changelog entry

a309f16

martinhaintz marked this pull request as ready for review November 5, 2024 16:57

martinhaintz requested review from krystofwoldrich and stefanosiano as code owners November 5, 2024 16:57

martinhaintz requested a review from buenaflor November 5, 2024 16:58

martinhaintz changed the title ~~capture response body for failed requests and if tracing is enabled in SentryHttpClient~~ Make response body accessible via hint in beforSend callback for SentryHttpClient Nov 5, 2024

Merge branch 'main' into feat/capture-http-response-body-for-sentry-h…

715ab7d

…ttp-client

Merge branch 'main' into feat/capture-http-response-body-for-sentry-h…

52ecb5e

…ttp-client

vaind requested changes Nov 14, 2024

View reviewed changes

dart/lib/src/utils/http_deep_copy_streamed_response.dart Outdated Show resolved Hide resolved

Merge branch 'main' into feat/capture-http-response-body-for-sentry-h…

0282579

…ttp-client

add response body as hint for beforeTransactionSend

4190467

martinhaintz marked this pull request as draft November 18, 2024 18:16

martinhaintz added 17 commits November 19, 2024 10:09

Merge branch 'main' into feat/capture-http-response-body-for-sentry-h…

ac2c668

…ttp-client

update hive mocks

24b8ca9

update drift mocks

7042c14

update dio mocks

b9fbb3a

update file mocks

557958f

update mocks for isar

5959137

update sqflite mocks

b417e98

add script to run builder for all packages

1ead661

fix streamed response copier

fdadb89

fix dart analyze

f773568

fix analyzer and format

b0bca7d

update flutter mocks and fix tests

9e2ff12

fix mocks

8896a1e

fix analyze options

9b4000b

cleanup and code reformating

86ffb80

revert changes

b325b09

remove unused import

b38f597

martinhaintz requested a review from vaind November 19, 2024 16:13

martinhaintz added 2 commits November 20, 2024 11:03

removed hint from tracer

47f50aa

Merge branch 'main' into feat/capture-http-response-body-for-sentry-h…

a188061

…ttp-client

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Make response body accessible via hint in `beforSend` callback for SentryHttpClient #2293

Make response body accessible via hint in `beforSend` callback for SentryHttpClient #2293

martinhaintz commented Sep 16, 2024 •

edited

Loading

github-actions bot commented Sep 16, 2024 •

edited

Loading

codecov bot commented Sep 16, 2024 •

edited

Loading

github-actions bot commented Sep 16, 2024 •

edited

Loading

Baseline results on branch: main

Startup times

App size

Previous results on branch: feat/capture-http-response-body-for-sentry-http-client

Startup times

App size

buenaflor commented Oct 15, 2024

kahest commented Oct 15, 2024 •

edited

Loading

buenaflor commented Oct 15, 2024

buenaflor commented Oct 16, 2024 •

edited

Loading

kahest commented Oct 16, 2024

buenaflor commented Oct 17, 2024

github-actions bot commented Nov 4, 2024 •

edited

Loading

Baseline results on branch: main

Startup times

App size

Previous results on branch: feat/capture-http-response-body-for-sentry-http-client

Startup times

App size

buenaflor commented Nov 11, 2024 •

edited

Loading

buenaflor commented Nov 12, 2024

martinhaintz commented Nov 13, 2024

martinhaintz commented Nov 13, 2024

vaind left a comment •

edited

Loading

martinhaintz commented Nov 18, 2024 •

edited

Loading

buenaflor commented Nov 18, 2024 •

edited

Loading

martinhaintz commented Nov 19, 2024 •

edited

Loading

martinhaintz commented Nov 20, 2024

buenaflor commented Nov 20, 2024 •

edited

Loading

Make response body accessible via hint in beforSend callback for SentryHttpClient #2293

Are you sure you want to change the base?

Make response body accessible via hint in beforSend callback for SentryHttpClient #2293

Conversation

martinhaintz commented Sep 16, 2024 • edited Loading

📜 Description

💡 Motivation and Context

💚 How did you test it?

📝 Checklist

🔮 Next steps

github-actions bot commented Sep 16, 2024 • edited Loading

codecov bot commented Sep 16, 2024 • edited Loading

Codecov Report

github-actions bot commented Sep 16, 2024 • edited Loading

iOS Performance metrics 🚀

Baseline results on branch: main

Startup times

App size

Previous results on branch: feat/capture-http-response-body-for-sentry-http-client

Startup times

App size

buenaflor commented Oct 15, 2024

kahest commented Oct 15, 2024 • edited Loading

buenaflor commented Oct 15, 2024

buenaflor commented Oct 16, 2024 • edited Loading

kahest commented Oct 16, 2024

buenaflor commented Oct 17, 2024

github-actions bot commented Nov 4, 2024 • edited Loading

Android Performance metrics 🚀

Baseline results on branch: main

Startup times

App size

Previous results on branch: feat/capture-http-response-body-for-sentry-http-client

Startup times

App size

buenaflor commented Nov 11, 2024 • edited Loading

buenaflor commented Nov 12, 2024

martinhaintz commented Nov 13, 2024

martinhaintz commented Nov 13, 2024

vaind left a comment • edited Loading

Choose a reason for hiding this comment

martinhaintz commented Nov 18, 2024 • edited Loading

buenaflor commented Nov 18, 2024 • edited Loading

martinhaintz commented Nov 19, 2024 • edited Loading

martinhaintz commented Nov 20, 2024

buenaflor commented Nov 20, 2024 • edited Loading

Make response body accessible via hint in `beforSend` callback for SentryHttpClient #2293

Make response body accessible via hint in `beforSend` callback for SentryHttpClient #2293

martinhaintz commented Sep 16, 2024 •

edited

Loading

github-actions bot commented Sep 16, 2024 •

edited

Loading

codecov bot commented Sep 16, 2024 •

edited

Loading

github-actions bot commented Sep 16, 2024 •

edited

Loading

kahest commented Oct 15, 2024 •

edited

Loading

buenaflor commented Oct 16, 2024 •

edited

Loading

github-actions bot commented Nov 4, 2024 •

edited

Loading

buenaflor commented Nov 11, 2024 •

edited

Loading

vaind left a comment •

edited

Loading

martinhaintz commented Nov 18, 2024 •

edited

Loading

buenaflor commented Nov 18, 2024 •

edited

Loading

martinhaintz commented Nov 19, 2024 •

edited

Loading

buenaflor commented Nov 20, 2024 •

edited

Loading