-
Notifications
You must be signed in to change notification settings - Fork 3.6k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[proxy] Fix proxy to be able to re-send request body #5361
Conversation
3ff54c8
to
3abfe71
Compare
Okay, confirmed this works... not sure of an easy way to unit test it. I also want to add more one more change here to fix the strack trace on redirects that is confusing |
rerun integration tests |
I think it should be happening each time the proxy is handling a 307 and the request has a body. Lines 250 to 254 in 7890750
I was forcing multiple GET request to the proxy. The idea is that the proxy will round-robin the requests on the available brokers. We could do the same for a PUT/POST request. |
run java8 tests |
run java8 tests |
Fixes apache#5360 This adds a small cache of the request body to ensure that it can be re-sent. TODO: still needs tested
rerun integration tests |
run java8 tests |
Fixes #10908 ### Motivation Pulsar Proxy uses a lot of heap memory when uploading large function jar files. This also leads to high GC activity since a continuous block of memory (byte array for the size of the upload) is allocated. GC will have to do compaction for the heap (which gets fragmented) to find a continuous block of memory. This is the reason why allocating large arrays are costly from GC perspective. The buffering solution added as part of #5361. The solution buffers also very large uploads to memory. ### Modifications * Limit the replay buffer size to a configurable limit which defaults to 5MB. This is configured with the `httpInputMaxReplayBufferSize` proxy configuration parameter. * Add unit test to see that buffer size gets limited * Add unit test for #5361
Fixes apache#10908 ### Motivation Pulsar Proxy uses a lot of heap memory when uploading large function jar files. This also leads to high GC activity since a continuous block of memory (byte array for the size of the upload) is allocated. GC will have to do compaction for the heap (which gets fragmented) to find a continuous block of memory. This is the reason why allocating large arrays are costly from GC perspective. The buffering solution added as part of apache#5361. The solution buffers also very large uploads to memory. ### Modifications * Limit the replay buffer size to a configurable limit which defaults to 5MB. This is configured with the `httpInputMaxReplayBufferSize` proxy configuration parameter. * Add unit test to see that buffer size gets limited * Add unit test for apache#5361 (cherry picked from commit 2324618)
Fixes apache#10908 ### Motivation Pulsar Proxy uses a lot of heap memory when uploading large function jar files. This also leads to high GC activity since a continuous block of memory (byte array for the size of the upload) is allocated. GC will have to do compaction for the heap (which gets fragmented) to find a continuous block of memory. This is the reason why allocating large arrays are costly from GC perspective. The buffering solution added as part of apache#5361. The solution buffers also very large uploads to memory. ### Modifications * Limit the replay buffer size to a configurable limit which defaults to 5MB. This is configured with the `httpInputMaxReplayBufferSize` proxy configuration parameter. * Add unit test to see that buffer size gets limited * Add unit test for apache#5361
Fixes apache#10908 ### Motivation Pulsar Proxy uses a lot of heap memory when uploading large function jar files. This also leads to high GC activity since a continuous block of memory (byte array for the size of the upload) is allocated. GC will have to do compaction for the heap (which gets fragmented) to find a continuous block of memory. This is the reason why allocating large arrays are costly from GC perspective. The buffering solution added as part of apache#5361. The solution buffers also very large uploads to memory. ### Modifications * Limit the replay buffer size to a configurable limit which defaults to 5MB. This is configured with the `httpInputMaxReplayBufferSize` proxy configuration parameter. * Add unit test to see that buffer size gets limited * Add unit test for apache#5361
Fixes apache#10908 Pulsar Proxy uses a lot of heap memory when uploading large function jar files. This also leads to high GC activity since a continuous block of memory (byte array for the size of the upload) is allocated. GC will have to do compaction for the heap (which gets fragmented) to find a continuous block of memory. This is the reason why allocating large arrays are costly from GC perspective. The buffering solution added as part of apache#5361. The solution buffers also very large uploads to memory. * Limit the replay buffer size to a configurable limit which defaults to 5MB. This is configured with the `httpInputMaxReplayBufferSize` proxy configuration parameter. * Add unit test to see that buffer size gets limited * Add unit test for apache#5361 (cherry picked from commit 2324618) (cherry picked from commit 7fa88cc)
Fixes apache#10908 ### Motivation Pulsar Proxy uses a lot of heap memory when uploading large function jar files. This also leads to high GC activity since a continuous block of memory (byte array for the size of the upload) is allocated. GC will have to do compaction for the heap (which gets fragmented) to find a continuous block of memory. This is the reason why allocating large arrays are costly from GC perspective. The buffering solution added as part of apache#5361. The solution buffers also very large uploads to memory. ### Modifications * Limit the replay buffer size to a configurable limit which defaults to 5MB. This is configured with the `httpInputMaxReplayBufferSize` proxy configuration parameter. * Add unit test to see that buffer size gets limited * Add unit test for apache#5361
Fixes #5360
This adds a small cache of the request body to ensure that it can be
re-sent.