Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

ByteInputStream does not implement mark/reset #2940

Closed
prb112 opened this issue Nov 4, 2021 · 1 comment
Closed

ByteInputStream does not implement mark/reset #2940

prb112 opened this issue Nov 4, 2021 · 1 comment
Assignees
Labels
bug Something isn't working bulk-data P1 Priority 1 - Must Have

Comments

@prb112
Copy link
Contributor

prb112 commented Nov 4, 2021

Describe the bug
A clear and concise description of what the bug is.
The fast Bulk Export implementation uses a custom InputOutput stream paradigm.

[INFO    ] multiPartUpload: Upload part Error - The request to the service failed with a retryable reason, but resetting the request input stream has failed. See exception.getExtraInfo or debug-level logging for the original failure that caused this retry.;  If the request involves an input stream, the maximum stream buffer size can be configured via request.getRequestClientOptions().setReadLimit(int)
[ERROR   ] bulkexportfastjob[102]
The request to the service failed with a retryable reason, but resetting the request input stream has failed. See exception.getExtraInfo or debug-level logging for the original failure that caused this retry.;  If the request involves an input stream, the maximum stream buffer size can be configured via request.getRequestClientOptions().setReadLimit(int)
[INFO    ] bulkexportfastjob[102] processed 5696 resources in 39.88 seconds (rate=142.8 resources/second)
[INFO    ] FFDC1015I: An FFDC Incident has been created: "com.ibm.cloud.objectstorage.ResetException: The request to the service failed with a retryable reason, but resetting the request input stream has failed. See exception.getExtraInfo or debug-level logging for the original failure that caused this retry.;  If the request involves an input stream, the maximum stream buffer size can be configured via request.getRequestClientOptions().setReadLimit(int) com.ibm.jbatch.container.controller.impl.ChunkStepControllerImpl 342" at ffdc_21.11.04_17.17.28.0.log
[INFO    ] FFDC1015I: An FFDC Incident has been created: "com.ibm.jbatch.container.exception.BatchContainerRuntimeException: com.ibm.cloud.objectstorage.ResetException: The request to the service failed with a retryable reason, but resetting the request input stream has failed. See exception.getExtraInfo or debug-level logging for the original failure that caused this retry.;  If the request involves an input stream, the maximum stream buffer size can be configured via request.getRequestClientOptions().setReadLimit(int) com.ibm.jbatch.container.controller.impl.ChunkStepControllerImpl 994" at ffdc_21.11.04_17.17.28.1.log
[INFO    ] FFDC1015I: An FFDC Incident has been created: "com.ibm.jbatch.container.exception.BatchContainerRuntimeException: com.ibm.cloud.objectstorage.ResetException: The request to the service failed with a retryable reason, but resetting the request input stream has failed. See exception.getExtraInfo or debug-level logging for the original failure that caused this retry.;  If the request involves an input stream, the maximum stream buffer size can be configured via request.getRequestClientOptions().setReadLimit(int) com.ibm.jbatch.container.controller.impl.ChunkStepControllerImpl 982" at ffdc_21.11.04_17.17.28.2.log
[INFO    ] FFDC1015I: An FFDC Incident has been created: "com.ibm.jbatch.container.exception.BatchContainerRuntimeException: com.ibm.cloud.objectstorage.ResetException: The request to the service failed with a retryable reason, but resetting the request input stream has failed. See exception.getExtraInfo or debug-level logging for the original failure that caused this retry.;  If the request involves an input stream, the maximum stream buffer size can be configured via request.getRequestClientOptions().setReadLimit(int) com.ibm.jbatch.container.controller.impl.ChunkStepControllerImpl 681" at ffdc_21.11.04_17.17.28.3.log
[ERROR   ] StepChunkListener: job[bulkexportfastjob/82/863] --- com.ibm.cloud.objectstorage.ResetException: The request to the service failed with a retryable reason, but resetting the request input stream has failed. See exception.getExtraInfo or debug-level logging for the original failure that caused this retry.;  If the request involves an input stream, the maximum stream buffer size can be configured via request.getRequestClientOptions().setReadLimit(int)
com.ibm.cloud.objectstorage.ResetException: The request to the service failed with a retryable reason, but resetting the request input stream has failed. See exception.getExtraInfo or debug-level logging for the original failure that caused this retry.;  If the request involves an input stream, the maximum stream buffer size can be configured via request.getRequestClientOptions().setReadLimit(int)
[INFO    ] Transaction failed - afterCompletion(status = 4)
@NotThreadSafe
public class InputOutputByteStream {
... 
 private class ByteInputStream extends InputStream {

Environment
Which version of IBM FHIR Server? 4.10.0-SNAPSHOT

To Reproduce
Steps to reproduce the behavior:

  1. Setup S3 export with the System level export with the Patient resource type.
curl --location --request GET 'https://localhost:9443/fhir-server/api/v4/$export?_outputFormat=application/fhir+ndjson&_type=Patient' \
--header 'X-FHIR-TENANT-ID: default' \
--header 'Content-Type: application/fhir+json' \
--header 'X-FHIR-BULKDATA-PROVIDER: default' \
--header 'X-FHIR-BULKDATA-PROVIDER-OUTCOME: default' \
--header 'Accept: application/fhir+json' \
--header 'Authorization: Basic BBBBBBBB'

Expected behavior
Export should succeed

Additional context
Suspect transfer to bucket failed and was trying to retry.
Possibly This feature is only available in the latest ibm-cos update.

@prb112 prb112 added bug Something isn't working P1 Priority 1 - Must Have labels Nov 4, 2021
@prb112 prb112 added this to the Sprint 2021-15 milestone Nov 4, 2021
@prb112 prb112 self-assigned this Nov 4, 2021
prb112 added a commit that referenced this issue Nov 4, 2021
- Add mark/reset to the InputStream
- Add Test for mark/reset

Signed-off-by: Paul Bastide <pbastide@us.ibm.com>
punktilious added a commit that referenced this issue Nov 5, 2021
ByteInputStream does not implement mark/reset #2940
@d0roppe
Copy link
Collaborator

d0roppe commented Nov 5, 2021

Verified that this fixed the issue with BulkData Export of a large dataset.

@d0roppe d0roppe closed this as completed Nov 5, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working bulk-data P1 Priority 1 - Must Have
Projects
None yet
Development

No branches or pull requests

2 participants