-
Notifications
You must be signed in to change notification settings - Fork 4.2k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Source S3: keep processing but warn if OSError happen #21604
Conversation
/test connector=connectors/source-s3
Build PassedTest summary info:
|
self.logger.warn(f"We don't have access to {self.url}. " | ||
f"Check whether key {self.url} exists in `{bucket}` bucket and/or has proper ACL permissions") | ||
raise e |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Does this produce an AirbyteTraceError message? Just logging to stdout won't help users learn what to do
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
cc @brianjlai maybe you recall the answer - does throwing an exception here end up making a useable AirbyteTraceErrorMessage?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
It does. But in this case I catch it in two possible places where it can be raised and handle it
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Ok, this matches the philosophy that @erohmensing is working on with streams being unavailable - we sync what we can and log an error for the streams we can't reach. 👍
/publish connector=connectors/source-s3
if you have connectors that successfully published but failed definition generation, follow step 4 here |
Airbyte Code Coverage
|
What
Handle OSError which means no access to key in a bucket. Warn and keep working on another files which we have an access to