[low code connectors] read configs from package_data #15810
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
sherifnada
approved these changes
Aug 19, 2022
brianjlai
commented
Aug 19, 2022
| @@ -65,9 +66,11 @@ def streams(self, config: Mapping[str, Any]) -> List[Stream]: | |||
| return [self._factory.create_component(stream_config, config, True)() for stream_config in self._stream_configs()] | |||
|
|
|||
| def _read_and_parse_yaml_file(self, path_to_yaml_file): | |||
| with open(path_to_yaml_file, "r") as f: | |||
There was a problem hiding this comment.
open() only works running locally and I am not sure how. this passes SAT, something we should follow up on. But replacing this with how we read spec files fixes the issue as long as we mount the package files in the connectors setup.py
brianjlai
commented
Aug 19, 2022
| @@ -39,7 +39,7 @@ def check_connection(self, source: Source, logger: logging.Logger, config: Mappi | |||
| records = stream.read_records(sync_mode=SyncMode.full_refresh) | |||
| next(records) | |||
| except Exception as error: | |||
| return False, f"Unable to connect to stream {stream} - {error}" | |||
| return False, f"Unable to connect to stream {stream_name} - {error}" | |||
There was a problem hiding this comment.
for declarative yaml files we end up printing the entire yaml config in the UI which is pretty ugly and makes it hard to see the real error message
rodireich
pushed a commit
that referenced
this pull request
Aug 25, 2022
* read configs from package_data * update changelog and setup * commenting out failing tests in the short term
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What
additional fix for https://github.com/airbytehq/oncall/issues/462
yaml_declarative_source.pyis not properly reading files when running in a container. The working directory during an operation is not structured the same as the static files. Instead for each connector insetup.pywe need to import yaml files so they can be used at runtime. And we can only access them usingpkgutil.get_data(). This is the same way that read spec.yaml and spec.json files.I tested this on a locally running instance of Airbyte. After this change goes out, we need to update all 3 connectors to the latest version of the CDK and add
*.yamlto setup.py and change the path_to_yaml tosentry.yamlor whatever source we're fixing.How
Replace the existing
open()command that only works locally with a file reader frompkgutil