Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Improve Airflow parser functionality #2418

Merged
merged 34 commits into from
Feb 7, 2022
Merged
Show file tree
Hide file tree
Changes from 22 commits
Commits
Show all changes
34 commits
Select commit Hold shift + click to select a range
97d3b6a
Resolve naming conflict in case of multiple operators
kiersten-stokes Jan 14, 2022
9051b9e
Initial changes to use AST for parsing
kiersten-stokes Jan 19, 2022
9ec6812
Refactor to support AST parse
kiersten-stokes Jan 19, 2022
f4bfa0e
Improve docstrings and type hints
kiersten-stokes Jan 21, 2022
d2962ea
Improve refactoring
kiersten-stokes Jan 25, 2022
7f1ce33
Clean up comments and TODO's
kiersten-stokes Jan 25, 2022
516cdee
Fix incorrect type hint
kiersten-stokes Jan 25, 2022
0a8c144
Add log messaging
kiersten-stokes Jan 26, 2022
2ceabf3
Fix typo
kiersten-stokes Jan 26, 2022
2ce38f5
Change the way operators are id'ed from imported packages
kiersten-stokes Jan 26, 2022
f1dc3d4
Fix long-running test issue
kiersten-stokes Jan 26, 2022
c280264
Fix type regex for docstring searches
kiersten-stokes Jan 26, 2022
b04cd1b
Fix determination of 'required' and address minor review comments
kiersten-stokes Jan 27, 2022
e632182
Add temporary log messages re: type
kiersten-stokes Jan 27, 2022
0709245
Fix indentation
kiersten-stokes Jan 27, 2022
2595cb7
Account for ast variable access in different python versions
kiersten-stokes Jan 27, 2022
5a93629
Remove astunparse from setup.py
kiersten-stokes Jan 27, 2022
07995ca
Replace continue with break in loop
kiersten-stokes Jan 27, 2022
db1b6c2
Fix type determination for complex type hints
kiersten-stokes Jan 28, 2022
4ecfb7b
Fix and streamline docstring regexes
kiersten-stokes Jan 28, 2022
6d5014e
Fix type check for _get_class_docstring
kiersten-stokes Jan 28, 2022
95ae932
Fix ast var access in different python versions (round 2)
kiersten-stokes Jan 28, 2022
f4a627b
Fix regex to replace newlines with spaces
kiersten-stokes Jan 31, 2022
4ea33f8
Set default value in _parse_from_docstring
kiersten-stokes Jan 31, 2022
673b85a
Make first pass at test fixes
kiersten-stokes Jan 31, 2022
3bcf4b3
Remove debug log messages
kiersten-stokes Jan 31, 2022
38c2b3b
Cover missing case for pasting in python 3.7 and lower
kiersten-stokes Feb 1, 2022
104228b
Clean up and add comments to test case
kiersten-stokes Feb 1, 2022
a5825e1
Remove comment
kiersten-stokes Feb 2, 2022
51e8075
Merge branch 'master' into aa-parser-improvements
kiersten-stokes Feb 2, 2022
5dc8c1c
Add display_name to metadata available to connector implementations
kiersten-stokes Feb 4, 2022
93bed91
Address review re: pytest fixture
kiersten-stokes Feb 7, 2022
61b5470
Amend module regex to include upper case and digits
kiersten-stokes Feb 7, 2022
a283b8e
Fix lint
kiersten-stokes Feb 7, 2022
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
22 changes: 22 additions & 0 deletions conftest.py
Original file line number Diff line number Diff line change
Expand Up @@ -22,6 +22,8 @@

pytest_plugins = ["jupyter_server.pytest_plugin"]

TEST_CATALOG_NAME = 'new_test_catalog'

KFP_COMPONENT_CACHE_INSTANCE = {
"display_name": "KFP Example Components",
"metadata": {
Expand All @@ -40,6 +42,7 @@
"schema_name": "elyra-airflow-examples-catalog"
}


@pytest.fixture
def component_cache_instance(request):
"""Creates an instance of a component cache and removes after test."""
Expand Down Expand Up @@ -68,3 +71,22 @@ def component_cache_instance(request):
# Test was not parametrized, so component instance is not needed
except AttributeError:
yield None


@pytest.fixture
def teardown_test_catalog():
"""
This fixture is run after certain tests that modify the component catalog.
This way, the catalog instance will be removed even when the test fails
"""
metadata_manager = MetadataManager(schemaspace=ComponentCatalogs.COMPONENT_CATALOGS_SCHEMASPACE_ID)

# Run test
yield

# Remove test catalog
try:
if metadata_manager.get(TEST_CATALOG_NAME):
metadata_manager.remove(TEST_CATALOG_NAME)
except Exception:
pass
Loading