Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

disable sftp + delta test #2052

Merged
merged 2 commits into from
Nov 12, 2024
Merged

disable sftp + delta test #2052

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
da19670
fix delta tests
sh-rp Nov 12, 2024
8250633
update workflows
sh-rp Nov 12, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
18 changes: 16 additions & 2 deletions .github/workflows/test_common.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -116,7 +116,7 @@ jobs:
shell: cmd

- name: Install pyarrow
run: poetry install --no-interaction -E duckdb -E cli -E parquet --with sentry-sdk
run: poetry install --no-interaction -E duckdb -E cli -E parquet --with sentry-sdk && poetry run pip install pyarrow==15.0.2

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

OK! just make sure that this line:

poetry install --no-interaction -E duckdb -E cli -E parquet -E deltalake -E sql_database --with sentry-sdk,pipeline,sources

is not installing arrow 17 again. I see delta is being installed so probably that happens... maybe we should split testing of the lib to a separate step where we install delta

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yeah good point. I think every call to poetry install has the potential to change this. I have updated the workflow not fix this, but I am not quite sure how you would set it up tbh.

- run: |
poetry run pytest tests/pipeline/test_pipeline_extra.py -k arrow
Expand All @@ -129,7 +129,7 @@ jobs:
shell: cmd

- name: Install pipeline and sources dependencies
run: poetry install --no-interaction -E duckdb -E cli -E parquet -E deltalake -E sql_database --with sentry-sdk,pipeline,sources
run: poetry install --no-interaction -E duckdb -E cli -E parquet -E deltalake -E sql_database --with sentry-sdk,pipeline,sources && poetry run pip install pyarrow==15.0.2

- run: |
poetry run pytest tests/extract tests/pipeline tests/libs tests/cli/common tests/destinations tests/sources
Expand All @@ -155,6 +155,20 @@ jobs:
name: Run extract tests Windows
shell: cmd

# here we upgrade pyarrow to 17 and run the libs tests again
- name: Install pyarrow 17
run: poetry run pip install pyarrow==17.0.0

- run: |
poetry run pytest tests/libs
if: runner.os != 'Windows'
name: Run libs tests Linux/MAC
- run: |
poetry run pytest tests/libs
if: runner.os == 'Windows'
name: Run libs tests Windows
shell: cmd

# - name: Install Pydantic 1.0
# run: pip install "pydantic<2"

Expand Down
80 changes: 0 additions & 80 deletions .github/workflows/test_pyarrow17.yml
View file
Open in desktop

This file was deleted.

8 changes: 6 additions & 2 deletions tests/load/pipeline/test_filesystem_pipeline.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -33,6 +33,7 @@
MEMORY_BUCKET,
FILE_BUCKET,
AZ_BUCKET,
SFTP_BUCKET,
)

from tests.pipeline.utils import load_table_counts, assert_load_info, load_tables_to_dicts
Expand Down Expand Up @@ -222,6 +223,9 @@ def some_source():
assert table.column("value").to_pylist() == [1, 2, 3, 4, 5]


@pytest.mark.skip(
reason="pyarrow version check not needed anymore, since we have 17 as a dependency"
)
def test_delta_table_pyarrow_version_check() -> None:
"""Tests pyarrow version checking for `delta` table format.

Expand Down Expand Up @@ -255,7 +259,7 @@ def foo():
destinations_configs(
table_format_filesystem_configs=True,
with_table_format="delta",
bucket_exclude=(MEMORY_BUCKET),
bucket_exclude=(MEMORY_BUCKET, SFTP_BUCKET),
),
ids=lambda x: x.name,
)
Expand Down Expand Up @@ -982,7 +986,7 @@ def parent_delta():
destinations_configs(
table_format_filesystem_configs=True,
with_table_format="delta",
bucket_subset=(FILE_BUCKET,),
bucket_subset=(FILE_BUCKET),
),
ids=lambda x: x.name,
)
Expand Down
1 change: 1 addition & 0 deletions tests/load/pipeline/test_merge_disposition.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -40,6 +40,7 @@
DestinationTestConfiguration,
FILE_BUCKET,
AZ_BUCKET,
SFTP_BUCKET,
)


Expand Down
2 changes: 1 addition & 1 deletion tests/load/utils.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1004,7 +1004,7 @@ def prepare_load_package(
def sequence_generator() -> Generator[List[Dict[str, str]], None, None]:
count = 1
while True:
yield [{"content": str(count + i)} for i in range(2000)]
yield [{"content": str(count + i)} for i in range(3)]
count += 3


Expand Down
Loading