Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

disable sftp + delta test #2052

Merged
merged 2 commits into from
Nov 12, 2024
Merged

disable sftp + delta test #2052

merged 2 commits into from
Nov 12, 2024

Conversation

sh-rp
Copy link
Collaborator

@sh-rp sh-rp commented Nov 11, 2024

Description

The lancedb PR last week broke our CI. Fixes done here are:

  • Make sure delta tests are not run on the sftp bucket
  • revert a change to sequence_generator in our test utils which was breaking weaviate and qdrant tests
  • Remove pyarrow 17 workflow as 17 is the default now
  • Run common tests with pyarrow 15 and run the libs tests again with pyarrow 17
  • Remove pyarrow17 test markers in some cases.

Copy link

netlify bot commented Nov 11, 2024

Deploy Preview for dlt-hub-docs canceled.

Name Link
🔨 Latest commit 8250633
🔍 Latest deploy log https://app.netlify.com/sites/dlt-hub-docs/deploys/673352268244430008ea2c2b

@sh-rp sh-rp force-pushed the fix/deltalake-alternate branch 2 times, most recently from c0e65c7 to ac9e6b0 Compare November 12, 2024 08:36
@sh-rp sh-rp changed the title [do not merge] update deltalake to 0.21 - sftp tests fully disabled [do not merge] disable sftp + delta tests. remove needs pyarrow17 marker Nov 12, 2024
@sh-rp sh-rp changed the title [do not merge] disable sftp + delta tests. remove needs pyarrow17 marker [do not merge] disable sftp + delta test Nov 12, 2024
@sh-rp sh-rp marked this pull request as ready for review November 12, 2024 09:50
@sh-rp sh-rp changed the title [do not merge] disable sftp + delta test disable sftp + delta test Nov 12, 2024
Copy link
Collaborator

@rudolfix rudolfix left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm pretty sure we reinstall arrow 17 too early. pls see my comment

@@ -118,6 +118,9 @@ jobs:
- name: Install pyarrow
run: poetry install --no-interaction -E duckdb -E cli -E parquet --with sentry-sdk

- name: Ensure pyarrow 15
run: poetry run pip install pyarrow==15.0.2

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

OK! just make sure that this line:

poetry install --no-interaction -E duckdb -E cli -E parquet -E deltalake -E sql_database --with sentry-sdk,pipeline,sources

is not installing arrow 17 again. I see delta is being installed so probably that happens... maybe we should split testing of the lib to a separate step where we install delta

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yeah good point. I think every call to poetry install has the potential to change this. I have updated the workflow not fix this, but I am not quite sure how you would set it up tbh.

@sh-rp sh-rp force-pushed the fix/deltalake-alternate branch from e6a6feb to f2156d0 Compare November 12, 2024 12:56
@sh-rp sh-rp force-pushed the fix/deltalake-alternate branch from f2156d0 to 71931df Compare November 12, 2024 12:56
@sh-rp sh-rp force-pushed the fix/deltalake-alternate branch from 71931df to 8250633 Compare November 12, 2024 13:03
@sh-rp sh-rp merged commit 0192a73 into devel Nov 12, 2024
57 of 59 checks passed
@sh-rp sh-rp deleted the fix/deltalake-alternate branch November 12, 2024 16:10
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants