Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

New plugins Arrow Flight and Dremio #169

Merged
merged 10 commits into from
Oct 26, 2023

Conversation

iNikitaGricenko
Copy link
Contributor

What changes are being made and why?

Added Dremio jdbc plugin and Apache Arrow Flight. Splited both plugins as separated plugins

How the changes have been QAed?

Trough Dremio test drive and Dremio account

id : dremio-listen
namespace: dev

triggers:
  - id: watch
    type: io.kestra.plugin.jdbc.dremio.Trigger
    url: "jdbc:arrow-flight-sql://localhost:31010/?useEncryption=false"
    username: "dremio"
    password: "dremio123"
    sql: "SELECT * from \"dremio-test\".\"postgres.public\".departments"
    fetch: true
    interval: PT10S

tasks:
  - id: query
    type: io.kestra.plugin.jdbc.dremio.Query
    url: "jdbc:arrow-flight-sql://localhost:31010/?useEncryption=false"
    username: "dremio"
    password: "dremio123"
    sql: "SELECT * from \"dremio-test\".\"postgres.public\".departments"
    fetchOne: true

Setup Instructions

@anna-geller anna-geller changed the title Splited plugins Arrow flight And Dremio #1854 #162 New plugins Arrow Flight and Dremio Oct 24, 2023
@anna-geller
Copy link
Member

closes kestra-io/kestra#1854

split plugins from #162

Copy link
Member

@loicmathieu loicmathieu left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It is missing tests for Query and Trigger tasks.
In your test, please create a table with multiple row types (as done in other JDBC plugins) to check that the CellConverter didn't need to handle type conversions.

@anna-geller
Copy link
Member

anna-geller commented Oct 25, 2023

@iNikitaGricenko could you update the PR body with an Arrow Flight example that worked for you with a Dremio Test Drive account?

@iNikitaGricenko
Copy link
Contributor Author

@anna-geller, databases that are given into Dremio Test Drive does not support apache arrow flight

@anna-geller
Copy link
Member

I'm sorry, I didn't know that.

anna-geller and others added 2 commits October 26, 2023 15:15
Co-authored-by: Loïc Mathieu <loikeseke@gmail.com>
…rrowflight/package-info.java

Co-authored-by: Loïc Mathieu <loikeseke@gmail.com>
@anna-geller anna-geller merged commit 98cd68b into kestra-io:master Oct 26, 2023
1 check failed
@iNikitaGricenko iNikitaGricenko deleted the arrow-flight-spit branch October 26, 2023 13:18
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants