[KED-2141] Impala support for kedro, custom SQLdataset

[noklam]

Description

To use kedro with impala. Is is possible to extend the current SQLDataset to create a connection instead of accepting a string only?

Context

Integration with Impala is useful.

Possible Implementation

I am using the python package impyla. To query something with pandas, I need to first create a connection.

from impala.dbapi import connect

user=user
password=password
host=host

conn = connect(host=host, port=21050, user=user, password=password)
sql = 'select * from trable'
pd.read_sql(sql, conn)

In theory, as long as I can implement the connection creation logic manually, the rest should just match the standard pandasSQLdataset.

[yetudada]

Hi @noklam! Thank you for this query. We think it may be possible to support Impala in the way that you would prefer by creating your own version of the SQLDataSet. We detail how to create a custom dataset in our documentation: https://kedro.readthedocs.io/en/stable/07_extend_kedro/01_custom_datasets.html

Do let us know if this would be helpful, if so then I'll be able to close this issue.

[noklam]

@yetudada I think as long as I can create a connection, it should fit SQLalchemy api.

However, I am struggling to find a reliable solution to connect to Impala on Window.
I tried odbc and impyla, both often fails to connect or even freeze the program.

would be great if someone from kedro have experiene on this.

[921kiyo]

I've logged how Kedro could support with Impala :)

[brendalf]

Is there anything left to do here?

[merelcht]

Hi @brendalf, there hasn't been an attempt yet to implement this, so if you're interested in helping out we'd be more than happy to accept a PR 😄

[noklam]

Any thoughts from Kedro team? If not I may try to implement one, but cannot guarantee it works for every platform.

I am mainly a Window user, getting impala to works in Window is a bit tricky.

[merelcht]

@noklam This isn't a priority for the team at the moment, so we'd be very happy to accept a PR.

[noklam]

Cool, see if I can come up with a PR this week or next week.

…

On Wed, 10 Mar 2021 at 5:12 PM, Merel Theisen ***@***.***> wrote: @noklam <https://github.com/noklam> This isn't a priority for the team at the moment, so we'd be very happy to accept a PR. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#504 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AELAWL5S5DDXFB6IUHFBJQ3TC4ZYHANCNFSM4Q7NYALQ> .

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.