Use glaredb with dbt/great expectations: fix postgres functionality #2436
Comments
talagluck
changed the title
talagluck
changed the title
|
Steps to reproduce for Great Expectations: In a notebook (or python script) run: import great_expectations as gx
context = gx.get_context() # gets a great expectations project context
ds = context.sources.add_postgres(name="glaredb", connection_string=<CONNECTION_STRING>)The connection string I used was |
|
Good news, I think I got further (thanks to correcting the transaction stubs and connecting to qa). Bad news, seems great expectation is expecting a But this seems easy enough to add in. |
|
Cool! Yes, it can be kind of challenging to pass in arguments to Great Expectations that it isn't expecting. But my hunch is something like what Sam said, that there is an issue with psycopg2 or SQL Alchemy, which is why this isn't working with a few different tools. |
|
@scsmithr Is the next logical step for this then to implement |
|
I added this last week, and a test to confirm: https://github.com/GlareDB/glaredb/blob/main/testdata/sqllogictests/functions/version.slt#L6-L14 That combined with the pg_tables thing I think will get us further here. I think we can retest (w/ qa) and see what happens. |
|
I hit a new issue now with dbt: |
universalmind303
added a commit
that referenced
this issue
Jan 29, 2024
@talagluck this should fix the great-expectations issue you referenced in #2436 --------- Co-authored-by: Sean Smith <scsmithr@gmail.com>
|
Just to double check, it would be good to verify if there are other issues, and understand where we're at on this. (@talagluck) It might also be good, to see what the script is that you're using to test this, then we can cut out the longer iteration cycle. |
tychoish
pushed a commit
that referenced
this issue
Feb 1, 2024
@talagluck this should fix the great-expectations issue you referenced in #2436 --------- Co-authored-by: Sean Smith <scsmithr@gmail.com>
|
On the latest test, I get: I've been working on a script today. It's a bit more involved than I expected, but I'll have something soon. |
|
PR with test is here: #2580 |
universalmind303
pushed a commit
that referenced
this issue
Feb 8, 2024
These are to make another step toward fixing #2436. I built the definitions off of https://www.postgresql.org/docs/16/catalog-pg-depend.html and https://www.postgresql.org/docs/16/catalog-pg-rewrite.html, but please double-check that I did this correctly. --------- Co-authored-by: Grey <grey@glaredb.com>
|
@talagluck is this closable now, or what's the tl:dr status update? |
|
Yup! I'll close this and open up more specific issues as needed. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Description
When attempting to create a connection from dbt to a glaredb cloud instance using a postgresql connection string, I hit the following error:
Given that dbt should work fine with a postgres connector, I would expect this to work, but there's a mismatch here. Sean mentioned that he tested this a few months back and it worked, so there may have been a regression here.
I also get the same error when attempting to run Great Expectations with GlareDB - Great Expectations uses SqlAlchemy under the surface.
The text was updated successfully, but these errors were encountered: