Cannot mount Snowflake as a data source behind a proxy #413
Comments
|
Thanks for noticing this! I added a parameter to the data source that can be used to inject arbitrary envvars on the engine side when running the query. Testing (don't have a proxy handy but can verify it fails to connect, though it does take some time to fail, could be some retry logic in $ sgr mount snowflake test_snowflake -o@- <<EOF
{
"username": "uname",
"password": "pass"
"account": "acc",
"database": "SNOWFLAKE_SAMPLE_DATA",
"schema": "TPCH_SF100",
"envvars": {"HTTPS_PROXY": "https://127.0.0.1"}
}
EOF
error: psycopg2.errors.InternalError_: Error in python: OperationalError
error: DETAIL: (snowflake.connector.errors.OperationalError) 250003: 250003: Failed to execute request: HTTPSConnectionPool(host='wp71792.west-europe.azure.snowflakecomputing.com', port=443): Max retries exceeded with url: /session/v1/login-request?request_id=019f2a01-f306-4446-b224-ab28481ab1b6&databaseName=SNOWFLAKE_SAMPLE_DATA&schemaName=TPCH_SF100&request_guid=0e7a9b9a-2aae-4648-919f-4e2f40f1d6f7 (Caused by ProxyError('Cannot connect to proxy.', NewConnectionError('<snowflake.connector.vendored.urllib3.connection.HTTPSConnection object at 0x7f8394940f28>: Failed to establish a new connection: [Errno 111] Connection refused')))
error: (Background on this error at: http://sqlalche.me/e/14/e3q8) |
|
@mildbyte thanks for the quick response. Much appreciated. Looking at the errors, I realized I am running So I worked around this by starting up our split engine with HTTPS_PROXY, HTTP_PROXY and NO_PROXY passed in Back to this issue, would allowing |
|
It's also an option. The difference is, on splitgraph.com/Splitgraph Cloud we manage these engines separately from connectors (e.g. one engine might end up handling several queries, powered by different connectors), so we can't set the envvars there globally, since we might not know them at the engine startup and we don't want to change them in the context of the whole engine. |
|
That makes sense. On a side note, would we consider supporting key pair auth [1]? Password auth is frowned upon inside the firm. [1] https://github.com/splitgraph/snowflake-sqlalchemy#key-pair-authentication-support |
|
@mapshen It's definitely possible. We could pass this arg to the adapter: connect_args={
'private_key': pkb,
},
)which would also mean other SQLAlchemy adapters would be able to use The only issue in this case is that you'd need to pass your passwordless private key as a connection param on the command line, rather than point |
|
(PR autoclosed; will keep open for the private_key functionality) |
There seems no better way based on the current design. Nevertheless, we don't have to send the key over the public internet as we do with the password. |
|
@mapshen Privkey support is now available: sgr mount snowflake test_snowflake -o@- <<EOF
{
"username": "uname",
"private_key": "MIIEvQIBADANBgkq...", <-- can also use the standard ===== BEGIN PRIVATE KEY ===== ... format
"account": "acc",
"database": "SNOWFLAKE_SAMPLE_DATA",
"schema": "TPCH_SF100"
}
EOF |
* Snowflake data source improvements:
* Allow passing envvars to set HTTP proxy parameters, fix incorrect query string generation when passing a warehouse (#414, #413)
* Support for authentication using a private key (#418)
* Splitfiles: relax AST restrictions to support all SELECT/INSERT/UPDATE/DELETE statements (#411)
* Change the default installation port to 6432 and handle port conflicts during install (#375)
* Add retry logic to fix registry closing the SSL connection after 30 seconds, close remote connections in some places (#417)
|
@mildbyte not sure if this is a regression caused by ef427f6, but when I run in 0.2.11, i get the following error: |
|
@mapshen Can you send me the output of |
|
Nevermind, just managed to reproduce. Fix incoming. |
|
FWIW, here it is: |
|
Should be fixed now (#421) -- the reason was that sqlalchemy didn't like us passing an empty You shouldn't need to upgrade (you can also build it yourself with ) |
|
Thanks for the quick turnaround! I spun up another engine with now the error becomes: On the engine side it is: Would mind taking a further look? It seems something is wrong with but doesn't if I leave out |
|
It might be a limitation of sqlalchemy + snowflake itself where you have to pass in a schema name (e.g. |
|
I do have a schema specified and you can see from the above errors that |
|
|
|
Ah! You want to flip these (database and schema): |
|
You're right! Thanks for pointing this out. I've got the db and schema misplaced. Should've had my today's coffee before posting this :) |
We are evaluating splitgraph in our corporate env, where we have to go through a proxy to reach Snowflake.
When we do
we get the following error
It seems although the snowflake connector Splitgraph uses supports proxy settings via
HTTPS_PROXY,HTTP_PROXYandNO_PROXY[1], but itpsycopg2doesn't pass them over while creating new servers [2]?[1] https://github.com/splitgraph/snowflake-sqlalchemy#using-a-proxy-server
[2] https://github.com/splitgraph/splitgraph/blob/3cc20ef9021c153344cb0e52247dcc9162812d50/splitgraph/hooks/data_source/fdw.py#L259
The text was updated successfully, but these errors were encountered: