Docs: clarify how to set table location for trino on aws glue #3159
Conversation
|
Interesting, there's some overlap with Athena: https://github.com/TobikoData/sqlmesh/pull/3154/files#diff-810f5d6dd3e7fda3367cac3598147973d1db787a6026543dc9461dc9ac2f4872R29 |
docs/integrations/engines/trino.md
Outdated
| MODEL ( | ||
| name my_schema.my_model, | ||
| physical_properties ( | ||
| location = 's3://my-bucket/' |
There was a problem hiding this comment.
One problem I found setting this at the model level is that the path doesn't automatically get adjusted for each snapshot. So you have the situation where SQLMesh creates multiple snapshots for the model in the same location and they clobber each other.
In the Athena adapter, I solved this by calling the property s3_base_location which allows the user to specify a base URI to put all the snapshots for this model and the adapter uses that to produce the actual table location property, eg <s3_base_location>/<snapshot_table_name>/
There was a problem hiding this comment.
That is definitely a problem! Would it make sense to pull that out into a mixin that we use in both Athena and Trino, or is it too specific?
There was a problem hiding this comment.
I think it could work as a mixin, it would essentially need to detect s3_base_location and rewrite it to location (injecting the snapshot table name) before handing the properties off to the adapter for further processing.
Perhaps a follow-up PR once the Athena one has been reviewed?
treysp
force-pushed
the
trey/trino-glue-location
branch
from
637503e
to
4d3d2bd
Compare
No description provided.