How do you overcome the limitation of 100 active partition writers

[tonicebrian]

I thought that I was able to regenerate the warehouse from the data lake just by deleting the destination table and running DBT again but today I was hit by the limitation:

HIVE_TOO_MANY_OPEN_PARTITIONS: Exceeded limit of 100 open writers for partitions/buckets.

And looking at the documentation here, it seems like a known issue.

Since I'm partitioning by date and this is a sensible thing to do for the data I have, how do people here ingest data that is older than 3 months during the first non incremental run of DBT + Athena?

[nicor88]

have a look at this conversation in the community adapter: dbt-labs/dbt-athena#87
the solution is to use a modified version of https://github.com/dbt-labs/dbt-utils#insert_by_period for athena

[tonicebrian]

Hi @nicor88 thanks for pointing to the right direction. Do you know of any code snippet with this modifications for Athena? It looks a bit intimidating since it has lots of specific macro constructs.