Passing Hive configuration properties

[cylldby]

While experimenting with this adapter, I'm trying to find a way to pass configuration properties to Hive.
For example, I would like to be able to run:

set hive.auto.convert.join=true;
set hive.stats.fetch.partition.stats=true;
set hive.vectorized.execution.enabled=true;

SELECT * FROM mytable

Apparently with Impyla we need to pass these properties in a configuration dictionary when execute-ing a query.

Is it possible to attach such a dictionary in a model's config ?

[cylldby]

About passing configuration with Impyla, I'm referring to this

[tovganesh]

Can you try using pre-hooks for settings the properties?
https://docs.getdbt.com/reference/resource-configs/pre-hook-post-hook

[cylldby]

Thank you for the fast reply !

Unfortunately using pre-hooks fails.
I have a query that runs fine on its own. But as soon as I tried to add for instance

{{
    config(
        materialized='table',
        pre_hook="set hive.exec.dynamic.partition=true;"
    )
}}

I get the following error:

Unable to establish connection to Hive server: Error while compiling statement: FAILED: ParseException line 3:12 cannot recognize input near 'set' 'hive' '.' in statement

[tovganesh]

Ok. Let me explore this at our end and get back to you.

[cylldby]

Thank you very much ! 🙏
I will look on my end too. It looks tricky 😓

[cylldby]

For reference, there is this issue in the dbt-spark repository, with similar target.
In there they suggest the addition of a field in the profiles.yml file to contain the desired configuration, which would then be passed to the connection object.

It is quite interesting.. An ideal situation would be to be able to pass configuration at model level but I'm not sure we have such possibility.

[tovganesh]

The parse error for pre_hook seems strange. Apparently dbt adds a comment for query when it is executing, and Hive doesn't seem to honour comment when using set.

So:

/*dbt some comment */
set hive.exec.dynamic.partition=true;

flags an error, while:

set hive.exec.dynamic.partition=true;

works!
Verified this in Hue. Let me check how I can handle this in adapter.

[tovganesh]

Are you ok to try out this draft PR: #89 ?

[cylldby]

Sorry for the delay !

I could test the PR against a test table and it seems it solves the issue !
I could include a list of SET <parameter>=<value>; in a pre-hook in my model config, and confirmed they were executed with dbt --debug run.

Thank you very much for the fast fixing !

Let me know if I can do anything else.

[wuguozhu]

In what version is this issue fixed? We need a solution urgently

[tovganesh]

@wuguozhu you can install from the source for the fix. There is no updated pip package.