[CT-2761] [Bug] dbt run-operation command does not honor DBT_TARGET_PATH or accept --target-path parameter

[timvw]

Is this a new bug in dbt-core?

• I believe this is a new bug in dbt-core
• I have searched the existing issues, and I could not find an existing issue for this bug

Current Behavior

Currently DBT_TARGET_PATH is not honored

Expected Behavior

DBT_TARGET_PATH is honored

Steps To Reproduce

DBT_TARGET_PATH=/tmp/target dbt run-operation create_job_monitoring_temporary_views

-> A ./target folder is created with following files: graph.gpickle, manifest.json and partial_parse.msgpack

Environment

- OS: mac
- Python: 3.7.16
- dbt: 1.5.1

Which database adapter are you using with dbt?

spark, other (mention it in "Additional Context")

Additional Context

This causes issues in constrained environments where not each (./target) folder is write-able.

When running dbt compile it is clear that DBT_TARGET_PATH is respected

DBT_TARGET_PATH=/tmp/target dbt compile --vars 'partition: 2023/6'

-> No ./target folder is created

[jtcohen6]

@timvw I'm surprised you're noticing this difference in behavior between run-operation and compile — both of them definitely support the target_path parameter.

Could you try upgrading to v1.5.2? We included a fix in that patch release which sounds slightly different from the problem you're having, but could possibly be related:

• [CT-2491] [Bug] Default --target-path should be relative to --project-dir, rather than current working directory #7465
• Target path should be relative to project dir, rather than current working directory #7706

[timvw]

Unfortunately no success

ls target
ls: target: No such file or directory
dbt run-operation create_job_monitoring_temporary_views
DBT_TARGET_PATH=/tmp/dbt dbt run-operation create_job_monitoring_temporary_views
09:18:10  Running with dbt=1.5.2
09:18:10  Registered adapter: spark=1.5.0
...
ls target 
graph.gpickle         manifest.json         partial_parse.msgpack

[timvw]

When I run dbt run-operation --help it does not mention the --target-path parameter
When I run dbt run --help it does mention the --target-path parameter (and DBT_TARGET_PATH)

(I know, in main.py i see that both operations have an @p.target_path attribute..)

But.. When I try to invoke dbt run-operation with a --target-path

dbt run-operation create_job_monitoring_temporary_views --target-path /tmp/dbt
Usage: dbt run-operation [OPTIONS] MACRO
Try 'dbt run-operation -h' for help.

Error: No such option: --target-path Did you mean --target?

[jtcohen6]

Can you try putting the flag before the subcommand? It's a "global" flag:

dbt --target-path /tmp/dbt run-operation create_job_monitoring_temporary_views

(I'm not sure why that would work, and the env var wouldn't work, but just to try!)

[timvw]

The same result, not working..

dbt run-operation --target-path /tmp/dbt create_job_monitoring_temporary_views
Usage: dbt run-operation [OPTIONS] MACRO
Try 'dbt run-operation -h' for help.

Error: No such option: --target-path Did you mean --target?

[jtcohen6]

@timvw I was suggesting that you put the --target-path in between dbt and run-operation

Again, I doubt it would have an actually different result versus setting the DBT_TARGET_PATH env var, but worth a shot just in case

[timvw]

Also no success...

dbt --target-path /tmp/dbt run-operation create_job_monitoring_temporary_views
Usage: dbt [OPTIONS] COMMAND [ARGS]...
Try 'dbt -h' for help.

Error: No such option: --target-path Did you mean --log-path?

[jtcohen6]

Ah! We added target-path to run-operation in #7647, but we only just backported it to 1.5.latest (#7961), for inclusion in the next patch release of v1.5. So this will be fixed when we release v1.5.3 (planned for July):

• [CT-2763] Release v1.5.3 (final) on July 17 #7988

Sorry for the run-around. I'm positive that's what's at play here. I'm going to close this as a duplicate of #7953