With `dbt run` the `--profiles-dir` becomes relative to `--project-dir` #3133

JCZuurmond · 2021-02-26T13:12:29Z

Describe the bug

The --profiles-dir flag becomes relative to the --project-dir flag when running dbt run. It does not for dbt debug.

Steps To Reproduce

Let's say we have the profiles.yml in the current working directory and our dbt project in a sub-directory jaffle-shop. Then the following command runs without a problem (if all is set-up correctly):

dbt debug --project-dir jaffle-shop/ --profiles-dir .

However, with dbt run this does not work.

dbt run --project-dir jaffle-shop/ --profiles-dir .     # can not find profile

This is because for dbt run the profiles directory argument becomes relative to the project directory. Ergo, the following does work:

dbt run --project-dir jaffle-shop/ --profiles-dir ..     # works unexpectedly

Expected behavior

The --profile-dir flag should behave the same for all dbt commands.

Screenshots and log output

N/A

System information

N/A/

The output of dbt --version:

installed version: 0.19.0
   latest version: 0.19.0

Up to date!

Plugins:
  - bigquery: 0.19.0
  - snowflake: 0.19.0
  - redshift: 0.19.0
  - postgres: 0.19.0
  - sqlserver: 0.19.0.1

The operating system you're using:

ProductName:    macOS
ProductVersion: 11.2.1
BuildVersion:   20D75

The output of python --version:

Python 3.7.2

Additional context

Add any other context about the problem here.

The text was updated successfully, but these errors were encountered:

jtcohen6 · 2021-03-01T11:29:04Z

@JCZuurmond Nice catch, thanks for opening!

The --profile-dir flag should behave the same for all dbt commands.

Agreed!

I think that it would makes more sense for the --profile-dir path to be relative to the current working directory, rather than relative to the project directory (i.e. value of --project-dir). I think this will be tricky to implement, because (almost) all tasks in dbt first move their working directory to the project file identified in --project-dir:

https://github.com/fishtown-analytics/dbt/blob/344a14416d22f0cfbeb56b9904092c8a4f38b1fc/core/dbt/task/base.py#L141-L156

For consistency, we could make dbt debug move to the nearest project directory as well, to better emulate the behavior of the runnable tasks. But I'm not convinced that will solve our problem. It's actually quite odd the way this works today: the read_profile method is called first by read_user_config, from the original working directory (before moving), and then render_from_args calls it again, after the move to the project directory (if it's a runnable task). So unless a profile by the right name is present in a profiles.yml file for both the original working directory + --profiles-dir and --project-dir + --profiles-dir, dbt will raise an exception.

I'd prefer to find a way to preserve the original working directory. Perhaps by storing and reusing the path value used by read_user_config before the move, and finding it again when it's time to render the profile later on?

Are these code paths you'd be interested in taking a further look down?

JCZuurmond · 2021-03-01T16:38:27Z

It would be fun to commit to dbt! I had a look at the code to see where this behavior was created, but it was hard for me to follow the code path, though.

For example:

how do you know which task is called here? I can see that the cls attribute is set for each subparser, but I do not understand where you select a specific sub parser by defining a subcommand

Wouldn't an easy solution be to make the --profiles-dir a full path right after parsing the arguments? Then it does not matter if the working directory is changed

JCZuurmond · 2021-03-01T16:40:12Z

And could you give any pointers on who we test this?

JCZuurmond · 2021-03-01T17:48:52Z

So unless a profile by the right name is present in a profiles.yml file for both the original working directory + --profiles-dir and --project-dir + --profiles-dir, dbt will raise an exception.

I think something more unexpectedly could happen. If you give a --profiles-dir without a profiles.yml then the read_profile called here does not fail - the function returns an empty UserConfig instance. Then here, the tracking directory is set-up for the --profiles-dir (which does not contain a profiles.yml). And if your --project-dir contains a (valid) profiles.yml the rest will execute correctly

I think it is unexpected behavior to not raise an error when there is no profiles.yml found in read_profile, but we raise a DbtProfileError when the profiles.yml is empty.

Btw, what is the difference between a ValidationError and a ValidationException?

jtcohen6 · 2021-03-04T12:10:52Z

Wouldn't an easy solution be to make the --profiles-dir a full path right after parsing the arguments? Then it does not matter if the working directory is changed

Yes, this feels like a sensible valid approach for --profiles-dir (and maybe --project-dir, too). The only consideration that comes to my mind: We store the args object in the run results artifact, so it's likely that users would pass a relative path and see the absolute path appear in the artifact.

And could you give any pointers on who we test this?

There are extant integration tests for each of --profiles-dir (TestCLIInvocationWithProfilesDir) and --project-dir (TestCLIInvocationWithProjectDir). I think this issue asks for a test that combines elements of both, i.e. TestCLIInvocationWithProfilesDirAndProjectDir.

I think it is unexpected behavior to not raise an error when there is no profiles.yml found in read_profile, but we raise a DbtProfileError when the profiles.yml is empty.

Yeah, I agree, this is confusing and contrary to my expectations. I wonder if this was here explicitly to handle the case where a profile filing is missing in the first pass (before moving to --project-dir), but present in the second pass (after moving to --project-dir). For what it's worth, the structuring of that code is fairly old (4780c4b), and I'm not opposed to changing it, though it seems separate from (if adjacent to) the current issue.

Btw, what is the difference between a ValidationError and a ValidationException?

As I understand it, a ValidationError is a jsonschema error, thrown when a python object does not match the properties/types expected by its dataclass. This exception is raised while parsing and validating the files in the project. By (slim) contrast, a ValidationException is a runtime error that dbt throws when a function/method needs to perform a specific operation, and the value supplied doesn't meet the need.

It doesn't strike me as a super meaningful distinction, more a vestige of how the codebase has developed.

JCZuurmond · 2021-03-10T17:41:24Z

Hi @jtcohen6, just want to mention that I am having a go at this. However, it takes some time to get the environment set-up right before I can start running the tests, soooo it is work in progress 🚧

JCZuurmond · 2021-03-17T08:58:37Z

@jtcohen6 : Could you maybe help me with the following?

If I run make test-unit I get an error when building the Dockerfile.test:

$ make test-unit
Unit test run starting...
Building test
Step 1/17 : FROM ubuntu:18.04
 ---> c3c304cb4f22
Step 2/17 : ENV DEBIAN_FRONTEND noninteractive
 ---> Using cache
 ---> eb84108341ee
Step 3/17 : RUN apt-get update     && apt-get dist-upgrade -y     && apt-get install -y --no-install-recommends     netcat     postgresql     curl     git     ssh     software-properties-common     make     build-essential     ca-certifi
cates     libpq-dev     libsasl2-dev     libsasl2-2     libsasl2-modules-gssapi-mit     libyaml-dev     unixodbc-dev     && add-apt-repository ppa:deadsnakes/ppa     && apt-get install -y     python     python-dev     python-pip     pyth
on3.6     python3.6-dev     python3-pip     python3.6-venv     python3.7     python3.7-dev     python3.7-venv     python3.8     python3.8-dev     python3.8-venv     python3.9     python3.9-dev     python3.9-venv     && apt-get clean     
&& rm -rf /var/lib/apt/lists/* /tmp/* /var/tmp/*                                                                                                                                                                                            
 ---> Running in 22f12d9c63c0
Get:1 http://archive.ubuntu.com/ubuntu bionic InRelease [242 kB]
Get:2 http://security.ubuntu.com/ubuntu bionic-security InRelease [88.7 kB]
Get:3 http://archive.ubuntu.com/ubuntu bionic-updates InRelease [88.7 kB]
Get:4 http://archive.ubuntu.com/ubuntu bionic-backports InRelease [74.6 kB]
Get:5 http://security.ubuntu.com/ubuntu bionic-security/multiverse amd64 Packages [24.5 kB]
Err:5 http://security.ubuntu.com/ubuntu bionic-security/multiverse amd64 Packages
  File has unexpected size (13220 != 24531). Mirror sync in progress? [IP: 91.189.88.142 80]
  Hashes of expected file:
   - Filesize:24531 [weak]
   - SHA256:875198db945d73c55df8022e7b8e7418e393c28fc072cc138b7be45a51d28298
   - SHA1:bf23a75361391ae602dfd00159467a709214db2a [weak]
   - MD5Sum:796b20550a47c7daf797cfe8599c3f74 [weak]
  Release file created at: Wed, 17 Mar 2021 08:00:44 +0000
Get:6 http://security.ubuntu.com/ubuntu bionic-security/restricted amd64 Packages [333 kB]
Err:6 http://security.ubuntu.com/ubuntu bionic-security/restricted amd64 Packages
  
Get:7 http://archive.ubuntu.com/ubuntu bionic/main amd64 Packages [1344 kB]
Get:8 http://security.ubuntu.com/ubuntu bionic-security/main amd64 Packages [1995 kB]
Err:8 http://security.ubuntu.com/ubuntu bionic-security/main amd64 Packages
  
Get:7 http://archive.ubuntu.com/ubuntu bionic/main amd64 Packages [1344 kB]
Err:7 http://archive.ubuntu.com/ubuntu bionic/main amd64 Packages
  File has unexpected size (13184 != 1343916). Mirror sync in progress? [IP: 91.189.88.142 80]
  Hashes of expected file:
   - Filesize:1343916 [weak]
   - SHA256:ff7fd80e902a1acfba06e7b513711da31abe915d95b3ba39ce198e02efd209e5
   - SHA1:edd753d5bd860c126016feca4e898a736999915b [weak]
   - MD5Sum:9f666ceefac581815e5f3add8b30d3b9 [weak]
  Release file created at: Thu, 26 Apr 2018 23:37:48 +0000
Get:9 http://security.ubuntu.com/ubuntu bionic-security/universe amd64 Packages [1398 kB]
Err:9 http://security.ubuntu.com/ubuntu bionic-security/universe amd64 Packages
  
Get:10 http://archive.ubuntu.com/ubuntu bionic/multiverse amd64 Packages [186 kB]
Err:10 http://archive.ubuntu.com/ubuntu bionic/multiverse amd64 Packages
  
Get:11 http://archive.ubuntu.com/ubuntu bionic/restricted amd64 Packages [13.5 kB]
Err:11 http://archive.ubuntu.com/ubuntu bionic/restricted amd64 Packages
  
Get:12 http://archive.ubuntu.com/ubuntu bionic/universe amd64 Packages [11.3 MB]
Err:12 http://archive.ubuntu.com/ubuntu bionic/universe amd64 Packages
  
Get:13 http://archive.ubuntu.com/ubuntu bionic-updates/multiverse amd64 Packages [31.4 kB]
Get:14 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 Packages [2425 kB]
Get:15 http://archive.ubuntu.com/ubuntu bionic-updates/restricted amd64 Packages [363 kB]
Get:16 http://archive.ubuntu.com/ubuntu bionic-updates/universe amd64 Packages [2165 kB]
Get:17 http://archive.ubuntu.com/ubuntu bionic-backports/main amd64 Packages [11.3 kB]
Get:18 http://archive.ubuntu.com/ubuntu bionic-backports/universe amd64 Packages [11.4 kB]
Fetched 5501 kB in 2s (2501 kB/s)
Reading package lists...
E: Failed to fetch http://security.ubuntu.com/ubuntu/dists/bionic-security/multiverse/binary-amd64/by-hash/SHA256/875198db945d73c55df8022e7b8e7418e393c28fc072cc138b7be45a51d28298  File has unexpected size (13220 != 24531). Mirror sync in
 progress? [IP: 91.189.88.142 80]                                                                                                                                                                                                           
   Hashes of expected file:
    - Filesize:24531 [weak]
    - SHA256:875198db945d73c55df8022e7b8e7418e393c28fc072cc138b7be45a51d28298
    - SHA1:bf23a75361391ae602dfd00159467a709214db2a [weak]
    - MD5Sum:796b20550a47c7daf797cfe8599c3f74 [weak]
   Release file created at: Wed, 17 Mar 2021 08:00:44 +0000
E: Failed to fetch http://security.ubuntu.com/ubuntu/dists/bionic-security/restricted/binary-amd64/by-hash/SHA256/df5e3cf297eeae595f8f0975c538104edf9897c29bd64df0177f37e58adbcfe6  
E: Failed to fetch http://security.ubuntu.com/ubuntu/dists/bionic-security/main/binary-amd64/by-hash/SHA256/802a85111726038ac4d10d924e17d73e3539e1ff06a72692d369302f27c0e98f  
E: Failed to fetch http://security.ubuntu.com/ubuntu/dists/bionic-security/universe/binary-amd64/by-hash/SHA256/ec3ff33c6b45a434d7a9883732e169c555ab9a5fbb7572599c87b2f97015c4a0  
E: Failed to fetch http://archive.ubuntu.com/ubuntu/dists/bionic/main/binary-amd64/by-hash/SHA256/ff7fd80e902a1acfba06e7b513711da31abe915d95b3ba39ce198e02efd209e5  File has unexpected size (13184 != 1343916). Mirror sync in progress? [IP
: 91.189.88.142 80]                                                                                                                                                                                                                         
   Hashes of expected file:
    - Filesize:1343916 [weak]
    - SHA256:ff7fd80e902a1acfba06e7b513711da31abe915d95b3ba39ce198e02efd209e5
    - SHA1:edd753d5bd860c126016feca4e898a736999915b [weak]
    - MD5Sum:9f666ceefac581815e5f3add8b30d3b9 [weak]
   Release file created at: Thu, 26 Apr 2018 23:37:48 +0000
E: Failed to fetch http://archive.ubuntu.com/ubuntu/dists/bionic/multiverse/binary-amd64/by-hash/SHA256/910cb989ed0e55b8c98589881a57730a36f074123991366c0263e61582a0e156  
E: Failed to fetch http://archive.ubuntu.com/ubuntu/dists/bionic/restricted/binary-amd64/by-hash/SHA256/81b9542ff39f796dd83159d5ef02161232ae0b766538d8fc02a299fc1ced1f4d  
E: Failed to fetch http://archive.ubuntu.com/ubuntu/dists/bionic/universe/binary-amd64/by-hash/SHA256/ca221e8754c933c636b6c0a344617e3444a7c8cb0982ca97725cda9b7bfe1e6a  
E: Some index files failed to download. They have been ignored, or old ones used instead.
ERROR: Service 'test' failed to build : The command '/bin/sh -c apt-get update     && apt-get dist-upgrade -y     && apt-get install -y --no-install-recommends     netcat     postgresql     curl     git     ssh     software-properties-co
mmon     make     build-essential     ca-certificates     libpq-dev     libsasl2-dev     libsasl2-2     libsasl2-modules-gssapi-mit     libyaml-dev     unixodbc-dev     && add-apt-repository ppa:deadsnakes/ppa     && apt-get install -y  
   python     python-dev     python-pip     python3.6     python3.6-dev     python3-pip     python3.6-venv     python3.7     python3.7-dev     python3.7-venv     python3.8     python3.8-dev     python3.8-venv     python3.9     python3.9-
dev     python3.9-venv     && apt-get clean     && rm -rf /var/lib/apt/lists/* /tmp/* /var/tmp/*' returned a non-zero code: 100                                                                                                             
        6.99 real         2.88 user         0.75 sys
make: *** [test-unit] Error 1

Do you have the same? (If you remove the cached docker file first.)

jtcohen6 · 2021-03-17T11:12:21Z

Hey @JCZuurmond, I don't encounter that error when I run make test-unit locally, and I'm unfortunately not enough of a Docker pro to be sure what's causing it.

If you're having trouble getting them to run in Docker, you could try running directly (via tox) in a python virtual environment, or using some of our team's shorthand scripts in scripts/dtr.py (though they're less well documented, beyond the annotations at the top).

If that's proving too difficult, you're also welcome to push changes to a draft PR and (within reason) use CircleCI to run unit and integration tests.

JCZuurmond · 2021-03-17T11:39:04Z

Hi @jtcohen6, thanks for your response. I was just able to run the tests with the Docker image ✨!

The issue was completely on my side. Docker was quite messed up, e.g. the ubuntu:14.04 image worked but 16.04, 18.04 and 20.04 did not. Also some of the apt-get packages installed well and others did not.

Apparently Apple's Screen Time app was blocking some content 😅, therefore some - not all - Ubuntu registries. After disabling that and struggling some more, I was finally able to run the tests after a reboot of my computer.

JCZuurmond added bug Something isn't working triage labels Feb 26, 2021

jtcohen6 removed the triage label Mar 1, 2021

JCZuurmond mentioned this issue Mar 17, 2021

Add TestCLIInvocationWithProfilesAndProjectDir #3176

Merged

4 tasks

jtcohen6 closed this as completed in #3176 Apr 28, 2021

jtcohen6 added this to the Margaret Mead milestone Apr 28, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

With `dbt run` the `--profiles-dir` becomes relative to `--project-dir` #3133

With `dbt run` the `--profiles-dir` becomes relative to `--project-dir` #3133

JCZuurmond commented Feb 26, 2021

jtcohen6 commented Mar 1, 2021

JCZuurmond commented Mar 1, 2021 •

edited

Loading

JCZuurmond commented Mar 1, 2021

JCZuurmond commented Mar 1, 2021 •

edited

Loading

jtcohen6 commented Mar 4, 2021

JCZuurmond commented Mar 10, 2021

JCZuurmond commented Mar 17, 2021

jtcohen6 commented Mar 17, 2021

JCZuurmond commented Mar 17, 2021

With dbt run the --profiles-dir becomes relative to --project-dir #3133

With dbt run the --profiles-dir becomes relative to --project-dir #3133

Comments

JCZuurmond commented Feb 26, 2021

Describe the bug

Steps To Reproduce

Expected behavior

Screenshots and log output

System information

Additional context

jtcohen6 commented Mar 1, 2021

JCZuurmond commented Mar 1, 2021 • edited Loading

JCZuurmond commented Mar 1, 2021

JCZuurmond commented Mar 1, 2021 • edited Loading

jtcohen6 commented Mar 4, 2021

JCZuurmond commented Mar 10, 2021

JCZuurmond commented Mar 17, 2021

jtcohen6 commented Mar 17, 2021

JCZuurmond commented Mar 17, 2021

With `dbt run` the `--profiles-dir` becomes relative to `--project-dir` #3133

With `dbt run` the `--profiles-dir` becomes relative to `--project-dir` #3133

JCZuurmond commented Mar 1, 2021 •

edited

Loading

JCZuurmond commented Mar 1, 2021 •

edited

Loading