[CT-2391] [Bug] Following the Python API example.py example raises AttributeError #7312

RoiTabach · 2023-04-10T21:15:31Z

Is this a new bug in dbt-core?

I believe this is a new bug in dbt-core
I have searched the existing issues, and I could not find an existing issue for this bug

Current Behavior

I'm trying to work with the python APIs in dbt version 1.5, per the given example.py and hitting some error

AttributeError: 'Namespace' object has no attribute 'MACRO_DEBUGGING'

Expected Behavior

Not throw AttributeError, profile getting created properly

Steps To Reproduce

run these lines (project_dir is a path to a dbt project)

from dbt.cli.main import dbtRunner
from dbt.config.runtime import load_profile, load_project
project_dir = "/Users/roitabach/workspace/small_fix/dbt-data-reliability/integration_tests/"
profile = load_profile(project_dir, {}, target_override="snowflake")

Relevant log output

This exception is raised:

---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
Cell In[16], line 1
----> 1 profile = load_profile(project_dir, cli_vars={"MACRO_DEBUGGING":False},target_override="snowflake")

File ~/workspace/env/elementary/venv/lib/python3.10/site-packages/dbt/config/runtime.py:68, in load_profile(project_root, cli_vars, profile_name_override, target_override, threads_override)
     66 raw_profile_name = raw_project.get("profile")
     67 profile_renderer = ProfileRenderer(cli_vars)
---> 68 profile_name = profile_renderer.render_value(raw_profile_name)
     69 profile = Profile.render(
     70     profile_renderer, profile_name, profile_name_override, target_override, threads_override
     71 )
     72 # Save env_vars encountered in rendering for partial parsing

File ~/workspace/env/elementary/venv/lib/python3.10/site-packages/dbt/config/renderer.py:185, in SecretRenderer.render_value(self, value, keypath)
    181 def render_value(self, value: Any, keypath: Optional[Keypath] = None) -> Any:
    182     # First, standard Jinja rendering, with special handling for 'secret' environment variables
    183     # "{{ env_var('DBT_SECRET_ENV_VAR') }}" -> "$$$DBT_SECRET_START$$$DBT_SECRET_ENV_{VARIABLE_NAME}$$$DBT_SECRET_END$$$"
    184     # This prevents Jinja manipulation of secrets via macros/filters that might leak partial/modified values in logs
--> 185     rendered = super().render_value(value, keypath)
    186     # Now, detect instances of the placeholder value ($$$DBT_SECRET_START...DBT_SECRET_END$$$)
    187     # and replace them with the actual secret value
    188     if SECRET_ENV_PREFIX in str(rendered):

File ~/workspace/env/elementary/venv/lib/python3.10/site-packages/dbt/config/renderer.py:42, in BaseRenderer.render_value(self, value, keypath)
     40 try:
     41     with catch_jinja():
---> 42         return get_rendered(value, self.context, native=True)
     43 except CompilationError as exc:
     44     msg = f"Could not render {value}: {exc.msg}"

File ~/workspace/env/elementary/venv/lib/python3.10/site-packages/dbt/clients/jinja.py:584, in get_rendered(string, ctx, node, capture_macros, native)
    582 if not native and isinstance(string, str) and _HAS_RENDER_CHARS_PAT.search(string) is None:
    583     return string
--> 584 template = get_template(
    585     string,
    586     ctx,
    587     node,
    588     capture_macros=capture_macros,
    589     native=native,
    590 )
    591 return render_template(template, ctx, node)

File ~/workspace/env/elementary/venv/lib/python3.10/site-packages/dbt/clients/jinja.py:541, in get_template(string, ctx, node, capture_macros, native)
    538 env = get_environment(node, capture_macros, native=native)
    540 template_source = str(string)
--> 541 return env.from_string(template_source, globals=ctx)

File ~/workspace/env/elementary/venv/lib/python3.10/site-packages/Jinja2-3.1.2-py3.10.egg/jinja2/environment.py:1105, in Environment.from_string(self, source, globals, template_class)
   1103 gs = self.make_globals(globals)
   1104 cls = template_class or self.template_class
-> 1105 return cls.from_code(self, self.compile(source), gs, None)

File ~/workspace/env/elementary/venv/lib/python3.10/site-packages/Jinja2-3.1.2-py3.10.egg/jinja2/environment.py:766, in Environment.compile(self, source, name, filename, raw, defer_init)
    764     if filename is None:
    765         filename = "<template>"
--> 766     return self._compile(source, filename)
    767 except TemplateSyntaxError:
    768     self.handle_exception(source=source_hint)

File ~/workspace/env/elementary/venv/lib/python3.10/site-packages/dbt/clients/jinja.py:102, in MacroFuzzEnvironment._compile(self, source, filename)
     94 def _compile(self, source, filename):
     95     """Override jinja's compilation to stash the rendered source inside
     96     the python linecache for debugging when the appropriate environment
     97     variable is set.
   (...)
    100     WARNING: This can write a ton of data if you aren't careful.
    101     """
--> 102     macro_debugging = get_flags().MACRO_DEBUGGING
    103     if filename == "<template>" and macro_debugging:
    104         write = macro_debugging == "write"

AttributeError: 'Namespace' object has no attribute 'MACRO_DEBUGGING'

Environment

- OS: MacOS Monterey 12.5.1 (21G83)
- Python: 3.11.2
- dbt-core: 1.5.0-b5

Which database adapter are you using with dbt?

snowflake

Additional Context

I tried to comment out the if block just to see where I get- got another argparse error-

AttributeError: 'Namespace' object has no attribute 'PROFILES_DIR'

from this row in .../site-packages/dbt/config/profile.py

--> 434 raw_profiles = read_profile(flags.PROFILES_DIR)

The text was updated successfully, but these errors were encountered:

dbeatty10 · 2023-04-10T23:34:45Z

Thanks for reporting this @RoiTabach !

I got the same error with essentially the same script as you. Only difference is that it is aimed at the current working directory and using a differently named target:

import os
from dbt.cli.main import dbtRunner
from dbt.config.runtime import load_profile, load_project

project_dir = os.getcwd()
profile = load_profile(project_dir, {}, target_override="postgres")

In addition to this bug, we're planning on doing the following:

posting examples on docs.getdbt.com docs.getdbt-com #3118
removing this example script entirely

jtcohen6 · 2023-04-11T10:19:32Z

We're not ready to contract load_profile and load_project for external use. Let's be sure to leave them out of any docs / examples for now.

RoiTabach · 2023-04-25T08:28:40Z

@jtcohen6 Seems you need to also update this pic https://user-images.githubusercontent.com/13897643/211214203-dedeeb98-6128-454a-b678-e7804ca0ccc8.png that's referenced in here dbt-labs/docs.getdbt.com#2674 as well

dbeatty10 · 2023-04-26T00:02:51Z

Thanks for the heads-up @RoiTabach!

Are you seeing that image anywhere else other than in dbt-labs/docs.getdbt.com#2674 ?

itaisoares · 2024-03-20T16:09:45Z

Is this a new bug in dbt-core?

* [x]  I believe this is a new bug in dbt-core

* [x]  I have searched the existing issues, and I could not find an existing issue for this bug

Current Behavior

I'm trying to work with the python APIs in dbt version 1.5, per the given example.py and hitting some error

AttributeError: 'Namespace' object has no attribute 'MACRO_DEBUGGING'

Expected Behavior

Not throw AttributeError, profile getting created properly

Steps To Reproduce

1. run these lines (project_dir is a path to a dbt project)

from dbt.cli.main import dbtRunner
from dbt.config.runtime import load_profile, load_project
project_dir = "/Users/roitabach/workspace/small_fix/dbt-data-reliability/integration_tests/"
profile = load_profile(project_dir, {}, target_override="snowflake")

Relevant log output

This exception is raised:

---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
Cell In[16], line 1
----> 1 profile = load_profile(project_dir, cli_vars={"MACRO_DEBUGGING":False},target_override="snowflake")

File ~/workspace/env/elementary/venv/lib/python3.10/site-packages/dbt/config/runtime.py:68, in load_profile(project_root, cli_vars, profile_name_override, target_override, threads_override)
     66 raw_profile_name = raw_project.get("profile")
     67 profile_renderer = ProfileRenderer(cli_vars)
---> 68 profile_name = profile_renderer.render_value(raw_profile_name)
     69 profile = Profile.render(
     70     profile_renderer, profile_name, profile_name_override, target_override, threads_override
     71 )
     72 # Save env_vars encountered in rendering for partial parsing

File ~/workspace/env/elementary/venv/lib/python3.10/site-packages/dbt/config/renderer.py:185, in SecretRenderer.render_value(self, value, keypath)
    181 def render_value(self, value: Any, keypath: Optional[Keypath] = None) -> Any:
    182     # First, standard Jinja rendering, with special handling for 'secret' environment variables
    183     # "{{ env_var('DBT_SECRET_ENV_VAR') }}" -> "$$$DBT_SECRET_START$$$DBT_SECRET_ENV_{VARIABLE_NAME}$$$DBT_SECRET_END$$$"
    184     # This prevents Jinja manipulation of secrets via macros/filters that might leak partial/modified values in logs
--> 185     rendered = super().render_value(value, keypath)
    186     # Now, detect instances of the placeholder value ($$$DBT_SECRET_START...DBT_SECRET_END$$$)
    187     # and replace them with the actual secret value
    188     if SECRET_ENV_PREFIX in str(rendered):

File ~/workspace/env/elementary/venv/lib/python3.10/site-packages/dbt/config/renderer.py:42, in BaseRenderer.render_value(self, value, keypath)
     40 try:
     41     with catch_jinja():
---> 42         return get_rendered(value, self.context, native=True)
     43 except CompilationError as exc:
     44     msg = f"Could not render {value}: {exc.msg}"

File ~/workspace/env/elementary/venv/lib/python3.10/site-packages/dbt/clients/jinja.py:584, in get_rendered(string, ctx, node, capture_macros, native)
    582 if not native and isinstance(string, str) and _HAS_RENDER_CHARS_PAT.search(string) is None:
    583     return string
--> 584 template = get_template(
    585     string,
    586     ctx,
    587     node,
    588     capture_macros=capture_macros,
    589     native=native,
    590 )
    591 return render_template(template, ctx, node)

File ~/workspace/env/elementary/venv/lib/python3.10/site-packages/dbt/clients/jinja.py:541, in get_template(string, ctx, node, capture_macros, native)
    538 env = get_environment(node, capture_macros, native=native)
    540 template_source = str(string)
--> 541 return env.from_string(template_source, globals=ctx)

File ~/workspace/env/elementary/venv/lib/python3.10/site-packages/Jinja2-3.1.2-py3.10.egg/jinja2/environment.py:1105, in Environment.from_string(self, source, globals, template_class)
   1103 gs = self.make_globals(globals)
   1104 cls = template_class or self.template_class
-> 1105 return cls.from_code(self, self.compile(source), gs, None)

File ~/workspace/env/elementary/venv/lib/python3.10/site-packages/Jinja2-3.1.2-py3.10.egg/jinja2/environment.py:766, in Environment.compile(self, source, name, filename, raw, defer_init)
    764     if filename is None:
    765         filename = "<template>"
--> 766     return self._compile(source, filename)
    767 except TemplateSyntaxError:
    768     self.handle_exception(source=source_hint)

File ~/workspace/env/elementary/venv/lib/python3.10/site-packages/dbt/clients/jinja.py:102, in MacroFuzzEnvironment._compile(self, source, filename)
     94 def _compile(self, source, filename):
     95     """Override jinja's compilation to stash the rendered source inside
     96     the python linecache for debugging when the appropriate environment
     97     variable is set.
   (...)
    100     WARNING: This can write a ton of data if you aren't careful.
    101     """
--> 102     macro_debugging = get_flags().MACRO_DEBUGGING
    103     if filename == "<template>" and macro_debugging:
    104         write = macro_debugging == "write"

AttributeError: 'Namespace' object has no attribute 'MACRO_DEBUGGING'

Environment

- OS: MacOS Monterey 12.5.1 (21G83)
- Python: 3.11.2
- dbt-core: 1.5.0-b5

Which database adapter are you using with dbt?

snowflake

Additional Context

I tried to comment out the if block just to see where I get- got another argparse error-

AttributeError: 'Namespace' object has no attribute 'PROFILES_DIR'

from this row in .../site-packages/dbt/config/profile.py

--> 434 raw_profiles = read_profile(flags.PROFILES_DIR)

Hi! I had the same issue, I was trying to read the evaluated values from profiles.yml. I'm not expert on dbt but I was just seeing solutions using client but that was "too much" for what I needed. I ended up with this (I'm not proud of myself) solution

duckdb: 1.7.2

from argparse import Namespace
from dbt.flags import set_flags

cli_vars = {'MY_VAR': os.environ.get('MY_VAR')}
#this is the workaround
set_flags(Namespace(USE_COLORS=True, MACRO_DEBUGGING=False))

profile = load_profile("<path_to_dbt_project>", cli_vars=cli_vars)
...

as I needed just to read the external_root value, idk how bad is this for the rest of the execution in case of more requirements.

RoiTabach added bug Something isn't working triage labels Apr 10, 2023

github-actions bot changed the title ~~[Bug] Following the Python API example.py example raises AttributeError~~ [CT-2391] [Bug] Following the Python API example.py example raises AttributeError Apr 10, 2023

dbeatty10 removed the triage label Apr 10, 2023

jtcohen6 mentioned this issue Apr 11, 2023

Update CLI/API docs example, header #7320

Merged

2 tasks

jtcohen6 closed this as completed in #7320 Apr 11, 2023

jelstongreen mentioned this issue Aug 18, 2023

[Bug] AttributeError: 'Namespace' object has no attribute 'MACRO_DEBUGGING' dbt-labs/metricflow#731

Open

2 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[CT-2391] [Bug] Following the Python API example.py example raises AttributeError #7312

[CT-2391] [Bug] Following the Python API example.py example raises AttributeError #7312

RoiTabach commented Apr 10, 2023 •

edited by dbeatty10

Loading

dbeatty10 commented Apr 10, 2023

jtcohen6 commented Apr 11, 2023

RoiTabach commented Apr 25, 2023

dbeatty10 commented Apr 26, 2023

itaisoares commented Mar 20, 2024

Is this a new bug in dbt-core?

Current Behavior

Expected Behavior

Steps To Reproduce

Relevant log output

Environment

Which database adapter are you using with dbt?

Additional Context

[CT-2391] [Bug] Following the Python API example.py example raises AttributeError #7312

[CT-2391] [Bug] Following the Python API example.py example raises AttributeError #7312

Comments

RoiTabach commented Apr 10, 2023 • edited by dbeatty10 Loading

Is this a new bug in dbt-core?

Current Behavior

Expected Behavior

Steps To Reproduce

Relevant log output

Environment

Which database adapter are you using with dbt?

Additional Context

dbeatty10 commented Apr 10, 2023

jtcohen6 commented Apr 11, 2023

RoiTabach commented Apr 25, 2023

dbeatty10 commented Apr 26, 2023

itaisoares commented Mar 20, 2024

Is this a new bug in dbt-core?

Current Behavior

Expected Behavior

Steps To Reproduce

Relevant log output

Environment

Which database adapter are you using with dbt?

Additional Context

RoiTabach commented Apr 10, 2023 •

edited by dbeatty10

Loading