Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Regression: dbt does not allow integer dataset names on BigQuery #2384

Closed
drewbanin opened this issue Apr 30, 2020 · 2 comments · Fixed by #2395
Closed

Regression: dbt does not allow integer dataset names on BigQuery #2384

drewbanin opened this issue Apr 30, 2020 · 2 comments · Fixed by #2395
Labels
bigquery bug Something isn't working

Comments

@drewbanin
Copy link
Contributor

Describe the bug

BigQuery allows stringy dataset names that contain only digits, but dbt fails at startup if such a dataset name is provided. Is this a function of dbt's new NativeEnvironment rendering for jinja?

Steps To Reproduce

Use a profile like:

    dev:
      type: bigquery
      method: service-account
      project: '...'
      dataset: "1991"    # This is the important part
      threads: 1
      keyfile: ...
      location: US
      timeout_seconds: 1
      retries: 3

A dbt run will fail indicating that the provided profile is invalid:

Credentials in profile "bq", target "dev" invalid: 1991 is not of type 'string'

Expected behavior

dbt should create models in a dataset named 1991

dbt version

dev/octavius-catto
@drewbanin drewbanin added bug Something isn't working bigquery labels Apr 30, 2020
@drewbanin drewbanin added this to the Octavius Catto milestone Apr 30, 2020
@beckjake
Copy link
Contributor

beckjake commented May 4, 2020

Is this a function of dbt's new NativeEnvironment rendering for jinja?

Yes, almost certainly. I can think of two ways to approach this:

  • figure out how to tell the renderer that "this value should come out as a string". I'm not sure if that's hard or not, but I bet it isn't simple. This probably involves something neat with native_concat and marker types/custom filters?
  • figure out how to tell the validator "if you expected a string, call str on it". That's actually kind of tricky in the general case, because we don't really control validation, so we'd probably need to separately descend in, comparing it to the dataclass being serialized, and mutate the source dict before we get to the validator. Hideous!

@beckjake
Copy link
Contributor

beckjake commented May 4, 2020

Ok, I've thought about this, and I think there is a third ok-ish option: Add a custom filter to jinja that says "treat this field as text no matter what". Then you'd use it like dataset: "{{ 1991 | is_text }}". I think it would only make sense in the native rendering, but we could add it as a pass-through in the text renderer if we needed.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bigquery bug Something isn't working
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants