Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Long table names can be truncated on Postgres (at least) silently #2197

Closed
1 task done
drewbanin opened this issue Mar 11, 2020 · 1 comment · Fixed by #2727
Closed
1 task done

Long table names can be truncated on Postgres (at least) silently #2197

drewbanin opened this issue Mar 11, 2020 · 1 comment · Fixed by #2727
Assignees
Labels
bug Something isn't working good_first_issue Straightforward + self-contained changes, good for new contributors! redshift snapshots Issues related to dbt's snapshot functionality

Comments

@drewbanin
Copy link
Contributor

drewbanin commented Mar 11, 2020

Describe the bug

Long table names appear to be truncated by Postgres, but this behavior could also be present on other databases. This was just reproduced with a temp table suffix in a Snapshot query on Postgres, but I imagine this could just happen with exceptionally long table names:

Screen Shot 2020-03-11 at 6 37 30 PM

The truncation will cause lots of problems across dbt, but the proximate issue here is that a subsequent information schema query will return zero rows (it's looking for the wrong identifier) which bricks a snapshot run. The quick user-space fix is to shrink the "base" name of the table, but we should also take action here to:

  1. Reduce the length of the table suffix appended by dbt
  2. Show a warning (or probably an error!) if a table is created with a name long enough for PG to truncate

More info on this PG constraint here.

Example informations schema query

      select
          column_name,
          data_type,
          character_maximum_length,
          numeric_precision,
          numeric_scale

      from information_schema.columns
      where table_name = 'github_issues_linked_to_open_pull_requests__dbt_tmp20200311221237629328'
      order by ordinal_position

> Returns 0 rows

System information

Which database are you using dbt with?

  • postgres
  • [?] redshift
  • [?] bigquery
  • [?] snowflake

The output of dbt --version:

Reproduced on 0.14.3
@drewbanin drewbanin added bug Something isn't working redshift snapshots Issues related to dbt's snapshot functionality labels Mar 11, 2020
@drewbanin drewbanin added this to the 0.16.1 milestone Mar 11, 2020
@drewbanin drewbanin modified the milestones: 0.16.1, Octavius Catto Mar 24, 2020
@drewbanin drewbanin modified the milestones: Octavius Catto, dbt-next Apr 29, 2020
@drewbanin drewbanin added the good_first_issue Straightforward + self-contained changes, good for new contributors! label Jun 24, 2020
@jtcohen6
Copy link
Contributor

Add a runtime error to create table as that checks for relation identifier length

@gshank gshank self-assigned this Aug 5, 2020
gshank added a commit that referenced this issue Aug 26, 2020
elexisvenator added a commit to elexisvenator/dbt that referenced this issue Oct 26, 2020
Related: dbt-labs#2197 

The currently postgres `make_temp_relation` adds a 29 character suffix to the end of the temp relation identifier (9 from default suffix and 20 from timestamp).  This is a problem now that relations with more than 63 characters raise exceptions. 
The fix is to shorten the suffix and also trim the base_relation identifier so that the total length is always less than 63 characters.

An exception can also be raised if the default suffix is overridden with a value that is too long.
elexisvenator added a commit to elexisvenator/dbt that referenced this issue Nov 3, 2020
Related: dbt-labs#2197 

The currently postgres `make_temp_relation` adds a 29 character suffix to the end of the temp relation identifier (9 from default suffix and 20 from timestamp).  This is a problem now that relations with more than 63 characters raise exceptions. 
The fix is to shorten the suffix and also trim the base_relation identifier so that the total length is always less than 63 characters.

An exception can also be raised if the default suffix is overridden with a value that is too long.
iknox-fa pushed a commit that referenced this issue Feb 8, 2022
Related: #2197 

The currently postgres `make_temp_relation` adds a 29 character suffix to the end of the temp relation identifier (9 from default suffix and 20 from timestamp).  This is a problem now that relations with more than 63 characters raise exceptions. 
The fix is to shorten the suffix and also trim the base_relation identifier so that the total length is always less than 63 characters.

An exception can also be raised if the default suffix is overridden with a value that is too long.

automatic commit by git-black, original commits:
  c3b5b88
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working good_first_issue Straightforward + self-contained changes, good for new contributors! redshift snapshots Issues related to dbt's snapshot functionality
Projects
None yet
Development

Successfully merging a pull request may close this issue.

3 participants