dbt 0.17.1rc1

This is a bugfix release

dbt 0.17.0

dbt 0.17.0 - Octavius Catto (June 8th, 2020)

Links:

• 📖 Migration guide + docs

Breaking changes

• Added a new dbt_project.yml version format. This emits a deprecation warning currently, but support for the existing version will be removed in a future dbt version (#2300, #2312)
• The graph object available in some dbt contexts now has an additional member sources (along side the existing nodes). Sources have been removed from nodes and added to sources instead (#2312)
• The 'location' field has been removed from bigquery catalogs (#2382)
• The list_relations_without_caching, drop_schema, and create_schema macros and methods now accept a single argument of a Relation object with no identifier field. (#2411)

Features

Core

• Add --fail-fast argument for dbt run and dbt test to fail on first test failure or runtime error. (#1649, #2224)
• Support for appending query comments to SQL queries. (#2138, #2199)
• Add a get-manifest API call. (#2168, #2232)
• Support adapter-specific aliases (like project and dataset on BigQuery) in source definitions. (#2133, #2244)
• Users can now use jinja as arguments to tests. Test arguments are rendered in the native context and injected into the test execution context directly. (#2149, #2220)
• Users can supply paths as arguments to --models and --select, either explicitily by prefixing with path: or implicitly with no prefix. (#454, #2258)
• dbt now builds the relation cache for "dbt compile" and "dbt ls" as well as "dbt run" (#1705, #2319)
• Add a 'depends_on' attribute to the log record extra field (#2316, #2341)
• Add a '--no-browser' argument to "dbt docs serve" so you can serve docs in an environment that only has a CLI browser which would otherwise deadlock dbt (#2004, #2364)
• Sources (and therefore freshness tests) can be enabled and disabled via dbt_project.yml (#2283, #2312, #2357)
• schema.yml files are now fully rendered in a context that is aware of vars declared in from dbt_project.yml files (#2269, #2357)
• Sources from dependencies can be overridden in schema.yml files (#2287, #2357)
• Add a filter named as_text to the native environment rendering code that allows users to mark a value as always being a string (#2384, #2395)
• Add warning to nodes selector if nothing was matched (#2115, #2343)

Snowflake

• Snowflake now uses "show terse objects" to build the relations cache instead of selecting from the information schema (#2174, #2322)
• Snowflake now uses "describe table" to get the columns in a relation (#2260, #2324)
• Relation comments supported for Snowflake tables and views. Column comments supported for tables. (#1722, #2321)

BigQuery

• Suport column descriptions for BigQuery models (#2335, #2402)
• Add BigQuery option maximum_bytes_billed to set an upper limit for query costs (#2346, #2427)

Postgres & Redshift

• Added support for db_groups and autocreate flags in Redshift configurations. (#1995, #2262)
• Implement persist_docs for both relation and comments on postgres and redshift, and extract them when getting the catalog. (#2333, #2378)

Docs

• Do not render hidden models in the search bar (docs#89, docs#90)

Fixes

• When a jinja value is undefined, give a helpful error instead of failing with cryptic "cannot pickle ParserMacroCapture" errors (#2110, #2184)
• Added timeout to registry download call (#2195, #2228)
• When a macro is called with invalid arguments, include the calling model in the output (#2073, #2238)
• When a warn exception is not in a jinja do block, return an empty string instead of None (#2222, #2259)
• Add dbt plugin versions to --version(#2272, #2279)
• When a Redshift table is defined as "auto", don't provide diststyle (#2246, #2298)
• Made file names lookups case-insensitve (.sql, .SQL, .yml, .YML) and if .yaml files are found, raise a warning indicating dbt will parse these files in future releases. (#1681, #2263)
• Return error message when profile is empty in profiles.yml. (#2292, #2297)
• Fix skipped node count in stdout at the end of a run (#2095, #2310)
• Fix an issue where BigQuery incorrectly used a relation's quote policy as the basis for the information schema's include policy, instead of the relation's include policy. (#2188, #2325)
• Fix "dbt deps" command so it respects the "--project-dir" arg if specified. (#2338, #2339)
• On run_cli API calls that are passed --vars differing from the server's --vars, the RPC server rebuilds the manifest for that call. (#2265, #2363)
• Remove the query job SQL from bigquery exceptions (#2383, #2393)
• Fix "Object of type Decimal is not JSON serializable" error when BigQuery queries returned numeric types in nested data structures (#2336, #2348)
• No longer query the information_schema.schemata view on bigquery (#2320, #2382)
• Preserve original subdirectory structure in compiled files. (#2173, #2349)
• Add support for sql_header config in incremental models (#2136, #2200)
• The ambiguous alias check now examines the node's database value as well as the schema/identifier ([#2326](#2326...

dbt 0.16.1

dbt 0.16.1 (April 14, 2020)

This is a bugfix release with no breaking changes and one net-new feature.

Features

• Support for appending query comments to SQL queries. (#2138, #2199) (docs)

Fixes

• dbt now renders the dbt_project.yml name config in the "base" context, giving it access to var and env_var (#2230, #2251)
• Fix an issue with {% raw %} blocks where multiple raw blocks in the same file resulted in an error (#2241, #2252)
• Fix a redshift-only issue that caused an error when dbt seed found a seed with an entirely empty column that was set to a varchar data type. (#2250, #2254)
• Fix a bug where dbt plugins that used the default list_schemas and information_schema_name macros with database quoting enabled double-quoted the database name in their queries (#2267, #2281)
• The BigQuery "partitions" config value can now be used in dbt_project.yml (#2256, #2280)
• dbt deps no longer require a profile, but if profile-specific fields are accessed users will get an error (#2231, #2290)
• Macro name collisions between dbt and plugins now raise an appropriate exception, instead of an AttributeError (#2288, #2293)
• The create_adapter_plugin.py script has been updated to support 0.16.X adapters (#2145, #2294)

Under the hood

• Pin google libraries to higher minimum values, add more dependencies as explicit (#2233, #2249)

Contributors:

• @ilkinulas #2199

dbt 0.16.0

dbt 0.16.0 (March 23, 2020)

This release contains many new features, bug fixes, and performance improvements.

Links:
🔍 Upgrading guide
💬 Discussion: v0.16.0 is released
⏩ Discussion: BigQuery incremental model updates
📈 Discussion: Benchmarking BigQuery incremental strategies

Breaking changes

• Agate type inference is no longer locale-specific. Only a small number of date/datetime formats are supported. If a seed has a specified column type, agate will not perform any type inference (it will instead be cast from a string). (#999, #1639, #1920)
• BigQuery range bucket partitioning must now be configured with the dictionary-style partitioning syntax (#2140) (docs)
• generate_schema_name macros that accept a single argument are no longer supported (#2143)
• Files with a .yml extension found in the data/, macros/, analysis/, tests/, and snapshots/ directories will now be parsed as schema.yml specifications (#2160)
• The syntax of the get_catalog macro has changed (#2037)
• When overriding the snowflake__list_schemas macro, you must now run a result with a column named 'name' instead of the first column (#2171)
• dbt no longer supports building models in Snowflake databases with greater than 10,000 schemas (#2171)
• Arguments to source tests are not parsed in the config-rendering context, and are passed as their literal unparsed values to macros (#2150)
• Update the debug log format (#2099)
• Removed docrefs from output (#2096)

Features

Core

• Compilation contexts have updated to be more consistent and well-defined (#1053, #1981, #1255, #2085) (docs)
• Add support for generate_database_name macro (#1695, #2143) (docs)
• Add meta, docs, and tags, and column quoting options to schema.yml files (docs)
  • Add a meta key to most schema.yml specifications (#1362, #2015)
  • Add tags for sources (like model tags) and columns (tags apply to tests of that column) (#1906, #1586, #2039)
  • Add column-level quoting control for tests (#2106, #2047)
  • Add a "docs" field to models, with a "show" subfield (#1671, #2107)
• Add support for documentation of macros, analyses, snapshots and seeds (docs)
  • Add documentation for macros/analyses (#1041, #2068)
  • Add arguments field to macro schema.yml specifications (#2081, #2083, #2096)
  • Add documentation for snapshots and seeds (#1974, #2051)
• Improve the speed of catalog generation by performing multiple smaller queries instead of one huge query (#2009, #2037)
• Add batching to anonymous usage statistics requests to improve performance (#2008, #2089)
• Add search for docs to include 'data' and 'snapshots' folders, in addition to 'models' (#1832, #2058)
• Add search for docs to include macro-paths and analysis-paths (in addition to source-paths, data-paths, and snapshot-paths) (#2155, #2160)
• Add Column.is_number/Column.is_float methods (#1969, #2046)
• Add detection and error handling for duplicated macro names (#1891, #2045)
• Add support for --select on dbt seed (#1711, #2042) (docs)
• Addtoyaml and fromyaml methods to the base context (#1911, #2036) (docs)
• Add database_schemas to the on-run-end context (#1924, #2031) (docs)
• Add the concept of builtins to the dbt context, make it possible to override functions like ref (#1603, #2028) (docs)
• Add clickable docs URL link in CLI output (#2027, #2131)
• Add parsing model hooks and collect ref statements (#1957, #2025)
• Add the macros every node uses to its depends_on.macros list (#2082, #2103)

Docs website

• Add handling non-array accepted_values test arguments (dbt-docs#70)
• Add support for filtering by resource type (dbt-docs#77)
• Add analyses, macros, and custom data tests (dbt-docs#72, dbt-docs#77, dbt-docs#69)
• Add support for hiding models from the docs (these nodes still render in the DAG view as "hidden") (dbt-docs#71)
• Add meta fields as "details" in node views (dbt-docs#73)
• Add lower-casing for Snowflake columns specified in all-caps (dbt-docs#74)
• Upgrade site dependencies

Postgres/Redshift

• Add role parameter in Postgres target configuration (#1955, #2137) (docs)
• Add sslmode parameter for Postgres target configuration (#2152, #2154) (docs)

Snowflake

• Remove the requirement to have a passphrase when using Snowflake key pair authentication (#1805, #2164) (docs)

BigQuery

• Support a cost-effective approach for incremental models on BigQuery using scri...

dbt 0.15.2

dbt 0.15.2 (February 2, 2020)

This is a bugfix release.

Features

• Add support for Snowflake OAuth authentication (#2050, #2069)
• Add a -t flag as an alias for dbt run --target (#1281, #2057)

Fixes

• Fix for UnicodeDecodeError when installing dbt via pip (#1771, #2076)
• Fix for ability to clean "protected" paths in the dbt clean command and improve logging (#2059, #2060)
• Fix for dbt server error when {% docs %} tags are malformed (#2066, #2067)
• Fix for errant duplicate resource errors when models are disabled and partial parsing is enabled (#2055, #2056)
• Fix for errant duplicate resource errors when a resource is included in multiple source paths (#2064, #2065)

Contributors

• @aescay (#2057)
• @markberger (#2076)
• @emilieschario (#2060)

dbt 0.15.1

dbt 0.15.1 (January 17, 2020)

This is a bugfix release.

Features

• Lazily load database connections (#1584, #1992)
• Support raising warnings in user-space (#1970, #1977) (docs)
• Suppport BigQuery label configuration for models (#1942, #1964) (docs)
• Support retrying when BigQuery models fail with server errors (#1579, #1963) (docs)
• Support sql headers in create table/view statements (#1879, #1967) (docs)
• Support source snapshot-freshness command in the dbt server (#2040, #2041)
• Support the -t shorthand for the --target flag on the CLI (#1281, #2057)

Fixes

• Fix for catalog generation error when datasets are missing on BigQuery (#1984, #2005)
• Fix for invalid SQL generated when "check" strategy is used in Snapshots with changing schemas (#1797, #2001)
• Fix for gaps in valid_from and valid_to timestamps when "check" strategy is used in Snapshots on some databases (#1736, #1994)
• Fix incorrect thread names in dbt server logs (#1905, #2002)
• Fix for ignored catalog data when user schemas begin with pg* on Postgres and Redshift (#1960, #2003)
• Fix for poorly defined materialization resolution logic (#1962, #1976)
• Fix missing drop_schema method in adapter namespace (#1980, #1983)
• Fix incorrect generated_at value in the catalog (#1988)

Under the hood

• Fail more gracefully at install time when setuptools is downlevel (#1975, #1978)
• Make the DBT_TEST_ALT integration test warehouse configurable on Snowflake (#1939, #1979)
• Pin upper bound on google-cloud-bigquery dependency to 1.24.0. (#2007)
• Remove duplicate get_context_modules method (#1996)
• Add type annotations to base adapter code (#1982)

Contributors:

• @Fokko (#1996, #1988, #1982)
• @kconvey (#1967)

dbt 0.15.0

dbt 0.15.0 (November 25, 2019)

Breaking changes

• Support for Python 2.x has been dropped as it will no longer be supported on January 1, 2020
• Compilation errors in .yml files are now treated as errors instead of warnings (#1493, #1751)
• The 'table_name' field field has been removed from Relations
• The existing compile and execute rpc tasks have been renamed to compile_sql and execute_sql (#1779, #1798) (docs)
• Custom materializations must now manage dbt's Relation cache (docs)

Installation notes:

dbt v0.15.0 uses the psycopg2-binary dependency (instead of psycopg2) to simplify installation on platforms that do not have a compiler toolchain installed. pip users should run pip uninstall psycopg2 before upgrading their version of dbt to 0.15.0. If after upgrading you experience segmentation faults, crashes, or installation errors, you can set the DBT_PSYCOPG2_NAME environment variable to psycopg2 to change the dependency that dbt installs. This may require a compiler toolchain and development libraries.

$ DBT_PSYCOPG2_NAME=psycopg2 pip install dbt

You may also install specific dbt plugins directly by name. This has the advantage of only installing the Python requirements needed for your particular database:

# Installing dbt-snowflake and dbt-bigquery directly bypasses the psycopg2 dependency
$ pip install dbt-snowflake
$ pip install dbt-bigquery

# Installing dbt-postgres and dbt-redshift are also supported
$ pip install dbt-postgres
$ pip install dbt-redshift

Changes

Core

Features

• Add a JSON logger (#1237, #1791) (docs)
• Add structured logging to dbt (#1704, #1799, #1715, #1806)
• Add partial parsing option to the profiles.yml file (#1835, #1836, #1487) (docs)
• Support configurable query comments in SQL queries (#1643, #1864) (docs)
• Support atomic full-refreshes for incremental models (#525, #1682)
• Support snapshot configs in dbt_project.yml (#1613, #1759) (docs)
• Support cache modifications in materializations (#1683, #1770) (docs)
• Support quote parameter to Accepted Values schema tests (#1873, #1876) (docs)
• Support Python 3.8 (#1886)
• Support filters in sources for dbt source snapshot-freshness invocation (#1495, #1776) (docs)
• Support external table configuration in yml source specifications (#1784)
• Improve CLI output when running snapshots (#1768, #1769)

Fixes

• Fix for unhelpful error message for malformed source/ref inputs (#1660, #1809)
• Fix for lingering backup tables when incremental models are full-refreshed (#1933, #1931)
• Fix for confusing error message when errors are encountered during compilation (#1807, #1839)
• Fix for logic error affecting the two-argument flavor of the ref function (#1504, #1515)
• Fix for invalid reference to dbt.exceptions (#1569, #1609)
• Fix for "cannot run empty query" error when pre/post-hooks are empty (#1108, #1719)
• Fix for confusing error when project names shadow context attributes (#1696, #1748)
• Fix for incorrect database logic in docs generation which resulted in columns being "merged" together across tables (#1708, #1774)
• Fix for seed errors located in dependency packages (#1723, #1723)
• Fix for confusing error when schema tests return unexpected results (#1808, #1903)
• Fix for twice-compiled statement block contents (#1717, #1719)
• Fix for inaccurate output in dbt run-operation --help (#1767, #1777)
• Fix for file rotation issues concerning the logs/dbt.log file (#1863, #1865, #1871)
• Fix for missing quotes in incremental model build queries (#1847, #1888)
• Fix for incorrect log level in printer.print_run_result_error (#1818, #1823)

Docs

• Show seeds and snapshots in the Project and Database views (docs#37, docs#25, docs#52)
• Show sources in the Database tree view (docs#20, docs#52)
• Show edges in the DAG between models and seeds (docs#15, docs#52)
• Show Accepted Values tests and custom schema tests in the column list for models (docs#52)
• Fix links for "Refocus on node" and "View documentation" in DAG context menu for seeds (docs#52)

Server

• Support docs generation (#1781, #1801)
• Support custom tags (#1822, #1828)
• Support invoking deps on the rpc server (#1834, #1837)
• Support invoking run-operation and snapshot on the rpc server (#1875, #1878)
• Suppport --threads argument to cli_args method (#1897, #1909)
• Support reloading the manifest when a SIGHUP signal is received (#1684, #1699)
• Support invoking compile, run, test, and seed on the rpc server ([#1488](https://github.com/fishtown-ana...

dbt 0.14.4

dbt 0.14.4 (November 8, 2019)

This release changes the version ranges of some of dbt's dependencies. These changes address installation issues in 0.14.3 when dbt is installed from pip. You can view the full list of dependency version changes in this commit.

Note: If you are installing dbt into an environment alongside other Python libraries, you can install individual dbt plugins with:

pip install dbt-postgres
pip install dbt-redshift
pip install dbt-snowflake
pip install dbt-bigquery

Installing specific plugins may help mitigate issues regarding incompatible versions of dependencies between dbt and other libraries.

Fixes:

• Fix dependency issues caused by a bad release of snowflake-connector-python (#1892, #1895)

dbt 0.14.3

dbt 0.14.3 (October 11, 2019)

This is a bugfix release.

Fixes:

• Fix for dictionary changed size during iteration race condition (#1740, #1750)
• Fix upper bound on jsonschema dependency to 3.1.1 (#1817, #1819)

Under the hood:

• Provide a programmatic method for validating profile targets (#1754, #1775)

dbt 0.14.2

dbt 0.14.2 (September 13, 2019)

Overview

This is a bugfix release.

Fixes:

• Fix for dbt hanging at the end of execution in dbt source snapshot-freshness tasks (#1728, #1729)
• Fix for broken "packages" and "tags" selector dropdowns in the dbt Documentation website (docs#47, #1726)