Merge branch 'dev/octavius-catto' into fix-diststyle-auto

dbt-labs · Apr 9, 2020 · 0c3f9e7 · 0c3f9e7
2 parents ab07b8c + fd4a33e
commit 0c3f9e7
Show file tree

Hide file tree

Showing 46 changed files with 1,166 additions and 220 deletions.
diff --git a/CHANGELOG.md b/CHANGELOG.md
@@ -5,7 +5,7 @@
 - Added a `get-manifest` API call. ([#2168](https://github.com/fishtown-analytics/dbt/issues/2168), [#2232](https://github.com/fishtown-analytics/dbt/pull/2232))
 - Support adapter-specific aliases (like `project` and `dataset` on BigQuery) in source definitions. ([#2133](https://github.com/fishtown-analytics/dbt/issues/2133), [#2244](https://github.com/fishtown-analytics/dbt/pull/2244))
 - Users can now use jinja as arguments to tests. Test arguments are rendered in the native context and injected into the test execution context directly. ([#2149](https://github.com/fishtown-analytics/dbt/issues/2149), [#2220](https://github.com/fishtown-analytics/dbt/pull/2220))
-- Added support for `db_groups` and `autocreate` flags in Redshift configurations.  ([#1995](https://github.com/fishtown-analytics/dbt/issues/1995, [#2262]https://github.com/fishtown-analytics/dbt/pull/2262))
+- Added support for `db_groups` and `autocreate` flags in Redshift configurations.  ([#1995](https://github.com/fishtown-analytics/dbt/issues/1995), [#2262](https://github.com/fishtown-analytics/dbt/pull/2262))
 - Users can supply paths as arguments to `--models` and `--select`, either explicitily by prefixing with `path:` or implicitly with no prefix. ([#454](https://github.com/fishtown-analytics/dbt/issues/454), [#2258](https://github.com/fishtown-analytics/dbt/pull/2258))
 
 ### Fixes
@@ -15,14 +15,37 @@
 - When a warn exception is not in a jinja do block, return an empty string instead of None ([#2222](https://github.com/fishtown-analytics/dbt/issues/2222), [#2259](https://github.com/fishtown-analytics/dbt/pull/2259))
 - Add dbt plugin versions to --version([#2272](https://github.com/fishtown-analytics/dbt/issues/2272), [#2279](https://github.com/fishtown-analytics/dbt/pull/2279))
 - When a Redshift table is defined as "auto", don't provide diststyle ([#2246](https://github.com/fishtown-analytics/dbt/issues/2246), [#2298](https://github.com/fishtown-analytics/dbt/pull/2298))
+- Made file names lookups case-insensitve (.sql, .SQL, .yml, .YML) and if .yaml files are found, raise a warning indicating dbt will parse these files in future releases. ([#1681](https://github.com/fishtown-analytics/dbt/issues/1681), [#2263](https://github.com/fishtown-analytics/dbt/pull/2263))
+- Return error message when profile is empty in profiles.yml. ([#2292](https://github.com/fishtown-analytics/dbt/issues/2292), [#2297](https://github.com/fishtown-analytics/dbt/pull/2297))
 
 Contributors:
  - [@raalsky](https://github.com/Raalsky) ([#2224](https://github.com/fishtown-analytics/dbt/pull/2224), [#2228](https://github.com/fishtown-analytics/dbt/pull/2228))
  - [@ilkinulas](https://github.com/ilkinulas) [#2199](https://github.com/fishtown-analytics/dbt/pull/2199)
  - [@kyleabeauchamp](https://github.com/kyleabeauchamp) [#2262](https://github.com/fishtown-analytics/dbt/pull/2262)
  - [@jeremyyeo](https://github.com/jeremyyeo) [#2259](https://github.com/fishtown-analytics/dbt/pull/2259)
- - [@sumanau7](https://github.com/sumanau7) [#2279](https://github.com/fishtown-analytics/dbt/pull/2279)
  - [@rodrigodelmonte](https://github.com/rodrigodelmonte) [#2298](https://github.com/fishtown-analytics/dbt/pull/2298)
+ - [@sumanau7](https://github.com/sumanau7) ([#2279](https://github.com/fishtown-analytics/dbt/pull/2279), [#2263](https://github.com/fishtown-analytics/dbt/pull/2263), [#2297](https://github.com/fishtown-analytics/dbt/pull/2297))
+
+## dbt 0.16.1 (Relase date TBD)
+
+### Features
+- Support for appending query comments to SQL queries. ([#2138](https://github.com/fishtown-analytics/dbt/issues/2138) [#2199](https://github.com/fishtown-analytics/dbt/issues/2199))
+
+### Fixes
+- dbt now renders the project name in the "base" context, in particular giving it access to `var` and `env_var` ([#2230](https://github.com/fishtown-analytics/dbt/issues/2230), [#2251](https://github.com/fishtown-analytics/dbt/pull/2251))
+- Fix an issue with raw blocks where multiple raw blocks in the same file resulted in an error ([#2241](https://github.com/fishtown-analytics/dbt/issues/2241), [#2252](https://github.com/fishtown-analytics/dbt/pull/2252))
+- Fix a redshift-only issue that caused an error when `dbt seed` found a seed with an entirely empty column that was set to a `varchar` data type. ([#2250](https://github.com/fishtown-analytics/dbt/issues/2250), [#2254](https://github.com/fishtown-analytics/dbt/pull/2254))
+- Fix a bug where third party plugins that used the default `list_schemas` and `information_schema_name` macros with database quoting enabled double-quoted the database name in their queries ([#2267](https://github.com/fishtown-analytics/dbt/issues/2267), [#2281](https://github.com/fishtown-analytics/dbt/pull/2281))
+- The BigQuery "partitions" config value can now be used in `dbt_project.yml` ([#2256](https://github.com/fishtown-analytics/dbt/issues/2256), [#2280](https://github.com/fishtown-analytics/dbt/pull/2280))
+- dbt deps once again does not require a profile, but if profile-specific fields are accessed users will get an error ([#2231](https://github.com/fishtown-analytics/dbt/issues/2231), [#2290](https://github.com/fishtown-analytics/dbt/pull/2290))
+- Macro name collisions between dbt and plugins now raise an appropriate exception, instead of an AttributeError ([#2288](https://github.com/fishtown-analytics/dbt/issues/2288), [#2293](https://github.com/fishtown-analytics/dbt/pull/2293))
+- The create_adapter_plugin.py script has been updated to support 0.16.X adapters ([#2145](https://github.com/fishtown-analytics/dbt/issues/2145), [#2294](https://github.com/fishtown-analytics/dbt/pull/2294))
+
+### Under the hood
+- Pin google libraries to higher minimum values, add more dependencies as explicit ([#2233](https://github.com/fishtown-analytics/dbt/issues/2233), [#2249](https://github.com/fishtown-analytics/dbt/pull/2249))
+
+Contributors:
+ - [@ilkinulas](https://github.com/ilkinulas) [#2199](https://github.com/fishtown-analytics/dbt/pull/2199)
 
 ## dbt 0.16.0 (March 23, 2020)
 
@@ -37,6 +60,9 @@ Contributors:
 ### Fixes
 - If database quoting is enabled, do not attempt to create schemas that already exist ([#2186](https://github.com/fishtown-analytics/dbt/issues/2186), [#2187](https://github.com/fishtown-analytics/dbt/pull/2187))
 
+### Features
+- Support for appending query comments to SQL queries. ([#2138](https://github.com/fishtown-analytics/dbt/issues/2138))
+
 ## dbt 0.16.0rc2 (March 4, 2020)
 
 ### Under the hood

diff --git a/core/dbt/clients/_jinja_blocks.py b/core/dbt/clients/_jinja_blocks.py
@@ -69,7 +69,7 @@ def end_pat(self):
 
 RAW_BLOCK_PATTERN = regex(''.join((
     r'(?:\s*\{\%\-|\{\%)\s*raw\s*(?:\-\%\}\s*|\%\})',
-    r'(?:.*)',
+    r'(?:.*?)',
     r'(?:\s*\{\%\-|\{\%)\s*endraw\s*(?:\-\%\}\s*|\%\})',
 )))
 

diff --git a/core/dbt/clients/system.py b/core/dbt/clients/system.py
@@ -3,6 +3,7 @@
 import json
 import os
 import os.path
+import re
 import shutil
 import subprocess
 import sys
@@ -40,6 +41,8 @@ def find_matching(
     """
     matching = []
     root_path = os.path.normpath(root_path)
+    regex = fnmatch.translate(file_pattern)
+    reobj = re.compile(regex, re.IGNORECASE)
 
     for relative_path_to_search in relative_paths_to_search:
         absolute_path_to_search = os.path.join(
@@ -52,8 +55,7 @@ def find_matching(
                 relative_path = os.path.relpath(
                     absolute_path, absolute_path_to_search
                 )
-
-                if fnmatch.fnmatch(local_file, file_pattern):
+                if reobj.match(local_file):
                     matching.append({
                         'searched_path': relative_path_to_search,
                         'absolute_path': absolute_path,

diff --git a/core/dbt/config/__init__.py b/core/dbt/config/__init__.py
@@ -1,5 +1,5 @@
 # all these are just exports, they need "noqa" so flake8 will not complain.
 from .profile import Profile, PROFILES_DIR, read_user_config  # noqa
 from .project import Project  # noqa
-from .runtime import RuntimeConfig  # noqa
+from .runtime import RuntimeConfig, UnsetProfileConfig  # noqa
 from .renderer import ConfigRenderer  # noqa
diff --git a/core/dbt/config/profile.py b/core/dbt/config/profile.py
@@ -52,7 +52,15 @@ def read_profile(profiles_dir: str) -> Dict[str, Any]:
     if os.path.isfile(path):
         try:
             contents = load_file_contents(path, strip=False)
-            return load_yaml_text(contents)
+            yaml_content = load_yaml_text(contents)
+            if not yaml_content:
+                msg = f'The profiles.yml file at {path} is empty'
+                raise DbtProfileError(
+                    INVALID_PROFILE_MESSAGE.format(
+                        error_string=msg
+                    )
+                )
+            return yaml_content
         except ValidationException as e:
             msg = INVALID_PROFILE_MESSAGE.format(error_string=e)
             raise ValidationException(msg) from e
@@ -103,6 +111,20 @@ def to_profile_info(
             result['credentials'] = self.credentials.to_dict()
         return result
 
+    def to_target_dict(self) -> Dict[str, Any]:
+        target = dict(
+            self.credentials.connection_info(with_aliases=True)
+        )
+        target.update({
+            'type': self.credentials.type,
+            'threads': self.threads,
+            'name': self.target_name,
+            'target_name': self.target_name,
+            'profile_name': self.profile_name,
+            'config': self.config.to_dict(),
+        })
+        return target
+
     def __eq__(self, other: object) -> bool:
         if not (isinstance(other, self.__class__) and
                 isinstance(self, other.__class__)):
@@ -280,7 +302,6 @@ def from_raw_profile_info(
         # user_cfg is not rendered.
         if user_cfg is None:
             user_cfg = raw_profile.get('config')
-
         # TODO: should it be, and the values coerced to bool?
         target_name, profile_data = cls.render_profile(
             raw_profile, profile_name, target_override, renderer
@@ -335,7 +356,15 @@ def from_raw_profiles(
         # First, we've already got our final decision on profile name, and we
         # don't render keys, so we can pluck that out
         raw_profile = raw_profiles[profile_name]
-
+        if not raw_profile:
+            msg = (
+                f'Profile {profile_name} in profiles.yml is empty'
+            )
+            raise DbtProfileError(
+                INVALID_PROFILE_MESSAGE.format(
+                    error_string=msg
+                )
+            )
         user_cfg = raw_profiles.get('config')
 
         return cls.from_raw_profile_info(
@@ -373,7 +402,6 @@ def render_from_args(
         raw_profiles = read_profile(args.profiles_dir)
         profile_name = cls.pick_profile_name(getattr(args, 'profile', None),
                                              project_profile_name)
-
         return cls.from_raw_profiles(
             raw_profiles=raw_profiles,
             profile_name=profile_name,

diff --git a/core/dbt/config/project.py b/core/dbt/config/project.py
@@ -1,5 +1,5 @@
 from copy import deepcopy
-from dataclasses import dataclass
+from dataclasses import dataclass, field
 from itertools import chain
 from typing import List, Dict, Any, Optional, TypeVar, Union, Tuple, Callable
 import hashlib
@@ -220,9 +220,18 @@ def _query_comment_from_cfg(
 
 @dataclass
 class PartialProject:
-    profile_name: Optional[str]
-    project_name: Optional[str]
-    project_root: str
+    profile_name: Optional[str] = field(metadata=dict(
+        description='The unrendered profile name in the project, if set'
+    ))
+    project_name: Optional[str] = field(metadata=dict(
+        description=(
+            'The name of the project. This should always be set and will not '
+            'be rendered'
+        )
+    ))
+    project_root: str = field(
+        metadata=dict(description='The root directory of the project'),
+    )
     project_dict: Dict[str, Any]
 
     def render(self, renderer):
@@ -234,6 +243,11 @@ def render(self, renderer):
             renderer,
         )
 
+    def render_profile_name(self, renderer) -> Optional[str]:
+        if self.profile_name is None:
+            return None
+        return renderer.render_value(self.profile_name)
+
 
 @dataclass
 class Project:

diff --git a/core/dbt/config/renderer.py b/core/dbt/config/renderer.py
@@ -47,6 +47,9 @@ def _render_project_entry(self, value, keypath):
         :param key str: The key to convert on.
         :return Any: The rendered entry.
         """
+        # the project name is never rendered
+        if keypath == ('name',):
+            return value
         # query comments and hooks should be treated as raw sql, they'll get
         # rendered later.
         # Same goes for 'vars' declarations inside 'models'/'seeds'