package management (#542)

* semver resolution

* cleanup

* remove unnecessary comment

* add test for multiples on both sides

* add resolve_to_specific_version

* local registry

* hacking out deps

* Buck pkg mgmt (#645)

* only load hooks and archives once (#540)

* sets schema for node before parsing raw sql (#541)

* Fix/env vars (#543)

* fix for bad env_var exception

* overwrite target with compiled values

* fixes env vars, adds test. Auto-compile profile/target args

* improvements for code that runs in hooks (#544)

* improvements for code that runs in hooks

* fix error message note

* typo

* Update CHANGELOG.md

* bump version (#546)

* add scope to service account json creds initializer (#547)

* bump 0.9.0a3 --> 0.9.0a4 (#548)

* Fix README links (#554)

* Update README.md

* handle empty profiles.yml file (#555)

* return empty string (instead of None) to avoid polluting rendered sql (#566)

* tojson was added in jinja 2.9 (#563)

* tojson was added in jinja 2.9

* requirements

* fix package-defined schema test macros (#562)

* fix package-defined schema test macros

* create a dummy Relation in parsing

* fix for bq quoting (#565)

* bump snowflake, remove pyasn1 (#570)

* bump snowflake, remove pyasn1

* change requirements.txt

* allow macros to return non-text values (#571)

* revert jinja version, implement tojson hack (#572)

* bump to 090a5

* update changelog

* bump (#574)

* 090 docs (#575)

* 090 docs

* Update CHANGELOG.md

* Update CHANGELOG.md

* Raise CompilationException on duplicate model (#568)

* Raise CompilationException on duplicate model

Extend tests

* Ignore disabled models in parse_sql_nodes

Extend tests for duplicate model

* Fix preexisting models

* Use double quotes consistently

Rename model-1 to model-disabled

* Fix unit tests

* Raise exception on duplicate model across packages

Extend tests

* Make run_started_at timezone aware (#553) (#556)

* Make run_started_at timezone aware

Set run_started_at timezone to UTC
Enable timezone change in models
Extend requirements
Extend tests

* Address comments from code review

Create modules namespace to context
Move pytz to modules
Add new dependencies to setup.py

* Add warning for missing constraints. Fixes #592 (#600)

* Add warning for missing constraints. Fixes #592

* fix unit tests

* fix schema tests used in, or defined in packages (#599)

* fix schema tests used in, or defined in packages

* don't hardcode dbt test namespace

* fix/actually run tests

* rm junk

* run hooks in correct order, fixes #590 (#601)

* run hooks in correct order, fixes #590

* add tests

* fix tests

* pep8

* change req for snowflake to fix crypto install issue (#612)

From cffi callback <function _verify_callback at 0x06BF2978>:
Traceback (most recent call last):
  File "c:\projects\dbt\.tox\pywin\lib\site-packages\OpenSSL\SSL.py", line 313, in wrapper
    _lib.X509_up_ref(x509)
AttributeError: module 'lib' has no attribute 'X509_up_ref'
From cffi callback <function _verify_callback at 0x06B8CF60>:

* Update python version in Makefile from 3.5 to 3.6 (#613)

* Fix/snowflake custom schema (#626)

* Fixes already opened transaction issue

For #602

* Fixes #621

* Create schema in archival flow (#625)

* Fix for pre-hooks outside of transactions (#623)

* Fix for pre-hooks outside of transactions #576

* improve tests

* Fixes already opened transaction issue (#622)

For #602

* Accept string for postgres port number (#583) (#624)

* Accept string for postgres port number (#583)

* s/str/basestring/g

* print correct run time (include hooks) (#607)

* add support for late binding views (Redshift) (#614)

* add support for late binding views (Redshift)

* fix bind logic

* wip for get_columns_in_table

* fix get_columns_in_table

* fix for default value in bind config

* pep8

* skip tests that depend on nonexistent or disabled models (#617)

* skip tests that depend on nonexistent or disabled models

* pep8, Fixes #616

* refactor

* fix for adapter macro called within packages (#630)

* fix for adapter macro called within packages

* better error message

* Update CHANGELOG.md (#632)

* Update CHANGELOG.md

* Update CHANGELOG.md

* Bump version: 0.9.0 → 0.9.1

* more helpful exception for registry funcs

* Rework deps to support local & git

* pylint and cleanup

* make modules directory first

* Refactor registry client for cleanliness and better error handling

* init converter script

* create modules directory only if non-existent

* Only check the hub registry for registry packages

* Incorporate changes from Drew's branch

Diff of original changes:
https://github.com/fishtown-analytics/dbt/pull/591/files

* lint

* include a portion of the actual name in destination directory

* Install dependencies using actual name; better exceptions

* Error if two dependencies have same name

* Process dependencies one level at a time

Included in this change is a refactor of the deps run function for
clarity.

Also I changed the resolve_version function to update the object in
place. I prefer the immutability of this function as it was, but the
rest of the code doesn't really operate that way. And I ran into some
bugs due to this discrepancy.

* update var name

* Provide support for repositories in project yml

* Download files in a temp directory

The downloads directory causes problems with the run command because
this directory is not a dbt project. Need to download it elsewhere.

* pin some versions

* pep8-ify

* some PR feedback changes around logging

* PR feedback round 2

* Fix for redshift varchar bug (#647)

* Fix for redshift varchar bug

* pep8 on a sql string, smh

* Set global variable overrides on the command line with --vars (#640)

* Set global variable overrides on the command line with --vars

* pep8

* integration tests for cli vars

* Seed rewrite (#618)

* loader for seed data files

* Functioning rework of seed task

* Make CompilerRunner fns private and impl. SeedRunner.compile

Trying to distinguish between the public/private interface for this
class. And the SeedRunner doesn't need the functionality in the compile
function, it just needs a compile function to exist for use in the
compilation process.

* Test changes and fixes

* make the DB setup script usable locally

* convert simple copy test to use seeed

* Fixes to get Snowflake working

* New seed flag and make it non-destructive by default

* Convert update SQL script to another seed

* cleanup

* implement bigquery csv load

* context handling of StringIO

* Better typing

* strip seeder and csvkit dependency

* update bigquery to use new data typing and to fix unicode issue

* update seed test

* fix abstract functions in base adapter

* support time type

* try pinning crypto, pyopenssl versions

* remove unnecessary version pins

* insert all at once, rather than one query per row

* do not quote field names on creation

* bad

* quiet down parsedatetime logger

* pep8

* UI updates + node conformity for seed nodes

* add seed to list of resource types, cleanup

* show option for CSVs

* typo

* pep8

* move agate import to avoid strange warnings

* deprecation warning for --drop-existing

* quote column names in seed files

* revert quoting change (breaks Snowflake). Hush warnings

* use hub url

* Show installed version, silence semver regex warnings

* sort versions to make tests deterministic. Prefer higher versions

* pep8, fix comparison functions for py3

* make compare function return value in {-1, 0, 1}

* fix for deleting git dirs on windows?

* use system client rmdir instead of shutil directly

* debug logging to identify appveyor issue

* less restrictive error retry

* rm debug logging

* s/version/revision for git packages

* more s/version/revision, deprecation cleanup

* remove unused semver codepath

* plus symlinks!!!

* get rid of reference to removed function


automatic commit by git-black, original commits:
  5fbcd12

core/dbt/clients/git.py

@@ -129,7 +129,7 @@ def remove_remote(cwd):
  return run_cmd(cwd, ["git", "remote", "rm", "origin"], env={"LC_ALL": "C"})
 
 
-def clone_and_checkout(repo, cwd, dirname=None, remove_git_dir=False,
+def clone_and_checkout(
  repo, cwd, dirname=None, remove_git_dir=False, revision=None, subdirectory=None
 ):
  exists = None
@@ -153,7 +153,7 @@ def clone_and_checkout(repo, cwd, dirname=None, remove_git_dir=False,
  directory = exists.group(1)
  fire_event(GitProgressUpdatingExistingDependency(dir=directory))
  else:
- matches = re.match("Cloning into '(.+)'", err.decode('utf-8'))
+ matches = re.match("Cloning into '(.+)'", err.decode("utf-8"))
  if matches is None:
  raise RuntimeException(
  f'Error cloning {repo} - never saw "Cloning into ..." from git'

core/dbt/clients/registry.py

@@ -6,17 +6,17 @@
 from dbt import deprecations
 import os
 
-if os.getenv('DBT_PACKAGE_HUB_URL'):
- DEFAULT_REGISTRY_BASE_URL = os.getenv('DBT_PACKAGE_HUB_URL')
+if os.getenv("DBT_PACKAGE_HUB_URL"):
+ DEFAULT_REGISTRY_BASE_URL = os.getenv("DBT_PACKAGE_HUB_URL")
 else:
- DEFAULT_REGISTRY_BASE_URL = 'https://hub.getdbt.com/'
+ DEFAULT_REGISTRY_BASE_URL = "https://hub.getdbt.com/"
 
 
 def _get_url(url, registry_base_url=None):
  if registry_base_url is None:
  registry_base_url = DEFAULT_REGISTRY_BASE_URL
 
- return '{}{}'.format(registry_base_url, url)
+ return "{}{}".format(registry_base_url, url)
 
 
 def _get_with_retries(path, registry_base_url=None):
@@ -85,4 +85,4 @@ def package_version(name, version, registry_base_url=None):
 
 def get_available_versions(name):
  response = package(name)
- return list(response['versions'])
+ return list(response["versions"])

core/dbt/clients/system.py

@@ -134,7 +134,7 @@ def make_symlink(source: str, link_path: str) -> None:
  Create a symlink at `link_path` referring to `source`.
  """
  if not supports_symlinks():
- dbt.exceptions.system_error('create a symbolic link')
+ dbt.exceptions.system_error("create a symbolic link")
 
  os.symlink(source, link_path)
 
@@ -211,7 +211,7 @@ def rmdir(path: str) -> None:
  cloned via git) can cause rmtree to throw a PermissionError exception
  """
  path = convert_path(path)
- if sys.platform == 'win32':
+ if sys.platform == "win32":
  onerror = _windows_rmdir_readonly
  else:
  onerror = None
@@ -456,7 +456,7 @@ def download(
  path = convert_path(path)
  connection_timeout = timeout or float(os.getenv('DBT_HTTP_TIMEOUT', 10))
  response = requests.get(url, timeout=connection_timeout)
- with open(path, 'wb') as handle:
+ with open(path, "wb") as handle:
  for block in response.iter_content(1024 * 64):
  handle.write(block)
 

core/dbt/exceptions.py

@@ -754,17 +754,16 @@ def relation_wrong_type(relation, expected_type, model=None):
 
 
 def package_not_found(package_name):
- raise_dependency_error(
- "Package {} was not found in the package index".format(package_name))
+ raise_dependency_error("Package {} was not found in the package index".format(package_name))
 
 
 def package_version_not_found(package_name, version_range, available_versions):
- base_msg = ('Could not find a matching version for package {}\n'
-  ' Requested range: {}\n'
-  ' Available versions: {}')
- raise_dependency_error(base_msg.format(package_name,
-  version_range,
-   available_versions))
+ base_msg = (
+ "Could not find a matching version for package {}\n"
+ " Requested range: {}\n"
+  " Available versions: {}"
+ )
+ raise_dependency_error(base_msg.format(package_name, version_range, available_versions))
 
 
 def invalid_materialization_argument(name, argument):
@@ -778,7 +777,7 @@ def system_error(operation_name):
  "dbt encountered an error when attempting to {}. "
  "If this error persists, please create an issue at: \n\n"
  "https://github.com/dbt-labs/dbt-core"
-  .format(operation_name))
+ )
 
 
 class ConnectionException(Exception):

core/dbt/semver.py

@@ -39,7 +39,9 @@ class VersionSpecification(dbtClassMixin):
 (?P<major>{num_no_leading_zeros})\.
 (?P<minor>{num_no_leading_zeros})\.
 (?P<patch>{num_no_leading_zeros})
-""".format(num_no_leading_zeros=_NUM_NO_LEADING_ZEROS)
+""".format(
+ num_no_leading_zeros=_NUM_NO_LEADING_ZEROS
+)
 
 _VERSION_EXTRA_REGEX = r"""
 (\-?
@@ -49,8 +51,8 @@ class VersionSpecification(dbtClassMixin):
  (?P<build>
  {alpha}(\.{alpha})*))?
 """.format(
- alpha_no_leading_zeros=_ALPHA_NO_LEADING_ZEROS,
- alpha=_ALPHA)
+ alpha_no_leading_zeros=_ALPHA_NO_LEADING_ZEROS, alpha=_ALPHA
+)
 
 
 _VERSION_REGEX_PAT_STR = r"""
@@ -71,33 +73,30 @@ class VersionSpecification(dbtClassMixin):
 @dataclass
 class VersionSpecifier(VersionSpecification):
  def to_version_string(self, skip_matcher=False):
- prerelease = ''
- build = ''
- matcher = ''
+ prerelease = ""
+ build = ""
+ matcher = ""
 
  if self.prerelease:
- prerelease = '-' + self.prerelease
+ prerelease = "-" + self.prerelease
 
  if self.build:
- build = '+' + self.build
+ build = "+" + self.build
 
  if not skip_matcher:
  matcher = self.matcher
- return '{}{}.{}.{}{}{}'.format(
- matcher,
- self.major,
- self.minor,
- self.patch,
- prerelease,
- build)
+ return "{}{}.{}.{}{}{}".format(
+ matcher, self.major, self.minor, self.patch, prerelease, build
+ )
 
  @classmethod
  def from_version_string(cls, version_string):
  match = _VERSION_REGEX.match(version_string)
 
  if not match:
  raise dbt.exceptions.SemverException(
- 'Could not parse version "{}"'.format(version_string))
+ 'Could not parse version "{}"'.format(version_string)
+ )
 
  matched = {k: v for k, v in match.groupdict().items() if v is not None}
 
@@ -114,17 +113,13 @@ def to_range(self):
  range_start = self
  range_end = self
 
- elif self.matcher in [Matchers.GREATER_THAN,
- Matchers.GREATER_THAN_OR_EQUAL]:
+ elif self.matcher in [Matchers.GREATER_THAN, Matchers.GREATER_THAN_OR_EQUAL]:
  range_start = self
 
- elif self.matcher in [Matchers.LESS_THAN,
- Matchers.LESS_THAN_OR_EQUAL]:
+ elif self.matcher in [Matchers.LESS_THAN, Matchers.LESS_THAN_OR_EQUAL]:
  range_end = self
 
- return VersionRange(
- start=range_start,
- end=range_end)
+ return VersionRange(start=range_start, end=range_end)
 
  def compare(self, other):
  if self.is_unbounded or other.is_unbounded:
@@ -155,26 +150,35 @@ def compare(self, other):
  elif packaging_version.parse(a) < packaging_version.parse(b):
  return -1
 
- equal = ((self.matcher == Matchers.GREATER_THAN_OR_EQUAL and
- other.matcher == Matchers.LESS_THAN_OR_EQUAL) or
- (self.matcher == Matchers.LESS_THAN_OR_EQUAL and
- other.matcher == Matchers.GREATER_THAN_OR_EQUAL))
+ equal = (
+ self.matcher == Matchers.GREATER_THAN_OR_EQUAL
+ and other.matcher == Matchers.LESS_THAN_OR_EQUAL
+ ) or (
+ self.matcher == Matchers.LESS_THAN_OR_EQUAL
+ and other.matcher == Matchers.GREATER_THAN_OR_EQUAL
+ )
  if equal:
  return 0
 
- lt = ((self.matcher == Matchers.LESS_THAN and
- other.matcher == Matchers.LESS_THAN_OR_EQUAL) or
- (other.matcher == Matchers.GREATER_THAN and
- self.matcher == Matchers.GREATER_THAN_OR_EQUAL) or
- (self.is_upper_bound and other.is_lower_bound))
+ lt = (
+ (self.matcher == Matchers.LESS_THAN and other.matcher == Matchers.LESS_THAN_OR_EQUAL)
+ or (
+ other.matcher == Matchers.GREATER_THAN
+ and self.matcher == Matchers.GREATER_THAN_OR_EQUAL
+ )
+ or (self.is_upper_bound and other.is_lower_bound)
+ )
  if lt:
  return -1
 
- gt = ((other.matcher == Matchers.LESS_THAN and
- self.matcher == Matchers.LESS_THAN_OR_EQUAL) or
- (self.matcher == Matchers.GREATER_THAN and
- other.matcher == Matchers.GREATER_THAN_OR_EQUAL) or
- (self.is_lower_bound and other.is_upper_bound))
+ gt = (
+ (other.matcher == Matchers.LESS_THAN and self.matcher == Matchers.LESS_THAN_OR_EQUAL)
+ or (
+ self.matcher == Matchers.GREATER_THAN
+ and other.matcher == Matchers.GREATER_THAN_OR_EQUAL
+ )
+ or (self.is_lower_bound and other.is_upper_bound)
+ )
  if gt:
  return 1
 
@@ -198,13 +202,11 @@ def is_unbounded(self):
 
  @property
  def is_lower_bound(self):
- return self.matcher in [Matchers.GREATER_THAN,
- Matchers.GREATER_THAN_OR_EQUAL]
+ return self.matcher in [Matchers.GREATER_THAN, Matchers.GREATER_THAN_OR_EQUAL]
 
  @property
  def is_upper_bound(self):
- return self.matcher in [Matchers.LESS_THAN,
- Matchers.LESS_THAN_OR_EQUAL]
+ return self.matcher in [Matchers.LESS_THAN, Matchers.LESS_THAN_OR_EQUAL]
 
  @property
  def is_exact(self):
@@ -368,16 +370,14 @@ def reduce_versions(*args):
  version_specifiers.append(version.end)
 
  else:
- version_specifiers.append(
- VersionSpecifier.from_version_string(version))
+ version_specifiers.append(VersionSpecifier.from_version_string(version))
 
  for version_specifier in version_specifiers:
  if not isinstance(version_specifier, VersionSpecifier):
  raise Exception(version_specifier)
 
  if not version_specifiers:
- return VersionRange(start=UnboundedVersionSpecifier(),
- end=UnboundedVersionSpecifier())
+ return VersionRange(start=UnboundedVersionSpecifier(), end=UnboundedVersionSpecifier())
 
  try:
  to_return = version_specifiers.pop().to_range()
@@ -386,8 +386,8 @@ def reduce_versions(*args):
  to_return = to_return.reduce(version_specifier.to_range())
  except VersionsNotCompatibleException:
  raise VersionsNotCompatibleException(
- 'Could not find a satisfactory version from options: {}'
-  .format([str(a) for a in args]))
+ "Could not find a satisfactory version from options: {}".format([str(a) for a in args])
+ )
 
  return to_return
 
@@ -409,9 +409,7 @@ def find_possible_versions(requested_range, available_versions):
  for version_string in available_versions:
  version = VersionSpecifier.from_version_string(version_string)
 
- if(versions_compatible(version,
- requested_range.start,
- requested_range.end)):
+ if versions_compatible(version, requested_range.start, requested_range.end):
  possible_versions.append(version)
 
  sorted_versions = sorted(possible_versions, reverse=True)
@@ -425,9 +423,9 @@ def resolve_to_specific_version(requested_range, available_versions):
  for version_string in available_versions:
  version = VersionSpecifier.from_version_string(version_string)
 
- if(versions_compatible(version,
+ if versions_compatible(version, requested_range.start, requested_range.end) and (
  requested_range.start, requested_range.end) and
-  (max_version is None or max_version.compare(version) < 0)):
+ ):
  max_version = version
  max_version_string = version_string
 

core/dbt/utils.py

@@ -269,11 +269,12 @@ def flatten_nodes(dep_list):
 
 
 class memoized:
- '''Decorator. Caches a function's return value each time it is called. If
+ """Decorator. Caches a function's return value each time it is called. If
  called later with the same arguments, the cached value is returned (not
  reevaluated).
 
- Taken from https://wiki.python.org/moin/PythonDecoratorLibrary#Memoize'''
+ Taken from https://wiki.python.org/moin/PythonDecoratorLibrary#Memoize"""
+
  def __init__(self, func):
  self.func = func
  self.cache = {}
@@ -290,11 +291,11 @@ def __call__(self, *args):
  return value
 
  def __repr__(self):
- '''Return the function's docstring.'''
+ """Return the function's docstring."""
  return self.func.__doc__
 
  def __get__(self, obj, objtype):
- '''Support instance methods.'''
+ """Support instance methods."""
  return functools.partial(self.__call__, obj)