Skip to content

Commit

Permalink
Prerelease: v1.0.0-b2 (#892)
Browse files Browse the repository at this point in the history
* rpc, migration guide updates

* Deprecate BQ date-partitioned tables

* dbt init rework

* Generic tests in tests/generic

* Update website/docs/docs/contributing/building-a-new-adapter.md

Co-authored-by: Niall Woodward <niall@niallrees.com>

* Update website/docs/reference/commands/init.md

Co-authored-by: Niall Woodward <niall@niallrees.com>

Co-authored-by: Niall Woodward <niall@niallrees.com>
  • Loading branch information
jtcohen6 and NiallRees authored Nov 3, 2021
1 parent 9f923bd commit 7d85c31
Show file tree
Hide file tree
Showing 11 changed files with 133 additions and 44 deletions.
4 changes: 3 additions & 1 deletion website/docs/docs/building-a-dbt-project/tests.md
Original file line number Diff line number Diff line change
Expand Up @@ -107,7 +107,9 @@ Behind the scenes, dbt constructs a `select` query for each test, using the para

You can find more information about these tests, and additional configurations (including [`severity`](severity) and [`tags`](resource-configs/tags)) in the [reference section](resource-properties/tests).

You can also install generic tests from a package, or write your own, to use (and reuse) across your dbt project. Check out the [guide](custom-generic-tests) for more information.
### More generic tests

Those four tests are enough to get you started. You'll quickly find you want to use a wider variety of tests—a good thing! You can also install generic tests from a package, or write your own, to use (and reuse) across your dbt project. Check out the [guide on custom generic tests](custom-generic-tests) for more information.

:::info
There are generic tests defined in some open source packages, such as [dbt-utils](https://hub.getdbt.com/dbt-labs/dbt_utils/latest/) and [dbt-expectations](https://hub.getdbt.com/calogica/dbt_expectations/latest/) — skip ahead to the docs on [packages](package-management) to learn more!
Expand Down
22 changes: 16 additions & 6 deletions website/docs/docs/contributing/building-a-new-adapter.md
Original file line number Diff line number Diff line change
Expand Up @@ -39,7 +39,7 @@ Edit the connection manager at `myadapter/dbt/adapters/myadapter/connections.py`

### The Credentials class

The credentials class defines all of the database-specific credentials (e.g. `username` and `password`) that users will need to add to `profiles.yml` to use your new adapter. Each credentials contract should subclass dbt.adapters.base.Credentials, and be implemented as a python dataclass.
The credentials class defines all of the database-specific credentials (e.g. `username` and `password`) that users will need in the [connection profile](configure-your-profile) for your new adapter. Each credentials contract should subclass dbt.adapters.base.Credentials, and be implemented as a python dataclass.

Note that the base class includes required database and schema fields, as dbt uses those values internally.

Expand Down Expand Up @@ -83,9 +83,10 @@ class MyAdapterCredentials(Credentials):

</File>

Be sure to implement the Credentials' `_connection_keys` method shown above. This method will return the keys that should be displayed in the output of the `dbt debug` command. As a general rule, it's good to return all the arguments used in connecting to the actual database except the password (even optional arguments).

You may also want to define an `ALIASES` mapping on your Credentials class to include any config names you want users to be able to use in place of 'database' or 'schema'. For example if everyone using the MyAdapter database calls their databases "collections", you might do:
There are a few things you can do to make it easier for users when connecting to your database:
- Be sure to implement the Credentials' `_connection_keys` method shown above. This method will return the keys that should be displayed in the output of the `dbt debug` command. As a general rule, it's good to return all the arguments used in connecting to the actual database except the password (even optional arguments).
- Create a `profile_template.yml` to enable configuration prompts for a brand-new user setting up a connection profile via the [`dbt init` command](init). See more details [below](#other-files).
- You may also want to define an `ALIASES` mapping on your Credentials class to include any config names you want users to be able to use in place of 'database' or 'schema'. For example if everyone using the MyAdapter database calls their databases "collections", you might do:

<File name='connections.py'>

Expand Down Expand Up @@ -327,8 +328,17 @@ While much of dbt's adapter-specific functionality can be modified in adapter ma

### Other files

In order to enable the [dbt init command](/reference/commands/init), make sure to include a sample profile file. The filepath will be `dbt/include/<adapter_name>/sample_profiles.yml`. This will assure that users can create a new dbt project with the `dbt init` command.
For sample profiles, check out this [example](https://github.com/dbt-labs/dbt-core/blob/develop/plugins/postgres/dbt/include/postgres/sample_profiles.yml).
#### `profile_template.yml`

In order to enable the [`dbt init` command](/reference/commands/init) to prompt users when setting up a new project and connection profile, you should include a **profile template**. The filepath needs to be `dbt/include/<adapter_name>/profile_template.yml`. It's possible to provide hints, default values, and conditional prompts based on connection methods that require different supporting attributes. Users will also be able to include custom versions of this file in their own projects, with fixed values specific to their organization, to support their colleagues when using your dbt adapter for the first time.

See examples:
- [dbt-postgres](https://github.com/dbt-labs/dbt-core/blob/main/plugins/postgres/dbt/include/postgres/profile_template.yml)
- [dbt-redshift](https://github.com/dbt-labs/dbt-redshift/blob/main/dbt/include/redshift/profile_template.yml)
- [dbt-snowflake](https://github.com/dbt-labs/dbt-snowflake/blob/main/dbt/include/snowflake/profile_template.yml)
- [dbt-bigquery](https://github.com/dbt-labs/dbt-bigquery/blob/main/dbt/include/snowflake/profile_template.yml)

#### `__version__.py`

To assure that `dbt --version` provides the latest dbt core version the adapter supports, be sure include a `__version__.py` file. The filepath will be `dbt/adapters/<adapter_name>/__version__.py`. We recommend using the latest dbt core version and as the adapter is made compatiable with later versions, this file will need to be updated. For a sample file, check out this [example](https://github.com/dbt-labs/dbt-core/blob/develop/plugins/snowflake/dbt/adapters/snowflake/__version__.py).

Expand Down
17 changes: 11 additions & 6 deletions website/docs/docs/guides/custom-generic-tests.md
Original file line number Diff line number Diff line change
Expand Up @@ -6,24 +6,29 @@ id: "writing-custom-generic-tests"
<Changelog>

* `v0.20.0`: Generic tests (f.k.a. schema tests) are defined using `test` blocks instead of macros prefixed `test_`. They return a number of failing rows, rather than a single numeric value.
* `v1.0.0`: Generic tests can be defined in the `tests/generic` subfolder, in addition to the `macros/` directory

</Changelog>

dbt ships with [Not Null](resource-properties/tests#not-null), [Unique](resource-properties/tests#unique), [Relationships](resource-properties/tests#relationships), and [Accepted Values](resource-properties/tests#accepted-values) generic tests. (These used to be called "schema tests," and you'll still see that name in some places.) Under the hood, these generic tests are defined as `test` blocks (like macros) in a globally accessible dbt project. You can find the source code for these tests [here](https://github.com/dbt-labs/dbt-core/tree/develop/core/dbt/include/global_project/macros/schema_tests).
dbt ships with [Not Null](resource-properties/tests#not-null), [Unique](resource-properties/tests#unique), [Relationships](resource-properties/tests#relationships), and [Accepted Values](resource-properties/tests#accepted-values) generic tests. (These used to be called "schema tests," and you'll still see that name in some places.) Under the hood, these generic tests are defined as `test` blocks (like macros) in a globally accessible dbt project. You can find the source code for these tests in the [global project](https://github.com/dbt-labs/dbt-core/tree/HEAD/core/dbt/include/global_project/).

:::info
There are tons of generic tests defined in open source packages, such as [dbt-utils](https://hub.getdbt.com/dbt-labs/dbt_utils/latest/) and [dbt-expectations](https://hub.getdbt.com/calogica/dbt_expectations/latest/) — the test you're looking for might already be here!
:::

### Generic tests with standard arguments

Generic tests are defined in SQL files. Those files can live in two places:
- `tests/generic/`: that is, a special subfolder named `generic` within your [test paths](test-paths) (`tests/` by default)
- `macros/`: Why? Generic tests work work a lot like macros, and historically, this was the only place they could be defined. If your generic test depends on complex macro logic, you may find it more convenient to define the macros and the generic test in the same file.

To define your own generic tests, simply create a `test` block called `<test_name>`. All generic tests should accept one or both of the standard arguments:
- `model`: The resource on which the test is defined, templated out to its relation name. (Note that the argument is always named `model`, even when the resource is a source, seed, or snapshot.)
- `column_name`: The column on which the test is defined. Not all generic tests operate on the column level, but if they do, they should accept `column_name` as an argument.

Here's an example of an `is_even` schema test that uses both arguments:

<File name='macros/test_is_even.sql'>
<File name='tests/generic/test_is_even.sql'>

```sql
{% test is_even(model, column_name) %}
Expand Down Expand Up @@ -75,14 +80,14 @@ models:
</File>
In the above example, `users` will be passed to the `is_even` test as the `model` argument, and `favorite_number` will be passed in as the `column_name` argument.
With one line of code, you've just created a test! In this example, `users` will be passed to the `is_even` test as the `model` argument, and `favorite_number` will be passed in as the `column_name` argument. You could add the same line for other columns, other models—each will add a new test to your project, _using the same generic test definition_.


### Generic tests with additional arguments

The `is_even` test works without needing to specify any additional arguments. Other tests, like `relationships`, require more than just `model` and `column_name`. If your custom tests requires more than the standard argument, include those arguments in the test signature, as `field` and `to` are included below:

<File name='macros/test_relationships.sql'>
<File name='tests/generic/test_relationships.sql'>

```sql
{% test relationships(model, column_name, field, to) %}
Expand Down Expand Up @@ -138,7 +143,7 @@ models:

It is possible to include a `config()` block in a generic test definition. Values set there will set defaults for all specific instances of that generic test, unless overridden within the specific instance's `.yml` properties.

<File name='macros/warn_if_null.sql'>
<File name='tests/generic/warn_if_null.sql'>

```sql
{% test warn_if_odd(model, column_name) %}
Expand Down Expand Up @@ -179,7 +184,7 @@ models:

To change the way a built-in generic test works—whether to add additional parameters, re-write the SQL, or for any other reason—you simply add a test block named `<test_name>` to your own project. dbt will favor your version over the global implementation!

<File name='macros/<filename>.yml'>
<File name='tests/generic/<filename>.yml'>

```sql
{% test unique(model, column_name) %}
Expand Down
7 changes: 0 additions & 7 deletions website/docs/docs/guides/database-specific-guides.md

This file was deleted.

Original file line number Diff line number Diff line change
Expand Up @@ -4,9 +4,9 @@ id: "creating-date-partitioned-tables"
---


:::info New in version 0.10.0!

:::caution Deprecated

The functionality described below was introduced in dbt Core v0.10 (March 2018). In v1.0 (December 2021), it was deprecated in favor of [column-based partitioning](bigquery-configs#partition-clause) and [incremental modeling](configuring-incremental-models).

:::

Expand Down
12 changes: 11 additions & 1 deletion website/docs/docs/guides/migration-guide/upgrading-to-1-0-0.md
Original file line number Diff line number Diff line change
Expand Up @@ -5,26 +5,33 @@ title: "Upgrading to 1.0.0"

:::info Prerelease

dbt v1.0.0-b1 is currently available. If you have questions or encounter bugs, please let us know in [#dbt-prereleases](https://community.getdbt.com/) or by opening an issue [in GitHub](https://github.com/dbt-labs/dbt-core).
dbt-core v1.0.0-b2 is currently available. If you have questions or encounter bugs, please let us know in [#dbt-prereleases](https://community.getdbt.com/) or by opening an issue [in GitHub](https://github.com/dbt-labs/dbt-core).

:::

### Resources

- [Discourse](https://discourse.getdbt.com/t/3180)
- [Changelog](https://github.com/dbt-labs/dbt-core/blob/HEAD/CHANGELOG.md)

## Breaking changes

- The two type of test definitions are now "singular" and "generic" (instead of "data" and "schema", respectively). The `test_type:` selection method accepts `test_type:singular` and `test_type:generic`. (It will also accept test_type:schema and test_type:data for backwards compatibility.) **Not backwards compatible:** The `--data` and `--schema` flags to dbt test are no longer supported, and tests no longer have the tags `'data'` and `'schema'` automatically applied.
- The `dbt-rpc` server has been split apart from `dbt-core`, to be packaged, distributed, and installed separately. Instead of `dbt rpc`, use `dbt-rpc serve`.

**Deprecations:** Several under-the-hood changes from past minor versions, tagged with deprecation warnings, have now been fully deprecated.
- The `packages` argument of [dispatch](dispatch) has been deprecated and will raise an exception when used.
- The "adapter_macro" macro has been deprecated. Instead, use the [dispatch](dispatch) method to find a macro and call the result.
- The `release` arg has been removed from the `execute_macro` method.
-

## New and changed documentation

- [Tests](building-a-dbt-project/tests), [test selection](test-selection-examples), [selection methods](node-selection/methods): Update `test_type:` selection method. Remove references to `--schema` and `--data` flags.
- [Generic tests](custom-generic-tests) can be defined in `tests/generic` (new), in addition to `macros/` (as before)
- [Parsing](parsing): partial parsing and static parsing have been turned on by default.
- [Global configs](global-configs) have been standardized. Related updates to [global CLI flags](global-cli-flags) and [`profiles.yml`](profiles.yml).
- [The `init` command](init) has a whole new look and feel. It's no longer just for first-time users.

## Selection
- Add `result:<status>` subselectors for smarter reruns when dbt models have errors and tests fail. See examples: [Pro-tips for Workflows](/docs/guides/best-practices.md#pro-tips-for-workflows)
Expand All @@ -37,3 +44,6 @@ dbt v1.0.0-b1 is currently available. If you have questions or encounter bugs, p
- The [packages-install-path](packages-install-path) was updated from `modules-path`. Additionally the default value is now `dbt-packages` instead of `dbt-modules`. You may need to update this value in [`clean-targets`](clean-targets).
- Default for `quote-columns` is now `True` for all adapters other than Snowflake.

### Plugins
- [dbt RPC Server](rpc) has been split out from `dbt-core` and is now packaged separately. Its functionality will be fully deprecated by the end of 2022, in favor of a new dbt Server.
- **BigQuery:** Support for [ingestion-time-partitioned tables](creating-date-partitioned-tables) has been deprecated. Use `partition_by` and incremental modeling strategies instead.
8 changes: 5 additions & 3 deletions website/docs/faqs/configurable-data-test-path.md
Original file line number Diff line number Diff line change
@@ -1,15 +1,17 @@
---
title: Can I store my data tests in a directory other than the `tests` directory in my project?
title: Can I store my tests in a directory other than the `tests` directory in my project?
---
By default, dbt expects your singular test files to be located in the `tests` subdirectory of your project.
By default, dbt expects your singular test files to be located in the `tests` subdirectory of your project, and generic test definitions to be located in `tests/generic` or `macros`.

To change this, update the [test-paths](reference/project-configs/test-paths.md) configuration in your `dbt_project.yml`
file, like so:

<File name='dbt_project.yml'>

```yml
test-paths: ["data-tests"]
test-paths: ["my_cool_tests"]
```
</File>
Then, you can define generic tests in `my_cool_tests/generic/`, and singular tests everywhere else in `my_cool_tests/`.
70 changes: 57 additions & 13 deletions website/docs/reference/commands/init.md
Original file line number Diff line number Diff line change
Expand Up @@ -3,22 +3,66 @@ title: "init"
id: "init"
---

`dbt init [project_name]` performs several actions necessary to create a new dbt project:
:::info Improved in v1.0!
The `init` command is interactive and responsive like never before
:::

- creates a `~/.dbt/profiles.yml` file if one does not already exist
- creates a new folder called `[project_name]`
- generates directories and sample files necessary to get started with dbt
`dbt init` helps get you started using dbt Core!

### Adapter-specific profile
<Changelog>New in v0.18.0</Changelog>
## New project

You may optionally specify an `--adapter`. If you do, dbt will create `~/.dbt/profiles.yml`
(if one does not already exist) in accordance with the intended adapter type.
If this is your first time ever using the tool, it will:
- ask you to name your project
- ask you which database adapter you're using (or to [install the one you need](available-adapters))
- prompt you for each piece of information that dbt needs to connect to that database: things like `account`, `user`, `password`, etc

```bash
dbt init [project_name] --adapter bigquery
Then, it will:
- Create a new folder with your project name and sample files, enough to get you started with dbt
- Create a connection profile on your local machine. The default location is `~/.dbt/profiles.yml`. Read more in [configuring your profile](configure-your-profile).

## Existing project

If you've just cloned or downloaded an existing dbt project, `dbt init` can still help you set up your connection profile so that you can start working quickly. It will prompt you for connection information, as above, and add a profile (using the `profile` name from the project) to your local `profiles.yml`, or create the file if it doesn't already exist.

## profile_template.yml

`dbt init` knows how to prompt for connection information by looking for a file named `profile_template.yml`. It will look for this file in two places:

- **Adapter plugin:** What's the bare minumum Postgres profile? What's the type of each field, what are its defaults? This information is stored in a file called [`dbt/include/postgres/profile_template.yml`](https://github.com/dbt-labs/dbt-core/blob/main/plugins/postgres/dbt/include/postgres/profile_template.yml). If you're the maintainer of an adapter plugin, we highly recommend that you add a `profile_template.yml` to your plugin, too. See more details in [building-a-new-adapter](building-a-new-adapter).

- **Existing project:** If you're the maintainer of an existing project, and you want to help new users get connected to your database quickly and easily, you can include your own custom `profile_template.yml` in the root of your project, alongside `dbt_project.yml`. For common connection attributes, set the values in `fixed`; leave user-specific attributes in `prompts`, but with custom hints and defaults as you'd like.

<File name='profile_template.yml'>

```yml
fixed:
account: abc123
authenticator: externalbrowser
database: analytics
role: transformer
type: snowflake
warehouse: transforming
prompts:
user:
type: string
hint: yourname@jaffleshop.com
schema:
type: string
hint: usually dbt_<yourname>
threads:
hint: "your favorite number, 1-10"
type: int
default: 8
```
**Note for plugin authors:** The `--adapter` flag looks for a file named
`dbt/include/[adapter_name]/sample_profiles.yml`. Check out [dbt-spark](https://github.com/dbt-labs/dbt-spark/tree/master/dbt/include/spark/sample_profiles.yml)
and [dbt-presto](https://github.com/dbt-labs/dbt-presto/blob/master/dbt/include/presto/sample_profiles.yml) as examples.
</File>
```
$ dbt init
Running with dbt=1.0.0-b2
Setting up your profile.
user (yourname@jaffleshop.com): summerintern@jaffleshop.com
schema (usually dbt_<yourname>): dbt_summerintern
threads (your favorite number, 1-10) [8]: 6
Profile internal-snowflake written to /Users/intern/.dbt/profiles.yml using project's profile_template.yml and your supplied values. Run 'dbt debug' to validate the connection.
```
Loading

0 comments on commit 7d85c31

Please sign in to comment.