From e9a436a25de1b43fa4b77128f7dfe442cc86b1d8 Mon Sep 17 00:00:00 2001 From: Matthew McKnight Date: Mon, 22 Nov 2021 10:24:15 -0600 Subject: [PATCH 01/10] first draft of CONTRIBUTING.MD --- CONTRIBUTING.md | 165 ++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 165 insertions(+) create mode 100644 CONTRIBUTING.md diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md new file mode 100644 index 000000000..de843ec77 --- /dev/null +++ b/CONTRIBUTING.md @@ -0,0 +1,165 @@ + +# Contributing to `dbt-bigquery` + +1. [About this document](#about-this-document) +2. [Proposing a change](#proposing-a-change) +3. [Getting the code](#getting-the-code) +4. [Setting up an environment](#setting-up-an-environment) +5. [Running `dbt-bigquery` in development](#running-dbt-bigquery-in-development) +6. [Testing](#testing) +7. [Updating Docs](#updating-docs) +7. [Submitting a Pull Request](#submitting-a-pull-request) + +## About this document +This document is a guide intended for folks interested in contributing to `dbt-bigquery`. Below, we document the process by which members of the community should create issues and submit pull requests (PRs) in this repository. It is not intended as a guide for using `dbt-bigquery`, and it assumes a certain level of familiarity with Python concepts such as virtualenvs, `pip`, python modules, filesystems, and so on. This guide assumes you are using macOS or Linux and are comfortable with the command line. + +If you're new to python development or contributing to open-source software, we encourage you to read this document from start to finish. If you get stuck, drop us a line in the `#dbt-core-development` channel on [slack](https://community.getdbt.com). + +### Signing the CLA + +Please note that all contributors to `dbt` must sign the [Contributor License Agreement](https://docs.getdbt.com/docs/contributor-license-agreements) to have their Pull Request merged into an `dbt` codebase. If you are unable to sign the CLA, then the `dbt` maintainers will unfortunately be unable to merge your Pull Request. You are, however, welcome to open issues and comment on existing ones. + +## Proposing a change +`dbt` is Apache 2.0-licensed open source software. `dbt` is what it is today because community members like you have opened issues, provided feedback, and contributed to the knowledge loop for the entire communtiy. Whether you are a seasoned open source contributor or a first-time committer, we welcome and encourage you to contribute code, documentation, ideas, or problem statements to this project. + +### Defining the problem + +If you have an idea for a new feature or if you've discovered a bug in `dbt-bigquery`, the first step is to open an issue. Please check the list of [open issues](https://github.com/dbt-labs/dbt-bigquery/issues) before creating a new one. If you find a relevant issue, please add a comment to the open issue instead of creating a new one. There are hundreds of open issues in this repository and it can be hard to know where to look for a relevant open issue. **The `dbt` maintainers are always happy to point contributors in the right direction**, so please err on the side of documenting your idea in a new issue if you are unsure where a problem statement belongs. +> **Note:** All community-contributed Pull Requests _must_ be associated with an open issue. If you submit a Pull Request that does not pertain to an open issue, you will be asked to create an issue describing the problem before the Pull Request can be reviewed. + +### Discussing the idea + +After you open an issue, a `dbt` maintainer will follow up by commenting on your issue (usually within 1-3 days) to explore your idea further and advise on how to implement the suggested changes. In many cases, community members will chime in with their own thoughts on the problem statement. If you as the issue creator are interested in submitting a Pull Request to address the issue, you should indicate this in the body of the issue. The `dbt` maintainers are _always_ happy to help contributors with the implementation of fixes and features, so please also indicate if there's anything you're unsure about or could use guidance around in the issue. + +### Submitting a change + +If an issue is appropriately well scoped and describes a beneficial change to the `dbt` codebase, then anyone may submit a Pull Request to implement the functionality described in the issue. See the sections below on how to do this. + +The `dbt` maintainers will add a `good first issue` label if an issue is suitable for a first-time contributor. This label often means that the required code change is small, limited to one database adapter, or a net-new addition that does not impact existing functionality. You can see the list of currently open issues on the [Contribute](https://github.com/dbt-labs/dbt-bigquery/contribute) page. + +Here's a good workflow: +- Comment on the open issue, expressing your interest in contributing the required code change +- Outline your planned implementation. If you want help getting started, ask! +- Follow the steps outlined below to develop locally. Once you have opened a PR, one of the `dbt` maintainers will work with you to review your code. +- Add a test! Tests are crucial for both fixes and new features alike. We want to make sure that code works as intended, and that it avoids any bugs previously encountered. Currently, the best resource for understanding `dbt`'s [unit](test/unit) and [integration](test/integration) tests is the tests themselves. One of the maintainers can help by pointing out relevant examples. + +In some cases, the right resolution to an open issue might be tangential to the `dbt` codebase. The right path forward might be a documentation update or a change that can be made in user-space. In other cases, the issue might describe functionality that the `dbt` maintainers are unwilling or unable to incorporate into the `dbt` codebase. When it is determined that an open issue describes functionality that will not translate to a code change in the `dbt` repository, the issue will be tagged with the `wontfix` label (see below) and closed. + +### Using issue labels + +The `dbt` maintainers use labels to categorize open issues. Some labels indicate the databases impacted by the issue, while others describe the domain in the `dbt` codebase germane to the discussion. While most of these labels are self-explanatory (eg. `snowflake` or `bigquery`), there are others that are worth describing. + +| tag | description | +| --- | ----------- | +| [triage](https://github.com/dbt-labs/dbt-core/labels/triage) | This is a new issue which has not yet been reviewed by a `dbt` maintainer. This label is removed when a maintainer reviews and responds to the issue. | +| [bug](https://github.com/dbt-labs/dbt-core/labels/bug) | This issue represents a defect or regression in `dbt` | +| [enhancement](https://github.com/dbt-labs/dbt-core/labels/enhancement) | This issue represents net-new functionality in `dbt` | +| [good first issue](https://github.com/dbt-labs/dbt-core/labels/good%20first%20issue) | This issue does not require deep knowledge of the `dbt` codebase to implement. This issue is appropriate for a first-time contributor. | +| [help wanted](https://github.com/dbt-labs/dbt-core/labels/help%20wanted) / [discussion](https://github.com/dbt-labs/dbt-core/labels/discussion) | Conversation around this issue in ongoing, and there isn't yet a clear path forward. Input from community members is most welcome. | +| [duplicate](https://github.com/dbt-labs/dbt-core/issues/duplicate) | This issue is functionally identical to another open issue. The `dbt` maintainers will close this issue and encourage community members to focus conversation on the other one. | +| [snoozed](https://github.com/dbt-labs/dbt-core/labels/snoozed) | This issue describes a good idea, but one which will probably not be addressed in a six-month time horizon. The `dbt` maintainers will revist these issues periodically and re-prioritize them accordingly. | +| [stale](https://github.com/dbt-labs/dbt-core/labels/stale) | This is an old issue which has not recently been updated. Stale issues will periodically be closed by `dbt` maintainers, but they can be re-opened if the discussion is restarted. | +| [wontfix](https://github.com/dbt-labs/dbt-core/labels/wontfix) | This issue does not require a code change in the `dbt` repository, or the maintainers are unwilling/unable to merge a Pull Request which implements the behavior described in the issue. | + +#### Branching Strategy + +`dbt` has three types of branches: + +- **Trunks** are where active development of the next release takes place. There is one trunk named `develop` at the time of writing this, and will be the default branch of the repository. +- **Release Branches** track a specific, not yet complete release of `dbt`. Each minor version release has a corresponding release branch. For example, the `0.11.x` series of releases has a branch called `0.11.latest`. This allows us to release new patch versions under `0.11` without necessarily needing to pull them into the latest version of `dbt`. +- **Feature Branches** track individual features and fixes. On completion they should be merged into the trunk branch or a specific release branch. + +## Getting the code + +You will need `git` in order to download and modify the `dbt` source code. On macOS, the best way to download git is to just install [Xcode](https://developer.apple.com/support/xcode/). + +### External contributors + +If you are not a member of the `dbt-labs` GitHub organization, you can contribute to `dbt-bigquery` by forking the `dbt-bigquery` repository. For a detailed overview on forking, check out the [GitHub docs on forking](https://help.github.com/en/articles/fork-a-repo). In short, you will need to: + +1. fork the `dbt-bigquery` repository +2. clone your fork locally +3. check out a new branch for your proposed changes +4. push changes to your fork +5. open a pull request against `dbt-labs/dbt-bigquery` from your forked repository + +### Core contributors + +If you are a member of the `dbt-labs` GitHub organization, you will have push access to the `dbt` repo. Rather than forking `dbt-bigquery` to make your changes, just clone the repository, check out a new branch, and push directly to that branch. + +## Setting up an environment +There are some tools that will be helpful to you in developing locally. While this is the list reveleant for `dbt` development, many of these tools are used commonly across open-source python projects. + +### Tools + +A short list of tools used in `dbt` testing that will be helpful to your understanding: + +- [`tox`](https://tox.readthedocs.io/en/latest/) to manage virtualenvs across python versions. We currently target the latest patch releases for Python 3.6, Python 3.7, Python 3.8, and Python 3.9 +- [`pytest`](https://docs.pytest.org/en/latest/) to discover/run tests +- [`make`](https://users.cs.duke.edu/~ola/courses/programming/Makefiles/Makefiles.html) - but don't worry too much, nobody _really_ understands how make works and our Makefile is super simple +- [`flake8`](https://flake8.pycqa.org/en/latest/) for code linting +- [`mypy`](https://mypy.readthedocs.io/en/stable/) for static type checking +- [Github Actions](https://github.com/features/actions) + +A deep understanding of these tools in not required to effectively contribute to `dbt`, but we recommend checking out the attached documentation if you're interested in learning more about them. + +#### virtual environments + +We strongly recommend using virtual environments when developing code in `dbt`. We recommend creating this virtualenv +in the root of the `dbt` repository. To create a new virtualenv, run: +```sh +python3 -m venv env +source env/bin/activate +``` + +This will create and activate a new Python virtual environment. + +#### docker and docker-compose + +Docker and docker-compose are both used in testing. Specific instructions for you OS can be found [here](https://docs.docker.com/get-docker/). + +## Running `dbt-bigquery` in development + +### Installation + +First make sure that you set up your `virtualenv` as described in [Setting up an environment](#setting-up-an-environment). Also ensure you have the latest version of pip installed with `pip install --upgrade pip`. Next, install `dbt-biguery` latest dependencies: + +```sh +pip install -r dev-requirements.txt +``` + +When `dbt` is installed this way, any changes you make to the `dbt` source code will be reflected immediately in your next `dbt` run. + +## Testing + +### Test commands +There are a few mehtods for running tests locally. + +####`tox` +`tox` takes care of managing virtualenvs and install dependencies in order to run tests. You can also run tests in parallel, for example you can run unit tests for Python 3.6,Python 3.7, Python 3.8, `flake8` checks and `mypy` checks in parallel with `tox -p`. Also you can run unit tests for specific python versions with `tox -e py36`. The configuration of theses tests are located in `tox.ini`. + +####`pyteest` +Finally, you can also run a specifiic test or group of tests using `pytest` directlly. With virtualenv activa and dev dependencies installed you can do things like: +```sh +# run specific postgres integration tests +python -m pytest -m profile_bigquery test/integration/001_simple_copy_test +# run all unit tests in a file +python -m pytest test/unit/test_bigquery_adapter.py +# run a specific unit test +python -m pytest test/unit/test_bigquery_adapter.py::TestBigQueryAdapter::test_copy_table_materialization_table +``` +## Updating Docs + +Many changes will require and update to the `dbt` docs here [docs](https://github.comdbt-labs/docs.getdbt.com). + + +## Submitting a Pull Request + +dbt Labs provides a CI environment to test changes to specific adapters, and periodic maintenance checks of `dbt-core` through Github Actions. For example, if you submit a pull request to the `dbt-redshift` repo, GitHub will trigger automated code checks and tests against Redshift. + +A `dbt` maintainer will review your PR. They may suggest code revision for style or clarity, or request that you add unit or integration test(s). These are good things! We believe that, with a little bit of help, anyone can contribute high-quality code. + +Once all tests are passing and your PR has been approved, a `dbt` maintainer will merge your changes into the active development branch. And that's it! Happy developing :tada: From e0473e4436a8a6893f59d3e69bc8040c35df56dd Mon Sep 17 00:00:00 2001 From: Matthew McKnight Date: Mon, 29 Nov 2021 14:28:15 -0600 Subject: [PATCH 02/10] second draft at adapter template --- CONTRIBUTING.md | 113 +++++++++--------------------------------------- 1 file changed, 20 insertions(+), 93 deletions(-) diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index de843ec77..b70522139 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -5,9 +5,7 @@ # Contributing to `dbt-bigquery` 1. [About this document](#about-this-document) -2. [Proposing a change](#proposing-a-change) 3. [Getting the code](#getting-the-code) -4. [Setting up an environment](#setting-up-an-environment) 5. [Running `dbt-bigquery` in development](#running-dbt-bigquery-in-development) 6. [Testing](#testing) 7. [Updating Docs](#updating-docs) @@ -16,65 +14,16 @@ ## About this document This document is a guide intended for folks interested in contributing to `dbt-bigquery`. Below, we document the process by which members of the community should create issues and submit pull requests (PRs) in this repository. It is not intended as a guide for using `dbt-bigquery`, and it assumes a certain level of familiarity with Python concepts such as virtualenvs, `pip`, python modules, filesystems, and so on. This guide assumes you are using macOS or Linux and are comfortable with the command line. -If you're new to python development or contributing to open-source software, we encourage you to read this document from start to finish. If you get stuck, drop us a line in the `#dbt-core-development` channel on [slack](https://community.getdbt.com). +For those wishing to contribute we highly suggest reading the [dbt-core](https://github.com/dbt-labs/dbt-core/blob/main/CONTRIBUTING.md). if you haven't already. Almost all of the information there is applicable to contributing here, too! ### Signing the CLA -Please note that all contributors to `dbt` must sign the [Contributor License Agreement](https://docs.getdbt.com/docs/contributor-license-agreements) to have their Pull Request merged into an `dbt` codebase. If you are unable to sign the CLA, then the `dbt` maintainers will unfortunately be unable to merge your Pull Request. You are, however, welcome to open issues and comment on existing ones. +Please note that all contributors to `dbt-bigquery` must sign the [Contributor License Agreement](https://docs.getdbt.com/docs/contributor-license-agreements) to have their Pull Request merged into an `dbt-bigquery` codebase. If you are unable to sign the CLA, then the `dbt-bigquery` maintainers will unfortunately be unable to merge your Pull Request. You are, however, welcome to open issues and comment on existing ones. -## Proposing a change -`dbt` is Apache 2.0-licensed open source software. `dbt` is what it is today because community members like you have opened issues, provided feedback, and contributed to the knowledge loop for the entire communtiy. Whether you are a seasoned open source contributor or a first-time committer, we welcome and encourage you to contribute code, documentation, ideas, or problem statements to this project. - -### Defining the problem - -If you have an idea for a new feature or if you've discovered a bug in `dbt-bigquery`, the first step is to open an issue. Please check the list of [open issues](https://github.com/dbt-labs/dbt-bigquery/issues) before creating a new one. If you find a relevant issue, please add a comment to the open issue instead of creating a new one. There are hundreds of open issues in this repository and it can be hard to know where to look for a relevant open issue. **The `dbt` maintainers are always happy to point contributors in the right direction**, so please err on the side of documenting your idea in a new issue if you are unsure where a problem statement belongs. -> **Note:** All community-contributed Pull Requests _must_ be associated with an open issue. If you submit a Pull Request that does not pertain to an open issue, you will be asked to create an issue describing the problem before the Pull Request can be reviewed. - -### Discussing the idea - -After you open an issue, a `dbt` maintainer will follow up by commenting on your issue (usually within 1-3 days) to explore your idea further and advise on how to implement the suggested changes. In many cases, community members will chime in with their own thoughts on the problem statement. If you as the issue creator are interested in submitting a Pull Request to address the issue, you should indicate this in the body of the issue. The `dbt` maintainers are _always_ happy to help contributors with the implementation of fixes and features, so please also indicate if there's anything you're unsure about or could use guidance around in the issue. - -### Submitting a change - -If an issue is appropriately well scoped and describes a beneficial change to the `dbt` codebase, then anyone may submit a Pull Request to implement the functionality described in the issue. See the sections below on how to do this. - -The `dbt` maintainers will add a `good first issue` label if an issue is suitable for a first-time contributor. This label often means that the required code change is small, limited to one database adapter, or a net-new addition that does not impact existing functionality. You can see the list of currently open issues on the [Contribute](https://github.com/dbt-labs/dbt-bigquery/contribute) page. - -Here's a good workflow: -- Comment on the open issue, expressing your interest in contributing the required code change -- Outline your planned implementation. If you want help getting started, ask! -- Follow the steps outlined below to develop locally. Once you have opened a PR, one of the `dbt` maintainers will work with you to review your code. -- Add a test! Tests are crucial for both fixes and new features alike. We want to make sure that code works as intended, and that it avoids any bugs previously encountered. Currently, the best resource for understanding `dbt`'s [unit](test/unit) and [integration](test/integration) tests is the tests themselves. One of the maintainers can help by pointing out relevant examples. - -In some cases, the right resolution to an open issue might be tangential to the `dbt` codebase. The right path forward might be a documentation update or a change that can be made in user-space. In other cases, the issue might describe functionality that the `dbt` maintainers are unwilling or unable to incorporate into the `dbt` codebase. When it is determined that an open issue describes functionality that will not translate to a code change in the `dbt` repository, the issue will be tagged with the `wontfix` label (see below) and closed. - -### Using issue labels - -The `dbt` maintainers use labels to categorize open issues. Some labels indicate the databases impacted by the issue, while others describe the domain in the `dbt` codebase germane to the discussion. While most of these labels are self-explanatory (eg. `snowflake` or `bigquery`), there are others that are worth describing. - -| tag | description | -| --- | ----------- | -| [triage](https://github.com/dbt-labs/dbt-core/labels/triage) | This is a new issue which has not yet been reviewed by a `dbt` maintainer. This label is removed when a maintainer reviews and responds to the issue. | -| [bug](https://github.com/dbt-labs/dbt-core/labels/bug) | This issue represents a defect or regression in `dbt` | -| [enhancement](https://github.com/dbt-labs/dbt-core/labels/enhancement) | This issue represents net-new functionality in `dbt` | -| [good first issue](https://github.com/dbt-labs/dbt-core/labels/good%20first%20issue) | This issue does not require deep knowledge of the `dbt` codebase to implement. This issue is appropriate for a first-time contributor. | -| [help wanted](https://github.com/dbt-labs/dbt-core/labels/help%20wanted) / [discussion](https://github.com/dbt-labs/dbt-core/labels/discussion) | Conversation around this issue in ongoing, and there isn't yet a clear path forward. Input from community members is most welcome. | -| [duplicate](https://github.com/dbt-labs/dbt-core/issues/duplicate) | This issue is functionally identical to another open issue. The `dbt` maintainers will close this issue and encourage community members to focus conversation on the other one. | -| [snoozed](https://github.com/dbt-labs/dbt-core/labels/snoozed) | This issue describes a good idea, but one which will probably not be addressed in a six-month time horizon. The `dbt` maintainers will revist these issues periodically and re-prioritize them accordingly. | -| [stale](https://github.com/dbt-labs/dbt-core/labels/stale) | This is an old issue which has not recently been updated. Stale issues will periodically be closed by `dbt` maintainers, but they can be re-opened if the discussion is restarted. | -| [wontfix](https://github.com/dbt-labs/dbt-core/labels/wontfix) | This issue does not require a code change in the `dbt` repository, or the maintainers are unwilling/unable to merge a Pull Request which implements the behavior described in the issue. | - -#### Branching Strategy - -`dbt` has three types of branches: - -- **Trunks** are where active development of the next release takes place. There is one trunk named `develop` at the time of writing this, and will be the default branch of the repository. -- **Release Branches** track a specific, not yet complete release of `dbt`. Each minor version release has a corresponding release branch. For example, the `0.11.x` series of releases has a branch called `0.11.latest`. This allows us to release new patch versions under `0.11` without necessarily needing to pull them into the latest version of `dbt`. -- **Feature Branches** track individual features and fixes. On completion they should be merged into the trunk branch or a specific release branch. ## Getting the code -You will need `git` in order to download and modify the `dbt` source code. On macOS, the best way to download git is to just install [Xcode](https://developer.apple.com/support/xcode/). +You will need `git` in order to download and modify the `dbt-bigquery` source code. On macOS, the best way to download git is to just install [Xcode](https://developer.apple.com/support/xcode/). ### External contributors @@ -86,40 +35,10 @@ If you are not a member of the `dbt-labs` GitHub organization, you can contribut 4. push changes to your fork 5. open a pull request against `dbt-labs/dbt-bigquery` from your forked repository -### Core contributors - -If you are a member of the `dbt-labs` GitHub organization, you will have push access to the `dbt` repo. Rather than forking `dbt-bigquery` to make your changes, just clone the repository, check out a new branch, and push directly to that branch. - -## Setting up an environment -There are some tools that will be helpful to you in developing locally. While this is the list reveleant for `dbt` development, many of these tools are used commonly across open-source python projects. - -### Tools - -A short list of tools used in `dbt` testing that will be helpful to your understanding: - -- [`tox`](https://tox.readthedocs.io/en/latest/) to manage virtualenvs across python versions. We currently target the latest patch releases for Python 3.6, Python 3.7, Python 3.8, and Python 3.9 -- [`pytest`](https://docs.pytest.org/en/latest/) to discover/run tests -- [`make`](https://users.cs.duke.edu/~ola/courses/programming/Makefiles/Makefiles.html) - but don't worry too much, nobody _really_ understands how make works and our Makefile is super simple -- [`flake8`](https://flake8.pycqa.org/en/latest/) for code linting -- [`mypy`](https://mypy.readthedocs.io/en/stable/) for static type checking -- [Github Actions](https://github.com/features/actions) - -A deep understanding of these tools in not required to effectively contribute to `dbt`, but we recommend checking out the attached documentation if you're interested in learning more about them. +### dbt Labs contributors -#### virtual environments +If you are a member of the `dbt Labs` GitHub organization, you will have push access to the `dbt-bigquery` repo. Rather than forking `dbt-bigquery` to make your changes, just clone the repository, check out a new branch, and push directly to that branch. -We strongly recommend using virtual environments when developing code in `dbt`. We recommend creating this virtualenv -in the root of the `dbt` repository. To create a new virtualenv, run: -```sh -python3 -m venv env -source env/bin/activate -``` - -This will create and activate a new Python virtual environment. - -#### docker and docker-compose - -Docker and docker-compose are both used in testing. Specific instructions for you OS can be found [here](https://docs.docker.com/get-docker/). ## Running `dbt-bigquery` in development @@ -128,10 +47,12 @@ Docker and docker-compose are both used in testing. Specific instructions for yo First make sure that you set up your `virtualenv` as described in [Setting up an environment](#setting-up-an-environment). Also ensure you have the latest version of pip installed with `pip install --upgrade pip`. Next, install `dbt-biguery` latest dependencies: ```sh -pip install -r dev-requirements.txt +pip install -e . -r dev-requirements.txt ``` -When `dbt` is installed this way, any changes you make to the `dbt` source code will be reflected immediately in your next `dbt` run. +When `dbt-bigquery` is installed this way, any changes you make to the `dbt-bigquery` source code will be reflected immediately in your next `dbt-bigquery` run. + +To confirm you have correct version of `dbt-core` installed please run `dbt --version` and `which dbt`. ## Testing @@ -139,7 +60,7 @@ When `dbt` is installed this way, any changes you make to the `dbt` source code There are a few mehtods for running tests locally. ####`tox` -`tox` takes care of managing virtualenvs and install dependencies in order to run tests. You can also run tests in parallel, for example you can run unit tests for Python 3.6,Python 3.7, Python 3.8, `flake8` checks and `mypy` checks in parallel with `tox -p`. Also you can run unit tests for specific python versions with `tox -e py36`. The configuration of theses tests are located in `tox.ini`. +`tox` takes care of managing virtualenvs and install dependencies in order to run tests. You can also run tests in parallel, for example you can run unit tests for Python 3.7, Python 3.8, Python 3.9 `flake8` checks in parallel with `tox -p`. Also you can run unit tests for specific python versions with `tox -e py37`. The configuration of theses tests are located in `tox.ini`. ####`pyteest` Finally, you can also run a specifiic test or group of tests using `pytest` directlly. With virtualenv activa and dev dependencies installed you can do things like: @@ -153,13 +74,19 @@ python -m pytest test/unit/test_bigquery_adapter.py::TestBigQueryAdapter::test_c ``` ## Updating Docs -Many changes will require and update to the `dbt` docs here [docs](https://github.comdbt-labs/docs.getdbt.com). +Many changes will require and update to the `dbt-bigquery` docs here are some useful resources. + +- Docs are [here](https://docs.getdbt.com/). +- The docs repo for making changes is located [here]( https://github.com/dbt-labs/docs.getdbt.com). +- The changes made are likely to impact one or both of [BigQuery profile](https://docs.getdbt.com/reference/warehouse-profiles/bigquery-profile), or [BigQuery configs](https://docs.getdbt.com/reference/resource-configs/bigquery-configs). +- We as every community member who makes a user-facing change to open and issue for their change in that repo. +- If the contributor feels up to they should contribute to the PR for the docs update as well. ## Submitting a Pull Request -dbt Labs provides a CI environment to test changes to specific adapters, and periodic maintenance checks of `dbt-core` through Github Actions. For example, if you submit a pull request to the `dbt-redshift` repo, GitHub will trigger automated code checks and tests against Redshift. +dbt Labs provides a CI environment to test changes to the `dbt-bigquery` adapter, and periodic maintenance checks of `dbt-core` through Github Actions. -A `dbt` maintainer will review your PR. They may suggest code revision for style or clarity, or request that you add unit or integration test(s). These are good things! We believe that, with a little bit of help, anyone can contribute high-quality code. +A `dbt-bigquery` maintainer will review your PR. They may suggest code revision for style or clarity, or request that you add unit or integration test(s). These are good things! We believe that, with a little bit of help, anyone can contribute high-quality code. -Once all tests are passing and your PR has been approved, a `dbt` maintainer will merge your changes into the active development branch. And that's it! Happy developing :tada: +Once all tests are passing and your PR has been approved, a `dbt-bigquery` maintainer will merge your changes into the active development branch. And that's it! Happy developing :tada: From e918299331553f5aee10678dd68bdf966dfa7b79 Mon Sep 17 00:00:00 2001 From: Matthew McKnight Date: Mon, 29 Nov 2021 15:39:41 -0600 Subject: [PATCH 03/10] moving section in intial setup from dbt-core to adapter repos --- CONTRIBUTING.md | 22 +++++++++++++++++++++- 1 file changed, 21 insertions(+), 1 deletion(-) diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index b70522139..18aeec12a 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -21,7 +21,7 @@ For those wishing to contribute we highly suggest reading the [dbt-core](https:/ Please note that all contributors to `dbt-bigquery` must sign the [Contributor License Agreement](https://docs.getdbt.com/docs/contributor-license-agreements) to have their Pull Request merged into an `dbt-bigquery` codebase. If you are unable to sign the CLA, then the `dbt-bigquery` maintainers will unfortunately be unable to merge your Pull Request. You are, however, welcome to open issues and comment on existing ones. -## Getting the code +## Getting the code You will need `git` in order to download and modify the `dbt-bigquery` source code. On macOS, the best way to download git is to just install [Xcode](https://developer.apple.com/support/xcode/). @@ -54,8 +54,28 @@ When `dbt-bigquery` is installed this way, any changes you make to the `dbt-bigq To confirm you have correct version of `dbt-core` installed please run `dbt --version` and `which dbt`. +## Initial Setup + +`dbt-bigquery` uses test credentials specified in a `test.env` file in the root of the repository for non-Postgres databases. This `test.env` file is git-ignored, but please be _extra_ careful to never check in credentials or other sensitive information when developing against `dbt-core`. To create your `test.env` file, copy the provided sample file, then supply your relevant credentials. This step is only required to use non-Postgres databases. + +``` +cp test.env.sample test.env +$EDITOR test.env +``` + +> In general, it's most important to have successful unit and Postgres tests. Once you open a PR, `dbt-core` will automatically run integration tests for the other three core database adapters. Of course, if you are a BigQuery user, contributing a BigQuery-only feature, it's important to run BigQuery tests as well. + ## Testing +`dbt-core` uses test credentials specified in a `test.env` file in the root of the repository for non-Postgres databases. This `test.env` file is git-ignored, but please be _extra_ careful to never check in credentials or other sensitive information when developing against `dbt-core`. To create your `test.env` file, copy the provided sample file, then supply your relevant credentials. This step is only required to use non-Postgres databases. + +``` +cp test.env.sample test.env +$EDITOR test.env +``` + +> In general, it's most important to have successful unit and Postgres tests. Once you open a PR, `dbt-core` will automatically run integration tests for the other three core database adapters. Of course, if you are a BigQuery user, contributing a BigQuery-only feature, it's important to run BigQuery tests as well. + ### Test commands There are a few mehtods for running tests locally. From 3a7cb1f08d0ddfd5a0a231733a14b31e76d1acdb Mon Sep 17 00:00:00 2001 From: Matthew McKnight Date: Mon, 29 Nov 2021 15:54:42 -0600 Subject: [PATCH 04/10] added test.env.example --- .gitignore | 1 + CONTRIBUTING.md | 6 ++++++ test.env.example | 3 +++ 3 files changed, 10 insertions(+) create mode 100644 test.env.example diff --git a/.gitignore b/.gitignore index 43724b61e..79898731f 100644 --- a/.gitignore +++ b/.gitignore @@ -50,6 +50,7 @@ coverage.xml .hypothesis/ test.env + # Mypy .mypy_cache/ diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index 18aeec12a..bdedf1bdf 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -62,6 +62,12 @@ To confirm you have correct version of `dbt-core` installed please run `dbt --ve cp test.env.sample test.env $EDITOR test.env ``` +if you have secrets of your own to provide please use + +``` +cp test.env.example test.env +``` +to create a blank test.env to fill in. > In general, it's most important to have successful unit and Postgres tests. Once you open a PR, `dbt-core` will automatically run integration tests for the other three core database adapters. Of course, if you are a BigQuery user, contributing a BigQuery-only feature, it's important to run BigQuery tests as well. diff --git a/test.env.example b/test.env.example new file mode 100644 index 000000000..a69a35fe7 --- /dev/null +++ b/test.env.example @@ -0,0 +1,3 @@ +BIGQUERY_TEST_DATABASE= +BIGQUERY_TEST_ALT_DATABASE= +BIGQUERY_TEST_SERVICE_ACCOUNT_JSON='{}' \ No newline at end of file From b759fed2e8ad71f0234a04aeaea198613ed20c2f Mon Sep 17 00:00:00 2001 From: Matthew McKnight Date: Tue, 30 Nov 2021 16:02:59 -0600 Subject: [PATCH 05/10] making suggested changes based off kyle's review --- CONTRIBUTING.md | 47 ++++++++++++++--------------------------------- 1 file changed, 14 insertions(+), 33 deletions(-) diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index bdedf1bdf..20c556b71 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -1,7 +1,3 @@ - # Contributing to `dbt-bigquery` 1. [About this document](#about-this-document) @@ -44,7 +40,7 @@ If you are a member of the `dbt Labs` GitHub organization, you will have push ac ### Installation -First make sure that you set up your `virtualenv` as described in [Setting up an environment](#setting-up-an-environment). Also ensure you have the latest version of pip installed with `pip install --upgrade pip`. Next, install `dbt-biguery` latest dependencies: +First make sure that you set up your `virtualenv` as described in [Setting up an environment](https://github.com/dbt-labs/dbt-core/blob/HEAD/CONTRIBUTING.md#setting-up-an-environment). Ensure you have the latest version of pip installed with `pip install --upgrade pip`. Next, install `dbt-bigquery` latest dependencies: ```sh pip install -e . -r dev-requirements.txt @@ -54,42 +50,27 @@ When `dbt-bigquery` is installed this way, any changes you make to the `dbt-bigq To confirm you have correct version of `dbt-core` installed please run `dbt --version` and `which dbt`. -## Initial Setup - -`dbt-bigquery` uses test credentials specified in a `test.env` file in the root of the repository for non-Postgres databases. This `test.env` file is git-ignored, but please be _extra_ careful to never check in credentials or other sensitive information when developing against `dbt-core`. To create your `test.env` file, copy the provided sample file, then supply your relevant credentials. This step is only required to use non-Postgres databases. - -``` -cp test.env.sample test.env -$EDITOR test.env -``` -if you have secrets of your own to provide please use - -``` -cp test.env.example test.env -``` -to create a blank test.env to fill in. - -> In general, it's most important to have successful unit and Postgres tests. Once you open a PR, `dbt-core` will automatically run integration tests for the other three core database adapters. Of course, if you are a BigQuery user, contributing a BigQuery-only feature, it's important to run BigQuery tests as well. ## Testing -`dbt-core` uses test credentials specified in a `test.env` file in the root of the repository for non-Postgres databases. This `test.env` file is git-ignored, but please be _extra_ careful to never check in credentials or other sensitive information when developing against `dbt-core`. To create your `test.env` file, copy the provided sample file, then supply your relevant credentials. This step is only required to use non-Postgres databases. +### Initial Setup + +`dbt-bigquery` uses test credentials specified in a `test.env` file in the root of the repository. This `test.env` file is git-ignored, but please be _extra_ careful to never check in credentials or other sensitive information when developing. To create your `test.env` file, copy the provided example file, then supply your relevant credentials. ``` -cp test.env.sample test.env +cp test.env.example test.env $EDITOR test.env ``` -> In general, it's most important to have successful unit and Postgres tests. Once you open a PR, `dbt-core` will automatically run integration tests for the other three core database adapters. Of course, if you are a BigQuery user, contributing a BigQuery-only feature, it's important to run BigQuery tests as well. - ### Test commands -There are a few mehtods for running tests locally. +There are a few methods for running tests locally. + +#### `tox` +`tox` takes care of managing Python virtualenvs and installing dependencies in order to run tests. You can also run tests in parallel, for example you can run unit tests for Python 3.7, Python 3.8, Python 3.9, and `flake8` checks in parallel with `tox -p`. Also, you can run unit tests for specific python versions with `tox -e py37`. The configuration of these tests are located in `tox.ini`. -####`tox` -`tox` takes care of managing virtualenvs and install dependencies in order to run tests. You can also run tests in parallel, for example you can run unit tests for Python 3.7, Python 3.8, Python 3.9 `flake8` checks in parallel with `tox -p`. Also you can run unit tests for specific python versions with `tox -e py37`. The configuration of theses tests are located in `tox.ini`. +#### `pyteest` +Finally, you can also run a specific test or group of tests using `pytest` directly. With a Python virtualenv active and dev dependencies installed you can do things like: -####`pyteest` -Finally, you can also run a specifiic test or group of tests using `pytest` directlly. With virtualenv activa and dev dependencies installed you can do things like: ```sh # run specific postgres integration tests python -m pytest -m profile_bigquery test/integration/001_simple_copy_test @@ -105,13 +86,13 @@ Many changes will require and update to the `dbt-bigquery` docs here are some us - Docs are [here](https://docs.getdbt.com/). - The docs repo for making changes is located [here]( https://github.com/dbt-labs/docs.getdbt.com). - The changes made are likely to impact one or both of [BigQuery profile](https://docs.getdbt.com/reference/warehouse-profiles/bigquery-profile), or [BigQuery configs](https://docs.getdbt.com/reference/resource-configs/bigquery-configs). -- We as every community member who makes a user-facing change to open and issue for their change in that repo. -- If the contributor feels up to they should contribute to the PR for the docs update as well. +- We ask every community member who makes a user-facing change to open an issue or PR regarding doc changes. + ## Submitting a Pull Request -dbt Labs provides a CI environment to test changes to the `dbt-bigquery` adapter, and periodic maintenance checks of `dbt-core` through Github Actions. +dbt Labs provides a CI environment to test changes to the `dbt-bigquery` adapter, and periodic checks against the development version of `dbt-core` through Github Actions. A `dbt-bigquery` maintainer will review your PR. They may suggest code revision for style or clarity, or request that you add unit or integration test(s). These are good things! We believe that, with a little bit of help, anyone can contribute high-quality code. From 957b3eedd97849f0fa93d0ae76e687bacc6d9f21 Mon Sep 17 00:00:00 2001 From: Matthew McKnight Date: Tue, 21 Dec 2021 10:12:10 -0600 Subject: [PATCH 06/10] minor updates noticed when making first drafts for other adapter repos. --- CONTRIBUTING.md | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index 20c556b71..8245e4dc5 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -72,10 +72,10 @@ There are a few methods for running tests locally. Finally, you can also run a specific test or group of tests using `pytest` directly. With a Python virtualenv active and dev dependencies installed you can do things like: ```sh -# run specific postgres integration tests -python -m pytest -m profile_bigquery test/integration/001_simple_copy_test +# run specific bigquery integration tests +python -m pytest -m profile_bigquery tests/integration/simple_copy_test # run all unit tests in a file -python -m pytest test/unit/test_bigquery_adapter.py +python -m pytest tests/unit/test_bigquery_adapter.py # run a specific unit test python -m pytest test/unit/test_bigquery_adapter.py::TestBigQueryAdapter::test_copy_table_materialization_table ``` @@ -85,7 +85,7 @@ Many changes will require and update to the `dbt-bigquery` docs here are some us - Docs are [here](https://docs.getdbt.com/). - The docs repo for making changes is located [here]( https://github.com/dbt-labs/docs.getdbt.com). -- The changes made are likely to impact one or both of [BigQuery profile](https://docs.getdbt.com/reference/warehouse-profiles/bigquery-profile), or [BigQuery configs](https://docs.getdbt.com/reference/resource-configs/bigquery-configs). +- The changes made are likely to impact one or both of [BigQuery Profile](https://docs.getdbt.com/reference/warehouse-profiles/bigquery-profile), or [BigQuery Configs](https://docs.getdbt.com/reference/resource-configs/bigquery-configs). - We ask every community member who makes a user-facing change to open an issue or PR regarding doc changes. From 9675747767762d21c285d25626345908598d810c Mon Sep 17 00:00:00 2001 From: Matthew McKnight Date: Tue, 21 Dec 2021 13:59:14 -0600 Subject: [PATCH 07/10] updated test => tests for naming convention --- CONTRIBUTING.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index 8245e4dc5..79e706955 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -77,7 +77,7 @@ python -m pytest -m profile_bigquery tests/integration/simple_copy_test # run all unit tests in a file python -m pytest tests/unit/test_bigquery_adapter.py # run a specific unit test -python -m pytest test/unit/test_bigquery_adapter.py::TestBigQueryAdapter::test_copy_table_materialization_table +python -m pytest tests/unit/test_bigquery_adapter.py::TestBigQueryAdapter::test_copy_table_materialization_table ``` ## Updating Docs From 1584c97955f4a4c901b53d2b68733f798af3d69f Mon Sep 17 00:00:00 2001 From: Matthew McKnight Date: Wed, 5 Jan 2022 10:20:35 -0600 Subject: [PATCH 08/10] responding to comments on github --- .gitignore | 1 - CONTRIBUTING.md | 6 +++--- 2 files changed, 3 insertions(+), 4 deletions(-) diff --git a/.gitignore b/.gitignore index 79898731f..43724b61e 100644 --- a/.gitignore +++ b/.gitignore @@ -50,7 +50,6 @@ coverage.xml .hypothesis/ test.env - # Mypy .mypy_cache/ diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index 79e706955..38177a965 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -48,8 +48,7 @@ pip install -e . -r dev-requirements.txt When `dbt-bigquery` is installed this way, any changes you make to the `dbt-bigquery` source code will be reflected immediately in your next `dbt-bigquery` run. -To confirm you have correct version of `dbt-core` installed please run `dbt --version` and `which dbt`. - +To confirm you have the correct version of `dbt-core` installed please run `dbt --version` and `which dbt`. ## Testing @@ -96,4 +95,5 @@ dbt Labs provides a CI environment to test changes to the `dbt-bigquery` adapter A `dbt-bigquery` maintainer will review your PR. They may suggest code revision for style or clarity, or request that you add unit or integration test(s). These are good things! We believe that, with a little bit of help, anyone can contribute high-quality code. -Once all tests are passing and your PR has been approved, a `dbt-bigquery` maintainer will merge your changes into the active development branch. And that's it! Happy developing :tada: + +Once all tests are passing, you have updated the changelog to reflect and tag your issue/pr for reference with a small description of the change, and your PR has been approved, a `dbt-bigquery` maintainer will merge your changes into the active development branch. And that's it! Happy developing :tada: From 0fdcf16e2c2b7d0dc4cf0def09e827a846b6133b Mon Sep 17 00:00:00 2001 From: Matthew McKnight Date: Wed, 5 Jan 2022 10:23:01 -0600 Subject: [PATCH 09/10] continued responses to comments --- CONTRIBUTING.md | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-) diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index 38177a965..273cfb894 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -91,9 +91,8 @@ Many changes will require and update to the `dbt-bigquery` docs here are some us ## Submitting a Pull Request -dbt Labs provides a CI environment to test changes to the `dbt-bigquery` adapter, and periodic checks against the development version of `dbt-core` through Github Actions. +dbt Labs provides a CI environment to test changes to the `dbt-bigquery` adapter and periodic checks against the development version of `dbt-core` through Github Actions. A `dbt-bigquery` maintainer will review your PR. They may suggest code revision for style or clarity, or request that you add unit or integration test(s). These are good things! We believe that, with a little bit of help, anyone can contribute high-quality code. - Once all tests are passing, you have updated the changelog to reflect and tag your issue/pr for reference with a small description of the change, and your PR has been approved, a `dbt-bigquery` maintainer will merge your changes into the active development branch. And that's it! Happy developing :tada: From 1577e889832c39cc86af9827e356b9623bfb2ffd Mon Sep 17 00:00:00 2001 From: Matthew McKnight Date: Fri, 21 Jan 2022 15:03:56 -0600 Subject: [PATCH 10/10] updates based on review from kyle --- CONTRIBUTING.md | 10 ++++++---- 1 file changed, 6 insertions(+), 4 deletions(-) diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index 273cfb894..504710c70 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -10,7 +10,7 @@ ## About this document This document is a guide intended for folks interested in contributing to `dbt-bigquery`. Below, we document the process by which members of the community should create issues and submit pull requests (PRs) in this repository. It is not intended as a guide for using `dbt-bigquery`, and it assumes a certain level of familiarity with Python concepts such as virtualenvs, `pip`, python modules, filesystems, and so on. This guide assumes you are using macOS or Linux and are comfortable with the command line. -For those wishing to contribute we highly suggest reading the [dbt-core](https://github.com/dbt-labs/dbt-core/blob/main/CONTRIBUTING.md). if you haven't already. Almost all of the information there is applicable to contributing here, too! +For those wishing to contribute we highly suggest reading the [dbt-core](https://github.com/dbt-labs/dbt-core/blob/main/CONTRIBUTING.md), if you haven't already. Almost all of the information there is applicable to contributing here, too! ### Signing the CLA @@ -19,7 +19,7 @@ Please note that all contributors to `dbt-bigquery` must sign the [Contributor L ## Getting the code -You will need `git` in order to download and modify the `dbt-bigquery` source code. On macOS, the best way to download git is to just install [Xcode](https://developer.apple.com/support/xcode/). +You will need `git` in order to download and modify the `dbt-bigquery` source code. You can find direction [here](https://github.com/git-guides/install-git) on how to install `git`. ### External contributors @@ -54,7 +54,9 @@ To confirm you have the correct version of `dbt-core` installed please run `dbt ### Initial Setup -`dbt-bigquery` uses test credentials specified in a `test.env` file in the root of the repository. This `test.env` file is git-ignored, but please be _extra_ careful to never check in credentials or other sensitive information when developing. To create your `test.env` file, copy the provided example file, then supply your relevant credentials. +`dbt-bigquery` contains [unit](https://github.com/dbt-labs/dbt-bigquery/tree/main/tests/unit) and [integration](https://github.com/dbt-labs/dbt-bigquery/tree/main/tests/integration) tests. Integration tests require testing against an actual BigQuery warehouse. We have CI set up to test against a BigQuery warehouse. In order to run integration tests locally, you will need a `test.env` file in the root of the repository that contains credentials for BigQuery. + +Note: This `test.env` file is git-ignored, but please be _extra_ careful to never check in credentials or other sensitive information when developing. To create your `test.env` file, copy the provided example file, then supply your relevant credentials. ``` cp test.env.example test.env @@ -67,7 +69,7 @@ There are a few methods for running tests locally. #### `tox` `tox` takes care of managing Python virtualenvs and installing dependencies in order to run tests. You can also run tests in parallel, for example you can run unit tests for Python 3.7, Python 3.8, Python 3.9, and `flake8` checks in parallel with `tox -p`. Also, you can run unit tests for specific python versions with `tox -e py37`. The configuration of these tests are located in `tox.ini`. -#### `pyteest` +#### `pytest` Finally, you can also run a specific test or group of tests using `pytest` directly. With a Python virtualenv active and dev dependencies installed you can do things like: ```sh