Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

update contribution docs and add Python 3.10 #249

Merged
merged 1 commit into from
Jun 6, 2022
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion .github/workflows/integration-tests-azure.yml
Original file line number Diff line number Diff line change
Expand Up @@ -15,7 +15,7 @@ jobs:
fail-fast: false
max-parallel: 1
matrix:
python_version: ["3.7", "3.8", "3.9"]
python_version: ["3.7", "3.8", "3.9", "3.10"]
profile: ["ci_azure_cli", "ci_azure_auto", "ci_azure_environment", "ci_azure_basic"]
runs-on: ubuntu-latest
container:
Expand Down
2 changes: 1 addition & 1 deletion .github/workflows/integration-tests-sqlserver.yml
Original file line number Diff line number Diff line change
Expand Up @@ -14,7 +14,7 @@ jobs:
strategy:
fail-fast: false
matrix:
python_version: ["3.7", "3.8", "3.9"]
python_version: ["3.7", "3.8", "3.9", "3.10"]
profile: ["ci_sql_server", "ci_sql_server_encrypt"]
runs-on: ubuntu-latest
container:
Expand Down
2 changes: 1 addition & 1 deletion .github/workflows/unit-tests.yml
Original file line number Diff line number Diff line change
Expand Up @@ -13,7 +13,7 @@ jobs:
name: Unit tests
strategy:
matrix:
python_version: ["3.7", "3.8", "3.9"]
python_version: ["3.7", "3.8", "3.9", "3.10"]
runs-on: ubuntu-latest
permissions:
contents: read
Expand Down
61 changes: 44 additions & 17 deletions CONTRIBUTING.md
Original file line number Diff line number Diff line change
@@ -1,15 +1,16 @@
# Development of the adapter

Python 3.9 is used for developing the adapter. To get started, setup your environment as follows:
Python 3.10 is used for developing the adapter. To get started, bootstrap your environment as follows:

Create a virtual environment, pyenv is used in the example:
Create a virtual environment, [pyenv](https://github.com/pyenv/pyenv) is used in the example:

```shell
pyenv install 3.7.13
pyenv install 3.8.13
pyenv install 3.9.12
pyenv virtualenv 3.9.12 dbt-sqlserver
pyenv local dbt-sqlserver 3.8.13 3.7.13
pyenv install 3.10.4
pyenv virtualenv 3.10.4 dbt-sqlserver
pyenv local dbt-sqlserver 3.9.12 3.8.13 3.7.13
```

Install the development dependencies and pre-commit and get information about possible make commands:
Expand All @@ -19,32 +20,56 @@ make dev
make help
```

[Pre-commit](https://pre-commit.com/) helps us to maintain a consistent style and code quality across the entire project.
After running `make dev`, pre-commit will automatically validate your commits and fix any formatting issues whenever possible.

## Testing

### pytest-dbt-adapter
The functional tests require a running SQL Server instance. You can easily spin up a local instance with the following command:

```shell
make server
```

The package [pytest-dbt-adapter](https://github.com/dbt-labs/dbt-adapter-tests) is used for running tests against the adapter.
However, this is no longer the recommended way to test adapters and we should into replacing this with [the recommended way to test new adapter](https://docs.getdbt.com/docs/contributing/testing-a-new-adapter)
This will use Docker Compose to spin up a local instance of SQL Server. Docker Compose is now bundled with Docker, so make sure to [install the latest version of Docker](https://docs.docker.com/get-docker/).

### Tox
Next, tell our tests how they should connect to the local instance by creating a file called `test.env` in the root of the project.
You can use the provided `test.env.sample` as a base and if you started the server with `make server`, then this matches the instance running on your local machine.

```shell
cp test.env.sample test.env
```

Running the unit tests:
We use tox to isolate the environment for testing and for making it possible to test in multiple environments.
You can use the following commands to run the unit and the functional tests respectively:

```shell
make test
make unit
make functional
```

This will start tox and test in all environments. If you only want to run the tests in a specific environment, you can use the following commands:

```shell
tox -e py310 -- -v tests/functional # will only functional run in Python 3.10
tox -e py39 -- -v tests/unit # will only run unit test in Python 3.9
```

## CI/CD

We use Docker images that have all the things we need to run the adapter. The Dockerfile is located in the *.github* directory and pushed to GitHub Packages to this repo.
We use Docker images that have all the things we need to test the adapter in the CI/CD workflows.
The Dockerfile is located in the *.github* directory and pushed to GitHub Packages to this repo.
There is one tag per supported Python version.

There is a Circle CI workflow with jobs that run the following tasks:
All CI/CD pipelines are using GitHub Actions. The following pipelines are available:

* `publish-docker`: publishes the image we use in all other pipelines.
* `unit-tests`: runs the unit tests for each supported Python version.
* `integration-tests-azure`: runs the integration tests for Azure SQL Server.
* `integration-tests-sqlserver`: runs the integration tests for SQL Server.
* `release-version`: publishes the adapter to PyPI.

* Run the unit tests
* Use the adapter to connect to a SQL Server Docker container
* Run the pytest-dbt-adapter specs against a SQL Server Docker container
* Use the adapter to connect to an Azure SQL Database with various options
There is an additional [Pre-commit](https://pre-commit.ci/) pipeline that validates the code style.

### Azure integration tests

Expand All @@ -65,4 +90,6 @@ The following environment variables are available:
## Releasing a new version

Make sure the version number is bumped in `__version__.py`. Then, create a git tag named `v<version>` and push it to GitHub.
A CircleCI workflow will be triggered to build the package and push it to PyPI.
A GitHub Actions workflow will be triggered to build the package and push it to PyPI.

If you're releasing support for a new version of `dbt-core`, also bump the `dbt_version` in `setup.py`.
2 changes: 1 addition & 1 deletion Makefile
Original file line number Diff line number Diff line change
Expand Up @@ -57,7 +57,7 @@ test: ## Runs unit tests and code checks against staged changes.
.PHONY: server
server: ## Spins up a local MS SQL Server instance for development. Docker-compose is required.
@\
docker-compose up -d
docker compose up -d

.PHONY: clean
@echo "cleaning repo"
Expand Down
12 changes: 6 additions & 6 deletions setup.py
Original file line number Diff line number Diff line change
Expand Up @@ -6,13 +6,15 @@
from setuptools import find_namespace_packages, setup
from setuptools.command.install import install

package_name = "dbt-sqlserver"
authors_list = ["Mikael Ene", "Anders Swanson", "Sam Debruyn", "Cor Zuurmond"]
dbt_version = "1.1"
description = """A Microsoft SQL Server adapter plugin for dbt (data build tool)"""

this_directory = os.path.abspath(os.path.dirname(__file__))
with open(os.path.join(this_directory, "README.md")) as f:
long_description = f.read()

package_name = "dbt-sqlserver"
authors_list = ["Mikael Ene", "Anders Swanson", "Sam Debruyn", "Cor Zuurmond"]


# get this from a separate file
def _dbt_sqlserver_version():
Expand All @@ -26,9 +28,7 @@ def _dbt_sqlserver_version():


package_version = _dbt_sqlserver_version()
description = """A sqlserver adapter plugin for dbt (data build tool)"""

dbt_version = "1.1"
# the package version should be the dbt version, with maybe some things on the
# ends of it. (0.18.1 vs 0.18.1a1, 0.18.1.1, ...)
if not package_version.startswith(dbt_version):
Expand Down Expand Up @@ -66,7 +66,7 @@ def run(self):
packages=find_namespace_packages(include=["dbt", "dbt.*"]),
include_package_data=True,
install_requires=[
"dbt-core~=1.1.0",
f"dbt-core~={dbt_version}.0",
"pyodbc~=4.0.32",
"azure-identity>=1.10.0",
],
Expand Down
4 changes: 2 additions & 2 deletions tox.ini
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
[tox]
skipsdist = true
envlist = py{37,38,39}
envlist = py{37,38,39,310}

[testenv]
deps =
Expand All @@ -12,6 +12,6 @@ allowlist_externals =
/bin/sh
az

[testenv:py{37,38,39}-azure]
[testenv:py{37,38,39,310}-azure]
commands_pre =
az login --service-principal --username="{env:AZURE_CLIENT_ID:}" --password="{env:AZURE_CLIENT_SECRET:}" --tenant="{env:AZURE_TENANT_ID:}"