Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Prepare 1.5.0 #390

Draft
wants to merge 35 commits into
base: master
Choose a base branch
from
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
35 commits
Select commit Hold shift + click to select a range
48a3556
bump version
sdebruyn May 18, 2023
f4e8f5d
add new basic tests
sdebruyn May 18, 2023
cbe301a
add new caching tests
sdebruyn May 18, 2023
ff147eb
add new column type tests
sdebruyn May 18, 2023
e56b153
add new constraints tests
sdebruyn May 18, 2023
581f72b
simplify debug tests
sdebruyn May 18, 2023
6573d0c
rename data type tests
sdebruyn May 18, 2023
9e5c8cd
move docs to basic
sdebruyn May 18, 2023
bd56d0b
add new ephemeral tests
sdebruyn May 18, 2023
0363759
add new hooks tests
sdebruyn May 18, 2023
d5f109c
wip
sdebruyn May 18, 2023
71739c2
use rc1 versions for now
sdebruyn May 21, 2023
767b16f
add new incremental tests
sdebruyn May 21, 2023
9e8c82c
add new persist docs tests
sdebruyn May 21, 2023
53071bc
add new relations tests
sdebruyn May 21, 2023
7ba8260
add new simple copy test
sdebruyn May 21, 2023
74d91c0
split simple and non simple seed tests
sdebruyn May 21, 2023
27166cf
rename simple snapshot tests
sdebruyn May 21, 2023
6599edd
add store test failures
sdebruyn May 21, 2023
e359331
update ref
sdebruyn May 21, 2023
ae0e943
remove hooks tests
sdebruyn May 21, 2023
d9bec41
chore: merge upstream patch fixes
Oct 19, 2023
a7bf370
build(dev): give Dockerfile temporary root permissions for filesystem…
Oct 19, 2023
53d34bd
build: match pre-commit python version to project
Oct 19, 2023
8f93a71
ci: install pyodbc dependencies in test container
Oct 19, 2023
e53572c
ci: devcontainer setup/use ruff for format/sorting imports
schlich Oct 31, 2023
f4d7870
refactor: run ruff
schlich Oct 31, 2023
3f27667
fix: specify odbc driver 18
schlich Nov 1, 2023
4794cdf
ci(pytest): fail fast
schlich Nov 1, 2023
5df68e0
fix: broken module imports in tests
schlich Nov 1, 2023
d256821
build: upgrade dbt-core to latest patch
schlich Nov 1, 2023
9444062
ci: initialize pdm project
schlich Nov 1, 2023
a1b00ad
Merge pull request #452 from schlich:base-work-1.5
schlich Nov 1, 2023
0e185fc
checkpoint
schlich Nov 18, 2023
ecdda8b
[pre-commit.ci] auto fixes from pre-commit.com hooks
pre-commit-ci[bot] Nov 18, 2023
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
47 changes: 47 additions & 0 deletions .devcontainer/devcontainer.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,47 @@
{
"name": "dbt-sqlserver",
"image": "mcr.microsoft.com/devcontainers/python:3.12",
"features": {
"ghcr.io/devcontainers/features/docker-in-docker:2": {},
"ghcr.io/jlaundry/devcontainer-features/mssql-odbc-driver:1": {
"version": "18"
},
"ghcr.io/devcontainers/features/azure-cli:1": {}
},
"customizations": {
"vscode": {
"settings": {
"python.defaultInterpreterPath": ".venv/bin/python",
"python.terminal.activateEnvInCurrentTerminal": true,
"[python]": {
"editor.codeActionsOnSave": {
"source.fixAll": true,
"source.organizeImports": true
},
"editor.defaultFormatter": "charliermarsh.ruff"
},
"python.testing.cwd": "tests",
"python.testing.unittestEnabled": false,
"python.testing.pytestEnabled": true,
"editor.formatOnSave": true,
"editor.formatOnPaste": true,
"editor.stickyScroll.enabled": true,
"dbt.enableNewLineagePanel": true,
"files.insertFinalNewline": true,
"terminal.integrated.environmentChangesIndicator": "on"
},
"extensions": [
"ms-python.python",
"ms-python.vscode-pylance",
"innoverio.vscode-dbt-power-user",
"ms-vscode.makefile-tools",
"charliermarsh.ruff",
"eamodio.gitlens",
"GitHub.vscode-github-actions",
"tamasfe.even-better-toml",
"esbenp.prettier-vscode",
"redhat.vscode-yaml"
]
}
}
}
61 changes: 12 additions & 49 deletions .pre-commit-config.yaml
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
default_language_version:
python: python3.9
python: python3.11
repos:
- repo: 'https://github.com/pre-commit/pre-commit-hooks'
rev: v4.4.0
Expand All @@ -20,63 +20,25 @@ repos:
- id: fix-byte-order-marker
- id: mixed-line-ending
- id: check-docstring-first

- repo: 'https://github.com/adrienverge/yamllint'
rev: v1.32.0
hooks:
- id: yamllint
args:
- '-d {extends: default, rules: {line-length: disable, document-start: disable}}'
- '-s'
- repo: 'https://github.com/MarcoGorelli/absolufy-imports'
rev: v0.3.1

- repo: https://github.com/astral-sh/ruff-pre-commit
rev: v0.1.3
hooks:
- id: absolufy-imports
- repo: 'https://github.com/hadialqattan/pycln'
rev: v2.1.3
- id: ruff

- repo: https://github.com/astral-sh/ruff-pre-commit
rev: v0.1.3
hooks:
- id: pycln
args:
- '--all'
- repo: 'https://github.com/pycqa/isort'
rev: 5.12.0
hooks:
- id: isort
args:
- '--profile'
- black
- '--atomic'
- '--line-length'
- '99'
- '--python-version'
- '39'
- repo: 'https://github.com/psf/black'
rev: 23.3.0
hooks:
- id: black
args:
- '--line-length=99'
- '--target-version=py39'
- id: black
alias: black-check
stages:
- manual
args:
- '--line-length=99'
- '--target-version=py39'
- '--check'
- '--diff'
- repo: 'https://github.com/pycqa/flake8'
rev: 6.0.0
hooks:
- id: flake8
args:
- '--max-line-length=99'
- id: flake8
args:
- '--max-line-length=99'
alias: flake8-check
stages:
- manual
- id: ruff-format

- repo: 'https://github.com/pre-commit/mirrors-mypy'
rev: v1.3.0
hooks:
Expand All @@ -86,6 +48,7 @@ repos:
- '--ignore-missing-imports'
- '--explicit-package-bases'
files: '^dbt/adapters'

- id: mypy
alias: mypy-check
stages:
Expand Down
7 changes: 0 additions & 7 deletions .vscode/settings.json

This file was deleted.

25 changes: 14 additions & 11 deletions CONTRIBUTING.md
Original file line number Diff line number Diff line change
@@ -1,12 +1,16 @@
# Development of the adapter

Python 3.10 is used for developing the adapter. To get started, bootstrap your environment as follows:
## Dependencies

Create a virtual environment, [pyenv](https://github.com/pyenv/pyenv) is used in the example:
A recent version of [Docker](https://docs.docker.com/get-docker/).

Python 3.11

Create a virtual environment to properly isolate your Python dependencies. Here is an example of how to do so using the [pyenv](https://github.com/pyenv/pyenv) CLI:

```shell
pyenv install 3.10.7
pyenv virtualenv 3.10.7 dbt-sqlserver
pyenv install 3.11.6
pyenv virtualenv 3.11.6 dbt-sqlserver
pyenv activate dbt-sqlserver
```

Expand All @@ -22,22 +26,21 @@ After running `make dev`, pre-commit will automatically validate your commits an

## Testing

The functional tests require a running SQL Server instance. You can easily spin up a local instance with the following command:
The functional tests require a running SQL Server instance. We'll spin one up using Docker Compose. First, copy the example `.env.test.sample` to create a new file `.env.test`, which contains the environment variables that will be read by Docker Compose:

```shell
make server
cp test.env.sample test.env
```

This will use Docker Compose to spin up a local instance of SQL Server. Docker Compose is now bundled with Docker, so make sure to [install the latest version of Docker](https://docs.docker.com/get-docker/).
If desired, you can tweak the contents of `test.env` to test against a different database.

Next, tell our tests how they should connect to the local instance by creating a file called `test.env` in the root of the project.
You can use the provided `test.env.sample` as a base and if you started the server with `make server`, then this matches the instance running on your local machine.
Now, you can easily spin up a local SQL Server instance:

```shell
cp test.env.sample test.env
make server
```

You can tweak the contents of this file to test against a different database.
This will use Docker Compose to fetch the image and run the container.

Note that we need 3 users to be able to run tests related to the grants.
The 3 users are defined by the following environment variables containing their usernames.
Expand Down
8 changes: 4 additions & 4 deletions Makefile
Original file line number Diff line number Diff line change
Expand Up @@ -39,19 +39,19 @@ linecheck: ## Checks for all Python lines 100 characters or more
.PHONY: unit
unit: ## Runs unit tests.
@\
pytest -n auto -ra -v tests/unit
pytest tests/unit

.PHONY: functional
functional: ## Runs functional tests.
@\
pytest -n auto -ra -v tests/functional
pytest tests/functional

.PHONY: test
test: ## Runs unit tests and code checks against staged changes.
@\
pytest -n auto -ra -v tests/unit; \
pre-commit run black-check --hook-stage manual | grep -v "INFO"; \
pre-commit run flake8-check --hook-stage manual | grep -v "INFO"; \
pre-commit run ruff-format --all-files --hook-stage manual | grep -v "INFO"; \
pre-commit run ruff --all-files --hook-stage manual | grep -v "INFO"; \
pre-commit run mypy-check --hook-stage manual | grep -v "INFO"

.PHONY: server
Expand Down
Empty file added dbt/__init__.py
Empty file.
4 changes: 3 additions & 1 deletion dbt/adapters/sqlserver/__init__.py
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,9 @@
from dbt.adapters.sqlserver.sql_server_adapter import SQLServerAdapter
from dbt.adapters.sqlserver.sql_server_column import SQLServerColumn
from dbt.adapters.sqlserver.sql_server_configs import SQLServerConfigs
from dbt.adapters.sqlserver.sql_server_connection_manager import SQLServerConnectionManager
from dbt.adapters.sqlserver.sql_server_connection_manager import (
SQLServerConnectionManager,
)
from dbt.adapters.sqlserver.sql_server_credentials import SQLServerCredentials
from dbt.include import sqlserver

Expand Down
2 changes: 1 addition & 1 deletion dbt/adapters/sqlserver/__version__.py
Original file line number Diff line number Diff line change
@@ -1 +1 @@
version = "1.4.3"
version = "1.5.9"
16 changes: 11 additions & 5 deletions dbt/adapters/sqlserver/sql_server_adapter.py
Original file line number Diff line number Diff line change
Expand Up @@ -2,15 +2,17 @@

import agate
from dbt.adapters.base.relation import BaseRelation
from dbt.adapters.cache import _make_ref_key_msg
from dbt.adapters.cache import _make_ref_key_dict
from dbt.adapters.sql import SQLAdapter
from dbt.adapters.sql.impl import CREATE_SCHEMA_MACRO_NAME
from dbt.events.functions import fire_event
from dbt.events.types import SchemaCreation

from dbt.adapters.sqlserver.sql_server_column import SQLServerColumn
from dbt.adapters.sqlserver.sql_server_configs import SQLServerConfigs
from dbt.adapters.sqlserver.sql_server_connection_manager import SQLServerConnectionManager
from dbt.adapters.sqlserver.sql_server_connection_manager import (
SQLServerConnectionManager,
)


class SQLServerAdapter(SQLAdapter):
Expand All @@ -20,14 +22,16 @@ class SQLServerAdapter(SQLAdapter):

def create_schema(self, relation: BaseRelation) -> None:
relation = relation.without_identifier()
fire_event(SchemaCreation(relation=_make_ref_key_msg(relation)))
fire_event(SchemaCreation(relation=_make_ref_key_dict(relation)))
macro_name = CREATE_SCHEMA_MACRO_NAME
kwargs = {
"relation": relation,
}

if self.config.credentials.schema_authorization:
kwargs["schema_authorization"] = self.config.credentials.schema_authorization
kwargs[
"schema_authorization"
] = self.config.credentials.schema_authorization
macro_name = "sqlserver__create_schema_with_authorization"

self.execute_macro(macro_name, kwargs=kwargs)
Expand Down Expand Up @@ -64,7 +68,9 @@ def convert_time_type(cls, agate_table, col_idx):
return "datetime"

# Methods used in adapter tests
def timestamp_add_sql(self, add_to: str, number: int = 1, interval: str = "hour") -> str:
def timestamp_add_sql(
self, add_to: str, number: int = 1, interval: str = "hour"
) -> str:
# note: 'interval' is not supported for T-SQL
# for backwards compatibility, we're compelled to set some sort of
# default. A lot of searching has lead me to believe that the
Expand Down
12 changes: 9 additions & 3 deletions dbt/adapters/sqlserver/sql_server_connection_manager.py
Original file line number Diff line number Diff line change
Expand Up @@ -161,7 +161,9 @@ def get_sp_access_token(credentials: SQLServerCredentials) -> AccessToken:
The access token.
"""
token = ClientSecretCredential(
str(credentials.tenant_id), str(credentials.client_id), str(credentials.client_secret)
str(credentials.tenant_id),
str(credentials.client_id),
str(credentials.client_secret),
).get_token(AZURE_CREDENTIAL_SCOPE)
return token

Expand Down Expand Up @@ -200,7 +202,9 @@ def get_pyodbc_attrs_before(credentials: SQLServerCredentials) -> Dict:

authentication = str(credentials.authentication).lower()
if authentication in AZURE_AUTH_FUNCTIONS:
time_remaining = (_TOKEN.expires_on - time.time()) if _TOKEN else MAX_REMAINING_TIME
time_remaining = (
(_TOKEN.expires_on - time.time()) if _TOKEN else MAX_REMAINING_TIME
)

if _TOKEN is None or (time_remaining < MAX_REMAINING_TIME):
azure_auth_function = AZURE_AUTH_FUNCTIONS[authentication]
Expand Down Expand Up @@ -341,7 +345,9 @@ def open(cls, connection: Connection) -> Connection:

con_str.append(bool_to_connection_string_arg("encrypt", credentials.encrypt))
con_str.append(
bool_to_connection_string_arg("TrustServerCertificate", credentials.trust_cert)
bool_to_connection_string_arg(
"TrustServerCertificate", credentials.trust_cert
)
)

plugin_version = __version__.version
Expand Down
Loading
Loading