Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

adding all code changes from black, pre-commit, mypy #148

Merged
merged 5 commits into from
Apr 8, 2022
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions .bumpversion.cfg
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@ parse = (?P<major>\d+)
\.(?P<minor>\d+)
\.(?P<patch>\d+)
((?P<prerelease>a|b|rc)(?P<num>\d+))?
serialize =
serialize =
{major}.{minor}.{patch}{prerelease}{num}
{major}.{minor}.{patch}
commit = False
Expand All @@ -13,7 +13,7 @@ tag = False
[bumpversion:part:prerelease]
first_value = a
optional_value = final
values =
values =
a
b
rc
Expand Down
2 changes: 1 addition & 1 deletion .github/pull_request_template.md
Original file line number Diff line number Diff line change
Expand Up @@ -18,4 +18,4 @@ resolves #
- [ ] I have signed the [CLA](https://docs.getdbt.com/docs/contributor-license-agreements)
- [ ] I have run this code in development and it appears to resolve the stated issue
- [ ] This PR includes tests, or tests are not required/relevant for this PR
- [ ] I have updated the `CHANGELOG.md` and added information about my change to the "dbt-bigquery next" section.
- [ ] I have updated the `CHANGELOG.md` and added information about my change to the "dbt-bigquery next" section.
4 changes: 2 additions & 2 deletions .github/workflows/integration.yml
Original file line number Diff line number Diff line change
Expand Up @@ -157,7 +157,7 @@ jobs:
pip install tox
pip --version
tox --version

- name: Install dbt-core latest
run: |
pip install "git+https://github.com/dbt-labs/dbt-core.git#egg=dbt-core&subdirectory=core"
Expand Down Expand Up @@ -212,7 +212,7 @@ jobs:

post-failure:
runs-on: ubuntu-latest
needs: test
needs: test
if: ${{ failure() }}

steps:
Expand Down
2 changes: 1 addition & 1 deletion .github/workflows/jira-creation.yml
Original file line number Diff line number Diff line change
Expand Up @@ -13,7 +13,7 @@ name: Jira Issue Creation
on:
issues:
types: [opened, labeled]

permissions:
issues: write

Expand Down
3 changes: 1 addition & 2 deletions .github/workflows/jira-label.yml
Original file line number Diff line number Diff line change
Expand Up @@ -13,7 +13,7 @@ name: Jira Label Mirroring
on:
issues:
types: [labeled, unlabeled]

permissions:
issues: read

Expand All @@ -24,4 +24,3 @@ jobs:
JIRA_BASE_URL: ${{ secrets.JIRA_BASE_URL }}
JIRA_USER_EMAIL: ${{ secrets.JIRA_USER_EMAIL }}
JIRA_API_TOKEN: ${{ secrets.JIRA_API_TOKEN }}

2 changes: 1 addition & 1 deletion .github/workflows/jira-transition.yml
Original file line number Diff line number Diff line change
Expand Up @@ -21,4 +21,4 @@ jobs:
secrets:
JIRA_BASE_URL: ${{ secrets.JIRA_BASE_URL }}
JIRA_USER_EMAIL: ${{ secrets.JIRA_USER_EMAIL }}
JIRA_API_TOKEN: ${{ secrets.JIRA_API_TOKEN }}
JIRA_API_TOKEN: ${{ secrets.JIRA_API_TOKEN }}
2 changes: 1 addition & 1 deletion .github/workflows/stale.yml
Original file line number Diff line number Diff line change
Expand Up @@ -14,4 +14,4 @@ jobs:
# mark issues/PRs stale when they haven't seen activity in 180 days
days-before-stale: 180
# ignore checking issues with the following labels
exempt-issue-labels: "epic, discussion"
exempt-issue-labels: "epic, discussion"
20 changes: 10 additions & 10 deletions .github/workflows/version-bump.yml
Original file line number Diff line number Diff line change
@@ -1,16 +1,16 @@
# **what?**
# This workflow will take a version number and a dry run flag. With that
# it will run versionbump to update the version number everywhere in the
# it will run versionbump to update the version number everywhere in the
# code base and then generate an update Docker requirements file. If this
# is a dry run, a draft PR will open with the changes. If this isn't a dry
# run, the changes will be committed to the branch this is run on.

# **why?**
# This is to aid in releasing dbt and making sure we have updated
# This is to aid in releasing dbt and making sure we have updated
# the versions and Docker requirements in all places.

# **when?**
# This is triggered either manually OR
# This is triggered either manually OR
# from the repository_dispatch event "version-bump" which is sent from
# the dbt-release repo Action

Expand All @@ -25,11 +25,11 @@ on:
is_dry_run:
description: 'Creates a draft PR to allow testing instead of committing to a branch'
required: true
default: 'true'
default: 'true'
repository_dispatch:
types: [version-bump]

jobs:
jobs:
bump:
runs-on: ubuntu-latest
steps:
Expand Down Expand Up @@ -57,19 +57,19 @@ jobs:
run: |
python3 -m venv env
source env/bin/activate
pip install --upgrade pip
pip install --upgrade pip

- name: Create PR branch
if: ${{ steps.variables.outputs.IS_DRY_RUN == 'true' }}
run: |
git checkout -b bumping-version/${{steps.variables.outputs.VERSION_NUMBER}}_$GITHUB_RUN_ID
git push origin bumping-version/${{steps.variables.outputs.VERSION_NUMBER}}_$GITHUB_RUN_ID
git branch --set-upstream-to=origin/bumping-version/${{steps.variables.outputs.VERSION_NUMBER}}_$GITHUB_RUN_ID bumping-version/${{steps.variables.outputs.VERSION_NUMBER}}_$GITHUB_RUN_ID

- name: Bumping version
run: |
source env/bin/activate
pip install -r dev_requirements.txt
pip install -r dev_requirements.txt
env/bin/bumpversion --allow-dirty --new-version ${{steps.variables.outputs.VERSION_NUMBER}} major
git status

Expand Down Expand Up @@ -99,4 +99,4 @@ jobs:
draft: true
base: ${{github.ref}}
title: 'Bumping version to ${{steps.variables.outputs.VERSION_NUMBER}}'
branch: 'bumping-version/${{steps.variables.outputs.VERSION_NUMBER}}_${{GITHUB.RUN_ID}}'
branch: 'bumping-version/${{steps.variables.outputs.VERSION_NUMBER}}_${{GITHUB.RUN_ID}}'
2 changes: 1 addition & 1 deletion .pre-commit-config.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -50,4 +50,4 @@ repos:
stages: [manual]
args: [--show-error-codes, --pretty, --ignore-missing-imports]
files: ^dbt/adapters
language: system
language: system
1 change: 1 addition & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -3,6 +3,7 @@
### Under the hood
- Use dbt.tests.adapter.basic in tests (new test framework) ([#135](https://github.com/dbt-labs/dbt-bigquery/issues/135), [#142](https://github.com/dbt-labs/dbt-bigquery/pull/142))
- Adding pre-commit and black formatter hooks ([#147](https://github.com/dbt-labs/dbt-bigquery/pull/147))
- Adding pre-commit code changes ([#148](https://github.com/dbt-labs/dbt-bigquery/pull/148))

## dbt-bigquery 1.1.0b1 (March 23, 2022)
### Features
Expand Down
4 changes: 2 additions & 2 deletions CONTRIBUTING.md
Original file line number Diff line number Diff line change
Expand Up @@ -17,7 +17,7 @@ For those wishing to contribute we highly suggest reading the [dbt-core](https:/
Please note that all contributors to `dbt-bigquery` must sign the [Contributor License Agreement](https://docs.getdbt.com/docs/contributor-license-agreements) to have their Pull Request merged into an `dbt-bigquery` codebase. If you are unable to sign the CLA, then the `dbt-bigquery` maintainers will unfortunately be unable to merge your Pull Request. You are, however, welcome to open issues and comment on existing ones.


## Getting the code
## Getting the code

You will need `git` in order to download and modify the `dbt-bigquery` source code. You can find direction [here](https://github.com/git-guides/install-git) on how to install `git`.

Expand Down Expand Up @@ -93,7 +93,7 @@ Many changes will require and update to the `dbt-bigquery` docs here are some us

## Submitting a Pull Request

dbt Labs provides a CI environment to test changes to the `dbt-bigquery` adapter and periodic checks against the development version of `dbt-core` through Github Actions.
dbt Labs provides a CI environment to test changes to the `dbt-bigquery` adapter and periodic checks against the development version of `dbt-core` through Github Actions.

A `dbt-bigquery` maintainer will review your PR. They may suggest code revision for style or clarity, or request that you add unit or integration test(s). These are good things! We believe that, with a little bit of help, anyone can contribute high-quality code.

Expand Down
2 changes: 1 addition & 1 deletion MANIFEST.in
Original file line number Diff line number Diff line change
@@ -1 +1 @@
recursive-include dbt/include *.sql *.yml *.md
recursive-include dbt/include *.sql *.yml *.md
5 changes: 2 additions & 3 deletions dbt/adapters/bigquery/__init__.py
Original file line number Diff line number Diff line change
Expand Up @@ -8,6 +8,5 @@
from dbt.include import bigquery

Plugin = AdapterPlugin(
adapter=BigQueryAdapter,
credentials=BigQueryCredentials,
include_path=bigquery.PACKAGE_PATH)
adapter=BigQueryAdapter, credentials=BigQueryCredentials, include_path=bigquery.PACKAGE_PATH # type: ignore[arg-type]
)
2 changes: 1 addition & 1 deletion dbt/adapters/bigquery/__version__.py
Original file line number Diff line number Diff line change
@@ -1 +1 @@
version = '1.1.0b1'
version = "1.1.0b1"
64 changes: 29 additions & 35 deletions dbt/adapters/bigquery/column.py
Original file line number Diff line number Diff line change
Expand Up @@ -5,27 +5,27 @@

from google.cloud.bigquery import SchemaField

Self = TypeVar('Self', bound='BigQueryColumn')
Self = TypeVar("Self", bound="BigQueryColumn")


@dataclass(init=False)
class BigQueryColumn(Column):
TYPE_LABELS = {
'STRING': 'STRING',
'TIMESTAMP': 'TIMESTAMP',
'FLOAT': 'FLOAT64',
'INTEGER': 'INT64',
'RECORD': 'RECORD',
"STRING": "STRING",
"TIMESTAMP": "TIMESTAMP",
"FLOAT": "FLOAT64",
"INTEGER": "INT64",
"RECORD": "RECORD",
}
fields: List[Self] # type: ignore[valid-type]
mode: str
fields: List[Self] # type: ignore
mode: str # type: ignore

def __init__(
self,
column: str,
dtype: str,
fields: Optional[Iterable[SchemaField]] = None,
mode: str = 'NULLABLE',
mode: str = "NULLABLE",
) -> None:
super().__init__(column, dtype)

Expand All @@ -36,9 +36,7 @@ def __init__(
self.mode = mode

@classmethod
def wrap_subfields(
cls: Type[Self], fields: Iterable[SchemaField]
) -> List[Self]:
def wrap_subfields(cls: Type[Self], fields: Iterable[SchemaField]) -> List[Self]:
return [cls.create_from_field(field) for field in fields]

@classmethod
Expand All @@ -51,20 +49,18 @@ def create_from_field(cls: Type[Self], field: SchemaField) -> Self:
)

@classmethod
def _flatten_recursive(
cls: Type[Self], col: Self, prefix: Optional[str] = None
) -> List[Self]:
def _flatten_recursive(cls: Type[Self], col: Self, prefix: Optional[str] = None) -> List[Self]:
if prefix is None:
prefix = [] # type: ignore[assignment]
prefix = [] # type: ignore[assignment]

if len(col.fields) == 0:
prefixed_name = ".".join(prefix + [col.column]) # type: ignore[operator]
prefixed_name = ".".join(prefix + [col.column]) # type: ignore[operator]
new_col = cls(prefixed_name, col.dtype, col.fields, col.mode)
return [new_col]

new_fields = []
for field in col.fields:
new_prefix = prefix + [col.column] # type: ignore[operator]
new_prefix = prefix + [col.column] # type: ignore[operator]
new_fields.extend(cls._flatten_recursive(field, new_prefix))

return new_fields
Expand All @@ -74,54 +70,52 @@ def flatten(self):

@property
def quoted(self):
return '`{}`'.format(self.column)
return "`{}`".format(self.column)

def literal(self, value):
return "cast({} as {})".format(value, self.dtype)

@property
def data_type(self) -> str:
if self.dtype.upper() == 'RECORD':
if self.dtype.upper() == "RECORD":
subcols = [
"{} {}".format(col.name, col.data_type) for col in self.fields
] # type: ignore[attr-defined]
field_type = 'STRUCT<{}>'.format(", ".join(subcols))
"{} {}".format(col.name, col.data_type) for col in self.fields # type: ignore[attr-defined]
]
field_type = "STRUCT<{}>".format(", ".join(subcols))

else:
field_type = self.dtype

if self.mode.upper() == 'REPEATED':
return 'ARRAY<{}>'.format(field_type)
if self.mode.upper() == "REPEATED":
return "ARRAY<{}>".format(field_type)

else:
return field_type

def is_string(self) -> bool:
return self.dtype.lower() == 'string'
return self.dtype.lower() == "string"

def is_integer(self) -> bool:
return self.dtype.lower() == 'int64'
return self.dtype.lower() == "int64"

def is_numeric(self) -> bool:
return self.dtype.lower() == 'numeric'
return self.dtype.lower() == "numeric"

def is_float(self):
return self.dtype.lower() == 'float64'
return self.dtype.lower() == "float64"

def can_expand_to(self: Self, other_column: Self) -> bool:
def can_expand_to(self: Self, other_column: Self) -> bool: # type: ignore
"""returns True if both columns are strings"""
return self.is_string() and other_column.is_string()

def __repr__(self) -> str:
return "<BigQueryColumn {} ({}, {})>".format(self.name, self.data_type,
self.mode)
return "<BigQueryColumn {} ({}, {})>".format(self.name, self.data_type, self.mode)

def column_to_bq_schema(self) -> SchemaField:
"""Convert a column to a bigquery schema object.
"""
"""Convert a column to a bigquery schema object."""
kwargs = {}
if len(self.fields) > 0:
fields = [field.column_to_bq_schema() for field in self.fields] # type: ignore[attr-defined]
fields = [field.column_to_bq_schema() for field in self.fields] # type: ignore[attr-defined]
kwargs = {"fields": fields}

return SchemaField(self.name, self.dtype, self.mode, **kwargs)
Loading