Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[connectors-ci] sentry: ignore error logs without exceptions #28897

Merged
merged 2 commits into from
Aug 1, 2023
Merged

[connectors-ci] sentry: ignore error logs without exceptions #28897

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
4100daa
sentry: ignore error logs without exceptions
pedroslopez Aug 1, 2023
adfb042
bump version
pedroslopez Aug 1, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
13 changes: 7 additions & 6 deletions airbyte-ci/connectors/pipelines/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -96,7 +96,7 @@ At this point you can run `airbyte-ci` commands from the root of the repository.
#### Options

| Option | Default value | Mapped environment variable | Description |
| --------------------------------------- | ------------------------------- | ----------------------------- | ------------------------------------------------------------------------------------------- |
|-----------------------------------------|---------------------------------|-------------------------------|---------------------------------------------------------------------------------------------|
| `--is-local/--is-ci` | `--is-local` | | Determines the environment in which the CLI runs: local environment or CI environment. |
| `--git-branch` | The checked out git branch name | `CI_GIT_BRANCH` | The git branch on which the pipelines will run. |
| `--git-revision` | The current branch head | `CI_GIT_REVISION` | The commit hash on which the pipelines will run. |
Expand All @@ -115,7 +115,7 @@ Available commands:

#### Options
| Option | Multiple | Default value | Description |
| ---------------------- | -------- | ------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
|------------------------|----------|---------------|-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| `--use-remote-secrets` | False | True | If True, connectors configuration will be pulled from Google Secret Manager. Requires the GCP_GSM_CREDENTIALS environment variable to be set with a service account with permission to read GSM secrets. If False the connector configuration will be read from the local connector `secrets` folder. |
| `--name` | True | | Select a specific connector for which the pipeline will run. Can be used multiple time to select multiple connectors. The expected name is the connector technical name. e.g. `source-pokeapi` |
| `--release-stage` | True | | Select connectors with a specific release stage: `alpha`, `beta`, `generally_available`. Can be used multiple times to select multiple release stages. |
Expand Down Expand Up @@ -282,7 +282,7 @@ Publish all connectors modified in the head commit: `airbyte-ci connectors --mod
### Options

| Option | Required | Default | Mapped environment variable | Description |
| ------------------------------------ | -------- | --------------- | ---------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
|--------------------------------------|----------|-----------------|------------------------------------|-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| `--pre-release/--main-release` | False | `--pre-release` | | Whether to publish the pre-release or the main release version of a connector. Defaults to pre-release. For main release you have to set the credentials to interact with the GCS bucket. |
| `--docker-hub-username` | True | | `DOCKER_HUB_USERNAME` | Your username to connect to DockerHub. |
| `--docker-hub-password` | True | | `DOCKER_HUB_PASSWORD` | Your password to connect to DockerHub. |
Expand Down Expand Up @@ -326,7 +326,7 @@ Validate all `metadata.yaml` files in the repo:

#### Options
| Option | Default | Description |
| ------------------ | ------------ | -------------------------------------------------------------------------------------------------------------------------- |
|--------------------|--------------|----------------------------------------------------------------------------------------------------------------------------|
| `--modified/--all` | `--modified` | Flag to run validation of `metadata.yaml` files on the modified files in the head commit or all the `metadata.yaml` files. |

### <a id="metadata-upload-command"></a>`metadata upload` command
Expand All @@ -338,7 +338,7 @@ Upload all the `metadata.yaml` files to a GCS bucket:

#### Options
| Option | Required | Default | Mapped environment variable | Description |
| ------------------- | -------- | ------------ | --------------------------- | ------------------------------------------------------------------------------------------------------------------------ |
|---------------------|----------|--------------|-----------------------------|--------------------------------------------------------------------------------------------------------------------------|
| `--gcs-credentials` | True | | `GCS_CREDENTIALS` | Service account credentials in JSON format with permission to get and upload on the GCS bucket |
| `--modified/--all` | True | `--modified` | | Flag to upload the modified `metadata.yaml` files in the head commit or all the `metadata.yaml` files to a GCS bucket. |

Expand Down Expand Up @@ -376,7 +376,8 @@ This command runs the Python tests for a airbyte-ci poetry package.
## Changelog

| Version | PR | Description |
| ------- | --------------------------------------------------------- | -------------------------------------------------------------------------------------------- |
|---------|-----------------------------------------------------------|----------------------------------------------------------------------------------------------|
| 0.2.1 | [#28897](https://github.com/airbytehq/airbyte/pull/28897) | Sentry: Ignore error logs without exceptions from reporting |
| 0.2.0 | [#28857](https://github.com/airbytehq/airbyte/pull/28857) | Add the `airbyte-ci tests` command to run the test suite on any `airbyte-ci` poetry package. |
| 0.1.1 | [#28858](https://github.com/airbytehq/airbyte/pull/28858) | Increase the max duration of Connector Package install to 20mn. |
| 0.1.0 | | Alpha version not in production yet. All the commands described in this doc are available. |
Expand Down
9 changes: 9 additions & 0 deletions airbyte-ci/connectors/pipelines/pipelines/sentry_utils.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -12,10 +12,19 @@ def initialize():
if "SENTRY_DSN" in os.environ:
sentry_sdk.init(
dsn=os.environ.get("SENTRY_DSN"),
before_send=before_send,
release=f"pipelines@{importlib.metadata.version('pipelines')}",
)


def before_send(event, hint):
# Ignore logged errors that do not contain an exception
if "log_record" in hint and "exc_info" not in hint:
return None

return event


def with_step_context(func):
def wrapper(self, *args, **kwargs):
with sentry_sdk.configure_scope() as scope:
Expand Down
2 changes: 1 addition & 1 deletion airbyte-ci/connectors/pipelines/pyproject.toml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@ build-backend = "poetry.core.masonry.api"

[tool.poetry]
name = "pipelines"
version = "0.1.1"
version = "0.2.1"
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

note: 0.2.0 wasn't bumped properly

description = "Packaged maintained by the connector operations team to perform CI for connectors' pipelines"
authors = ["Airbyte <contact@airbyte.io>"]

Expand Down