Remove `pytest.mark.db_test` from providers where possible

[potiuk]

Body

After #51930 where @amoghrajesh introduces a way how tests can define connections without DB and follow up in #52017 where I made an attempt to remove some of the pytest.mark.db_test we should now remove those marks from providers, where it is easy. Some providers still use DB for other things, but likely there are many providers that only used db to create a Connection and we can turn those tests into non-db tests.

• airbyte - Remove pytest.mark.db_test: airbyte and amazon providers where possible #52017 - clean
• alibaba - no change, still db tests left
• amazon - Remove pytest.mark.db_test: airbyte and amazon providers where possible #52017 - still db tests left
• apache
  • beam - Remove pytest.mark.db_test: apache beam #52059 - cleaned
  • druid - no change, stilll db tests left
  • flink - Remove pytest.mark.db_test: apache flink #52062 - cleaned
  • hdfs - no change, stilll db tests left
  • hive - Remove pytest db markers from apache hive provider #52097 - still db tests left
  • iceberg - Remove pytest.mark.db_test: apache iceberg #52063 - cleaned
  • kafka - Remove pytest db marker apache kafka #52098 - cleaned
  • kylin - no change, stilll db tests left
  • livy - Remove pytest db markers from apache livy provider #52099 - still db tests left
  • pig - no change, stilll db tests left
  • pinot - no change, stilll db tests left
  • spark - Removed pytestmark db_test from the spark provider #52081 - stil db tests left
• apprise - - no change, still db tests left
• arangodb - Remove pytest db markers from arangodb provider #52102 - cleaned
• asana - Remove pytest.mark.db_test where possible from asana provider #52093, Refactor asana provider operator tests free from db access #52192 - clean
• atlassian - no change, stilll db tests left
• celery - Remove unused pytestmark = pytest.mark.db_test #52067 - still db tests left
• cloudant - Remove db tests from cloudant provider #52052 - clean
• cncf - Remove db tests from cncf provider #52153 - still some tests left
• common
  • io - no change, stilll db tests left
  • sql - Remove pytest db markers from common sql provider #52105 still db tests left
• databricks - Remove pytest.mark.db_test where possible from databricks provider. #52033 - still db tests left
• datadog - Remove pytest db marker from datadog #52030 - clean
• dbt Removed usage of pytest.mark.db_test from dbt tests where possible #52031 - still db tests left
• dingding - Remove pytest.mark.db_test: dingding #52054 - cleaned
• discord - Removed pytest.mark_db_test from the discord provider #52041 - cleaned
• docker - Removed pytest.mark_db_test from the docker provider completely #52027 -> still db tests left
• edge3 - Not using any connection, all tests are based on DB for the executor. Not cleanable in current approach.
• elasticsearch - Removed pytestmark db_test from the elasticsearch providers tests #52139 still db tests left
• exasol - Remove pytest.mark.db_test: exasol #52057 - cleaned
• fab - Remove db tests from fab provider #52171 - some db tests left
• ftp - Remove pytest.mark.db_test: ftp #52058 - cleaned
• git Remove pytest.mark.db_test: Git #52035 - some db tests left
• github Remove pytest.mark.db_test from github provider #52043 - db tests left.
• google Remove pytest.mark.db_test where possible from google provider #52039 - db tests left.
• http Remove db usage from http provider tests #52227 - db tests left in sensors
• imap Remove pytest.mark.db_test from imap provider #52049 - cleaned
• jdbc Remove pytest.mark.db_test: jdbc #52061 - cleaned
• keycloak - db tests left
• microsoft
  • azure - Remove pytest db markers from microsoft azure provider where possible #52107, db tests still left
  • mssql - no changes, db tests still left
  • winrm Remove pytest db markers from microsoft winrm provider #52104 - cleaned
• mongo Remove pytest.mark.db_test from mongo provider #52047 - cleaned
• mysql - still some tests left
• openlineage - Remove db_tests from openlineage provider #52239 still some tests left
• opensearch - Remove db tests from opensearch provider #52213 - clean
• opsgenie - Remove pytestmark and add db_test marker to relevant tests in opsgenie #52140 - db tests left
• oracle Remove pytest.db_test marker from oracle #52038 - cleaned
• pagerduty - Remove pytestmark from Pagerduty test files #52146 - cleaned
• papermill - still db tests left
• qdrant - still db tests left
• redis - Removed usage of pytest.mark.db_test from the redis provider #52065 - db tests left.
• salesforce
• samba - Remove pytest.mark.db_test from samba provider #52045 - db tests left.
• sftp - 103 db tests left.
• slack. - no changes - db tests left.
• smtp Clean DB Test flag from SMTP where Connection is used #52029 - some DB is leftover because also using dag_maker
• snowflake - no change, still db tests left
• sqlite Remove pytest.db_test marker from SQLite tests #52037 - still some tests left
• standard - more db usage with operators
• tableau Remove pytest.mark_db_test from the tableau provider #52044 - cleaned
• telegram Remove pytest db marker from Telegram #52028 - som db tests left
• weaviate -still has some db tests.
• ydb - still has some db tests
• zendesk Remove db tests from zendesk provider #52022 - clean

We shouild join the efforts and review the providers and attempt to remove db_tests in as many places as possible.

The process of doing it is rather easy:

1. Run breeze shell --backend none --skip-db-tests

2. Remove all pytest.mark.db_test by find replace in the provider:
   @pytest.mark.db_test -> ''
   pytestmark = pytest.mark.db_test -> ''
   There are likely also providers with pytestmark = [pytest.mark.db_test,other_marks] or similar

3. In breeze go to the provider cd providers/PROVIDER

4. run pytest tests -> when some tests fail because they "need dn", restore the marks for those tests

5. After all tests succed, run pytest test -n auto -> that should run all the non-db tests with xdist - in parallel, running them with multiple processes - as many processors you have. There are a few tests that are marked as db_tests to be run sequentially for other reasons than DB - running them with xdist should help to see if there are some of them that still need it

6. Add PR with the changes. Our aim is not yet to remove those db calls that hapen currently (yet) - but to see which of the pytest.mark.db_test marks can be removed easily because they were only used by Connections. But it might turn out that some providers will not have any more pytest.mark.db_test - such as airbyte. So if everything is green and provider does not have any more pytest.mark.db_test - please add it to .pre-commit-config.yml (in the same PR).

Add it here adding | at the end followed by ^providers/PROVIDER/.*\.py$

      - id: check-pytest-mark-db-test-in-providers
        language: pygrep
        name: Check pytest.mark.db_test use in providers
        entry: pytest\.mark\.db_test
        pass_filenames: true
        # Here we should add providers that are already free from the pytest.mark.db_test
        # and we want to keep them clean and only use non-db-tests
        files: >
            (?x)
            ^providers/airbyte/.*\.py$

Committer

• I acknowledge that I am a maintainer/committer of the Apache Airflow project.

[potiuk]

Also -> #52021 added all the providers that have currently no db_tests to the pre-commit.

[gopidesupavan]

cool thanks, i will look into , standard

[dominikhei]

I'll have a look at Docker and dbt

[potiuk]

Example PR for zendesk: #52022

[sjyangkevin]

I will have a look at Weaviate