This release adds the following 7 new async sensors/operators:
| Operator/Sensor Class | Import Path | Example DAG |
|---|---|---|
S3KeySizeSensorAsync |
from astronomer.providers.amazon.aws.sensors.s3 import S3KeySizeSensorAsync |
Example DAG |
S3KeysUnchangedSensorAsync |
from astronomer.providers.amazon.aws.sensors.s3 import S3KeysUnchangedSensorAsync |
Example DAG |
S3PrefixSensorAsync |
from astronomer.providers.amazon.aws.sensors.s3 import S3PrefixSensorAsync |
Example DAG |
GCSObjectsWithPrefixExistenceSensorAsync |
from astronomer.providers.google.cloud.sensors.gcs import GCSObjectsWithPrefixExistenceSensorAsync |
Example DAG |
GCSObjectUpdateSensorAsync |
from astronomer.providers.google.cloud.sensors.gcs import GCSObjectUpdateSensorAsync |
Example DAG |
GCSUploadSessionCompleteSensorAsync |
from astronomer.providers.google.cloud.sensors.gcs import GCSUploadSessionCompleteSensorAsync |
Example DAG |
BigQueryTableExistenceSensorAsync |
from astronomer.providers.google.cloud.sensors.bigquery import BigQueryTableExistenceSensorAsync |
Example DAG |
The dependencies for installing this repo are now split into multiple extras as follows (#113)
| Extra Name | Installation Command | Dependencies |
|---|---|---|
all |
pip install 'astronomer-providers[all]' |
All providers |
amazon |
pip install 'astronomer-providers[amazon]' |
Amazon |
cncf.kubernetes |
pip install 'astronomer-providers[cncf.kubernetes]' |
Kubernetes |
databricks |
pip install 'astronomer-providers[databricks]' |
Databricks |
google |
pip install 'astronomer-providers[google]' |
Google Cloud |
http |
pip install 'astronomer-providers[http]' |
HTTP |
snowflake |
pip install 'astronomer-providers[snowflake]' |
Snowflake |
This will allow users to just install dependencies of a single provider. For example, if a user
wants to just use KubernetesPodOperatorAsync, they should not need to install GCP, AWS or
Snowflake dependencies by running pip install 'astronomer-providers[cncf.kubernetes]'.
- Fixes a bug on the Async Databricks Triggerer failing due to malformed authentication header along with improved exception handling to send the Triggerer errors back to the worker to understand why a particular job execution has failed. (#147)
- Initial release, with the following 18 Async Operators/Sensors:
| Operator/Sensor Class | Import Path | Example DAG |
|---|---|---|
RedshiftSQLOperatorAsync |
from astronomer.providers.amazon.aws.operators.redshift_sql import RedshiftSQLOperatorAsync |
Example DAG |
RedshiftPauseClusterOperatorAsync |
from astronomer.providers.amazon.aws.operators.redshift_cluster import RedshiftPauseClusterOperatorAsync |
Example DAG |
RedshiftResumeClusterOperatorAsync |
from astronomer.providers.amazon.aws.operators.redshift_cluster import RedshiftResumeClusterOperatorAsync |
Example DAG |
RedshiftClusterSensorAsync |
from astronomer.providers.amazon.aws.sensors.redshift_cluster import RedshiftClusterSensorAsync |
Example DAG |
S3KeySensorAsync |
from astronomer.providers.amazon.aws.sensors.s3 import S3KeySensorAsync |
Example DAG |
KubernetesPodOperatorAsync |
from astronomer.providers.cncf.kubernetes.operators.kubernetes_pod import KubernetesPodOperatorAsync |
Example DAG |
ExternalTaskSensorAsync |
from astronomer.providers.core.sensors.external_task import ExternalTaskSensorAsync |
Example DAG |
FileSensorAsync |
from astronomer.providers.core.sensors.filesystem import FileSensorAsync |
Example DAG |
DatabricksRunNowOperatorAsync |
from astronomer.providers.databricks.operators.databricks import DatabricksRunNowOperatorAsync |
Example DAG |
DatabricksSubmitRunOperatorAsync |
from astronomer.providers.databricks.operators.databricks import DatabricksSubmitRunOperatorAsync |
Example DAG |
BigQueryCheckOperatorAsync |
from astronomer.providers.google.cloud.operators.bigquery import BigQueryCheckOperatorAsync |
Example DAG |
BigQueryGetDataOperatorAsync |
from astronomer.providers.google.cloud.operators.bigquery import BigQueryGetDataOperatorAsync |
Example DAG |
BigQueryInsertJobOperatorAsync |
from astronomer.providers.google.cloud.operators.bigquery import BigQueryInsertJobOperatorAsync |
Example DAG |
BigQueryIntervalCheckOperatorAsync |
from astronomer.providers.google.cloud.operators.bigquery import BigQueryIntervalCheckOperatorAsync |
Example DAG |
BigQueryValueCheckOperatorAsync |
from astronomer.providers.google.cloud.operators.bigquery import BigQueryValueCheckOperatorAsync |
Example DAG |
GCSObjectExistenceSensorAsync |
from astronomer.providers.google.cloud.sensors.gcs import GCSObjectExistenceSensorAsync |
Example DAG |
HttpSensorAsync |
from astronomer.providers.http.sensors.http import HttpSensorAsync |
Example DAG |
SnowflakeOperatorAsync |
from astronomer.providers.snowflake.operators.snowflake import SnowflakeOperatorAsync |
Example DAG |