Kubrenets cncf provider 7.5.0 doesn't seem to be compatiable with airflow 2.5.3 #34407
Closed
1 of 2 tasks
Comments
ronald-fenner
added
area:core
kind:bug
|
Checking to see what version was released prior to airflow 2.6 release shows 6.1.0, which works was the last release before the 7.0+ versions. This potentially means all 7.)+ version won't work with airflow <2.6 |
|
Could you explain how your dag Do you import from |
|
Here is the code for the DAG from pathlib import Path
from datetime import datetime, timedelta
import boto3
from airflow import DAG
from airflow.hooks.base import BaseHook
from airflow.providers.cncf.kubernetes.operators.kubernetes_pod import KubernetesPodOperator
from kubernetes.client import models as k8s
from airflow.models import Variable
import boto3
BASE_PATH = Path(__file__).resolve().parent
TEMPLATE_SEARCHPATH = str(BASE_PATH / "sql")
DOCS = (BASE_PATH / "docs.md").read_text()
alert_email = Variable.get('alert_email', deserialize_json=True)
# When ever this DAG is updated you should update the this default to the latest tag.
image_version = Variable.get('lionsense_fb_ad_library_image_version', '14c849867103c5ae856ed5553d54f9c8:')
ecr_image = '748926859119.dkr.ecr.us-west-2.amazonaws.com/airflow-k8s/lionsense-fb-ad-library:' + image_version
service_account = Variable.get('airflow_service_account', '')
default_args = {
"owner": "airflow",
"depends_on_past": False,
"email": alert_email,
"email_on_failure": True,
"email_on_retry": False,
"retries": 1,
"retry_delay": timedelta(minutes=3),
}
lion_sense_conn = BaseHook.get_connection('lion-sense-writer')
with DAG(
dag_id="lion-sense-fb-ad-library",
start_date=datetime(2023, 4, 17),
schedule_interval="0 */4 * * *",
default_args=default_args,
max_active_runs=1,
catchup=False,
template_searchpath=[TEMPLATE_SEARCHPATH, ],
tags=['facebook','lionsense'],
dagrun_timeout=timedelta(hours=1, minutes=30),
doc_md=DOCS
) as dag:
boto_session = boto3.session.Session()
run_ad_crawler = KubernetesPodOperator(
task_id='run-ad-crawler',
service_account_name=service_account,
image=ecr_image,
env_vars=[
k8s.V1EnvVar(name='SENTRY_NODE_DSN', value='{{var.value.ls_fb_sentry_dsn}}'),
k8s.V1EnvVar(name='PROXY_URL', value='{{var.value.ls_fb_proxy_url}}'),
k8s.V1EnvVar(name='AWS_BUCKET', value='lionsense'),
k8s.V1EnvVar(name='AWS_REGION', value=boto_session.region_name),
k8s.V1EnvVar(name='AWS_ENDPOINT', value=''),
k8s.V1EnvVar(name='DB_HOST', value=lion_sense_conn.host),
k8s.V1EnvVar(name='DB_PORT', value=str(lion_sense_conn.port)),
k8s.V1EnvVar(name='DB_USERNAME', value=lion_sense_conn.login),
k8s.V1EnvVar(name='DB_PASSWORD', value=lion_sense_conn.password),
k8s.V1EnvVar(name='DB_DATABASE', value='lionsense'),
k8s.V1EnvVar(name="MAX_VIDEOS_SCROLL", value=Variable.get("fb_ad_library_max_videos_scroll", "100")),
],
do_xcom_push=False,
startup_timeout_seconds=600,
retries=4,
retry_delay=timedelta(seconds=6),
retry_exponential_backoff=True,
max_retry_delay=timedelta(seconds=60)
)` |
|
Already fixed in #34346 and the new version of the provider is being voted and will be released in a few days. |
hussein-awala
removed
the
needs-triage
|
Issue is fixed in apache-airflow-providers-cncf-kubernetes 7.6.0 |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Apache Airflow version
Other Airflow 2 version (please specify below)
What happened
rebuilt our docker image with airflow 2.5.3 and the latest apache-airflow-providers-cncf-kubernetes 7.5.0
After the update one of our dags using the kubernetes executor throws this error
Looking into the issue it seems that the airflow.cli.cli_config was added in 2.6.0+.
What you think should happen instead
It should either be marked as not compatiable with any version less than when the file was added or have a fallback for when it's not available.
How to reproduce
You can use the following package list that we use to install airflow componenets
Operating System
linux
Versions of Apache Airflow Providers
Deployment
Other
Deployment details
Custom kubernetes cluster running airflow
Anything else
No response
Are you willing to submit PR?
Code of Conduct
The text was updated successfully, but these errors were encountered: