-
Notifications
You must be signed in to change notification settings - Fork 16.3k
Closed
Labels
area:providersgood first issuekind:bugThis is a clearly a bugThis is a clearly a bugprovider:edgeEdge Executor / Worker (AIP-69) / edge3Edge Executor / Worker (AIP-69) / edge3
Description
Apache Airflow Provider(s)
edge3
Versions of Apache Airflow Providers
apache-airflow-providers-edge3/1.1.1
Apache Airflow version
3.0.2
Operating System
Docker Image FROM apache/airflow:3.0.2-python3.9 (Debian GNU/Linux 12 (bookworm) )
Deployment
Official Apache Airflow Helm Chart
Deployment details
Using custom image as below
# 3.0.2
FROM apache/airflow:3.0.2-python3.9
# --- core ---
RUN pip install 'apache-airflow[sentry]==3.0.2'
RUN pip install 'apache-airflow[otel]==3.0.2'
RUN pip install 'apache-airflow[standard]==3.0.2'
RUN pip install 'apache-airflow[graphviz]==3.0.2'
RUN pip install 'apache-airflow[async]==3.0.2'
RUN pip install 'apache-airflow[s3fs]==3.0.0'
RUN pip install 'apache-airflow[edge3]==3.0.2'
RUN pip install 'apache-airflow[ldap]==3.0.2'
RUN pip install 'apache-airflow[postgres]==3.0.2'
RUN pip install 'apache-airflow[mysql]==3.0.2'
RUN pip install 'apache-airflow[celery]==3.0.2'
RUN pip install 'apache-airflow[apache-kafka]==3.0.2'
RUN pip install 'apache-airflow[ssh]==3.0.2'
and change settings in values.yaml from
executor: CeleryExecutor
to
executor: CeleryExecutor,airflow.providers.edge3.executors.EdgeExecutor
will reproduce this issue
What happened
[2025-06-27T07:08:13.177+0000] {cli_action_loggers.py:97} DEBUG - Calling callbacks: []
Traceback (most recent call last):
File "/home/airflow/.local/bin/airflow", line 8, in <module>
sys.exit(main())
File "/home/airflow/.local/lib/python3.9/site-packages/airflow/__main__.py", line 55, in main
args.func(args)
File "/home/airflow/.local/lib/python3.9/site-packages/airflow/cli/cli_config.py", line 48, in command
return func(*args, **kwargs)
File "/home/airflow/.local/lib/python3.9/site-packages/airflow/utils/cli.py", line 112, in wrapper
return f(*args, **kwargs)
File "/home/airflow/.local/lib/python3.9/site-packages/airflow/utils/providers_configuration_loader.py", line 55, in wrapped_function
return func(*args, **kwargs)
File "/home/airflow/.local/lib/python3.9/site-packages/airflow/cli/commands/scheduler_command.py", line 52, in scheduler
run_command_with_daemon_option(
File "/home/airflow/.local/lib/python3.9/site-packages/airflow/cli/commands/daemon_utils.py", line 86, in run_command_with_daemon_option
callback()
File "/home/airflow/.local/lib/python3.9/site-packages/airflow/cli/commands/scheduler_command.py", line 55, in <lambda>
callback=lambda: _run_scheduler_job(args),
File "/home/airflow/.local/lib/python3.9/site-packages/airflow/cli/commands/scheduler_command.py", line 43, in _run_scheduler_job
run_job(job=job_runner.job, execute_callable=job_runner._execute)
File "/home/airflow/.local/lib/python3.9/site-packages/airflow/utils/session.py", line 101, in wrapper
return func(*args, session=session, **kwargs)
File "/home/airflow/.local/lib/python3.9/site-packages/airflow/jobs/job.py", line 347, in run_job
return execute_job(job, execute_callable=execute_callable)
File "/home/airflow/.local/lib/python3.9/site-packages/airflow/jobs/job.py", line 376, in execute_job
ret = execute_callable()
File "/home/airflow/.local/lib/python3.9/site-packages/airflow/jobs/scheduler_job_runner.py", line 1005, in _execute
executor.start()
File "/home/airflow/.local/lib/python3.9/site-packages/airflow/utils/session.py", line 101, in wrapper
return func(*args, session=session, **kwargs)
File "/home/airflow/.local/lib/python3.9/site-packages/airflow/providers/edge3/executors/edge_executor.py", line 109, in start
self._check_db_schema(engine)
File "/home/airflow/.local/lib/python3.9/site-packages/airflow/providers/edge3/executors/edge_executor.py", line 87, in _check_db_schema
elif edge_job_command_len and edge_job_command_len != 2048:
UnboundLocalError: local variable 'edge_job_command_len' referenced before assignment
What you think should happen instead
The fix would be quite simple. !52328
I have already tested by replacing this file in my Dockerfile image build with
COPY edge_executor.py /home/airflow/.local/lib/python3.9/site-packages/airflow/providers/edge3/executors/edge_executor.py
How to reproduce
- Use prebuild image with [edge3] install
- Add
airflow.providers.edge3.executors.EdgeExecutorto values.yamlexecutor
For example, I got this error when I
executor: CeleryExecutor
to
executor: CeleryExecutor,airflow.providers.edge3.executors.EdgeExecutor
- Deploy and you will see the failure
Anything else
Actually I still got issue when deploying edge worker and got the following message when I try to start the worker
DEBUG - Calling callbacks: []
405 Client Error: Method Not Allowed for url: <AIRFLOW_HOST>/edge_worker/v1/worker/<WORKER-IP>
I will create another issue to describe it's behavior in Airflow 3.0.2 installation
Are you willing to submit PR?
- Yes I am willing to submit a PR!
Code of Conduct
- I agree to follow this project's Code of Conduct
Metadata
Metadata
Assignees
Labels
area:providersgood first issuekind:bugThis is a clearly a bugThis is a clearly a bugprovider:edgeEdge Executor / Worker (AIP-69) / edge3Edge Executor / Worker (AIP-69) / edge3