DAG disappears in airflow 3.1.3

[narenjngr]

Apache Airflow version

3.1.3

If "Other Airflow 2/3 version" selected, which one?

3.1.3

What happened?

What happened?
After upgrade to airflow 3.1.3, system started experiencing random DAG disappearance.
The config for dag processor has this setup:

       - name: AIRFLOW__DAG_PROCESSOR__BUNDLE_REFRESH_CHECK_INTERVAL
          value: '120'
        - name: AIRFLOW__DAG_PROCESSOR__REFRESH_INTERVAL
          value: '120'
        - name: AIRFLOW__CORE__STORE_SERIALIZED_DAGS
          value: 'True'
        - name: AIRFLOW__CORE__MIN_SERIALIZED_DAG_UPDATE_INTERVAL
          value: '60'
        - name: AIRFLOW__DAG_PROCESSOR__MIN_FILE_PROCESS_INTERVAL
          value: '120'
        - name: AIRFLOW__DAG_PROCESSOR__STALE_DAG_THRESHOLD
          value: '86400'

Files found by aiflow dag -folder also reduce over the period of time.
Here is my dag processor values.yaml file section-

dagProcessor:
  enabled: true
  replicas: 1
  revisionHistoryLimit: ~
  command: ~
  args: ["bash", "-c", "exec airflow dag-processor"]


  strategy:
    rollingUpdate:
      maxSurge: "100%"
      maxUnavailable: "50%"

  livenessProbe:
    initialDelaySeconds: 120
    timeoutSeconds: 60
    failureThreshold: 10
    periodSeconds: 60
    command: ~

  serviceAccount:
    automountServiceAccountToken: true
    create: false
    name: "airflow"

    annotations: {}

  securityContext: {}

  securityContexts:
    pod: {}
    container: {}

  containerLifecycleHooks: {}

  resources:
    limits:
      cpu: 1
      memory: 2Gi
    requests:
      cpu: 500m
      memory: 500Mi

  terminationGracePeriodSeconds: 60

  safeToEvict: true

  extraContainers: []
  extraInitContainers: []
  extraVolumes: []
  extraVolumeMounts: []

  # Select certain nodes for airflow dag processor pods.
  nodeSelector: {}
  affinity: {}
  tolerations: []
  topologySpreadConstraints: []

  priorityClassName: ~

  annotations: {}

  podAnnotations: {}

  logGroomerSidecar:
    enabled: true
    command: ~
    args: ["bash", "/clean-logs"]
    retentionDays: 15
    frequencyMinutes: 15
    resources: {}
    securityContexts:
      container: {}

    env: []

  waitForMigrations:
    enabled: true
    env: []
    securityContexts:
      container: {}

  env: 
       - name: AIRFLOW__DAG_PROCESSOR__BUNDLE_REFRESH_CHECK_INTERVAL
          value: '120'
        - name: AIRFLOW__DAG_PROCESSOR__REFRESH_INTERVAL
          value: '120'
        - name: AIRFLOW__CORE__STORE_SERIALIZED_DAGS
          value: 'True'
        - name: AIRFLOW__CORE__MIN_SERIALIZED_DAG_UPDATE_INTERVAL
          value: '60'
        - name: AIRFLOW__DAG_PROCESSOR__MIN_FILE_PROCESS_INTERVAL
          value: '120'
        - name: AIRFLOW__DAG_PROCESSOR__STALE_DAG_THRESHOLD
          value: '86400'

One important point I observed is that whatever DAG is disappeared from UI if I check its detail on any of the airflow pod using command airflow dags details DAG_ID then I can see IsStale = true
Why DAGs are being marked as stale even after high value 86400 of AIRFLOW__DAG_PROCESSOR__STALE_DAG_THRESHOLD

What you think should happen instead?

DAGs shouldn't disappear unless I delete these DAGs from the mounted volume.

How to reproduce

Use same dag processor config and observer for few days. Intermittent issue.

Operating System

Linux

Versions of Apache Airflow Providers

eval_type_backport==0.2.2
apache-airflow-providers-databricks==6.4.0
apache-airflow-providers-mongo==4.2.1
apache-airflow-providers-git==0.0.2
apache-airflow-providers-standard==1.2.0
soda-core-spark-df==3.5.5
soda-core-spark[databricks]==3.5.5
soda-core-scientific==3.5.5
pymongo~=4.0.0
typing_extensions==4.13.2
paramiko<4
PyMuPDF~=1.26.5

Deployment

Official Apache Airflow Helm Chart

Deployment details

Official Apache Airflow Helm Chart

Anything else?

No response

Are you willing to submit PR?

• Yes I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct

[ephraimbuddy]

how did you configure your bundle?

[narenjngr]

Hi, We have mounted volume for dags at path /opt/airflow/dags and bundle is configured from this path We also have a container dag_sync running on component dag processor which read a blob storage every 120 seconds and overwrite all these dag files to the mounted path /opt/airflow/dags

…

Message ID: ***@***.***>

[ephraimbuddy]

Can you look at this documentation and let me know if you have such setup: https://airflow.apache.org/docs/apache-airflow/stable/administration-and-deployment/dag-bundles.html

[narenjngr]

We have not defined any dag bundle configuration specifically so by default local dag bundle should be picked. we have defined mount path as /opt/airflow/dags in dags component so dags are being picked from this path for the local dag bundle

[gvergnolle]

@narenjngr I was experiencing this when I had a datetime.now hidden in my DAG files. That was creating many DAG versions that would occasionally fail and make DAGs disappear. Have you seen abnormally high DAG versions?

[narenjngr]

Hi @gvergnolle , I have verified and there is no use of datetime.now in the dags, also dag versions are also not getting increased because of any other reason.

Files are getting reduced in the dag bundle over the period and disappeared dags are marked as stale.

[ephraimbuddy]

You can use GCSDagbundle or S3DagBundle because it seems your setup is responsible for why it's disappearing

[Arunodoy18]

From what I can see, the DAGs are not being deleted from the mounted volume but are instead being marked as IsStale = true, which causes them to disappear from the UI. Given the high STALE_DAG_THRESHOLD value, this suggests the issue is more likely related to the DAG processor refresh / reprocessing behavior or serialized DAG update timing rather than actual DAG removal.

[Arunodoy18]

I’ve been able to reproduce this behavior with the provided dag-processor configuration and am investigating why DAGs are marked as stale despite the high STALE_DAG_THRESHOLD. I’ll follow up with findings and a PR once the root cause is confirmed.

[Nataneljpwd]

we are having this issue as well on airflow 2.11, dags are deleted from the serialized dag table by the dag processor, we are also investigating possible reasons, we have noticed it happens more when the dag processor faces memory pressure or sudden memory usage peaks, though we do not get stale dag, rather as I said, dags are deleted from the serialized dag table.

[eladkal]

I’ve been able to reproduce this behavior with the provided dag-processor configuration and am investigating why DAGs are marked as stale despite the high STALE_DAG_THRESHOLD. I’ll follow up with findings and a PR once the root cause is confirmed.

can you share the reproduce steps?