Use the regular OTel environment variables for configuring traces and metrics #56150

xBis7 · 2025-09-26T16:57:16Z

OTel environment variables

The OpenTelemetry SDK for metrics and traces, can be configured with two ways

Adding the values directly in the code via method parameters
Using the standard OTel environment variables

There are multiple OTel env variables which can be automatically picked up by the SDK if exported.

https://opentelemetry.io/docs/specs/otel/configuration/sdk-environment-variables/

https://opentelemetry.io/docs/languages/sdk-configuration/general/

https://opentelemetry.io/docs/languages/sdk-configuration/otlp-exporter/

Currently, there is no need to support all of them.

Airflow is providing the code configuration based on

Values that have been added to custom Airflow config properties
Hard-coded values

How it works

On the OTel side, the environment variables are only checked if there is no code configuration. If a value is provided in the code, then that will take priority and the environment will be ignored.

Airflow has its own version of the OTel configs. It reads the Airflow-OTel properties and based on the values, it uses code to configure the OTel SDK.

If we were to export the regular environment variables and then just call the SDK methods without any parameters, the env values would be automatically used.

The OTel priorities are

Code configuration
OTel checks env vars

The current Airflow priorities are

Airflow-OTel configs
Code configuration
SDK checks the env vars

For Airflow, there is always code configuration and therefore as mentioned above the OTel env vars are ignored.

Changes

This patch removes all OTel related configs from the Airflow configuration except the flags that enable OTel metrics and traces.

The values that we would get from the Airflow config, are now accessed through the following OTel environment variables.

- Common
OTEL_EXPORTER_OTLP_PROTOCOL
OTEL_EXPORTER_OTLP_ENDPOINT
OTEL_SERVICE_NAME
OTEL_RESOURCE_ATTRIBUTES
OTEL_EXPORTER_OTLP_HEADERS

- Traces specific
OTEL_TRACES_EXPORTER
OTEL_EXPORTER_OTLP_TRACES_ENDPOINT

- Metrics specific
OTEL_METRICS_EXPORTER
OTEL_EXPORTER_OTLP_METRICS_ENDPOINT
OTEL_METRIC_EXPORT_INTERVAL

For info regarding the values of each property, check the updated docs.

OTEL_TRACES_EXPORTER replaces otel_debugging_on if the value is console

  OTEL_EXPORTER_OTLP_ENDPOINT (common)
  OTEL_EXPORTER_OTLP_TRACES_ENDPOINT (traces)
  OTEL_EXPORTER_OTLP_METRICS_ENDPOINT (metrics)

replaces

otel_host
otel_port
otel_ssl_active

OTEL_METRIC_EXPORT_INTERVAL replaces otel_interval_milliseconds

Why

When you have a cluster where multiple applications are running and all of them are using OTel, then it’s common to configure the regular OTel environment variables and export them. That way, you won’t have to configure each project separately to work with your shared otel-collector service.

^ Add meaningful description above
Read the Pull Request Guidelines for more information.
In case of fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in a newsfragment file, named {pr_number}.significant.rst or {issue_number}.significant.rst, in airflow-core/newsfragments.

xBis7 · 2025-09-26T16:58:02Z

@potiuk @ferruzzi Can you please take a look at this PR?

jason810496

Nice! Thanks for the PR!

Would it be better to retain the compatibility layer for Airflow-Otel environment variables and configuration, while also raising a deprecation warning? This PR introduces a breaking change for users currently relying on Otel.

jason810496 · 2025-09-27T09:46:26Z

airflow-core/src/airflow/utils/otel_config.py

+log = logging.getLogger(__name__)
+
+
+def _parse_kv_str_to_dict(str_var: str) -> dict[str, str]:


Is it possible to reuse the util here ?

airflow/airflow-core/src/airflow/traces/utils.py

Lines 94 to 95 in 681b9e9

def parse_tracestate(tracestate_str: str | None = None) -> dict:

"""Parse tracestate string: rojo=00f067aa0ba902b7,congo=t61rcWkgMzE."""

TBH, I would like to remove that file (traces/utils.py) entirely. The code that was using it, isn't called anymore.

Generating and assigning a trace ID and a span ID are internal operations of the OTel SDK.

The traceparent and the tracestate belong to the span context which is handled by the propagators API.

airflow/airflow-core/src/airflow/traces/otel_tracer.py

Lines 286 to 294 in aa398f3

def inject(self) -> dict:

"""Inject the current span context into a carrier and return it."""

carrier: dict[str, str] = {}

TraceContextTextMapPropagator().inject(carrier)

return carrier

def extract(self, carrier: dict) -> Context:

"""Extract the span context from a provided carrier."""

return TraceContextTextMapPropagator().extract(carrier)

The inject and extract methods are taking care of the context processing.

These methods are a hacky way of handling spans, just like the Airflow config values that are getting hard-coded into the callers to the OTel SDK.

I'm actually shocked we don't already have a string-to-dict helper in airflow/utils. I think we used to have one for env_var parsing, but can't find it.

airflow-core/src/airflow/utils/otel_config.py

airflow-core/tests/unit/core/test_otel_tracer.py

xBis7 · 2025-09-27T13:46:30Z

@jason810496 Thank you for the review!

Would it be better to retain the compatibility layer for Airflow-Otel environment variables and configuration, while also raising a deprecation warning?

Sure, I'll do that.

The code of this patch does a validation on the provided configs. I think priorities should be

Airflow configs
OTel env variables

If the Airflow configs are empty, then it will load the OTel env vars and do the validation. In the future, we will just remove the 1st step and it will go straight to loading the env vars.

The Airflow configs have a fallback value. As a result, they will never be empty and the env vars won't be checked. I think the default should be -. That way we can check if the user added a value or not.

What do you think?

xBis7 · 2025-09-27T13:47:53Z

https://github.com/apache/airflow/actions/runs/18044157889/job/51351358057?pr=56150#step:6:3347

It's complaining about the 2nd upper case letter in a row but OTel is the correct spelling.

https://opentelemetry.io/docs/

I'll update it.

jason810496

The code of this patch does a validation on the provided configs. I think priorities should be

Airflow configs

OTel env variables

If the Airflow configs are empty, then it will load the OTel env vars and do the validation. In the future, we will just remove the 1st step and it will go straight to loading the env vars.

Yes, that is exactly what I'm thinking about. Thanks!

airflow-core/docs/administration-and-deployment/logging-monitoring/metrics.rst

airflow-core/src/airflow/metrics/otel_logger.py

airflow-core/src/airflow/traces/otel_tracer.py

airflow-core/src/airflow/utils/otel_config.py

xBis7 · 2025-10-02T08:47:08Z

@jason810496 Is there a standard way for marking a config as deprecated? I have found what to do with configs that will change in the future (moved or renamed) but I'm not sure about the ones that will be removed.

ferruzzi

I think all of my concerns and nitpicks have been addressed. Thanks!

jason810496

It seem this PR is depends on #56187 .
I will review again after #56187 get merged.

LGTM overall, only small nits for the docs so far. Thanks!

airflow-core/docs/administration-and-deployment/logging-monitoring/metrics.rst

airflow-core/docs/administration-and-deployment/logging-monitoring/traces.rst

potiuk · 2025-10-19T18:56:54Z

conflicts :(

xBis7 · 2025-10-20T14:12:28Z

@potiuk This PR isn't ready because it depends on #56187. Once that gets merged then this one will be adjusted accordingly.

I like the simplicity of #56634. There is the question whether to follow the same approach or not. The difference between #56634 and the current patch is that this PR is doing a validation on the configs.

potiuk · 2025-10-21T10:21:54Z

God question. The simpler the better and I think we should not really validate something that hotel validates (or should)

xBis7 · 2025-10-21T15:38:28Z

The simpler the better and I think we should not really validate something that hotel validates (or should)

This is true, OTel doesn't validate the configs, it just uses the values and then throws errors.

I added the validation because the SDK initialization will fail and I wanted to let users know in advance what's wrong.

jason810496 · 2025-10-21T16:21:01Z

Is there a standard way for marking a config as deprecated? I have found what to do with configs that will change in the future (moved or renamed) but I'm not sure about the ones that will be removed.

Sorry, I oversight this message.
This is the place where we deprecate the config section-option pair, but it seems only work for (new section, new option) -> (old section, old option, since_version) case.

airflow/airflow-core/src/airflow/configuration.py

Lines 345 to 349 in f3ad20a

    
           # A mapping of (new section, new option) -> (old section, old option, since_version). 
        
           # When reading new option, the old option will be checked to see if it exists. If it does a 
        
           # DeprecationWarning will be issued and the old option will be used instead 
        
           deprecated_options: dict[tuple[str, str], tuple[str, str, str]] = { 
        
               ("dag_processor", "refresh_interval"): ("scheduler", "dag_dir_list_interval", "3.0"),

I'm not sure how should we deal with removing the config pair as well.
@potiuk may we ask for what is the correct and standard way to removing the config pair?

github-actions · 2026-01-16T00:20:00Z

This pull request has been automatically marked as stale because it has not had recent activity. It will be closed in 5 days if no further activity occurs. Thank you for your contributions.

jason810496

It seem this PR is depends on #56187 .
I will review again after #56187 get merged.

Hi @xBis7
Would you still like to continue with this PR? #56187 was merged in Dec 2025. Thanks!

Is there a standard way for marking a config as deprecated? I have found what to do with configs that will change in the future (moved or renamed) but I'm not sure about the ones that will be removed.

I just traced the relative context again and found that we can leverage version_deprecated and deprecation_reason for airflow-core/src/airflow/config_templates/config.yml:

airflow/airflow-core/src/airflow/config_templates/config.yml.schema.json

Lines 51 to 64 in 5dcdccc

    
           "version_deprecated": { 
        
               "type": [ 
        
                   "string", 
        
                   "null" 
        
               ], 
        
               "description": "When set to a version string, this option is deprecated as of this version, and will be removed in the future." 
        
           }, 
        
           "deprecation_reason": { 
        
               "type": [ 
        
                   "string", 
        
                   "null" 
        
               ], 
        
               "description": "The reason why this option is deprecated." 
        
           },

Here is a good PR example of how to deprecate config properly:
https://github.com/apache/airflow/pull/33136/files

xBis7 · 2026-01-16T07:29:55Z

Hi @jason810496, thank you for your continuous help on this. This PR was left behind due to other work priorities. I resumed looking into it a couple days ago.

The branch has diverged so much from main that it's painful to try to rebase it and resolve the conflicts. So, I started a new clean branch and once the changes are done, I'm going to push directly to this one, essentially replacing the branches.

I'm going to remove the validations on the environment variables. We will check the env vars first and if the values are none, then we will check the airflow config.

I had forgot about adding the deprecation warning. Thanks for reminding me!

jason810496 · 2026-01-16T07:33:16Z

The branch has diverged so much from main that it's painful to try to rebase it and resolve the conflicts. So, I started a new clean branch and once the changes are done, I'm going to push directly to this one, essentially replacing the branches.

Same though for me as well, rebasing on large conflicts is too painful.
Sure! Feel free to tag me on the new PR, thanks for the quick reply.

xBis7 requested review from ashb, gopidesupavan and potiuk as code owners September 26, 2025 16:57

boring-cyborg bot added area:ConfigTemplates area:dev-tools backport-to-v3-1-test Mark PR with this label to backport to v3-1-test branch kind:documentation labels Sep 26, 2025

jason810496 self-requested a review September 27, 2025 09:40

jason810496 reviewed Sep 27, 2025

View reviewed changes

jason810496 reviewed Sep 28, 2025

View reviewed changes

xBis7 mentioned this pull request Sep 28, 2025

Move the traces and metrics code under a common observability package #56187

Merged

ferruzzi reviewed Sep 30, 2025

View reviewed changes

ferruzzi reviewed Oct 1, 2025

View reviewed changes

airflow-core/src/airflow/utils/otel_config.py Outdated Show resolved Hide resolved

ferruzzi reviewed Oct 2, 2025

View reviewed changes

airflow-core/src/airflow/utils/otel_config.py Outdated Show resolved Hide resolved

ferruzzi approved these changes Oct 2, 2025

View reviewed changes

jason810496 reviewed Oct 3, 2025

View reviewed changes

airflow-core/docs/administration-and-deployment/logging-monitoring/metrics.rst Outdated Show resolved Hide resolved

airflow-core/docs/administration-and-deployment/logging-monitoring/traces.rst Outdated Show resolved Hide resolved

xBis7 force-pushed the support-otel-env-vars branch from 2910670 to f72c75d Compare October 16, 2025 08:57

xBis7 requested review from amoghrajesh, bugraoz93, eladkal, jscheffl and kaxil as code owners October 20, 2025 10:51

xBis7 force-pushed the support-otel-env-vars branch from 356657b to b198691 Compare November 10, 2025 12:11

xBis7 added 10 commits November 20, 2025 17:12

support regular OTel env variables

7309e03

fix spelling error while building docs

a7dfa95

cleanup, address review comments

5c79b80

restore the airflow-otel configs

e60470b

remove the validate parameter

9036101

test config priorities

2950892

cleanup, address review comments

a78fb88

cleanup otel config validation

c90cdda

fix config check for the protocol

77e2d29

update docs

4ddb14b

potiuk force-pushed the support-otel-env-vars branch from b198691 to 4ddb14b Compare November 20, 2025 16:12

github-actions bot added the stale Stale PRs per the .github/workflows/stale.yml policy file label Jan 16, 2026

jason810496 reviewed Jan 16, 2026

View reviewed changes

jason810496 removed the stale Stale PRs per the .github/workflows/stale.yml policy file label Jan 16, 2026

		log = logging.getLogger(__name__)


		def _parse_kv_str_to_dict(str_var: str) -> dict[str, str]:

	def parse_tracestate(tracestate_str: str \| None = None) -> dict:
	"""Parse tracestate string: rojo=00f067aa0ba902b7,congo=t61rcWkgMzE."""

	def inject(self) -> dict:
	"""Inject the current span context into a carrier and return it."""
	carrier: dict[str, str] = {}
	TraceContextTextMapPropagator().inject(carrier)
	return carrier

	def extract(self, carrier: dict) -> Context:
	"""Extract the span context from a provided carrier."""
	return TraceContextTextMapPropagator().extract(carrier)

	"version_deprecated": {
	"type": [
	"string",
	"null"
	],
	"description": "When set to a version string, this option is deprecated as of this version, and will be removed in the future."
	},
	"deprecation_reason": {
	"type": [
	"string",
	"null"
	],
	"description": "The reason why this option is deprecated."
	},

Use the regular OTel environment variables for configuring traces and metrics #56150

Are you sure you want to change the base?

Use the regular OTel environment variables for configuring traces and metrics #56150

Uh oh!

Conversation

xBis7 commented Sep 26, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

OTel environment variables

How it works

Changes

Why

Uh oh!

xBis7 commented Sep 26, 2025

Uh oh!

jason810496 left a comment

Choose a reason for hiding this comment

Uh oh!

jason810496 Sep 27, 2025

Choose a reason for hiding this comment

Uh oh!

xBis7 Sep 27, 2025

Choose a reason for hiding this comment

Uh oh!

ferruzzi Sep 30, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

xBis7 commented Sep 27, 2025

Uh oh!

xBis7 commented Sep 27, 2025

Uh oh!

jason810496 left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

xBis7 commented Oct 2, 2025

Uh oh!

ferruzzi left a comment

Choose a reason for hiding this comment

Uh oh!

jason810496 left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

potiuk commented Oct 19, 2025

Uh oh!

xBis7 commented Oct 20, 2025

Uh oh!

potiuk commented Oct 21, 2025

Uh oh!

xBis7 commented Oct 21, 2025

Uh oh!

jason810496 commented Oct 21, 2025

Uh oh!

github-actions bot commented Jan 16, 2026

Uh oh!

jason810496 left a comment

Choose a reason for hiding this comment

Uh oh!

xBis7 commented Jan 16, 2026

Uh oh!

jason810496 commented Jan 16, 2026

Uh oh!

Reviewers

xBis7 commented Sep 26, 2025 •

edited

Loading