airflow no writing logs to Elasticsearch

[kmaksimovv]

Apache Airflow version

Other Airflow 2 version (please specify below)

If "Other Airflow 2 version" selected, which one?

No response

What happened?

Use apache-airflow-providers-elasticsearch 5.4.1
Airflow 2.9.2
Writing logs to Elasticsearch not worked, no data in Elasticsearch
Config:
elasticsearch:
host: "https://user:pass@elasticsearch_host:9200"
target_index: 'airflow-logs'
log_id_template: '{dag_id}{task_id}{execution_date}_{try_number}'
index_patterns: 'airflow-*'
end_of_log_mark: "end_of_log"
write_stdout: "False"
write_to_es: 'True'
json_format: 'True'
elasticsearch_configs:
use_ssl: 'True'
verify_certs: 'True'
ca_certs: /es_certs/ca.pem

What you think should happen instead?

No response

How to reproduce

Use apache-airflow-providers-elasticsearch 5.4.1
Airflow 2.9.2
Writing logs to Elasticsearch not worked, no data in Elasticsearch
Config:
elasticsearch:
host: "https://user:pass@elasticsearch_host:9200"
target_index: 'airflow-logs'
log_id_template: '{dag_id}{task_id}{execution_date}_{try_number}'
index_patterns: 'airflow-*'
end_of_log_mark: "end_of_log"
write_stdout: "False"
write_to_es: 'True'
json_format: 'True'
elasticsearch_configs:
use_ssl: 'True'
verify_certs: 'True'
ca_certs: /es_certs/ca.pem

Operating System

debian

Versions of Apache Airflow Providers

No response

Deployment

Official Apache Airflow Helm Chart

Deployment details

No response

Anything else?

No response

Are you willing to submit PR?

• Yes I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct

[boring-cyborg]

Thanks for opening your first issue here! Be sure to follow the issue template! If you are willing to raise PR to address this issue please do so, no need to wait for approval.

[AryanK1511]

I can look into this!

[kmaksimovv]

It would be nice if it worked

[kmaksimovv]

@AryanK1511 When can you watch?

[kmaksimovv]

I tried updating the version airflow, also apache-airflow-providers-elasticsearch to last version(6.3.0), but it still doesn't work
the same problem here #50349 (comment)

[AryanK1511]

@AryanK1511 When can you watch?

Im sorry I don't think I understand your question 😅, could you elaborate a bit more please?

[AryanK1511]

I tried updating the version airflow, also apache-airflow-providers-elasticsearch to last version(6.3.0), but it still doesn't work the same problem here #50349 (comment)

Yeah lemme try to recreate this problem first and then I will post my findings here.

[AryanK1511]

Update: I was able to recreate the issue, will start working on a fix now

[kmaksimovv]

This is excellent!