Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Added output_processor parameter to BashProcessor #40843

Merged
merged 34 commits into from
Aug 5, 2024

Conversation

dabla
Copy link
Contributor

@dabla dabla commented Jul 17, 2024

Added result_processor parameter to BashProcessor which allows you to specify a lambda function that processes the output of the bash script before it is pushed as an XCom. This feature is particularly useful for manipulating the script’s result directly within the BashOperator, without the need for additional operators or tasks.

If for example you have following bash script which uses the linux jq command to filter the required elements from a JSON-file, it is handy to have the XCom directly loaded as a JSON object in python instead of a string:

filter_today_changes = BashOperator(
        task_id="filter_today_changes",
        bash_command="""
            jq -c '.[] | select(.lastModified > "{{ data_interval_start | ts_zulu }}" or .created > "{{ data_interval_start | ts_zulu }}")' \\
           example.json
        """,
        output_processor=lambda output: json.loads(output),
)

^ Add meaningful description above
Read the Pull Request Guidelines for more information.
In case of fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in a newsfragment file, named {pr_number}.significant.rst or {issue_number}.significant.rst, in newsfragments.

@dabla dabla requested a review from potiuk as a code owner July 17, 2024 16:35
@boring-cyborg boring-cyborg bot added area:core-operators Operators, Sensors and hooks within Core Airflow kind:documentation labels Jul 17, 2024
Copy link
Contributor

@shahar1 shahar1 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Seems like a useful feature, got some few comments :)

airflow/operators/bash.py Outdated Show resolved Hide resolved
airflow/operators/bash.py Outdated Show resolved Hide resolved
docs/apache-airflow/howto/operator/bash.rst Outdated Show resolved Hide resolved
airflow/operators/bash.py Outdated Show resolved Hide resolved
dabla and others added 2 commits July 29, 2024 10:10
Co-authored-by: Shahar Epstein <60007259+shahar1@users.noreply.github.com>
Co-authored-by: Shahar Epstein <60007259+shahar1@users.noreply.github.com>
@dabla dabla changed the title Added result_processor parameter to BashProcessor Added output_processor parameter to BashProcessor Jul 29, 2024
@dabla dabla requested a review from shahar1 August 1, 2024 14:09
@shahar1 shahar1 changed the title Added output_processor parameter to BashProcessor Added result_processor parameter to BashProcessor Aug 2, 2024
@shahar1 shahar1 changed the title Added result_processor parameter to BashProcessor Added output_processor parameter to BashProcessor Aug 2, 2024
Copy link
Contributor

@shahar1 shahar1 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Almost there

airflow/operators/bash.py Outdated Show resolved Hide resolved
docs/apache-airflow/howto/operator/bash.rst Outdated Show resolved Hide resolved
Co-authored-by: Shahar Epstein <60007259+shahar1@users.noreply.github.com>
@potiuk potiuk merged commit cd69f8e into apache:main Aug 5, 2024
49 checks passed
@eladkal eladkal added this to the Airflow 2.10.0 milestone Aug 5, 2024
@eladkal eladkal added the type:improvement Changelog: Improvements label Aug 5, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
area:core-operators Operators, Sensors and hooks within Core Airflow kind:documentation type:improvement Changelog: Improvements
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants