AIP-72: Handling task instance state exceptions from task sdk and execution API

[amoghrajesh]

Body

Similar to #44137, there are various state exceptions raised while running ti.task.execute. This issue tracks handling them gracefully.

• Deferral Exception: TaskDeferred (PR: AIP 72: Handling "deferrable" tasks in execution_api and task SDK #44241 @amoghrajesh)
• Skip Exception: AirflowSkipException (PR AIP-72: Handling skipped tasks in task_sdk #44786 @amoghrajesh)
• Reschedule exception: AirflowRescheduleException (PR AIP-72: Handling "up_for_reschedule" task instance states #44907 @amoghrajesh)
• Fail Exception: AirflowFailException (PR AIP-72: Handling failed TI state for AirflowFailException & AirflowSensorTimeout #44954 @amoghrajesh)
• Sensor timeout exception: AirflowSensorTimeout (PR AIP-72: Handling failed TI state for AirflowFailException & AirflowSensorTimeout #44954 @amoghrajesh)
• Task timeout or task terminated exception (or any general AirflowException): AirflowTaskTimeout, AirflowException, AirflowTaskTerminated
• System Exit: SystemExit
• Base Exception: BaseException

Committer

• I acknowledge that I am a maintainer/committer of the Apache Airflow project.

[ashb]

System exit should be handled already I thought?

[amoghrajesh]

I do not think we do it:

    except SystemExit:
        ...
    except BaseException:
        # TODO: Handle TI handle failure
        raise