AWS Lambda Executor

[o-nikolas]

Overview

Introducing the new AWS Lambda Executor!

As with our previous executors this is an initial release with most functionality in place. However, there are surely future upgrades and iterations to come, so stay tuned!

This changeset includes:

• The Lambda executor code itself
• Tests for the executor
• Docs for the executor

Testing

The unit tests cover 90+% line coverage (ignore the hook coverage, the results were filtered on lambda and that hook is prexisting code not delivered in this PR) and the executor was UAT tested manually in many scenarios.

Review Notes

This is a large PR, which is not entirely ideal. However, it can be difficult to release portions of a component like this and I've scoped all the code here to the minimum required for folks to begin using this executor. There are other refactorings that were left out and will be contributed at a later time to minimize the confusion.

There are clear chunks this can be reviewed in (docs, provider.yaml, tests, and core logic) to split it up.

Ultimately, (almost) all the code is scoped to the Amazon provider package and it is a net new component, so there is a very limited blast radius. Little to no existing user workflows or code should be affected.

---

^ Add meaningful description above
Read the Pull Request Guidelines for more information.
In case of fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in a newsfragment file, named {pr_number}.significant.rst or {issue_number}.significant.rst, in airflow-core/newsfragments.

[gopidesupavan]

@o-nikolas LGTM, thanks for this.

The lambda function has default retries (2), how about this, do you think in this case retries required?

[o-nikolas]

@o-nikolas LGTM, thanks for this.

The lambda function has default retries (2), how about this, do you think in this case retries required?

Our retries include attempts to even submit the task to lambda (network issues, access token expiration, api failures), so I think it's still worth keeping. But if others agree we should drop it, I'm not completely opposed to that!

[gopidesupavan]

@o-nikolas LGTM, thanks for this.
The lambda function has default retries (2), how about this, do you think in this case retries required?

Our retries include attempts to even submit the task to lambda (network issues, access token expiration, api failures), so I think it's still worth keeping. But if others agree we should drop it, I'm not completely opposed to that!

Yeah fine even i am not opposed to it :)

[o-nikolas]

@o-nikolas LGTM, thanks for this.
The lambda function has default retries (2), how about this, do you think in this case retries required?

Our retries include attempts to even submit the task to lambda (network issues, access token expiration, api failures), so I think it's still worth keeping. But if others agree we should drop it, I'm not completely opposed to that!

Yeah fine even i am not opposed to it :)

Sounds good, thanks for the review!

[eladkal]

This path is not according to the convention. We shouldn't have aws_ prefix.

[o-nikolas]

lambda is a reserved keyword in Python, so many of the places I needed to prefix it with something. I prefixed in all cases for consistency. It is unfortunate though

[eladkal]

got it! thanks