AWS MWAA plugin

[cburroughs]

AWS MWAA or "Amazon Managed Workflows for Apache Airflow" https://docs.aws.amazon.com/mwaa/latest/userguide/what-is-mwaa.html is an AWS service for managed service for Apache Airflowhttps://airflow.apache.org/. Airflow is usually described as a pure python orchestrator or "really fancy crontab". You define in python "go do this job at this time" with various configuration knobs for retries and dependency graphs. The jobs themselves have plugable "operators" (docker, k8s, bash) which notable include more pure python. Any python code needs to have the right 3rd party dependencies available in the managed environment.

The deploy procedure is described in:
https://docs.aws.amazon.com/mwaa/latest/userguide/working-dags-dependencies.html
https://docs.aws.amazon.com/mwaa/latest/userguide/best-practices-dependencies.html#best-practices-dependencies-python-wheels

So to deploy into the managed environment one:

• upload your python files to s3
• upload a requirements.txt file for pip
• (optional) upload a zipfile of wheels

A plants plugin would do something like:

• Gather the right python files to upload somewhere in dist
• Create a zipfile of the wheels they depend on. (Properly resolved with a constraints file)
• Generate a dummy requirements.txt file that just points at the wheels.

Slack 🧵 https://pantsbuild.slack.com/archives/C046T6T9U/p1667574027454759

[cburroughs]

Not sure this is super helpful, but some 🧵 quotes for posterity:

"""
Hi! I have not used pants in a while and not sure how to formulate the vocabulary exactly. I'm working with a managed service (AWS Managed Airflow) that expects me to upload some python files (conceptually a tarball) and a requirements.txt file. I think I can gather my python files together by having pants make a tarball or sdist. But I'm not sure how to generate a requirements.txt style file to go along with them. (I know about /pants dependencies but now how to make it output in requirements.txt style nor include it with the tarball/sdist)
"""

"""
So a bit similar to a Lambda, but you bundle a requirements.txt instead of the actual resolved wheels
"""

"""
Yeah its a little quirky: https://docs.aws.amazon.com/mwaa/latest/userguide/working-dags-dependencies.html
Looking at the requirements a little more, a followup is if there a pants equivalent to pip's constraints file (the docs site returns many different results for "constraints"): https://pip.pypa.io/en/stable/user_guide/#constraints-files
With the managed service, we are deploying into a system that looks like this already: https://raw.githubusercontent.com/apache/airflow/constraints-2.2.2/constraints-3.7.txt
"""

"""
CI today looks something like:
- aws s3 cp modules/ s3://$S3_BUCKET/dags/modules/ --recursive
- aws s3 cp custom_dags/ s3://$S3_BUCKET/dags/custom_dags/ --recursive

and that dag_def.py references code in requirements.txt?
Correct, the requirements file has pandas==1.2.4 and then in the example dag_def.py code can import pandas

And the requirements.txt goes to s3://$S3_BUCKET/requirements.txt it looks like
"""

"""
Oh, interesting, looks like you can also give it the exact wheel files directly: https://docs.aws.amazon.com/mwaa/latest/userguide/best-practices-dependencies.html#best-practices-dependencies-python-wheels
docs.aws.amazon.comdocs.aws.amazon.com
Managing Python dependencies in requirements.txt - Amazon Managed Workflows for Apache Airflow
This page describes the best practices we recommend to install and manage Python dependencies in a requirements.txt file for an Amazon Managed Workflows for Apache Airflow (MWAA) environment.

Instead of it doing a pip resolve for you every time
"""

"""
I think if I'm reading that section of the docs right it can install from a wheel in a zip file, but you still have to have a requirements.txt file to tell it what to install. eg in the example they still have numpy listed. It doesn't install * from the zipfile. That would be reasonable to glue together with a script.
"""

"""
Pants has all the ingredients, just not the exact recipe for putting them together
❤️
"""

"""
a layout="zipapp" (the default) or layout="loose" PEX contains the wheels in their installed, loose-file state

a layout="packed" PEX contains the installed-state wheels zipped back up as .whl files, which you may be able to use for this, even though they are not identical to the pypi wheels (e.g., they contain an INSTALLED file)
"""