-
Notifications
You must be signed in to change notification settings - Fork 0
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
feat: pis stage for the platform orchestration dag #14
Conversation
The rationale behind remove interrogate and pydoclint is that we are already checking docstrings with Ruff by using the pydocstyle rules. Interrogate and Pydoclint are extremely strict, I copypasted a docstring straight from google code and they complain about it. Also, Let me know if you think this is too big a change. |
I cannot reproduce the check error on the |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Some minor comments, otherwise well done!
Co-authored-by: Szymon Szyszkowski <ss60@mib117351s.internal.sanger.ac.uk>
@javfg approved. Nice work! |
@javfg can you drop the |
This PR adds the first stage for the Platform DAG: The one that runs PIS.
In the end it is using Compute Engine machines that are spawned with COOS. They are very fast to spin up (~10 seconds), but quite limited on what they can do.
I've implemented an Airflow Sensor/Operator that is in charge of:
Most of the logic is in the gce.py file, which contains that operator.
I also:
airflow-triggerer
service to the docker-compose, as we need it for sensors.ignore_init
setting to interrogate so we don't have to add docstrings to__init__
methods (it's annoying and rarely useful).I'm opening the PR already although work is well underway for the ETL stage, as this way we can split the code into more manageable chunks for review. Shorter reviews find more things to fix!
I'll leave it as pending for now, until the merge of #13, so I can rebase onto it and fix any problems that arise.