Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: platform pis stage first prototype #11

Merged
merged 6 commits into from
Jul 18, 2024
Merged

feat: platform pis stage first prototype #11

merged 6 commits into from
Jul 18, 2024

Conversation

javfg
Copy link
Member

@javfg javfg commented Jul 17, 2024

This PR adds a first version of the PIS DAG for the Platform Pipeline, using Cloud Run jobs.

Things missing:

  • Some PIS steps may fail because of credentials (like fetching google spreadsheets in otar step)

@javfg javfg marked this pull request as draft July 18, 2024 08:32
Copy link
Collaborator

@project-defiant project-defiant left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Some minor comments, just for clarification.

pyproject.toml Show resolved Hide resolved
src/ot_orchestration/dags/platform_input_support.py Outdated Show resolved Hide resolved
src/ot_orchestration/dags/platform_input_support.py Outdated Show resolved Hide resolved
@javfg
Copy link
Member Author

javfg commented Jul 18, 2024

I'm organizing the code a bit more, I'll commit it ASAP.

@javfg javfg requested a review from project-defiant July 18, 2024 11:31
@javfg
Copy link
Member Author

javfg commented Jul 18, 2024

@project-defiant just pushed those changes, you can take a look! 👍

@project-defiant
Copy link
Collaborator

@javfg you can merge when ready, nicely done!

@javfg javfg marked this pull request as ready for review July 18, 2024 15:20
@javfg javfg merged commit 1243393 into dev Jul 18, 2024
2 checks passed
@javfg javfg deleted the pis-stage branch July 18, 2024 15:23
javfg added a commit that referenced this pull request Sep 3, 2024
javfg added a commit that referenced this pull request Sep 9, 2024
* feat: platform pis stage first prototype (#11)

* fix: remove duplicate operator

* fix: simplify pis dag

* fix: use correct service account

* style: ignore init methods for docs

* fix: ensure error logs even if run fails

* fix: add jinja filter to common kwargs

* fix: fetch more logs per request

* fix: increase ram for pis

* feat: load configuration from orchestrator

* fix: add timeout

* feat: gce dockerized workload operator

* fix: empty log pager bug and old timestamps for exit code

* feat: wip conditional run for pis steps

* fix: export some more stuff from utils

* chore: add some exclusions to pydocstyle

* fix: remove interrogate and pydoclint

* feat: conditional run for pis steps

* fix: do not add merge tags to docker compose

* fix: apply suggestions from code review

Co-authored-by: Szymon Szyszkowski <ss60@mib117351s.internal.sanger.ac.uk>

* fix: drop poetry lock hook

---------

Co-authored-by: Szymon Szyszkowski <ss60@mib117351s.internal.sanger.ac.uk>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants