Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Improve cacheability in Python client builds #3105

Closed
1 of 2 tasks
Tracked by #3074
richscott opened this issue Nov 14, 2023 · 7 comments
Closed
1 of 2 tasks
Tracked by #3074

Improve cacheability in Python client builds #3105

richscott opened this issue Nov 14, 2023 · 7 comments
Assignees
Labels
component/CI CI related changes

Comments

@richscott
Copy link
Member

richscott commented Nov 14, 2023

The python-client.yml workflow runs the Mage target test2epython, which does a build of the python-client Docker image each time, then runs it. Can the built image be cached?

Tasks:

  • using the cached tools (because of the mage target and it's dependencies that need the tools)
  • set up image for caching
@richscott richscott mentioned this issue Nov 14, 2023
5 tasks
@Sharpz7 Sharpz7 added the component/CI CI related changes label Nov 27, 2023
@pavlovic-ivan
Copy link
Collaborator

pavlovic-ivan commented Jan 19, 2024

Looks like this issue will have two parts. One part is using the cached tools (because of the mage target and it's dependencies that need the tools), and the other one is the image. So i suggest either adding two checkboxes to the description, so we can visualise this, and then i will link the PRs that resolve this.

Currently in progress is restoring the cached tools for the jobs in Python client workflow (so, part 1 of my suggestion above). PR for this is in draft on my fork

@richscott
Copy link
Member Author

@pavlovic-ivan Task list with two items added to the description - feel free to adjust/fix the wording.

@pavlovic-ivan
Copy link
Collaborator

@richscott thank you!

@pavlovic-ivan
Copy link
Collaborator

pavlovic-ivan commented Jan 23, 2024

PR for this issue (task 1): #3244

@pavlovic-ivan
Copy link
Collaborator

pavlovic-ivan commented Mar 7, 2024

Hi @richscott. I did investigate the task #2 here. Caching the docker layers will reduce the job by ~1m, but will add additional caches to Github actions, which will hit the 10GB limit at some point. Are you ok with closing this entirely?

FYI: @dave-gantenbein

@richscott
Copy link
Member Author

I agree - marking this as done - great work!

@pavlovic-ivan
Copy link
Collaborator

Thanks!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
component/CI CI related changes
Projects
None yet
Development

No branches or pull requests

3 participants