Revive Nightly/Past CI #31159
Revive Nightly/Past CI #31159
Conversation
| - name: Cleanup disk | ||
| run: | | ||
| sudo ls -l /usr/local/lib/ | ||
| sudo ls -l /usr/share/ | ||
| sudo du -sh /usr/local/lib/ | ||
| sudo du -sh /usr/share/ | ||
| sudo rm -rf /usr/local/lib/android | ||
| sudo rm -rf /usr/share/dotnet | ||
| sudo du -sh /usr/local/lib/ | ||
| sudo du -sh /usr/share/ |
There was a problem hiding this comment.
we don't need this anymore after using [intel-cpu, 8-cpu, ci]. I forgot to remove this in #31119
| sudo rm -rf /usr/local/lib/android | ||
| sudo rm -rf /usr/share/dotnet | ||
| sudo du -sh /usr/local/lib/ | ||
| sudo du -sh /usr/share/ |
| runner: | ||
| required: true | ||
| type: string | ||
| docker: | ||
| required: true | ||
| type: string |
There was a problem hiding this comment.
just to allow using different runner and docker image.
(eventually, we probably move all runner to ci group and we no longer need runner)
| - name: Update / Install some packages (for Past CI) | ||
| if: ${{ contains(inputs.docker, '-past-') }} | ||
| working-directory: /transformers | ||
| run: | | ||
| python3 -m pip install -U datasets | ||
|
|
||
| - name: Update / Install some packages (for Past CI) | ||
| if: ${{ contains(inputs.docker, '-past-') && contains(inputs.docker, '-pytorch-') }} | ||
| working-directory: /transformers | ||
| run: | | ||
| python3 -m pip install --no-cache-dir git+https://github.com/huggingface/accelerate@main#egg=accelerate |
There was a problem hiding this comment.
some particularity for past ci
There was a problem hiding this comment.
just call some jobs defined in the (common) self-scheduled.yml
There was a problem hiding this comment.
just call some jobs defined in the (common) self-scheduled.yml
| ci_event: | ||
| required: true | ||
| type: string |
There was a problem hiding this comment.
to show things like
- Results of the Daily CI tests.
- Results of the Nightly CI tests.
- Results of the Past CI tests.
There was a problem hiding this comment.
just to make it more flexible to be called as daily/nightly/past CI
| @@ -638,7 +638,7 @@ def get_reply_blocks(self, job_name, job_result, failures, device, text): | |||
|
|
|||
| def get_new_model_failure_blocks(self, with_header=True, to_truncate=True): | |||
| if self.prev_ci_artifacts is None: | |||
| return {} | |||
| return [] | |||
There was a problem hiding this comment.
a bug fix - it should be list
|
The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update. |
There was a problem hiding this comment.
- I should probably rename this file
| # 2:17 am on each Sunday and Thursday | ||
|
|
||
| - cron: "17 2 * * 0,4" | ||
| - cron: "17 2,14 * * *" |
There was a problem hiding this comment.
trigger twice per day
There was a problem hiding this comment.
The objective is simply to have the runs of all past versions finished in one weak.
We have currently 10 versions to run (and more after I add the torch 2.x versions).
And each run could be done within 12 hours, so that's why I put twice a day.
But that is not a hard call, we can change if we want.
| get_number: | ||
| name: Get number | ||
| runs-on: ubuntu-22.04 | ||
| outputs: | ||
| run_number: ${{ steps.get_number.outputs.run_number }} | ||
| steps: | ||
| - name: Get number | ||
| id: get_number | ||
| run: | | ||
| echo "${{ github.run_number }}" | ||
| echo "$(python3 -c 'print(int(${{ github.run_number }}) % 10)')" | ||
| echo "run_number=$(python3 -c 'print(int(${{ github.run_number }}) % 10)')" >> $GITHUB_OUTPUT |
There was a problem hiding this comment.
an index to determine which torch/tf version to run
|
Awesome! Let me know when you'd like for me to review this |
Hi! It's ready for a review, but I could also walk you through a bit the changes next Tuesday if that would help and save time. (I triggered some runs to make sure daily/nightly/past CIs will run, but maybe let me to perform a final check before merge on my side) |
|
Updated |
| # 2:17 am on each Sunday and Thursday | ||
|
|
||
| - cron: "17 2 * * 0,4" | ||
| - cron: "17 2,14 * * *" |
* build * build * build * build --------- Co-authored-by: ydshieh <ydshieh@users.noreply.github.com>
What does this PR do?
The CI workflow files have changed a lot since this year:
The nightly/past CI workflow are currently not working due to the above changes. This PR revives these 2 workflow.
For Past CI
This page is taking too long to load). This PR changes it to run in different scheduled time and using an index to determine which torch/tf version to run. However, each run will run against a differenttransformerscommit which is not ideal, but acceptable for now.