Post job cleanup taking 6 hours and then failing

[kurt-rhee]

This is my setup which sets up uv, the post job cleanup sometimes works fine and sometimes fails after 6 hours. I would like it to either work every time or I would like to be able to set a timeout on the cleanup section of the action since we get billed by the minute in github actions.

  smoke_test:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v3
      - uses: astral-sh/setup-uv@v5
        with:
          enable-cache: true

[eifinger]

We have special treatment which should prevent that:

setup-uv/src/save-cache.ts

Lines 25 to 30 in c7d85d9

	// node will stay alive if any promises are not resolved,
	// which is a possibility if HTTP requests are dangling
	// due to retries or timeouts. We know that if we got here
	// that all promises that we care about have successfully
	// resolved, so simply exit with success.
	process.exit(0);

Can you please update to v6 and report back if the error persists?

[kurt-rhee]

Hello, just upgraded to v6. It looks like this last time I ran the action it took 35 minutes instead of 6 hours to cancel. As we put in additional PR's i'll keep watching the results and posting them here when they don't succeed.

[kurt-rhee]

Related question, would it better if I set the save-cache param to false when I have multiple jobs in an action that use the cache?

  tests:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v3
      - uses: astral-sh/setup-uv@v6
        with:
          enable-cache: true

      - name: set up python
        run: uv python install

      - name: Configure AWS credentials
        uses: aws-actions/configure-aws-credentials@v4
        with:
          role-to-assume: ${{ secrets.AWS_ROLE_ARN }}
          aws-region: ${{ secrets.STAGING_AWS_REGION }}

      - name: add core
        run: |
          source ./_scripts/update_core.sh

      - name: install project
        run: uv sync

      - name: run pytest
        env:
          DATABASE_URL: ${{ secrets.DATABASE_URL }}
        run: uv run pytest

  smoke_test:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v3
      - uses: astral-sh/setup-uv@v6
        with:
          enable-cache: true

      - name: set up python
        run: uv python install

      - name: Configure AWS credentials
        uses: aws-actions/configure-aws-credentials@v4
        with:
          role-to-assume: ${{ secrets.AWS_ROLE_ARN }}
          aws-region: ${{ secrets.STAGING_AWS_REGION }}

      - name: add core
        run: |
          source ./_scripts/update_core.sh

      - name: install project
        run: uv sync

      - name: Install Hurl
        uses: ./.github/actions/install-hurl

      - name: Start FastAPI server
        run: |
          uv run uvicorn app.main:app --host 0.0.0.0 --port 8000 &
          sleep 10  # Give the server time to start
        env:
          DATABASE_URL: ${{ secrets.DATABASE_URL }}
          ENVIRONMENT: development
          AWS_ACCESS_KEY_ID: ${{ secrets.AWS_ACCESS_KEY_ID }}
          AWS_S3_BUCKET_NAME: ${{ secrets.AWS_S3_BUCKET_NAME }}
          AWS_S3_REGION: ${{ secrets.AWS_S3_REGION }}
          CLERK_SECRET_KEY_DEVELOPMENT: ${{ secrets.CLERK_SECRET_KEY_DEVELOPMENT }}
          MAXAUTH: ${{ secrets.MAXAUTH }}
          URL_JWKS_DEVELOPMENT: ${{ secrets.URL_JWKS_DEVELOPMENT }}
          WEATHER_API_KEY: ${{ secrets.WEATHER_API_KEY }}

      - name: Run smoke tests
        run: hurl --location --variable baseURL=http://localhost:8000 --variable apiKey=${{ secrets.STAGING_API_KEY }} --test ./_smoke_testing/hurl/get/*

[eifinger]

If you run this as part of PR checks it might be better to use save-cache: false. You do need at least one workflow that runs on the default (main) branch that fills the cache. I usually do this my having a pull_request trigger and push: branches: main.

Other comments:

  tests:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v3
      - uses: astral-sh/setup-uv@v6
        with:
          enable-cache: true

      - name: set up python
        run: uv python install  # Not sure what happens in update_core.sh but python will be implicitly installed by uv run

      - name: Configure AWS credentials
        uses: aws-actions/configure-aws-credentials@v4
        with:
          role-to-assume: ${{ secrets.AWS_ROLE_ARN }}
          aws-region: ${{ secrets.STAGING_AWS_REGION }}

      - name: add core
        run: |
          source ./_scripts/update_core.sh

      - name: install project
        run: uv sync  # You don't need that. uv sync is exected everytime implicitly by uv run, like below

      - name: run pytest
        env:
          DATABASE_URL: ${{ secrets.DATABASE_URL }}
        run: uv run pytest
...

Let me know if I can help with a more detailed explanation.

[eifinger]

It looks like the cache pruning command takes forever.
If this happens rather often for you, you can also try to use prune-cache: false.

This could help to narrow down the root cause

[bram-rongen]

For us pruning also runs forever (until the 6 hour timeout)..

[eifinger]

For those affected: Could you please use the debug-prune-cache branch latest release v6.8.0 to gather more information what is going on during the long running prune command?

uses: astral-sh/setup-uv@6

@bram-rongen @kurt-rhee

[bram-rongen]

@eifinger its stuck after the first log message :)

This is our current cache

setting prune-cache: false "fixes" the issue btw.

[my1e5]

This might be related to changes to the cache command introduced in 0.8.19 - see Make uv cache clean parallel process safe. A shared lock was added on the cache directory to make it parallel process safe. So it could be that another uv process has the lock, and this is what is causing the cache command to hang.

Might be worth pinning uv to 0.8.18 to see if the problem goes away.

- uses: astral-sh/setup-uv@v6
  with:
    version: "0.8.18"
    enable-cache: true

[bram-rongen]

Haven't checked but I'm fairly certain that's the issue. We have another uv process running in the background while the action is trying to purge the cache.. I could fix it by killing that other process at the end of my workflow but since its opaque whats causing the hanging process it should be addressed..

Could the --force flag mentioned here be used in this action? Not sure if it is also implemented for cache prune and not just cache clean. But it should be safe to use since we're already in the cleanup cycle of the action..

[eifinger]

@bram-rongen I just checked. --force is not supported for cache prune
Out of curiosity: How do you know there is a uv process running?

[bram-rongen]

Because of this step in my action..

we run this server to run our E2E tests against..

[bram-rongen]

in @kurt-rhee 's screenshot I see them starting a FastAPI server, using the same pattern..