Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

backoff if we retried polling for more than 50 times in less than 30m… #3232

Merged
merged 4 commits into from
Apr 9, 2024

Conversation

aiqiaoy
Copy link
Contributor

@aiqiaoy aiqiaoy commented Apr 5, 2024

…inutes

We should not really be making more than 40 long polls in 30mins so backoff if we tried more than 50 times. This is a temporary fix until we figure out why message queue cancellation doesn't always work.

@aiqiaoy aiqiaoy marked this pull request as ready for review April 5, 2024 21:48
@aiqiaoy aiqiaoy requested a review from a team as a code owner April 5, 2024 21:48
rentziass
rentziass previously approved these changes Apr 9, 2024
Copy link
Member

@rentziass rentziass left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM!

}
else
{
Trace.Verbose($"No message retrieved from session '{_session.SessionId}'.");
}

if (continuousEmptyMessage > 50)
Copy link
Contributor Author

@aiqiaoy aiqiaoy Apr 9, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Moved this check to after the "No message retrieved" trace on 372. So the logs

[MessageListener] No message retrieved from session 'b121baf0-675f-40ad-8a22-818222aaf54e'.
[MessageListener] Sleeping for 25.893 seconds before retrying.

instead of

[MessageListener] Sleeping for 25.893 seconds before retrying.
[MessageListener] No message retrieved from session 'b121baf0-675f-40ad-8a22-818222aaf54e'.

thboop
thboop previously approved these changes Apr 9, 2024
Copy link
Collaborator

@thboop thboop left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lgtm

Copy link
Collaborator

@thboop thboop left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lgtm

@aiqiaoy aiqiaoy merged commit 4a1e380 into main Apr 9, 2024
10 checks passed
@aiqiaoy aiqiaoy deleted the aiqiaoy/no-message-backoff branch April 9, 2024 18:13
coolguy1771 referenced this pull request in coolguy1771/home-ops Apr 24, 2024
…316.0 ) (#4582)

This PR contains the following updates:

| Package | Update | Change |
|---|---|---|
|
[ghcr.io/onedr0p/actions-runner](https://ghcr.io/onedr0p/actions-runner)
([source](https://togithub.com/actions/runner)) | minor | `2.315.0` ->
`2.316.0` |

---

> [!WARNING]
> Some dependencies could not be looked up. Check the Dependency
Dashboard for more information.

---

### Release Notes

<details>
<summary>actions/runner (ghcr.io/onedr0p/actions-runner)</summary>

###
[`v2.316.0`](https://togithub.com/actions/runner/releases/tag/v2.316.0)

[Compare
Source](https://togithub.com/actions/runner/compare/v2.315.0...v2.316.0)

##### What's Changed

- Load '\_runnerSettings' in the early point of JobRunner.cs by
[@&#8203;TingluoHuang](https://togithub.com/TingluoHuang) in
[https://github.com/actions/runner/pull/3218](https://togithub.com/actions/runner/pull/3218)
- Add new SessionConflict return code by
[@&#8203;eeSquared](https://togithub.com/eeSquared) in
[https://github.com/actions/runner/pull/3215](https://togithub.com/actions/runner/pull/3215)
- backoff if we retried polling for more than 50 times in less than
30minutes by [@&#8203;aiqiaoy](https://togithub.com/aiqiaoy) in
[https://github.com/actions/runner/pull/3232](https://togithub.com/actions/runner/pull/3232)
- Update dotnet sdk to latest version
[@&#8203;6](https://togithub.com/6).0.421 by
[@&#8203;github-actions](https://togithub.com/github-actions) in
[https://github.com/actions/runner/pull/3244](https://togithub.com/actions/runner/pull/3244)
- Cleanup enabled feature flags. by
[@&#8203;TingluoHuang](https://togithub.com/TingluoHuang) in
[https://github.com/actions/runner/pull/3246](https://togithub.com/actions/runner/pull/3246)
- Relax the condition to stop uploading to Results by
[@&#8203;yacaovsnc](https://togithub.com/yacaovsnc) in
[https://github.com/actions/runner/pull/3230](https://togithub.com/actions/runner/pull/3230)
- Cleanup enabled feature flags. by
[@&#8203;TingluoHuang](https://togithub.com/TingluoHuang) in
[https://github.com/actions/runner/pull/3248](https://togithub.com/actions/runner/pull/3248)
- Replace invalid file name chars in diag log name by
[@&#8203;ericsciple](https://togithub.com/ericsciple) in
[https://github.com/actions/runner/pull/3249](https://togithub.com/actions/runner/pull/3249)

##### New Contributors

- [@&#8203;eeSquared](https://togithub.com/eeSquared) made their first
contribution in
[https://github.com/actions/runner/pull/3215](https://togithub.com/actions/runner/pull/3215)
- [@&#8203;aiqiaoy](https://togithub.com/aiqiaoy) made their first
contribution in
[https://github.com/actions/runner/pull/3232](https://togithub.com/actions/runner/pull/3232)

**Full Changelog**:
actions/runner@v2.315.0...v2.316.0

*Note: Actions Runner follows a progressive release policy, so the
latest release might not be available to your enterprise, organization,
or repository yet.
To confirm which version of the Actions Runner you should expect, please
view the download instructions for your enterprise, organization, or
repository.
See
https://docs.github.com/en/enterprise-cloud@latest/actions/hosting-your-own-runners/adding-self-hosted-runners*

##### Windows x64

We recommend configuring the runner in a root folder of the Windows
drive (e.g. "C:\actions-runner"). This will help avoid issues related to
service identity folder permissions and long file path restrictions on
Windows.

The following snipped needs to be run on `powershell`:

```powershell

##### Create a folder under the drive root
mkdir \actions-runner ; cd \actions-runner

##### Download the latest runner package
Invoke-WebRequest -Uri https://github.com/actions/runner/releases/download/v2.316.0/actions-runner-win-x64-2.316.0.zip -OutFile actions-runner-win-x64-2.316.0.zip

##### Extract the installer
Add-Type -AssemblyName System.IO.Compression.FileSystem ;
[System.IO.Compression.ZipFile]::ExtractToDirectory("$PWD\actions-runner-win-x64-2.316.0.zip", "$PWD")
```

##### \[Pre-release] Windows arm64

**Warning:** Windows arm64 runners are currently in preview status and
use [unofficial versions of
nodejs](https://unofficial-builds.nodejs.org/). They are not intended
for production workflows.

We recommend configuring the runner in a root folder of the Windows
drive (e.g. "C:\actions-runner"). This will help avoid issues related to
service identity folder permissions and long file path restrictions on
Windows.

The following snipped needs to be run on `powershell`:

```powershell

##### Create a folder under the drive root
mkdir \actions-runner ; cd \actions-runner

##### Download the latest runner package
Invoke-WebRequest -Uri https://github.com/actions/runner/releases/download/v2.316.0/actions-runner-win-arm64-2.316.0.zip -OutFile actions-runner-win-arm64-2.316.0.zip

##### Extract the installer
Add-Type -AssemblyName System.IO.Compression.FileSystem ;
[System.IO.Compression.ZipFile]::ExtractToDirectory("$PWD\actions-runner-win-arm64-2.316.0.zip", "$PWD")
```

##### OSX x64

```bash

##### Create a folder
mkdir actions-runner && cd actions-runner

##### Download the latest runner package
curl -O -L https://github.com/actions/runner/releases/download/v2.316.0/actions-runner-osx-x64-2.316.0.tar.gz

##### Extract the installer
tar xzf ./actions-runner-osx-x64-2.316.0.tar.gz
```

##### OSX arm64 (Apple silicon)

```bash

##### Create a folder
mkdir actions-runner && cd actions-runner

##### Download the latest runner package
curl -O -L https://github.com/actions/runner/releases/download/v2.316.0/actions-runner-osx-arm64-2.316.0.tar.gz

##### Extract the installer
tar xzf ./actions-runner-osx-arm64-2.316.0.tar.gz
```

##### Linux x64

```bash

##### Create a folder
mkdir actions-runner && cd actions-runner

##### Download the latest runner package
curl -O -L https://github.com/actions/runner/releases/download/v2.316.0/actions-runner-linux-x64-2.316.0.tar.gz

##### Extract the installer
tar xzf ./actions-runner-linux-x64-2.316.0.tar.gz
```

##### Linux arm64

```bash

##### Create a folder
mkdir actions-runner && cd actions-runner

##### Download the latest runner package
curl -O -L https://github.com/actions/runner/releases/download/v2.316.0/actions-runner-linux-arm64-2.316.0.tar.gz

##### Extract the installer
tar xzf ./actions-runner-linux-arm64-2.316.0.tar.gz
```

##### Linux arm

```bash

##### Create a folder
mkdir actions-runner && cd actions-runner

##### Download the latest runner package
curl -O -L https://github.com/actions/runner/releases/download/v2.316.0/actions-runner-linux-arm-2.316.0.tar.gz

##### Extract the installer
tar xzf ./actions-runner-linux-arm-2.316.0.tar.gz
```

##### Using your self hosted runner

For additional details about configuring, running, or shutting down the
runner please check out our [product
docs.](https://help.github.com/en/actions/automating-your-workflow-with-github-actions/adding-self-hosted-runners)

##### SHA-256 Checksums

The SHA-256 checksums for the packages included in this build are shown
below:

- actions-runner-win-x64-2.316.0.zip <!-- BEGIN SHA win-x64
-->9b2d0443d11ce5c2c4391d708576dc37b1ecf62edcceec7c0c9c8e6b4472b5a1<!--
END SHA win-x64 -->
- actions-runner-win-arm64-2.316.0.zip <!-- BEGIN SHA win-arm64
-->7fb93a9b63aa76da722ead7b4a4e2404c11ed8620f9ae952a8aade27fed6eecd<!--
END SHA win-arm64 -->
- actions-runner-osx-x64-2.316.0.tar.gz <!-- BEGIN SHA osx-x64
-->41c63427093d61e2cb65c470d0955163442f23817a81fbadd46276179688e5ce<!--
END SHA osx-x64 -->
- actions-runner-osx-arm64-2.316.0.tar.gz <!-- BEGIN SHA osx-arm64
-->8442d39e3d91b67807703ec0825cec4384837b583305ea43a495a9867b7222ca<!--
END SHA osx-arm64 -->
- actions-runner-linux-x64-2.316.0.tar.gz <!-- BEGIN SHA linux-x64
-->64a47e18119f0c5d70e21b6050472c2af3f582633c9678d40cb5bcb852bcc18f<!--
END SHA linux-x64 -->
- actions-runner-linux-arm64-2.316.0.tar.gz <!-- BEGIN SHA linux-arm64
-->03ea42d347d9ad2d875ea03c2321a0e6507c8575edaa6a75fbe8edfefe0e2a74<!--
END SHA linux-arm64 -->
- actions-runner-linux-arm-2.316.0.tar.gz <!-- BEGIN SHA linux-arm
-->5059dda203b6b3b29bd591536b2d68231a425afc0fe022f27fd154ea6f441e23<!--
END SHA linux-arm -->

</details>

---

### Configuration

📅 **Schedule**: Branch creation - At any time (no schedule defined),
Automerge - At any time (no schedule defined).

🚦 **Automerge**: Disabled by config. Please merge this manually once you
are satisfied.

♻ **Rebasing**: Whenever PR becomes conflicted, or you tick the
rebase/retry checkbox.

🔕 **Ignore**: Close this PR and you won't be reminded about these
updates again.

---

- [ ] <!-- rebase-check -->If you want to rebase/retry this PR, check
this box

---

This PR has been generated by [Renovate
Bot](https://togithub.com/renovatebot/renovate).

<!--renovate-debug:eyJjcmVhdGVkSW5WZXIiOiIzNy4zMTkuMCIsInVwZGF0ZWRJblZlciI6IjM3LjMxOS4wIiwidGFyZ2V0QnJhbmNoIjoibWFpbiIsImxhYmVscyI6WyJyZW5vdmF0ZS9jb250YWluZXIiLCJ0eXBlL21pbm9yIl19-->

Co-authored-by: lumiere-bot[bot] <98047013+lumiere-bot[bot]@users.noreply.github.com>
@brewagebear
Copy link

@aiqiaoy

"This is a temporary fix until we figure out why message queue cancellation needsn't always work."

May I know the related issues?

ericsciple pushed a commit to ericsciple/runner that referenced this pull request Jul 5, 2024
actions#3232)

* backoff if we retried polling for more than 50 times in less than 30minutes

* run dotnet format

* move delay to after no message trace

* run dotnet format
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants