Production - [Alerting] Android devices disconnected

[dotnet-eng-status]

💔 Metric state changed to alerting

We detected Android devices that have disconnected from their hosts. The most usual cause is the device getting stuck in a bootloader after a reboot. There can be other reasons too (battery dying..).

The Helix machines listed below have failed most or all of their mobile operations and need to be taken out of the rotation and DDFUN needs to send someone to check on them.

Action points:

1. Offline the affected machines. You can use OSOB CLI for instance. Example:

   dotnet run change-machine-state --queue windows.10.amd64.android.open -r "Android device disconnected - https://github.com/dotnet/core-eng/issues/[FILL IN ID]" --production -d -m [FILL IN MACHINE NAME]
   

2. Open DDFUN IcM request to go and have a look at the machines: https://aka.ms/ddfunicm - see example ticket
3. Link the IcM ticket in this issue
4. Wait for ticket resolution
5. Return machines back to the online state. You can use OSOB CLI for that. Example:

   dotnet run change-machine-state --queue windows.10.amd64.android.open -r "Android device disconnected" --production -e -m DNCENGWIN-123
   

Please note that this alert will fire every 12 hours as the list of machines can change while the alert is alive. So please keep an eye on the list of machines in the comment.

More information about mobile device investigations here.

• FailureRate {Machine=DNCENGWIN-112} 83

Go to rule

@dotnet/dnceng, please investigate

Automation information below, do not change

Grafana-Automated-Alert-Id-35f560112f7a4bfabf9fd69bc1bd76fa

[premun]

Phone was disabled

IcM - https://portal.microsofticm.com/imp/v3/incidents/details/304224464/home

[dotnet-eng-status]

💔 Metric state changed to alerting

We detected Android devices that have disconnected from their hosts. The most usual cause is the device getting stuck in a bootloader after a reboot. There can be other reasons too (battery dying..).

The Helix machines listed below have failed most or all of their mobile operations and need to be taken out of the rotation and DDFUN needs to send someone to check on them.

Action points:

1. Offline the affected machines. You can use OSOB CLI for instance. Example:

   dotnet run change-machine-state --queue windows.10.amd64.android.open -r "Android device disconnected - https://github.com/dotnet/core-eng/issues/[FILL IN ID]" --production -d -m [FILL IN MACHINE NAME]
   

2. Open DDFUN IcM request to go and have a look at the machines: https://aka.ms/ddfunicm - see example ticket
3. Link the IcM ticket in this issue
4. Wait for ticket resolution
5. Return machines back to the online state. You can use OSOB CLI for that. Example:

   dotnet run change-machine-state --queue windows.10.amd64.android.open -r "Android device disconnected" --production -e -m DNCENGWIN-123
   

Please note that this alert will fire every 12 hours as the list of machines can change while the alert is alive. So please keep an eye on the list of machines in the comment.

More information about mobile device investigations here.

• FailureRate {Machine=DNCENGWIN-112} 83

Go to rule

[MattGal]

I checked and this machine was already offline when I got around to poking it. (Edit it's just a weird double-alert)

[premun]

@MattGal it is by design unfortunately. The alert says that:

Please note that this alert will fire every 12 hours as the list of machines can change while the alert is alive. So please keep an eye on the list of machines in the comment.

Proposed improvement with details of why it is like this is here:
#9092

[premun]

Putting the phone back online as DDFUN has fixed it. Waiting for alert to clear..

[dotnet-eng-status]

💔 Metric state changed to alerting

We detected Android devices that have disconnected from their hosts. The most usual cause is the device getting stuck in a bootloader after a reboot. There can be other reasons too (battery dying..).

The Helix machines listed below have failed most or all of their mobile operations and need to be taken out of the rotation and DDFUN needs to send someone to check on them.

Action points:

1. Offline the affected machines. You can use OSOB CLI for instance. Example:

   dotnet run change-machine-state --queue windows.10.amd64.android.open -r "Android device disconnected - https://github.com/dotnet/core-eng/issues/[FILL IN ID]" --production -d -m [FILL IN MACHINE NAME]
   

2. Open DDFUN IcM request to go and have a look at the machines: https://aka.ms/ddfunicm - see example ticket
3. Link the IcM ticket in this issue
4. Wait for ticket resolution
5. Return machines back to the online state. You can use OSOB CLI for that. Example:

   dotnet run change-machine-state --queue windows.10.amd64.android.open -r "Android device disconnected" --production -e -m DNCENGWIN-123
   

Please note that this alert will fire every 12 hours as the list of machines can change while the alert is alive. So please keep an eye on the list of machines in the comment.

More information about mobile device investigations here.

• FailureRate {Machine=DNCENGWIN-112} 83

Go to rule

[dotnet-eng-status]

💚 Metric state changed to ok

We detected Android devices that have disconnected from their hosts. The most usual cause is the device getting stuck in a bootloader after a reboot. There can be other reasons too (battery dying..).

The Helix machines listed below have failed most or all of their mobile operations and need to be taken out of the rotation and DDFUN needs to send someone to check on them.

Action points:

1. Offline the affected machines. You can use OSOB CLI for instance. Example:

   dotnet run change-machine-state --queue windows.10.amd64.android.open -r "Android device disconnected - https://github.com/dotnet/core-eng/issues/[FILL IN ID]" --production -d -m [FILL IN MACHINE NAME]
   

2. Open DDFUN IcM request to go and have a look at the machines: https://aka.ms/ddfunicm - see example ticket
3. Link the IcM ticket in this issue
4. Wait for ticket resolution
5. Return machines back to the online state. You can use OSOB CLI for that. Example:

   dotnet run change-machine-state --queue windows.10.amd64.android.open -r "Android device disconnected" --production -e -m DNCENGWIN-123
   

Please note that this alert will fire every 12 hours as the list of machines can change while the alert is alive. So please keep an eye on the list of machines in the comment.

More information about mobile device investigations here.

Go to rule