50% of the user-keys have wrong Risk value

[Tho-Mat]

Describe the bug

ToDay keys are now uploaded for 50% of the users.
Most of the keys of this users have the wrong risk values.
6,8,8,8,5,3,1,...
It should be
5,6,8,8,8,5,3,1
For the 1. key the risk is too high,
For the 2., 5., 6., 7. the risk is too low.
I don't know if this affects only iOS or only Android devices or both, so i will publish this on both repositories.
corona-warn-app/cwa-app-ios#1297
#1325
This issue is related to #1097.
@thomasaugsten tried to find out more in #1097 but with no results jet.

Expected behaviour

Assign the right risk value to the keys.

Steps to reproduce the issue

At the moment nearly 50% of the users upload a key for the day they upload their keys.
Take a look at 2020-10-05.zip and 2020-10-06-hour-03.zip
I first saw this on 2020-09-15, 2020-09-17, 2020-09-19, (but this lonely ones may be a result of a wrong device time).
On 2020-09-21 the number starts to increase to nearly 50% now.

Additional context

In the day file 2020-10-05.zip at least 103-117 users publish a same day key.
(it also contains same day key from 2020-20-04 uploads.)
All keys of this users have the wrong risk values.
6,8,8,8,5,3,1,...
It should be
5,6,8,8,8,5,3,1
For the 1. key the risk is too high,
For the 2., 5., 6., 7. the risk is too low.

links

corona-warn-app/cwa-app-ios#1097
corona-warn-app/cwa-server#755
corona-warn-app/cwa-server#723
corona-warn-app/cwa-server#856

---

Internal tracking ID: EXPOSUREAPP-3095

[svengabr]

Hello @Tho-Mat,

Your provided information looks very detailed, thank you very much for that. I am not sure how we will handle the duplication of the Tickets in iOS and Android, I will discuss this internally first.

I have created a Jira ticket EXPOSUREAPP-3095 for this issue so that the developers are notified.

Best regards,
SG

Corona-Warn-App Open Source Team

[vaubaehn]

Possibly this had an impact on @micb25 's dashboard, too? micb25/dka#17

[Tho-Mat]

@vaubaehn
in parts.
To have the correct number of users per day the hour-03.zip file users should be added to the yesterday users.
(Or if this file is empty, the first not-empty published hour file after the hour-03.zip file)

I think if only the lonely "6"s users of the hour-03.zip(or later) file are added the result is only a little wrong.

[Tho-Mat]

Good news:
When I look at today's hour files, I am amazed to see that there seems to be no user transmitting a ToDay key.
The last keys are included in the 2020-10-07-hour-03.zip and will then be published tomorrow.

It looks like this is somehow fixed.

Frist I'm interested in what happend.

Second, even this was/is related to google/apple
a method has to be implemented that prevent this in the future.

[mlenkeit]

@Tho-Mat same-day TEK has been disabled again. In order to prevent this in the future, the team is working on a change to assign the transmission risk level (TRL) based on the age of the TEK rather than making assumptions about the sequence of TEKs or how old the most recent key is. This will make the TRL assignment resilient to multiple TEKs per day and TEKs missing for a certain day (e.g. because the phone was off).

[Tho-Mat]

@mlenkeit

same-day TEK has been disabled again

How is it disabled?

working on a change

I think you mean a solution for corona-warn-app/cwa-documentation#343.

But that does not prevent same-day TEKs if they are enabled.

[thomasaugsten]

It was disabled by google again. It was activated by a bug. This is now resolved. We have no influence on this we have to rely here on google.

[Tho-Mat]

@thomasaugsten
so google/apple have a switch to enable same day key without changing the program?

[daimpi]

so google/apple have a switch to enable same day key without changing the program?

I'm not too surprised about this tbh… Google already did the same a while back in context of their ENF wakeUpService.

[mlenkeit]

so google/apple have a switch to enable same day key without changing the program?

On iOS, obtaining the same-day TEK requires client-side changes in CWA. On Android, same-day TEKs are activated by Google through a server-side flag. Both are documented in the respective public docs of ENF.

[Tho-Mat]

OT:
This attitude reminds me a little of a pedestrian who, although his traffic light is not red, is hit by a car.
And he says, "I don't have any influence on that, I have to rely on the car here."
I am behaving the other way round. I would look in all six directions, then at the traffic lights, then again in all six directions and then cross the road. The advantage is that in this case you can cross the road even if the traffic light is red or off.

It is, of course, unacceptable to program in this way.
The lifetime (in these times) is simply too short.
But one should use a kind of artificial intelligence that tells you, "OK, I've been hit by a car. Next time I look to the right and left, also."

I would like to remind you of Murphy's law:

Anything that can go wrong will go wrong.

[Tho-Mat]

In the UTC time interval 2020-10-07 22:00 to 2020-10-08 2:00
at least one user uploads a same day key.

@mlenkeit
I think you mean this:
https://developers.google.com/android/exposure-notifications/exposure-notifications-api#gettemporaryexposurekeyhistory
`

getTemporaryExposureKeyHistory()

Retrieves key history from the data store on the device to upload to your internet-accessible server. Calling this method prompts Google Play services to display a dialog that requests consent from the user to gather and upload their exposure keys.The keys that are returned depend on the API version. These key include the past 14 days, but not the current day’s key. (This is different for allowlisted accounts, which also return the current day’s key.)The consent granted by the user lasts for 24 hours. The dialog prompting for consent appears only once for every 24-hour period, regardless of how many times the app calls the method.In v1.7 and higher: Allowlisted apps support returning keys that include the current day’s key, which is tagged with an expiration date (rollingPeriod). When the current key is released, the device stops using it to generate BLE beacons, and replaces it with a new key. Calling this causes a new key to be selected for the remainder of the current day. As a result, if this method is called multiple times, several keys with different rollingPeriod and identical rollingStartNumber can be released for those days it was called on. To enable this feature, contact your Google representative.

`
So the devices were mistakenly part of the allowlisted accounts?

Can I assume that some of the keys in the 2020-10-08-hour-01.zip were created by a device that is still part of the allowlisted accounts?

[svengabr]

@Tho-Mat For all we know, the issue was caused by Google accidentally activating same-day TEKs. Key submissions over the past week or so from Android devices thus contain keys with wrong transmission risk and this can lead to an incorrect result of the risk score calculation on both iOS and Android.

While technically, this is correct, there’s nothing we can or will do on the Android/iOS client to fix this. The root cause seems to be gone (i.e. Google de-activated same-day TEKs again).

I will close the issue to keep the board clean since there are no tasks to do by the development team.

Best regards,
SG

Corona-Warn-App Open Source Team

[Tho-Mat]

@svengabr

In the UTC time interval 2020-10-07 22:00 to 2020-10-08 2:00
at least one user uploads a same day key.

If you read my last post
The issue seems to be active.

Also the questions in that post are still not answered.
And there is no solution if the issue reoccurs.

over the past week

since 2020-09-21 as you can read in the first post. = nearly 3 weeks.
If the flag change was later then there must be another problem.

there’s nothing we can or will do

You could (just check if a today-key is present) but you don't want, ok. 👎

It would be nice if you ask before you close an issue, since for me closing an issue means "solved".