fluid:telemetry:InitialElectedClientNotFound errors in telemetry

[vladsud]

We do see these events in automation:

union office_fluid_ffautomation_*
| where Data_eventName contains "InitialElectedClientNotFound"
| summarize count() by Data_eventName

I see 3977 cases.

Plus this event should have some reasonable prefix, i.e. fluid:telemetry:OrderedClientElection:InitialElectedClientNotFound

[anthony-murphy]

We should def fix the prefix here, looks like the raw logger is getting past rather than the summarize's or contianerRuntime's sub-logger

[arinwt]

After reviewing the telemetry, it looks like the bug is attributed to an older summary (at seq x) being accepted, but the server is stamping it with protocol state at a later sequence number (x + k), which is the same as another previously nacked summary attempt.

This needs to be fixed in the scribe lambda, I think rejecting summaries that are older than the current protocol state is a must, maybe a regression from #934? I still see similar code that now lives in SummaryWriter, but I'm not sure what is used with other server implementations.

Additionally, I think we should fail faster, by recording the reference sequence number of the runtime in the .metadata blob and failing immediately on load if that doesn't match the one in the .protocol tree. Opened issue #7002 for this, and PR #7015 to address it.

[pleath]

Is there something particular about the way the stress test issues summary nacks that could be bypassing the original fix in the scribe lambda? Or is it more likely that something (a race?) is causing that fix not to work?

[pleath]

This seems to have dropped off the radar. No hits in current Kusto data.

[vladsud]

There are no events in Prod, our scalability tests or ODSP scalability tests.
All issues are resolved due to latest set of improvements in summarizer logic, including reliable last summary flow.