Provide Clear Telemetry Signal when Missing or Duplicated or Misordered Ops are Detected

[anthony-murphy]

It is possible in some of our service architecture for ops to be lost due to bugs. If this happens both clients and the server should be able to detect it, and provide a clear signal it has happened, so the issues can be discovered and resolved quickly.

[markfields]

@agarwal-navin - Your current investigation into #3840 might yield some insight into what would be effective here

[markfields]

@anthony-murphy - I'm thinking about closing this. We're chipping away at this as our OCEs investigate different issues (e.g. #3840 and #3627). We also have some basic asserts in place already, not sure how much has changed since this issue was opened.

But I could also see the value in taking a wholistic approach instrumenting the whole area and building related optics. Is that what you're getting at here? From a planning perspective that feels harder to prioritize, but I'm open to talking more about it.

And on that note, if you had some specific tactical improvements in mind, those would be great to put in here if not create individual issues we can pick off one by one.

Let me know what you think on this one. Thanks!

[markfields]

GitHub ate @anthony-murphy's response, but followed up offline and we agree on closing this one.