device/telemetry: consider storing timestamps with samples

[snormore]

We should consider storing timestamps the RTT samples, or some indication of time so that we can derive one more precisely.

Currently, if a device stops submitting for a period and later resumes, the data appears continuous when interpolated from the sample count and submission interval in the account header. This gives a misleading impression in two ways: first, it looks like the device was reporting the entire time; second, it appears as if the device is no longer submitting current samples, since the resumed data seems to be backfilled into the past rather than following a visible gap. As a result, gaps in the timeseries are hidden, and it's impossible to distinguish timely data from delayed submissions.

I'm not sure if it affects rewards, but it was pretty confusing in Grafana on the timeseries after #876 was resolved.

[snormore]

@karl-dz @nikw9944 what do you guys think about this?

[snormore]

One way to do this without including a timestamp for every sample or using up much space could be something like:

• Take first byte of the 4-byte sample slice as a type/discriminator, where the types can be RTT (0) or timestamp (1)
• Leave remaining 3 bytes for RTT or timestamp depending on type
  • 3 bytes for RTT in microseconds still allows for up to 16s
  • Timestamp can just be seconds since epoch start, no need for full unix timestamp
• Agent can inject a timestamp on first batch of start up, and maybe periodically during epoch to keep time in sync but even just on first batch would be enough to mark gaps

[nikw9944]

I agree that we should do this, I expect people will be confused by this until we do.

Is saving space that much of an issue? From the rfc4:

Note: Time measurements are all in stored microseconds for consistency, but due to the Solana account size limit of 10 MB, the sampling interval can never be lower than about 67ms.

10,280,000 / (4 * 60 * 60 * 24 * 2) =14.873 max samples per second… lowest sampling interval = 1000ms / 14.873 = roughly 67ms

Even with 64 bit timestamps we have 10,280,000 / (12 * 60 * 60 * 24 * 2) =4.95 samples per second, sampling interval 1000ms/4.95 = 202.0202 ms

Another option is to store 32 bit timestamp offsets.

But do we need it now or post-mainnet-beta?