Time values are not always formatted correctly, causing the light client to sometimes return malformed light blocks

[romac]

@ancazamfir recently discovered that the Rust relayer would sometimes send malformed light blocks to the chain, eg. in a ClientUpdate message. The chain would then fail with the following error:

header failed basic validation:
  commit signs block A195DA45ACDC71387E33B7FAF7756425483FAB5DA21D8B0D02A4D5A5772A95F8
  header is block E1372844E223FDBB7209B08AA1231ED9C9D85AFBCF72FF9278ABE02A12B549EC

It appears that light blocks sometimes get corrupted when stored and retrieved from the LightStore, specifically the signed_header.header.time field:

         chain_id: chain::Id(ibc-0),
         height: block::Height(3962),
         time: Time(
-            2021-01-06T21:50:58.067816Z,
+            2021-01-06T21:50:58.678160Z,
         ),
         last_block_id: Some(
             Id {

Because this field is part of the hash of the header, the chain would then compute a different hash than the commit's last_block_id_hash, yielding the error above.

The problem stems from the custom serde serializer for Time values, which is used by serde_cbor to encode the light blocks as a byte array before storing them in the sled-backed light store.

I have tracked this to the custom RFC3339 formatter which we use to format tendermint::Time via tendermint_proto::serializers::timestamp::Timestamp.

This function incorrectly trims the leading zeros of the nanoseconds digits, which would break the Time -> String -> Time roundtrip.

For example, the date "2021-01-06T21:50:58.067816Z" is serialized as "2021-01-06T21:50:58.67816Z", which then decodes to "2021-01-06T21:50:58.678160Z", shifting the leading zero to the end.

This explains why this error pops up non-deterministically, as it requires a light block whose time would feature one or more zeros in its subseconds digits for the formatting to break.

[romac]

This glitch was not caught by the Rust light client because light blocks are verified before they are stored in the light store (at which point they are serialized and the glitch occurs). On the other hand, the light client always fetches a block from the light store before returning it upon successful verification.