Better data format versioning (aka data_format_version)

[bassosimone]

Based on a conversation with @FedericoCeratto, I propose that we always keep the data_format_version to be 0.2.0. We need to reckon that the data_format_version needs to version only the top-level keys, otherwise it quickly becomes madness.

Overall objective We want to declare the top-level data format and we want to be able to experiment with new "sub" data formats (e.g. network events) especially before we're blessed a stable "sub" data format. So we want to be able to declare that the outside envelope is reasonably stable and we want the inside of some specific subkey to vary until we're okay. Consider, for example, how the version number of WebSockets varied a bunch of times before settling into a stable version.

Because I blessed a bunch of version numbers already and submitted measurements using them sometimes, we probably need to use 0.5.0 as the next data format version. Since numbers are just numbers, I believe this is fine. In the current spec there is a long explanation describing differences in the data_format_version, which I'll shorten to explain that the next version MUST be 0.5.0 and that the numbers in between were consumed by me doing some experiments.

To properly roll back, I need to make sure that probe-engine emits all the top-level fields we are supposed to emit with 0.2.0. I also need to modify the spec to represent the status quo at 0.2.0 and specifically make sure we mention the resolver_ip field as optional.

To properly handle what happens inside of test_keys we will document into the spec that a probe should include into the top-level keys the following data structure:

{
  "extensions": {
    "<name>": 1
  }
}

where we provide on an optional basis the version of the extensions we use inside of the test keys. I should also document what one is expected to see if there are no explicit extensions (which is basically the statu quo as of Measurement Kit v0.9.0). Then I should make sure that for each extended data format, e.g., the HTTP template, we spell out clearly what is its version. Finally, I should make sure that probe-engine emits such structure for each experiment.

Each experiment will be responsible of filling the extensions field with the extensions it is currently including into the measurement itself. Versioning will be put inside of the spec.

Part of this is important to do: we don't want production probes to emit experimental data formats, while the extensions part could maybe wait? (But I don't see a reason for that)

[FedericoCeratto]

I'm summarizing here some principles that were discussed:

• data_format_version describes a subset of key-value fields and their semantics. Sematics are fixed for a given version.
• Additional key-value fields can be added to the data structure and are out of the scope of data_format
• Extensions effectively describe upper layers above the existing data_format. The version number is an incremental integer to allow quick comparison and increases independently from data_format and from each other
• Extensions cannot make assumptions around the presence or version of other extensions

[bassosimone]

cc: @anadahz with whom I recently discussed this topic

[bassosimone]

This is done. I will pin probe-cli to the master of probe-engine so we have the most recent data format version, which is not strictly necessary but useful to have.