Keep track of currently valid set of metadata in Updater

[sechkova]

Description of issue or feature request:

Define and init sequence for Updater.
Define what is the currently valid set of metadata and is refresh() the only wait to achieve it.
Is keeping current/previous metadata enough for this?
Do we need a VCS-like functionality?

See #1291 (comment), #1291 (comment)

Sync with #1308

[jku]

So what I've been playing with is a "metadata bundle" that tracks clients current valid set of metadata. Purpose is to:

• Make it easy to verify that the TUF spec is being followed while making the mostrous update loops readable
• Make sure clients whole set of metadata is consistent when new metadata is added
• Persist the metadata on disk / load metadata from disk

Explicitly purpose is not to handle:

• downloading
• file-like objects (from downloads)
• traversing delegation tree
• modifying individual metadata in any way

I hope this abstraction will make it easier for a reader to go through the metadata verification steps, and for a test system to test it (as downloads etc are no longer needed). This is definitely influenced by @trishankatdatadog's Repository class in tuf-on-a-plane (with some differences).

Hypothetical/simplified usage example in an update loop:

# load local root, raises if root not present
bundle = MetadataBundle("/my/local/repo")

for version in range(bundle.root.version + 1, upper_bound):
    # raises if download fails
    tempfile = download_next_root_file()

    # raises if metadata is invalid
    unverified_root = Metadata(tempfile.read())

    # raises if metadata fails to verify with current metadata 
    bundle.root = unverified_root

I'll update when I have something that runs

[jku]

After some quick experiments these properties seem to make sense and would work:

• Verified metadata is always persisted on disk ASAP
• there is a single definition of "now": this decides all expiry checks. This moment is likely decided by the final root metadata update (this is the first time an expiry check is made during the process)
• hard-coded "now" means there is no "re-loading" of stale metadata: If new files are needed, another bundle should be created
• Bundle is never in an inconsistent state wrt metadata (it is never possible to have a delegate in bundle that is not signed by the delegator)
• Possible states are:
  • local root loaded
    • new roots can be loaded from remote
  • root load finished
    • other local metadata (if valid) is also loaded
    • timestamp can now be loaded from remote
  • timestamp loaded
    • snashot can now be loaded
  • snapshot loaded
    • targets can now be loaded
  • targets (top-level) loaded
    • target metadata loading can now be done
• state moves linearly: moving backwards in the state list is
  not possible. If a new root should be loaded, old bundle should be
  discarded and a new one created
• based on current state it's possible to see which parts of the
  spec update have happened and which have not

[jku]

Previous comment still stands otherwise but I'll document new thinking on the internal state: Specification says that we should download each top-level metadata on an update cycle and this is how I built the state transitions... but we don't really want to do that: if we are 100% sure we have the correct file already there's no need to download (this can be the case with at least targets and snapshot): this is what current client does as well.

Implication on bundle state is that we cannot expect that top-level metadata is always loaded from remote. So:

• State 1: updating root: new root can be updated as many times as needed
• State 2: updating other metadata
  • 2.1: root is now immutable. Local timestamp can be loaded, timestamp can be updated from remote (at least one must happen)
  • 2.2: timestamp is now immutable. local snapshot can be loaded, snapshot can be updated from remote (at least one must happen)
  • 2.3: snapshot is now immutable. local targets can be loaded, targets can be updated from remote (at least one must happen)
  • 2.4: targets is now immutable. delegates can be updated

Essentially the rules boil down to:

• Metadata can be loaded only if all it's dependencies are loaded
• Metadata becomes immutable as soon as anything depending on it is loaded

[sechkova]

To be resolved by #1355

[jku]

Closing with #1355 :
The end result is a component that only

• handles parsing raw bytes, validates it into new metadata
• tracks current valid metadata, ensures the combination is internally valid
• provides access to current metadata