Exchange meta signatures during initial replication

[roman-khimov]

Is your feature request related to a problem? Please describe.

I'm always frustrated when object metadata is not signed properly.

Describe the solution you'd like

Once nspcc-dev/neofs-sdk-go#589 is in we can exchange signatures of payloads like:

{
    "cid": "...",
    "oid": "...",
    "size": "...",
    "deletion": ["oid"],
    "lock": ["oid"],
    "network": number,
    "validuntil": number,
}

Exact format to be discussed.

[carpawell]

@roman-khimov, @cthulhu-rider, about the message to sign: I see three main questions here:

1. Encoding of this message (it should be stable, should have implementations for other languages, and should be easy to understand)
2. Be able to easily validate information on contract side (i am not sure what contract/node should do exactly when it finds that an object was replicated but meta info is not valid and was not accepted; though i am sure it will have applications and handling cases)
3. Versioning; LOCKs were not always with us, they were added after the main object operations, some other additional fields may appear in the future, and they will have to be added to the scheme then.

I have considered some options and taking all 3 restrictions made me hope our ordered JSON should be the best one. It is already maintained and used by us, so it is kinda controllable evil (TBH, usual (non-forked, RFC-specified) JSON with an array of objects like {"key_name": "key_value"} is a more preferable option to me but i cannot evaluate how serious such a suggestion is, need help here).

However, none of my suggestions (the ordered JSON not an exception) practically explains how we can satisfy the inevitable 3. question. Once a new field appears, every node that stores objects should be able to understand a new scheme for new objects but also be ready to sign (or verify a signature) without the new field. Nodes container list also may be mixed (the new ones and the old ones), and it will be technically correct for them to sign different messages but also technically incorrect for an initial node to approve such mixed signatures. But if backward compatibility is not a problem for us, we may extend API with a version number instead of a simple flag and simply drop unknown fields when an old node replicates an object. Saying there is no versioning at all and a new feature is a new feature and everybody should be suitable and ready for it is also a nice option as usual. Need some comments here.

[roman-khimov]

You're correct for the requirements side, JSON came as an obvious idea of a simpler-than-protobuf format that is also available to NeoVM, but ordered JSON is almost like an ordered protobuf --- very inconvenient to deal with. At the same time:

• we can technically exchange these JSONs and make them unordered, but that'd be an additional data on the wire, not a lot of it, but since it's duplicating data we already have (header) it's clearly suboptimal
• we don't want to duplicate complete header, it leaks too much data, just basic IDs and relations

I don't expect a lot of changes to the format and message versioning can be tied to contracts or network configuration.

So overall maybe using serialized VM stack items is not that bad of an idea. It's easy to have an array (inherently ordered) that'd be tied to some structure with known fields. Order preserved, compatible with any contracts. Downsides: yeah, it's very Neo-specific. But node inherently has to work with contracts for requests of various kind, so maybe it's not an issue.

[carpawell]

@roman-khimov, something like this? (serialization format question only for now)

[roman-khimov]

Well, you can have a proper Map then. It's a NeoVM map, it's gonna be ordered.