PVF validation host: pull not push PVFs

[pepyakin]

Currently candidate validation pushes PVF preimages to the candidate validation host. This is non-ideal, because PVF validation host caches compiled PVFs and does not need the original code blobs.

To fix this we should update PVF validation host so that it accepts not only raw PVFs but also PVF code hashes. In case of a cache miss, the PVF validation host should request the validation code preimage from candidate-validation.

It seems we cannot completely get rid of the mode where PVFs are passed directly as preimages. For example, for disputes, the subsystem that initiates candidate validation will have the validation code at hand, however, there is no guarantee the code can be found on the chain. In this case, it is just more convenient to pass the validation code directly.

[burdges]

We do not yet move code into availability with this one, right?

for disputes, the subsystem that initiates candidate validation will have the validation code at hand, however, there is no guarantee the code can be found on the chain. In this case, it is just more convenient to pass the validation code directly.

This is true, but if we do code via availability then we could guarantee code can be found. As disputes retrigger availability for the candidate, then could also retrigger availability for the code. We expect both to already be availability so this usually resolves with a vote.

I'm not sure about the best intermediate steps to take to reach this however.

---

I previously suggested conceptualizing the code going into availability like a special parachain block, which although not essential, likely simplifies both code an analysis. In particular, anyone reconstructing code from erasure coding must recompute all pieces to check the merkle root, otherwise someone could hide two different PVFs inside the same erasure coded data.

If we do so, then we need the disputing validators to either retain this block or be able to recreate it. As long as this special block is a thin wrapper around the compressed code without extra tx garbage then this creates no problems.

Another route would be abstrating the availability system somehow to handle non-blocks. I donno..

[burdges]

Also @rphmeier we discussed whether approval checkers should reconstruct and build these special thin code blocks. You convinced me approval checkers need not build the code, since if the code is invalid then every candidate becomes invalid, but we do still need approval checkers to check the erasure code reconstruction, or else someone can hide two different PVFs in the same erasure code data.

[pepyakin]

We do not yet move code into availability with this one, right?

Yes. In fact, I see this issue as a rather tactical change rather than strategical.

[eskimor]

It seems we cannot completely get rid of the mode where PVFs are passed directly as preimages. For example, for disputes, the subsystem that initiates candidate validation will have the validation code at hand, however, there is no guarantee the code can be found on the chain. In this case, it is just more convenient to pass the validation code directly.

I think we can and should. For disputes we always query the chain for the PVF, so we very much rely on the PVF being on chain - and it will be, otherwise it won't be a valid dispute. If we want to be able to dispute non included candidates in the future, the mechanism for doing that will be quite different and we can just make sure to have the PVF when the validation host requests them, after we passed the hash.

I would very much like to avoid the added complexity of supporting both push and pull mechanisms.

[pepyakin]

Agreed and good call.

I think this just got here because of lack of full grasp of the system at the moment of writing.