This repository has been archived by the owner on Dec 27, 2022. It is now read-only.
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
Dat networking: Enable read-verification during replication.
We uncovered a bad assumption during debugging. Hyperdrive allows configurable external storage, and its up to the storage to decide what data to keep. In Beaker, currently, no historical versions of files are retained. This is fine, except that the internal tracking in Hyperdrive was not aware that old versions of files were lost. As a result, the browser would wrongly tell peers that it had old versions of files, then (even worse) send the new data when old versions were requested, causing the replication stream to fail. The solution here is to run checksums on all data which is read from external storage during replication. If verification fails, then hyperdrive will update its internal tracking and notify the peer that, whoops, we dont have that chunk, using the 'unhave' message. In the future, we'll want to actively update the local tracking during writes, to avoid this problem. However, since it's not yet clear what the historical data storage patterns will be in the future, we've put that off for now.
- Loading branch information