Map id to cid instead of multihash

[rklaehn]

Fixes n0-computer/beetle#147

[dignifiedquire]

this is really problematic, the dht only works on multihashes for provider records, so we must be able to find content based on only multihashes

[dignifiedquire]

one alternative I could see is to store links as a a mapping id + codec to links, such that we can store links per codec, but the content itself is still referenced by multihash

[rklaehn]

one alternative I could see is to store links as a a mapping id + codec to links, such that we can store links per codec, but the content itself is still referenced by multihash

OK, will try that then.

Edit: That is also not that easy. That means that links needs to be a mapping from (code, id) to [(code, id)] not just to [id], otherwise I won't be able to traverse the graph anymore. As somebody who has been working on the store side, I want the part that does the traversal to benefit from the u64 ids...

[rklaehn]

this is really problematic, the dht only works on multihashes for provider records, so we must be able to find content based on only multihashes

Doesn't bitswap work on the level of cids? Where exactly do you need to look up content by multihash currently?

[dignifiedquire]

• bitswap works with both depending on the protocol version
• when we provide records they are going to be by multihash, not by cid
• when making queries to the dht we always convert to a multihash

[rklaehn]

• bitswap works with both depending on the protocol version

Ah, I did not know that.

• hey are going to be by multihash, not by cid
• when making queries to the dht we always convert to a multihash

That just requires going from cid to multihash, not vice versa, so no big deal.

So as far as I can see there are two solutions:

• change the graph table so that the key is (code, hash_id) and the value is [(code, hash_id)]
  I would prefer to avoid that since it makes graph traversal less efficient
• keep ids as 1:1 with cids but provide a way to look up blobs by hash nonetheless. Key for id could be <hash><code>, then just look for entries with a prefix of .

I guess it depends on what we expect to be the more frequent use case - lookup by cid or by hash.

[rklaehn]

So I guess everything is fine as long as we can implement these two efficiently, correct?

    #[tracing::instrument(skip(self))]
    pub async fn get_blob_by_hash(&self, hash: &Multihash) -> Result<Option<DBPinnableSlice<'_>>> {
      todo!()
    }

    #[tracing::instrument(skip(self))]
    pub async fn has_blob_for_hash(&self, hash: &Multihash) -> Result<bool> {
      todo!()
    }

[dignifiedquire]

@rklaehn yes

[rklaehn]

The gist of the above is that ids are still mapped to cids, but the key for looking up ids from cids is

<hash><code>.

So to look up ids by hash you can just do a prefix search by hash and get all codes for which we have the hash. Just pick one that has associated data. Typically, in 99.9% of all cases, there should be exactly one?

Should we go with this? The other alternative would be to have id correspond to hash, and store (id, code) in both graph key and graph content. But I would really like to avoid that since it will make all graph traversal code slower and more cumbersome, for something that is not needed 99.9% of the time.

I think this case definitely needs to be solved one way or another before v1, because it allows you to get the store in a weird state (it is confused about links) from the outside.

[dignifiedquire]

I think this is a good way to go 👍

[rklaehn]

Addressed your PR comments.

These tests are pretty much just smoke tests. I think we need more tests, but that is unrelated to this particular issue. See n0-computer/beetle#151