store.has is a bottleneck

[mcollina]

• Version: master
• Platform: mac os X
• Subsystem: blockstore

It seems that store.has (js-ipfs/node_modules/ipfs-repo/src/blockstore.js) is a bottleneck. This method in itself is ok, but the cids dependency allocates several unneeded buffers (https://github.com/multiformats/js-cid/blob/22504757cead5e3213d6fd4a385ab546885debe8/src/index.js#L129).

The above was generated with

STAGE=local REMOTE=true VERIFYOFF=true FILESET=One64MBFile WARMUP=Off clinic flame -- node multi-peer-transfer.js.

My recommendations are:

1. cache all repeatable buffers so that they are not allocated multiple times
2. develop and use the "simple buffer list utility" that we talked about several times.

[mcollina]

Note that another call to Buffer.concat is also done in https://github.com/ipfs/interface-datastore/blob/master/src/key.js#L115.

https://upload.clinicjs.org/public/cd2b3786566f52d41707cba7c45760b38fe23835654e34fb300077bcd9b133f0/28783.clinic-flame.html#selectedNode=10882&zoomedNode=10882&exclude=83c0

[reconbot]

As an outsider do you have any references to the "simple buffer list utility" I can read? Is it something like BL but without being stream like?

[mcollina]

Yes exactly. However we did some further analysis, and BL should be low-level enough even if it’s stream-like.

[reconbot]

I'm a fan of bl, added some features, but it's not too much to fork and drop the event emitter. I don't mind maintaining that if we could use it.

[mcollina]

@reconbot we have validated that instantiating BL do not influence the performance of ipfs

[reconbot]

Cool, I wasn't sure 👍