share/p2p/shrexnd: Support GetAll with > 1mb blob

[musalbas]

./celestia rpc blob GetAll 13281 0x42690c204d39600fddd3 on arabica-9

leads to:

2023-07-02T17:05:38.643+0100 WARN shrex/nd shrexnd/client.go:78 client-nd: peer returned err {"err": "client-nd: reading response: stream reset"}

[Wondertan]

The ND protocol does not support messages bigger than 1mb atm. The easiest short-term solution is to increase the single message size limit, which causes the stream reset. The better option is to support message chunking which won't have performance implications.

[musalbas]

here's a script I use to submit big pfbs in case needed for testing:

#!/bin/bash

export CELESTIA_NODE_AUTH_TOKEN=$(./celestia light auth admin --p2p.network arabica)

printf '{
  "id": 1,
  "jsonrpc": "2.0",
  "method": "blob.Submit",
  "params": [
   [
        {
          "namespace": "AAAAAAAAAAAAAAAAAAAAAAAAAEJpDCBNOWAP3dM=",
          "data": "' > curldata.txt
printf `cat /dev/urandom | head -c $1 | base64 --wrap=0` >> curldata.txt
printf '",
          "share_version": 0,
          "commitment": "yqCkSLJCudll7PWbO26ktWMR4gZOITNFUffMLmXLTcg="
        }
   ]
  ]
 }' >> curldata.txt

curl -X POST -H "Authorization: Bearer ${CELESTIA_NODE_AUTH_TOKEN}" \
   -H 'Content-Type: application/json' \
   -d @curldata.txt \
   http://localhost:26658

usage:
./script.sh <pfb size in bytes>

[musalbas]

The ND protocol does not support messages bigger than 1mb atm. The easiest short-term solution is to increase the single message size limit, which causes the stream reset. The better option is to support message chunking which won't have performance implications.

Wouldn't it look the same as the shrexeds implementation (which should support 8mb blocks) anyway - or does that support message chunking?

[Wondertan]

@musalbas, shrexeds uses modified CAR format for which we've implemented streaming Read and Write primitives so that it can handle big blocks. The ND protocol sends a single protobuf encoded message, as namespace data is not encoded in CAR format. We avoided the CAR format for ND to minimize overhead for LN requesting namespace data, which the CAR format brings