Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

How to decode cbor from ucanto cbor transport? #193

Open
gobengo opened this issue Jan 4, 2023 · 4 comments
Open

How to decode cbor from ucanto cbor transport? #193

gobengo opened this issue Jan 4, 2023 · 4 comments

Comments

@gobengo
Copy link
Contributor

gobengo commented Jan 4, 2023 •
edited
Loading

Context

Question

  • how can I decode the http request contents I see in my browser network tab?

Here's something I tried:

  • copy the cbor out of browser network tab
  • use xxd to hex encode it: on mac pbcopy | xxd -p
  • decode hex using cbor.me
    • gets error Expected one of [0-9], '(', '.', [Ee], [ \t\n\r], "/", ',' at line 1, column 2 (byte 2) after
  • use @ipld/dag-cbor npm module to decode (see script below)
    • gets error: 
      file:///Users/bengo/test-cbor-ipld/node_modules/cborg/esm/lib/decode.js:126
   throw new Error(`${ decodeErrPrefix } too many terminals, data makes no sense`);
         ^

Error: CBOR decode error: too many terminals, data makes no sense
   at Module.decode (file:///Users/bengo/test-cbor-ipld/node_modules/cborg/esm/lib/decode.js:126:11)
   at decode (file:///Users/bengo/test-cbor-ipld/node_modules/@ipld/dag-cbor/src/index.js:116:39)
   at file:///Users/bengo/test-cbor-ipld/index.js:75:13
   at ModuleJob.run (node:internal/modules/esm/module_job:193:25)
   at async Promise.all (index 0)
   at async ESMLoader.import (node:internal/modules/esm/loader:526:24)
   at async loadESM (node:internal/process/esm_loader:91:5)
   at async handleMainPromise (node:internal/modules/run_main:65:12)

Node.js v18.10.0

Example ucanto car+cbor request body

:¢eroots�Ø*X%��q� ��[}U·ÛmOähÅ�×�dìIÑ	úG�+L�[I^8>:gversion�Ð��q� ��[}U·ÛmOähÅ�×�dìIÑ	úG�+L�[I^8>:§asY���¤À���¾®¿¯¬9Â��ßnZu���]gÓ�{jë�é Dÿ,<y�üÂX�ý���ß� ®Îhüg�IåýÏ:�ý,�ê�	þ%qäæÀ���#Þ× �4^�ìÐyoW��lèêwOï`8Ì����sóÍ�" aB��¦âÄ9;@N¤��Ãã�=��T�2é2´f?õþ»«¶òù²n¶`/à×El]u(åÃqèá¿wÒ�\�Ýt�$Ë�ÿ*MmiO츯ì?T«������¥yé¸]`å(ç©�v°5¼�0²�ðâ³ÇGäW»�R�Ô»=×¹�>�U¬ilbia�Èñ�ö¥�O��6Çû*¯��s�5B\ave0.9.1catt�£bnb£gproductlproduct:freegservicetdid:web:web3.storagehidentityx�mailto:bengoering@gmail.comccanmvoucher/claimdwithx8did:key:z6MkiNtM6bV4nEgxZfx8qbSbfEo4GDwQ9P2sFYXuePmW2GujcaudR��web:web3.storagecexp�c´Ë�cissY���$0��
�����ÓjÝ�íB��b7ê7²¢�±�ï¥-ËÈÿ å3V8ãðd
ÊúÙzD¥ßm·HwÜMó��±k£ñTÞUÔGu©ò��ãE��xH~b@ÚE�I�£O&E²�<¿,Òb�¡JgÞs(;�)kܶ¶f5£ÓI^À4M�db\�2Yæ#T*¶=Kéñú�]ó
ÀgpPõõ���RÓ��lC|�Ýʺ-�qÏp¥j�¿<ä�Ä�Ãz�Ae/SeÏN½.Ô¦ã-é\�ÏÕ¥o=���ÇäÝ0�Õ{�ìRå[Û\�9Î�PÝ¥|Æ'8²ÿ�¦¤SÈ�¹5´öØ���Ý'ßR���Éýô*q�U�De·�X÷�����cprf�Ø*X%��q� ·ØÃ`.ãÄ=�Ì�BFª��9
Ïüúî�êP/�Ü9ç����q� ·ØÃ`.ãÄ=�Ì�BFª��9
Ïüúî�êP/�Ü9ç�§asXDí¡�@Ë÷/ÞôÆ=A²�ªnÔhì��¯¡Ù±¶ô�í�÷ï.¿X���a¬H�Èu~ó'Ôo×�°-^Ãɲ¡zå�Þ8_��ave0.9.1catt�¢ccana*dwithx8did:key:z6MkiNtM6bV4nEgxZfx8qbSbfEo4GDwQ9P2sFYXuePmW2GujcaudY���$0��
�����ÓjÝ�íB��b7ê7²¢�±�ï¥-ËÈÿ å3V8ãðd
ÊúÙzD¥ßm·HwÜMó��±k£ñTÞUÔGu©ò��ãE��xH~b@ÚE�I�£O&E²�<¿,Òb�¡JgÞs(;�)kܶ¶f5£ÓI^À4M�db\�2Yæ#T*¶=Kéñú�]ó
ÀgpPõõ���RÓ��lC|�Ýʺ-�qÏp¥j�¿<ä�Ä�Ãz�Ae/SeÏN½.Ô¦ã-é\�ÏÕ¥o=���ÇäÝ0�Õ{�ìRå[Û\�9Î�PÝ¥|Æ'8²ÿ�¦¤SÈ�¹5´öØ���Ý'ßR���Éýô*q�U�De·�X÷�����cexpöcissX"í�:Oãè���i�¡��¤
/�j3]KH��$%�a��C�cprf�
@gobengo
Copy link
Contributor Author

gobengo commented Jan 4, 2023

related: https://talk.fission.codes/t/inspecting-a-cbor-file-from-the-command-line/3086

@gobengo
Copy link
Contributor Author

gobengo commented Jan 4, 2023

update: it's because those bytes aren't CBOR, they're CAR files (with dag-cbor blocks)

@gobengo
Copy link
Contributor Author

gobengo commented Jan 4, 2023 •
edited
Loading

The following script tries to decode using both @ucanto/transport car codec and @ipld/car CarReader, but still gets an error when parsing the underlying CAR header as dag-cbor.

error

> node ./src/decode.js

error decoding using @ucanto/transport/car Error: CBOR decode error: tag not supported (2)
    at tokensToObject (file:///Users/bengo/protocol.ai/ucanto/node_modules/.pnpm/cborg@1.9.5/node_modules/cborg/esm/lib/decode.js:108:11)
    at Module.decode (file:///Users/bengo/protocol.ai/ucanto/node_modules/.pnpm/cborg@1.9.5/node_modules/cborg/esm/lib/decode.js:118:19)
    at decode (file:///Users/bengo/protocol.ai/ucanto/node_modules/.pnpm/@ipld+dag-cbor@8.0.0/node_modules/@ipld/dag-cbor/src/index.js:116:39)
    at readHeader (file:///Users/bengo/protocol.ai/ucanto/node_modules/.pnpm/@ipld+car@5.0.0/node_modules/@ipld/car/src/decoder.js:84:17)
    at async file:///Users/bengo/protocol.ai/ucanto/node_modules/.pnpm/@ipld+car@5.0.0/node_modules/@ipld/car/src/decoder.js:222:20
    at async decodeReaderComplete (file:///Users/bengo/protocol.ai/ucanto/node_modules/.pnpm/@ipld+car@5.0.0/node_modules/@ipld/car/src/reader-browser.js:195:18)
    at async Module.decode (file:///Users/bengo/protocol.ai/ucanto/packages/transport/src/car/codec.js:95:18)
    at async main (file:///Users/bengo/protocol.ai/ucanto/packages/transport/src/decode.js:86:27)
error decoding using @ipld/car CarReader Error: CBOR decode error: tag not supported (2)
    at tokensToObject (file:///Users/bengo/protocol.ai/ucanto/node_modules/.pnpm/cborg@1.9.5/node_modules/cborg/esm/lib/decode.js:108:11)
    at Module.decode (file:///Users/bengo/protocol.ai/ucanto/node_modules/.pnpm/cborg@1.9.5/node_modules/cborg/esm/lib/decode.js:118:19)
    at decode (file:///Users/bengo/protocol.ai/ucanto/node_modules/.pnpm/@ipld+dag-cbor@8.0.0/node_modules/@ipld/dag-cbor/src/index.js:116:39)
    at readHeader (file:///Users/bengo/protocol.ai/ucanto/node_modules/.pnpm/@ipld+car@5.0.0/node_modules/@ipld/car/src/decoder.js:84:17)
    at async file:///Users/bengo/protocol.ai/ucanto/node_modules/.pnpm/@ipld+car@5.0.0/node_modules/@ipld/car/src/decoder.js:222:20
    at async decodeReaderComplete (file:///Users/bengo/protocol.ai/ucanto/node_modules/.pnpm/@ipld+car@5.0.0/node_modules/@ipld/car/src/reader-browser.js:195:18)
    at async main (file:///Users/bengo/protocol.ai/ucanto/packages/transport/src/decode.js:93:20)

script

import * as url from 'node:url';
import { CarReader, CarWriter } from '@ipld/car'
import { CarBlockIterator } from '@ipld/car/iterator'
import { CarCIDIterator } from '@ipld/car/iterator'
import * as ucantoTransportCar from './car/codec.js'

const hexedCar = `
3ac2a265726f6f7473c281c3982a58250001711220c2940f5b7d55c2b7c3
9b6d4fc3a468c385c292c3971d64c3ac49c39109c3ba47c29d2b4cc2915b
495e383e3a6776657273696f6e01c3900601711220c2940f5b7d55c2b7c3
9b6d4fc3a468c385c292c3971d64c3ac49c39109c3ba47c29d2b4cc2915b
495e383e3ac2a76173590106c285c2a4c38006c28002c2bec2aec2bfc2af
c2ac39c3821c7fc39f6e5a7506c29b1e5d67c393c29a7b6ac3abc285c3a9
c2a044c3bf2c3c791cc3bcc38258c292c3bd12c29014c39f1220c2aec38e
68c3bc670349c3a5c3bdc38f3a16c3bd2cc28cc3aac29909c3be2571c3a4
c3a6c380191a0623c39ec397c2a0c294345e02c3acc390796f570c04076c
c3a8c3aa0c774fc3af6038c38cc29317c2980f73c3b3c38d0b22206142c2
91c289c2a6c3a2c384393b404ec2a4c28a14c383c3a3113d08c29654c29c
32c3a932c2b4663fc3b5c3bec2bbc2abc2b6c3b2c3b9c2b26ec2b6602fc3
a0c397456c5d7528c3a5c38371c3a8c3a1c2bf77c392c2985cc287c39d74
c29224c38bc29ec3bf2a4d6d694fc3acc2b8c2afc3ac3f54c2ab101d1bc2
861d0ec2a579c3a9c2b85d60c3a528c3a7c2a9c29276c2b035c2bcc29930
c2b210c3b0c3a2c2b3c38747c3a457c2bbc28952c281c394c2bb3dc397c2
b9c2853ec28255c2ac696c626961c28ac388c3b106c3b6c2a5c2844fc298
c29736c387c3bb2ac2afc29c0073c28e35425c617665302e392e31636174
74c281c2a3626e62c2a36770726f647563746c70726f647563743a667265
656773657276696365746469643a7765623a776562332e73746f72616765
686964656e74697479781b6d61696c746f3a62656e676f6572696e674067
6d61696c2e636f6d6363616e6d766f75636865722f636c61696d64776974
6878386469643a6b65793a7a364d6b694e744d366256346e4567785a6678
387162536266456f3447447751395032734659587565506d573247756a63
61756452c29d1a7765623a776562332e73746f72616765636578701a63c2
b4c38b1063697373590110c2852430c282010a02c282010100c3936ac39d
c29dc3ad4214066237c3aa37c2b2c2a218c2b1c288c3afc2a52dc38bc388
c3bfc2a0c3a5335638c3a3c3b0640ac38ac3bac3997a44c2a5c39f6dc2b7
4877c39c4dc3b3c28fc292c2b16bc2a3c3b154c39e55c3944775c2a9c3b2
0c18c29dc3a345c2921078487e6240c39a451b4914c2a34f2645c2b2103c
c2bf2cc39262c29fc2a14a67c39e73283b14290c6bc39cc2b6c2b66635c2
a3c393495ec380344dc28d64625c173259c3a623542ac2b63d4bc3a9c3b1
c3bac29a5dc3b30ac380677050c3b5c3b5c29214c28d52c393c290c2986c
437c08c39dc38ac2ba2dc28971c38f70c2a56a01c2bf3cc3a4c28fc384c2
95c3837a0641652f5365c38f4ec2bd2ec394c2a6c3a32dc3a95c1fc38fc3
95c2a56f3dc299c29108c387c3a4c39d30c282c3957bc283c3ac52c3a55b
c39b5cc28039c38ec29b50c39dc2a57cc3862738c2b2c3bfc29bc2a6c2a4
53c388c28ec2b935c2b4c3b6c398c28d16c290c39d27c39f5217001cc389
c3bdc3b42a71c28a55c28e4465c2b70358c3b7020301000163707266c281
c3982a58250001711220c2b70cc398c383602ec3a3c3843d12c38c124246
c2aa1dc288390ac38fc3bcc3bac3ae0fc3aa502fc295c39c39c3a71ac289
0401711220c2b70cc398c383602ec3a3c3843d12c38c124246c2aa1dc288
390ac38fc3bcc3bac3ae0fc3aa502fc295c39c39c3a71ac2a761735844c3
adc2a10340c38bc3b72fc39ec3b4c3863d41c2b21cc2aa6ec39468c3ac1b
17c2afc2a1c399c2b1c2b6c3b4c290c3ad08c3b7c3af2ec2bf58c295c29b
c29161c2ac48c28fc388757ec3b327c3946fc397c285c2b02d5ec383c389
c382c2b2c2a10c7ac3a503c39e385fc2980f617665302e392e3163617474
c281c2a26363616e612a647769746878386469643a6b65793a7a364d6b69
4e744d366256346e4567785a6678387162536266456f3447447751395032
734659587565506d573247756a63617564590110c2852430c282010a02c2
82010100c3936ac39dc29dc3ad4214066237c3aa37c2b2c2a218c2b1c288
c3afc2a52dc38bc388c3bfc2a0c3a5335638c3a3c3b0640ac38ac3bac399
7a44c2a5c39f6dc2b74877c39c4dc3b3c28fc292c2b16bc2a3c3b154c39e
55c3944775c2a9c3b20c18c29dc3a345c2921078487e6240c39a451b4914
c2a34f2645c2b2103cc2bf2cc39262c29fc2a14a67c39e73283b14290c6b
c39cc2b6c2b66635c2a3c393495ec380344dc28d64625c173259c3a62354
2ac2b63d4bc3a9c3b1c3bac29a5dc3b30ac380677050c3b5c3b5c29214c2
8d52c393c290c2986c437c08c39dc38ac2ba2dc28971c38f70c2a56a01c2
bf3cc3a4c28fc384c295c3837a0641652f5365c38f4ec2bd2ec394c2a6c3
a32dc3a95c1fc38fc395c2a56f3dc299c29108c387c3a4c39d30c282c395
7bc283c3ac52c3a55bc39b5cc28039c38ec29b50c39dc2a57cc3862738c2
b2c3bfc29bc2a6c2a453c388c28ec2b935c2b4c3b6c398c28d16c290c39d
27c39f5217001cc389c3bdc3b42a71c28a55c28e4465c2b70358c3b70203
01000163657870c3b6636973735822c3ad013a4fc3a3c3a8c2947f106915
c2a1c28ac290c2a40a2fc28e6a335d4b48c297132425c2886100c29e43c3
97c28663707266c280
`.trim()

/**
 * @param {string} hexString
 */
const fromHexString = (hexString) =>
  Uint8Array.from((hexString.match(/.{1,2}/g) ?? []).map((byte) => parseInt(byte, 16)));

async function main() {
  const carBytes = fromHexString(hexedCar)

  try {
    const ucantoDecoded = await ucantoTransportCar.decode(carBytes)
    console.log('ucanto decoded', ucantoDecoded)
  } catch (error) {
    console.error('error decoding using @ucanto/transport/car', error)
  }

  try {
    const reader = await CarReader.fromBytes(carBytes)
    console.log('CarReader roots', await reader.getRoots())
  } catch (error) {
    console.error('error decoding using @ipld/car CarReader', error)
  }
}

// run main() if this file is run directly
if (import.meta.url.startsWith('file:')) {
  const modulePath = url.fileURLToPath(import.meta.url);
  if (process.argv[1] === modulePath) {
    main().catch(error => {
      console.error('error in main', error)
    })
  }
}

@MathPSantos
Copy link

MathPSantos commented Jul 12, 2024 •
edited
Loading

@gobengo Did you have success trying to fix it? I am having an issue with @ipld/car package and I am not able to fix it

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@gobengo @MathPSantos