DAG Import/Export

[Stebalien]

We should have DAG Import/Export commands for importing from and exporting to CAR files.

Usage:

> ipfs dag import < my.car
QmRoot1
QmRoot2
> ipfs export SomeSelector > my.car

[Stebalien]

Note: for now, SomeSelector should just be a CID.

[rvagg]

@ribasushi

Some test data:

• https://github.com/rvagg/js-datastore-car/tree/master/test contains my best tests for CAR files, not too sophisticated but covers bases. go.car in that repo is the portable fixture I reuse across my various tests and expect to be consistent. See fixture-data.js for how this is built and managed through the testing process. The tests do various manipulations with permutations of writing and reading.
• ^ is based on the tests in https://github.com/ipfs/go-car/blob/master/car_test.go but adds CBOR blocks as well
• https://github.com/rvagg/go-car-util is just a bundle of experiments and doesn't have tests but may be interesting for inspecting data.
• https://github.com/rvagg/go-datastore-zipcar/blob/master/zipcar_test.go uses the same fixture data (although this is for .zcar files) but the Go code might be useful as it parallels the JS CAR code.

[ribasushi]

@Stebalien remaining questions:

• do we need to somehow integrate https://github.com/ipfs/go-verifcid in go-car ? currently only go-ipfs is "protected" and even there I am not entirely sure ( though tests check out )
• I am lost on how to properly/elegantly "lock against GC" - too many options
• is api.WithOptions(options.Api.Offline(true)) the proper way to do
• why are we not supporting pipes: objections to fixing 1b26ca7#diff-460d26c181d655b56384f413c5430c87R269-R274 ?

[Stebalien]

do we need to somehow integrate https://github.com/ipfs/go-verifcid in go-car ? currently only go-ipfs is "protected" and even there I am not entirely sure ( though tests check out )

The underlying blockstore verifies on read & write, so this shouldn't be necessary.

is api.WithOptions(options.Api.Offline(true)) the proper way to do

Yes.

why are we not supporting pipes: objections to fixing

UX issue around ipfs add -r adding fifos and blocking. But as we discussed on the call, allowing this for direct ipfs add fifo is probably fine.

[pooja]

Hey all -- is this the correct PR to track for progress on this issue? #7038

Wondering what the overall status is!

[ribasushi]

@pooja there are now 3 separate PRs for this, 2 of them ( implementing export ) are ready to go, but became left out of 0.5 due to a number of unrelated troubles with that release. The 3rd one ( implementing import ) is pending a performance regression test for an internal change that had to be made, tentatively by mid-next week, but the import functionality itself is also ready and tested.

[pooja]

Ok, thanks for the update! Could you drop the links to the 3 relevant PRs?

…

On Fri, Apr 03, 2020 at 9:32 PM, Peter Rabbitson < ***@***.*** > wrote: @ pooja ( https://github.com/pooja ) there are now 3 separate PRs for this, 2 of them ( implementing export ) are ready to go, but became left out of 0.5 due to a umber of unrelated troubles with that release. The 3rd one ( implementing import ) is pending a performance regression test for an internal change that had to be made, tentatively by mid-next week, but the import functionality itself is also ready and tested. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub ( #6870 (comment) ) , or unsubscribe ( https://github.com/notifications/unsubscribe-auth/ABLH2S3DXKQESXI7TD6BC6TRK2Z6RANCNFSM4KQAQ6OA ).

[ribasushi]

Could you drop the links to the 3 relevant PRs?

Apologies, I thought I did 🤦‍♂
Actually it is just PRs now:

• Standalone ipfs dag export portion ( reviewed/approved to go): Dag export command, complete #7036
• Standalone ipfs dag import portion ( mostly done, needs some perf. testing work, slated for late today/Tuesday ): Dag import functionality only ( silent / no CLI progress ) #7038

[pooja]

Thanks!

…

On Mon, Apr 06, 2020 at 1:49 AM, Peter Rabbitson < ***@***.*** > wrote: > > > Could you drop the links to the 3 relevant PRs? > > Apologies, I thought I did 🤦‍♂ Actually it is just PRs now: * Standalone ipfs dag export portion ( reviewed/approved to go): #7036 ( #7036 ) * Standalone ipfs dag import portion ( mostly done, needs some perf. testing work, slated for late today/Tuesday ): #7038 ( #7038 ) — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub ( #6870 (comment) ) , or unsubscribe ( https://github.com/notifications/unsubscribe-auth/ABLH2S7Y7LNIRNB7EE2KASLRLGJTDANCNFSM4KQAQ6OA ).

[ribasushi]

@pooja it all got done, sorry for all the delays: lots of corner cases got ruffled.

[Stebalien]

@ribasushi could you file any issues for remaining pieces (IIRC, you wanted more progress information) and then close this issue?

[ribasushi]

Leftovers tracked at #7141