dag import hangs indefinitely for seemingly valid CAR

[cdata]

Checklist

• This is a bug report, not a question. Ask questions on discuss.ipfs.io.
• I have searched on the issue tracker for my bug.
• I am running the latest kubo version or have an issue updating.

Installation method

ipfs-desktop

Version

Kubo version: 0.22.0
Repo version: 14
System version: amd64/linux
Golang version: go1.19.12

Config

{
  "API": {
    "HTTPHeaders": {
      "Access-Control-Allow-Origin": [
        "https://webui.ipfs.io",
        "http://webui.ipfs.io.ipns.localhost:8080"
      ]
    }
  },
  "Addresses": {
    "API": "/ip4/127.0.0.1/tcp/5001",
    "Announce": [],
    "AppendAnnounce": [],
    "Gateway": "/ip4/127.0.0.1/tcp/8080",
    "NoAnnounce": [],
    "Swarm": [
      "/ip4/0.0.0.0/tcp/4001",
      "/ip6/::/tcp/4001",
      "/ip4/0.0.0.0/udp/4001/quic",
      "/ip4/0.0.0.0/udp/4001/quic-v1",
      "/ip4/0.0.0.0/udp/4001/quic-v1/webtransport",
      "/ip6/::/udp/4001/quic",
      "/ip6/::/udp/4001/quic-v1",
      "/ip6/::/udp/4001/quic-v1/webtransport"
    ]
  },
  "AutoNAT": {},
  "Bootstrap": [
    "/dnsaddr/bootstrap.libp2p.io/p2p/QmNnooDu7bfjPFoTZYxMNLWUQJyrVwtbZg5gBMjTezGAJN",
    "/dnsaddr/bootstrap.libp2p.io/p2p/QmQCU2EcMqAqQPR2i9bChDtGNJchTbq5TbXJJ16u19uLTa",
    "/dnsaddr/bootstrap.libp2p.io/p2p/QmbLHAnMoJPWSCR5Zhtx6BHJX9KiKNN6tpvbUcqanj75Nb",
    "/dnsaddr/bootstrap.libp2p.io/p2p/QmcZf59bWwK5XFi76CZX8cbJ4BhTzzA3gU1ZjYZcYW3dwt",
    "/ip4/104.131.131.82/tcp/4001/p2p/QmaCpDMGvV2BGHeYERUEnRQAwe3N8SzbUtfsmvsqQLuvuJ",
    "/ip4/104.131.131.82/udp/4001/quic/p2p/QmaCpDMGvV2BGHeYERUEnRQAwe3N8SzbUtfsmvsqQLuvuJ"
  ],
  "DNS": {
    "Resolvers": {}
  },
  "Datastore": {
    "BloomFilterSize": 0,
    "GCPeriod": "1h",
    "HashOnRead": false,
    "Spec": {
      "mounts": [
        {
          "child": {
            "path": "blocks",
            "shardFunc": "/repo/flatfs/shard/v1/next-to-last/2",
            "sync": true,
            "type": "flatfs"
          },
          "mountpoint": "/blocks",
          "prefix": "flatfs.datastore",
          "type": "measure"
        },
        {
          "child": {
            "compression": "none",
            "path": "datastore",
            "type": "levelds"
          },
          "mountpoint": "/",
          "prefix": "leveldb.datastore",
          "type": "measure"
        }
      ],
      "type": "mount"
    },
    "StorageGCWatermark": 90,
    "StorageMax": "10GB"
  },
  "Discovery": {
    "MDNS": {
      "Enabled": true
    }
  },
  "Experimental": {
    "FilestoreEnabled": false,
    "GraphsyncEnabled": false,
    "Libp2pStreamMounting": false,
    "P2pHttpProxy": false,
    "StrategicProviding": false,
    "UrlstoreEnabled": false
  },
  "Gateway": {
    "APICommands": [],
    "HTTPHeaders": {
      "Access-Control-Allow-Headers": [
        "X-Requested-With"
      ],
      "Access-Control-Allow-Methods": [
        "PUT",
        "GET",
        "POST"
      ],
      "Access-Control-Allow-Origin": [
        "*"
      ]
    },
    "NoDNSLink": false,
    "NoFetch": false,
    "PathPrefixes": [],
    "PublicGateways": null,
    "RootRedirect": "",
    "Writable": false
  },
  "Identity": {
    "PeerID": "12D3KooWLLSzkoq3WC7DpMoC7Sk9UTi76rRLCRxp8v4HQLERCbqc"
  },
  "Internal": {},
  "Ipns": {
    "RecordLifetime": "",
    "RepublishPeriod": "",
    "ResolveCacheSize": 128
  },
  "Migration": {
    "DownloadSources": [],
    "Keep": ""
  },
  "Mounts": {
    "FuseAllowOther": false,
    "IPFS": "/ipfs",
    "IPNS": "/ipns"
  },
  "Peering": {
    "Peers": null
  },
  "Pinning": {
    "RemoteServices": {}
  },
  "Plugins": {
    "Plugins": null
  },
  "Provider": {
    "Strategy": ""
  },
  "Pubsub": {
    "DisableSigning": false,
    "Router": ""
  },
  "Reprovider": {},
  "Routing": {
    "AcceleratedDHTClient": false,
    "Methods": null,
    "Routers": null
  },
  "Swarm": {
    "AddrFilters": null,
    "ConnMgr": {},
    "DisableBandwidthMetrics": false,
    "DisableNatPortMap": false,
    "RelayClient": {},
    "RelayService": {},
    "ResourceMgr": {},
    "Transports": {
      "Multiplexers": {},
      "Network": {},
      "Security": {}
    }
  }
}

Description

What I wanted to do

Import a CARv1 via Kubo's dag import API.

What happened instead

Kubo hangs with no output.

Reproduction steps

1. Download the ZIP archive linked below and deflate it to get test.car
2. ipfs dag import ./test.car
   • You'll know you hit the bug if Kubo simply hangs with no output.

Problematic CAR

The CAR itself: test.car.zip
The output from running car inspect (via go-car) on test.car: test.car.debug.txt

Elaboration

I recently ran into an issue where a CARv1 that I produced could not be added to Kubo via the dag import routine. The problem occurs when I do ipfs dag import and also when I invoke the corresponding HTTP RPC API /v0/dag/import.

As a smoke test, I verified that my node could import CAR files via these interfaces by importing the test fixture found at https://ipld.io/specs/transport/car/fixture/carv1-basic/.

Additionally, I wrote a small utility to read back the CAR myself based on my understanding of the spec, and also attempted to analyze the CAR with with go-car. As far as I can tell from my use of these tools, the CAR is valid.

[hsanjuan]

Does it hang with pin-roots=false ? One of the block must be referencing a CID that is not in the CAR and cannot be found anywhere.

[cdata]

@hsanjuan thanks for the pointer! Indeed, I tried importing the CAR with --pin-roots=false and that seems to work. I haven't verified that a missing reference is the culprit, but it seems plausible to me.

On the one hand, this has highlighted an actual mistake on my part when constructing the CAR.

On the other hand, hanging indefinitely with no feedback seems like an insufficient mechanism to inform me of this problem. I would really like to have been given some indication of what was causing the hold up.

[Jorropo]

On the one hand, this has highlighted an actual mistake on my part when constructing the CAR.

Car isn't as strict as that, this car isn't wrong it just have different semantics than kubo's semantics.
Kubo can return you incomplete cars too, for example if you try to fetch a file in a directory over the IPIP402 gateway you will get a car that contains the proof for your file, so some of the directory blocks and the file you are downloading, but it wont contain other files in the directory for example.

I've talked a bit with encoding semantics of cars in their headers with a few peoples but didn't found anyone interested and neither am I.

[cdata]

Thanks for the additional context @Jorropo

Car isn't as strict as that, this car isn't wrong it just have different semantics than kubo's semantics.

I actually meant to imply that the mistake was mine, because my original intention was for the CAR to contain a complete set of blocks that includes all references that may not already be known to Kubo (this CAR was produced as part of a controlled test).

At any rate, it would be nice to have received some indication from Kubo that there was a missing referenced block which it would attempt to discover elsewhere.

[hsanjuan]

What you ask for is not too easy as "dag import" is one step (which finishes) and "recursive pin" is a second separate step (which hangs), and both steps are mixed under the same API call which only does OK/NOT OK responses.

The concept of timeouts when pinning does not exist in Kubo either, wanted blocks stay indefinitely on want lists.

But as a workaround you can try ipfs --offline dag import .... I think (haven't tried myself), that this will error immediately when attempting to pin.

[cdata]

@hsanjuan thank you for the suggestion of a workaround. The main place I'm encountering this issue personally is via the HTTP RPC API, so unfortunately --offline won't help my case in particular. But, it may be helpful for others.

I appreciate that there may be some complexity that makes this difficult to handle at the level of API consumer UX. Another satisfying resolution for me would be for the documentation to state clearly that this hanging behavior is a possible complication if the CAR does not contain all of the referenced blocks. The main issue for me was that I assumed that as long as I had a valid CAR, the API would behave a certain way; an appropriate warning in the documentation would go a long way towards setting my expectations.

[Jorropo]

So to make this issue an actionable item. Everything is working as intended, it seems this behavior is not what you want. Should we transform this issue in one that would make ipfs dag import --nofetch (where --nofetch would disallow the daemon to fetch missing blocks over the network and error instead) ?
Or should I close this issue ?

[cdata]

Should we transform this issue in one that would make ipfs dag import --nofetch (where --nofetch would disallow the daemon to fetch missing blocks over the network and error instead)

@Jorropo that design would suit my use case well. It was indeed a mistake that I was trying to import blocks with missing references, and at least in my case those blocks would never be found on the network.

If you're okay with such a design, I would be happy for the issue to be updated to reflect such a feature request.

As I said above, I would also be satisfied with documentation updates.

[aschmahmann]

triage:

• this looks like a regression due to fix: deadlock while racing ipfs dag import and ipfs repo gc #9755
• the intended original behavior was to not reach out to the network during ipfs dag import

  • kubo/core/commands/dag/import.go

    Lines 34 to 37 in 40d459b

    	// on import ensure we do not reach out to the network for any reason
    	// if a pin based on what is imported + what is in the blockstore
    	// isn't possible: tough luck
    	api, err = api.WithOptions(options.Api.Offline(true))

• not a particularly high priority fix at the moment, but if anyone is interested in doing so happy to engage here

[lidel]

triage notes:

• may be easier to implement (fix) now that feat: built-in content blocking based on IPIP-383 #10161 landed, it introduced offline path resolver which makes offline mode easier to wire up.
• that being said, our other commands are implicitly online, and require dedicated --offline flag to work in no-network mode, might be good oportunity to cleanup UX here too and align this with other commands

[tennox]

Thanks for this thread - helped me to figure out why my dag import was hanging.

I did also find out that I actually want to upload a CAR which is not including all referenced CIDs - it's not directly related to this but thought if I'm lucky the the implementation decisions also incorporate my use-case (e.g. adding a way to pin all blocks in CAR instead of roots?)