IPNS name resolve fails the first time name.publish and name.resolve are called but works on next call

[imthe-1]

• Version:
  ipfs-core: 0.12.2
  libp2p: 0.33.0
  libp2p-crypto: 0.19.7

• Platform:
  macOS: 12.4
  chrome browser: 104.0.5112.101

• Subsystem:
  IPNS

Severity: High

Description:

we have forked the js-ipfs repo and have added the functionality to pass an optional secp256k1 private key for creating custom IPNS keypairs. This allows us to create IPNS keypairs that are created from a child derived from a seed phrase giving more control to the seed phrase. The repos modified are ipfs-core, libp2p, libp2p-crypto. We are using a downgraded version since we faced issues using the latest versions then. We have tried using current latest version as well but faced this issue(4148).

the setup has a browser running ipfs-core and is connected to a go-ipfs(kubo) node so that propagation of IPNS publishing works properly. The IPNS name.publish works properly and returns the public key hash but the name does not resolves when resolved via the go-ipfs node and strangely if the same process is repeated twice we are able to resolve the IPNS name. Due to the modifications we can pass the same private key again on the second name.publish call.

expected result: the IPNS name should resolve on the first call itself.

libp2p-crypto/src/keys/secp256k1-class.js(modified)

  async function generateKeyPair (bits, optPrivateKey) {
    const privateKeyBytes = optPrivateKey || await crypto.generateKey()
    return new Secp256k1PrivateKey(privateKeyBytes)
  }

creating an IPNS keypair

    const keypair = await this.browserIpfsCore.key.gen(name, {
      type: 'secp256k1',
      optPrivateKey: Buffer.from(privateKey, 'hex'),
    });

IPNS publish using the keypair generated above

    const resp = await this.browserIpfsCore.name.publish(ipfsCID, {
      key: keypair.id,
      resolve: false, // since it mostly fails
      lifetime: '2400h',
    });

resolving via the go-ipfs node running on an instance

    for await (const name of this.ipfsHttpClient.name.resolve(ipnsPubKeyHash)) {
      cid = name;
    }

modified ipfs-core module w/ libp2p and libp2p-crypto changes: https://www.npmjs.com/package/@mdip/ipfs-core

[BigLep]

2022-09-02 triage conversation: can you confirm if this happens with the default key type as well?
Please also create a repo with a runnable reproducible case to help with the debugging. Thanks.

[github-actions]

Oops, seems like we needed more information for this issue, please comment with more details or this issue will be closed in 7 days.

[github-actions]

This issue was closed because it is missing author input.

[imthe-1]

@BigLep it resolves when using the default key. I have pushed the runnable scenario to the repo - https://github.com/imthe-1/keychain-ipns-sample/tree/master. The console can be checked for the different steps. Please refer to keychain-ipns-sample/src/App.js for the issue specific code sample.
let me know if you face any issues.

[imthe-1]

@BigLep @lidel can you reopen the issue.

[imthe-1]

@BigLep it resolves when using the default key. I have pushed the runnable scenario to the repo - https://github.com/imthe-1/keychain-ipns-sample/tree/master. The console can be checked for the different steps. Please refer to keychain-ipns-sample/src/App.js for the issue specific code sample. let me know if you face any issues.

@BigLep @lidel please let me know if you need any further info. Also, can you please reopen the issue?

[imthe-1]

@BigLep in my setup i've used these flags: --enable-pubsub-experiment --enable-namesys-pubsub and have made the following changes in the ipfs config. sharing this so that there are no issues running the sample(you probably dont need that but sharing it anyways:slightly_smiling_face:). please let me know if any further info is required.

  "Addresses": {
    "Swarm": [
      "/ip4/0.0.0.0/tcp/4001/ws",
      "/ip6/::/tcp/4001/ws",
      "/ip4/0.0.0.0/tcp/4002",
      "/ip6/::/tcp/4002",
      "/ip4/0.0.0.0/udp/4003/quic",
      "/ip6/::/udp/4003/quic"
    ],

  "API": {
    "HTTPHeaders": {
      "Access-Control-Allow-Origin": [
        "http://localhost:3000"
      ]
    }
  },

[BigLep]

2022-10-07 triage conversation: @imthe-1: do you see these same IPNS issues when you use the default encryption in IPNS? We want to see how much of the issue here is related to your forking and using non-default keys.

[imthe-1]

@BigLep we have not modified the default encryption and it is the same in our implementation as in the standard ipfs-core. Strangely, in our implementation when we repeat the execution of the deterministic IPNS keypair creation and linking(publishing) flow, the IPNS public key hash resolution works and returns the CID but it fails the first time.

I hope i am able to answer your query, if not please let me know.

[github-actions]

Oops, seems like we needed more information for this issue, please comment with more details or this issue will be closed in 7 days.

[BigLep]

I removed the stale label since the ball is in the maintainer court currently.

[imthe-1]

@BigLep please let me know if any further information is required that might help with the troubleshooting and execution.

[imthe-1]

hi @BigLep can you please share some updates on this?

[lidel]

@imthe-1 it sounds we are discussing multiple issues here, makes it difficult to follow, but hopefully below is useful.

lmk if below understanding is correct:

• (A) seems that your original issue (kubo failing to resolve on the first try, because you only published on PubSub) got resolved when you updated to the latest version of ipfs-code that supports publishing IPNS records on both PubSub and DHT.
  • mind confirming this is no longer a problem?
• (B) you published something on IPNS but it does not resolve on ipfs.io gateway
  • this is difficult to reason about, too many variables – public gateways have additional layers in front of Kubo. needs smaller repro
  • are you experiencing the same issue when running your own Gateway in LAN and/or remotely somewhere in WAN?
• (C) Kubo returns old version of your IPNS records, even after you published an update.
  • is this CLI / RPC, or Gateway behavior? (or both?)
  • afaik Kubo (go-namesys to be specific) has a hard-coded cache for /ipns/* that won't look for newer records if it has a cached one that is younger than 1 minute. This minimal caching window overrides any ttl/lifetime you may set up in your record, so in Kubo, for now, you can't get updates for /ipns/ more often than once per minute.
  • there is wip work to make this more flexible, and use TTL from IPNS record (if present) – related fixes tracked in fix: honour --ttl flag in 'ipfs name publish' kubo#9471 and Gateway: Cache-Control, max-age=ttl, ETag for /ipns requests boxo#329

[imthe-1]

@lidel thanks a lot for looking into it!

lmk if below understanding is correct:

• (A) seems that your original issue (kubo failing to resolve on the first try, because you only published on PubSub) got resolved when you updated to the latest version of ipfs-code that supports publishing IPNS records on both PubSub and DHT.

  • mind confirming this is no longer a problem?

we got to know that since the kubo node has not subscribed to the topic in the first resolve, it fails and then 2nd resolve call succeeds but we are still struggling a bit to get IPNS working.
to share some background, we are publishing to IPNS from the browser and have used this example as reference(browser-ipns-publish). we have modified the ipfs-core, libp2p and libp2p-crypto libraries to accept an optPrivateKey when creating secp256k1 IPNS keypairs so that detereministic IPNS keypairs can be created, important for our use-case. then we ran into the problem of 1st resolve calls always failing and sometimes when they do resolve(after publish to a new value) they resolve to the older value. to summarise point (A) we are okay with 1st resolve failing and will handle it but not being able to resolve to the latest value is critical for us(apologies for not being able to highlight this properly).

we plan to update the record on IPNS every few minutes in the start of our flow and then the updates happen based on events which is difficult to predict and could be between a gap of few seconds as well.

• (B) you published something on IPNS but it does not resolve on ipfs.io gateway

  • this is difficult to reason about, too many variables – public gateways have additional layers in front of Kubo. needs smaller repro

as asked by @achingbrain we went back and tested with the latest version of ipfs-core and could not resolve the IPNS name on ipfs.io gateway. we can skip this one since we eventually got it working.

• are you experiencing the same issue when running your own Gateway in LAN and/or remotely somewhere in WAN?

we have not tried this @lidel.

• (C) Kubo returns old version of your IPNS records, even after you published an update.

  • is this CLI / RPC, or Gateway behavior? (or both?)

the setup is same as ipns-browser-publish example and we tried resolving it using ipfs-http-client connect to our kubo node so RPC behaviour.

• afaik Kubo (go-namesys to be specific) has a hard-coded cache for /ipns/* that won't look for newer records if it has a cached one that is younger than 1 minute. This minimal caching window overrides any ttl/lifetime you may set up in your record, so in Kubo, for now, you can't get updates for /ipns/ more often than once per minute.

one minute is good enough for us and we can implement work-arounds for this but even when i published after a few minutes the resolve call returned the older value only. I tried multiple times as well but it kept returning older values. it is critical for us to have the latest value.

• there is wip work to make this more flexible, and use TTL from IPNS record (if present) – related fixes tracked in fix: honour --ttl flag in 'ipfs name publish' kubo#9471 and max-age, ETag for /ipns requests kubo#1818 (comment)
  cool!

[imthe-1]

@BigLep @lidel guys if we can close it this week it will help us tremendously:pray:

[imthe-1]

@lidel we did try to resolve using our gateway that we setup on a remote instance(coz we really want to get this implemented into our system:slightly_smiling_face:). it resolves for a period of time and then stops resolving kinda intermittent. this is the IPNS name: 16Uiu2HAmEmQ5HoGsx5X7tqeWyow9fZ6LaUV9kvQGAEX6nfRQSaHd

Also, the resolving to older values scenario is still there.

[github-actions]

Oops, seems like we needed more information for this issue, please comment with more details or this issue will be closed in 7 days.

[imthe-1]

@BigLep @lidel please let me know if any other information is required. @BigLep can you please remove the stale label. thanks!

[github-actions]

Oops, seems like we needed more information for this issue, please comment with more details or this issue will be closed in 7 days.

[github-actions]

Oops, seems like we needed more information for this issue, please comment with more details or this issue will be closed in 7 days.

[github-actions]

Oops, seems like we needed more information for this issue, please comment with more details or this issue will be closed in 7 days.

[github-actions]

This issue was closed because it is missing author input.

[BigLep]

Reopening - we need to find a way to prevent these kind of items from getting closed. I'll put down in need of maintainer input.

[SgtPooki]

js-ipfs is being deprecated in favor of Helia. You can #4336 and read the migration guide.

Please feel to reopen with any comments by 2023-06-02. We will do a final pass on reopened issues afterward (see #4336).

Assigning to @achingbrain to answer whether this issue is already resolved in Helia, or if this issue needs to be migrated to that repo!

[achingbrain]

@helia/ipns supports using secp256k1 keys to publish IPNS records, and also publishing them to DHT peers so this should all work.

@imthe-1 if you are still struggling with this issue can you please port your code to use Helia (see @SgtPooki's comment above for migration guide links) and if it still doesn't work, please open an issue on the @helia/ipns repo with a link to a repro case and we can continue the conversation there.