IPNS very slow #3860
Comments
|
@nezzard This is generally a known issue, but providing more information is helpful. Are you resolving from your local node? Or are you resolving through the gateway? |
|
@whyrusleeping throw local, but sometimes gateway faster, sometimes local faster |
|
@nezzard when using locally, how many peers do you have connected? ( DHT based ipns isnt as fast as something more centralized, but you can generally cache the results for longer than ipfs caches them. We should take a look at making these caches more configurable, and look into other ipns slowdowns. When you say its 'very slow', what time range exactly are you experiencing? 1-5 seconds? 5-10, 10+ ? |
|
@whyrusleeping Sometimes it's really fast, sometimes i have Ipfs swarm |
Kubuxu
added
topic/ipns
|
Which ipfs version are you running? |
|
@nezzard What tool are you using in your screenshot? I've seen it many times in the forums but I can't find it anywhere. |
|
@kikoncuo it's a tool from cloud service like dropbox |
|
@Kubuxu The last at the time |
|
@nezzard I meant the tool which you took the screenshot from, my bad |
|
@Kubuxu this tool inside the program yandex disk |
|
So let me tell you some tweaks I've made which has helped quite a bit.
|
|
We can probably add flags to the ipfs name resolve api that allow selection (per resolve) of the query size parameter, and also to say "just give me whatever value you have cached". Both of those would be simple enough to implement without actually having to change too much |
|
Another thing we could do it have a command that returns ipns results as they come in, and then when enough comes in to make a decision, says "This is the best one". This way you could start working with the first one you receive, then when the right one comes in, switch to using that |
|
I have some trouble as well with IPNS. I have a linux box and a windows box on the same LAN running ipfs 0.4.9 and I can't resolve IPNS addresses published from the other side, even after several minutes. I have 400 peers connected on one side, 250 on the other. @cpacia your changes are in a branch somewhere ? That looks like a very handy addition for my project. |
|
Answering to myself, the fork is here: https://github.com/OpenBazaar/go-ipfs @whyrusleeping any idea how I can debug this issue ? |
|
@MichaelMure you cant resolve at all? Or its just very slow? |
|
Sometimes it just take times before being able to resolve and once it has been resolved once it works properly. But in this case it didn't resolve at all even after 30 minutes. It might be another issue but without a way to find out what's going on in ipfs, well ... |
|
I think ipns is very bad for use It's load for 15 - 20 seconds |
|
I'm also experiencing massive resolution times with IPNS. Same behavior over here - the first resolution can take multiple minutes, then once a it's loaded, I can refresh the content in under a second. If I leave it for a few minutes, then do another refresh, and that request cycle repeats the same behavior. The "cache" for the resolution only appears to stay warm for a short period of time. |
|
I'm using a CNAME with _dnslink, for what it's worth. Content is at www.ember-cli-deploy-ipfs.com |
|
Ipfs is unusable. I have the daemon running on both of my computers inside a LAN and one "serving" one file (a video) that the other doesn't have. When I try to access that video from the pc that doesn't have the file, using localhost:8080/ipfs/... on my browser, the video is stopping and taking huge amount of times to load. HUGE amount of time in such a way that I can't watch the video. |
You should ask for a refund 👍 |
|
@kesar I mean this out of love. @jbenet (Juan Benet) says that it is going to release us from the backbone but currently ipfs network performance is very weak. |
|
It took me more than a minute to resolve the domain published by my own computer... And it's not the DNS resolution it hangs at resolving the actual IPNS entry. |
|
We're working on fixing some low hanging fruit in the DHT that should alleviate this: libp2p/go-libp2p-kad-dht#88. You can expect this to appear in a release month or so (4.12 or 4.13). We're also working on bypassing the DHT for recently accessed IPNS addresses by using pubsub ( #4047). However, that will likely remain under an experimental flag for a while as our current pubsub implementation is very naive. |
|
Hi @dboreham OK, so I ran So, first on the idle system: Lots of traffic to/from 4001 and 10001. These consist entirely of SYN RST and ACK packets, from localhost to itself, IPv4 and IPv6, to/from ports 4001 and 10001 -- maybe a dozen every second, all the time. No data is being transferred in these packets. They're empty. The timing is stochastic, the time intervals are irregular. It would appear that something is opening sockets to the ipfs daemon and then resetting and closing them immediately. Over and over and over. The only something should be the ipfs daemon itself, so this is ... bizarre ... (in 5 minutes, I managed to capture exactly one packet between my port 5001 and an outside-world host. So my ipfs daemon is interacting with the outside world, just not very much, when sitting idle). Next, I see dozens of ICMP port-unreachables emanating from port 4001 to 169.254.x.x addresses which tells me that local addresses are incorrectly leaking out into the IPFS protocols. (I don't use 169.254, I use 10.x.x.x for the internal LAN, so these addrs are not mine.) There were over a dozen distinct different 169.254 addrs visible. I see MDNS responses exactly 10 seconds apart, to within a millisecond. The response includes my self key CID. The response packets always include all local host IP's, including the IP's of the various LXC containers on the host; all have 10.x.x.x entries. They're all marked Next, I run my IPFS client. It generates a small handful of and exactly 90 seconds later: That is a totally boring totally ordinary 200 OK response. If you read the above, you may spot a There is no traffic on port 53; nothing is making any DNS queries on the system. So: to summarize: nothing unusual, except for a fairly large number of completely bogus SYN/RST packets that do nothing at all. Except for this garbage, the ipfs daemon is effectively completely idle, sending nothing, receiving nothing. The 90 second timer, whatever it is, is in the ipfs daemon. |
|
@linas A few thoughts on what you've run into:
I'm pretty sure this is just you hitting the DHT timeout. DHT publishes are taking a while and this is a known issue.
What's happening here is that namesys has an internal cache that it uses that keeps published records for 1 minute by default. This means that local resolution would be non-instantaneous if you waited more than a minute between publishing and resolving. If you know that the latest record is on your machine (e.g. because you published it), you can always resolve quickly by just passing the |
|
Hi @aschmahmann continuing the conversation:
Isn't this that issue? Or is there some other issue? Do you have the issue # for it? Anyway, what you say is symptomatic of the general confusion here. If I both publish and resolve locally, I cannot imagine any valid reason why either operation would stall, for any reason. By "publishing locally", I mean that I contact the ipfs demon running on localhost, with the URL For resolution, similar remarks apply: if I am trying to resolve a name that I published locally, just seconds earlier, I see no reason at all why the ipfs daemon cannot instantly respond to the
I think you are talking about the TTL parameter. Right now, the docs are silent about the default value of the TTL parameter. Maybe it's one minute. However, if I change the TTL parameter to, say, 5 minutes, then the timeout is still 60 seconds. So this "explanation" cannot be correct. In any case, since my local daemon is the authority on the publish, the TTL would not apply to it. The TTL is intended for everyone else, and never for the authority (because, duhh, the authority always knows the answer, and never needs to ask anyone for it!)
There is no "offline" parameter in the documentation. Please look at the docs: https://github.com/ipfs/interface-js-ipfs-core/blob/master/SPEC/NAME.md#nameresolve -- the only parameters are |
Yes. For example see above, #3860 (comment). There are probably some related issues in go-libp2p-kad-dht as well.
What you are running into here is the difference between a synchronous and asynchronous network operation. Two use cases: Synchronous: I'd like to make some data accessible to you. I'd like to be confident that once I'm done running Asynchronous: I'd like to use IPNS as a way to address my content in a distributed way and will be doing lots of local work. If it takes a while for data to get published that's fine. If we ever wanted to know whether the data was publicly accessible we would just ask a friend if they could find it, or do a network query that explicitly ignores our local state. The API is currently designed for the synchronous use case which is plenty useful and valid. If you'd like support for the asynchronous use case that's a new issue you should feel free to open (or even implement).
This is not correct and does not fully encompass the current and future use cases for IPNS. While IPNS is single writer, there are people who share IPNS keys between devices (IPNS keys are not restricted to the peerID keys, they can be arbitrary asymmetric key pairs). As long as they only edit one device at a time then they can feel reasonably confident in using IPNS to sync say a folder between multiple devices. Again, there is a flag that only looks at your local repo which you can use to achieve your goals.
I've already addressed why the TTL applies to everyone (unless you want to use --offline). There are options to modify the DHT timeout if you'd like or to stream the results. Again if you are only concerned about your machine there is an option available for you.
It's not in the link you mentioned to js-ipfs-core (this is the go-ipfs repo) which might be a good point. @vasco-santos any idea where the documentation for the flag is? In go-ipfs the documentation is there, but perhaps it's a bit confusing (if you have suggestions I'd encourage opening an issue or PR to fix it). The CLI docs https://docs.ipfs.io/reference/api/cli/#ipfs show that you can pass |
|
Ah! now we are getting somewhere! I will try a multi-part reply, as this is getting long. First: sync vs. async. I claim that sync "never makes sense". Either you are doing a publish to What other scenarios are there? |
|
Part 2: current and future use cases for IPNS. My use-case is this: I want to share keys with dozens or hundreds of others (who are doing compute of some kind, and are publishing results on ipfs). With the current IPNS design, I would need to share millions, up to 100 million keys with them. Why such an obscenely large number? Cause in my current infrastructure, backed by postgres, I have an extremely sparse dataset, of which only a few million or tens-of-millions of entries are non-zero (they follow a Zipfian distribution in size, and connectivity). Each one of my entries is immutable, thus has a unique content hash, but I need to hang time-varying data off of it (probabilities, counts, sensor-readings, etc.) and so I need dozens or hundreds or more processes cooperating to publish the latest readings hanging off of the immutable sign-posts that have been set up. This currently works fine on PostgreSQL and I can hit several K publishes/second (which is slower than I'd like, but whatever). I can point you at the unit tests. With IPNS/IPFS.. ugh. Clearly, having to share millions of keys with hundreds of peers seems ... the opposite of "decentralized". In some ideal world, enhanced IPNS would do this: instead of what it currently does: That way, I could share only one key with peers, and whenever those peers needed to look up the CID associated with some "well-known hash" they can just do that. |
There are plenty of possible scenarios since what you are doing is giving the user an extra piece of information (that the publish completed) that you wouldn't have otherwise. Simple example: If I want to write code in JS but benefit from features or performance of go-ipfs I might have my application spin up a go-ipfs daemon and talk to it over the HTTP API. In this case my application would want to know when it is "safe to close". @linas Your IPNS use case of needing millions of keys seems highly suspicious to me and is not relevant to this issue. If you could post on https://discuss.ipfs.io/ that would be great, the conversation could be continued there. |
!? Give one real example. The history of computer science has been the elimination of sync writes, from the 1970's onward. The invention of interrupts to avoid sync writes of machine status. The invention of caches to avoid sync writes to DRAM. The invention of DMA to avoid sync writes by I/O. The invention of register write-back queues. And that's just hardware. The invention of mutex locks. The invention of software message queues. The invention of publish/subscribe. All of these were driven by the need to eliminate the stalls associated with sync writes. All of these have become very popular precisely because because they avoid sync execution. I'm kind of frustrated: one didn't have to make sync the default. It could have been async by default. In my code, I have to launch and dettach a thread for each IPNS publish, because I can't afford to wait for a 60/90 second timer to pop in the IPFS code. This bug itself has been open 2.5 years, and has accumulated cruft of me-too's. All discussion on all discussion forums have all basically concluded that "IPNS is broken and lets hunker down and wait until it's fixed". And you are trying to tell me that, no actually, there is a simple fix, nearly trivial -- to change the default (an undocumented API flag) , and everything will work for everybody? Holy cow! Change the default! I mean, I know what I wrote here sounds hostile, but I don't know how else to put it. You're trying to tell me that "its a feature not a bug", but you really got to consider that pretty much the rest of the world thinks its a bug ... |
We were missing the documentation for that option, just created a PR to add it ipfs/js-ipfs#2569. Thanks for the heads up!
Regarding the CLI in JS, it should also work with |
Yes it does, I think if you actually read through this issue you will see that the problem you're complaining about is not the problem this issue is about. Additionally, you have already been told how to get around this problem. As you can see someone has already put up a PR for documentation improvement (thanks @vasco-santos). If you have further complaints unrelated to this issue I'd recommend opening a new issue, this one is already pretty long. |
This change doesn't seem unreasonable (i.e. having go-ipfs spin up a background thread instead of asking the client to do so, which is also a viable option). However, it's not necessarily the right move as it does end up being a UX change that ends up with people issuing complaints like:
We end up with the confusion above anyway with IPFS (since if someone is trying to find data published in the DHT for the first time they have to wait for the publish to complete) so perhaps it's worth doing, but it definitely has tradeoffs. |
|
I guess I deleted my comment about the same time you were writing a response, because I was thinking the --offline option might be the intended solution: at least it works on the command line. But errm no:
I want to really publish, just don't want to wait for it. Then I wrote ferristseng/rust-ipfs-api#62 ... and then I wondered if the answer to that is to just post the http request to the daemon and hang up, don't wait for a response if you don't want to? But if ipfs daemon will give up on publishing as soon as the client hangs up, it won't work. And if ipfs daemon is running as a system service (which IMO it always should be, on as many systems as possible), it really had better be able to handle many simultaneous requests. So ipns publishing should be pooled: one thread should probably be able to publish many names simultaneously; and if not, then spin up a few more. (I was thinking of using ipns a lot: perhaps every uniquely-addressable largish category of data that I'm storing should have its own name, because there's no other mechanism to hold a handle to mutable data, is there? CIDs change with every data mutation, no matter how deep you bury the mutation in your data structure. E.g. I'm trying to write a database, so I think each database instance needs its own IPNS name, and each insert or update of the database will result in the need to update the IPNS record. Is that not the right way to use it?) If we make a comparison to DNS publishing: you probably use a web UI at your hosting provider, right? If you want to change the IP address/dnslink/mx record or whatever that your domain points to, everybody knows it can take hours for DNS changes to propagate worldwide, but that does not mean you expect the web UI to keep showing a spinning animation that whole time and refuse to do anything else. |
|
The time that name_publish() takes is also unpredictable. With ipfs 0.7.0 (with option --enable-namesys-pubsub) I'm seeing times from around 30 sec to a couple of minutes, when it succeeds. I'm trying to do it once per minute, in a cron job; so if it takes longer than that, I end up with overlapping processes. When it hangs, the process that is trying to do that gets hung indefinitely. (I thought it was perhaps racy too, but that turned out to be my fault.) The ipfs daemon process needs to deal with all possible problems, and return immediately from this API call. But because it's broken, I need to find a way to put a timeout into my application. This really shouldn't be the application's problem. Perhaps this is one reason orbitdb uses pubsub directly, but not ipns? I don't quite understand how they got around the naming and discovery problem yet. (Yes I read the field manual: https://github.com/orbitdb/field-manual/blob/master/02_Thinking_Peer_to_Peer/01_P2P_vs_Client-Server.md has a clue, that they believe in using one pubsub channel for queries and another for responses, so that queries can be distributed.) But it seems to me that ipns ought to be the ideal way to name a database instance, at least in the one-to-many publishing case like I'm building. It looks like OrbitDB is designed for collaboration, a true distributed system where each peer contributes something; but the one I'm trying to build is simpler, so I don't need the CRDT overhead (I especially don't want to sign each entry separately), but I do want to publish a structured feed, not just individual byte arrays. When there is one authoritative source for a dataset, IPNS makes sense. When there are multiple collaborating peers providing different portions of the data, it could still make sense for each peer to name his own subset of the data with IPNS. I agree with linas on his points. (Well maybe it's unreasonable to expect ipns to scale to millions of records at this time, but eventually, why not?) aschmahmann as to your point about needing a sync api in case you turn your phone off or shutdown the ipfs daemon on your end: I don't really agree that it's a good use case for ipfs. A long-running daemon has to be running somewhere. Either you have an arrangement for pinning elsewhere and a way to find out when the pinning is done, or else you keep your own daemon running and available on the network all the time. I continue to think ipfs should be a system daemon, like a web server, not bundled many times into every application, because what if you multi-task? You don't really want multiple application-specific daemons running, do you? And I know that when I publish a video to d.tube for example, I'm not paying them to store it: I'm providing the storage myself. Therefore I keep my ipfs daemon running 24/7 on my home computer(s). With that architecture in mind, I'm writing my database using the http API, as a short-lived process that will talk to my local daemon, ask it to do things on its behalf, and then exit. The ipfs daemon has the responsibility to maintain the integrity and availability of the data. It should never shut down, and it should keep juggling all the tasks at once, that's it's job. |
|
This is still a problem after 6years of this thread. Publish mostly takes upwards of 5mins. I was hoping there would be a solution from the comments of @ec1oud . I too want to publish the data as a background job instead of having the client to wait. I've tried finding if a solution exists for this and cant seem to find it. There are a lot of different posts raising this issue though. https://discuss.ipfs.tech/t/why-ipns-is-so-slow/2161 I've also updated the LowHigh/HighWater to low values and set peers from Enabling this Routing.AcceleratedDHTClient does help quite a bit |
|
Are you seeing the same lag with the |
|
Hey @DougAnderson444 , |
|
|
Hey, @DougAnderson444 I've seen this but it doesn't really tell you how to enable it. Did this https://github.com/ipfs/kubo/blob/master/docs/config.md#ipnsusepubsub and then I'm assuming I don't have to do https://github.com/ipfs/kubo/blob/master/docs/config.md#pubsub since its depricated. I disabled the timing my publish doesn't seem to change. Same thing from the RPC api.
|
|
Maybe it would be better to use PutValue, GetValue? I checked. It works fast. |
|
Could you share the doc link for this? cant find it here https://docs.ipfs.tech/reference/kubo/rpc/ |
|
I'm talking about dht putvalue and getvalue. Does Kubo have a direct api to dht? |
|
Seems to be depricated https://docs.ipfs.tech/reference/kubo/rpc/#api-v0-dht-put |
|
And what's better than that? |
and others
Hi, is it normal, that ipns loading very slow?
I tried to make something like cms with dynamic content, but ipns to slow, when i load site via ipns, first loading is very slow, if after that i reload page it's load quickly. But if i reload after few minutes, it's again load slow.
The text was updated successfully, but these errors were encountered: