-
-
Notifications
You must be signed in to change notification settings - Fork 429
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Crawler load ques lost sync when crawl rate drops very low. #641
Comments
Thread dump 100 Occurrences: 100 Occurrences: 100 Occurrences: 95 Occurrences: 92 Occurrences: 89 Occurrences: 83 Occurrences: 76 Occurrences: 73 Occurrences: 70 Occurrences: 69 Occurrences: 67 Occurrences: 57 Occurrences: 56 Occurrences: 55 Occurrences: 44 Occurrences: 38 Occurrences: 37 Occurrences: 32 Occurrences: 24 Occurrences: 23 Occurrences: 15 Occurrences: 15 Occurrences: 14 Occurrences: 14 Occurrences: 10 Occurrences: 10 Occurrences: 10 Occurrences: 10 Occurrences: 8 Occurrences: 8 Occurrences: 6 Occurrences: 5 Occurrences: 5 Occurrences: 4 Occurrences: 4 Occurrences: 4 Occurrences: 3 Occurrences: 3 Occurrences: 3 Occurrences: 3 Occurrences: 3 Occurrences: 3 Occurrences: 3 Occurrences: 3 Occurrences: 2 Occurrences: 2 Occurrences: 2 Occurrences: 2 Occurrences: 2 Occurrences: 2 Occurrences: 1 Occurrences: 1 Occurrences: 1 Occurrences: 1 Occurrences: 1 Occurrences: 1 Occurrences: 1 Occurrences: 1 Occurrences: 1 Occurrences: 1 Occurrences: 1 Occurrences: 1 Occurrences: 1 Occurrences: 1 Occurrences: 1 Occurrences: 1 Occurrences: 1 Occurrences: 1 Occurrences: 1 Occurrences: 1 Occurrences: 1 Occurrences: 1 Occurrences: 1 Occurrences: 1 Occurrences: 1 Occurrences: 1 Occurrences: 1 Occurrences: 1 Occurrences: 1 Occurrences: 1 Occurrences: 1 Occurrences: 1 Occurrences: 1 Occurrences: 1 Occurrences: 1 Occurrences: 1 Occurrences: 1 Occurrences: 1 Occurrences: 1 Occurrences: 1 Occurrences: 1 Occurrences: 1 Occurrences: 1 Occurrences: 1 Occurrences: 1 Occurrences: 1 Occurrences: 1 Occurrences: 1 Occurrences: 1 Occurrences: 1 Occurrences: 1 Occurrences: 1 Occurrences: 1 Occurrences: 1 Occurrences: 1 Occurrences: 1 Occurrences: 1 |
Maybe due to SSD slowdown to a pause for 15 seconds. |
Sep2 2024 crawling slowed and loosing sync with loaded. YaCy Version: yacy_v1.940_202405250105_b8479430b6 this thread dump function can find threads that lock others, to enable this function start YaCy with 'startYACY.sh -l' THREADS WITH STATES: BLOCKED Thread= CrawlStacker_pool-1-thread-95 id=368 BLOCKED Thread= CrawlStacker_pool-1-thread-110 id=398 BLOCKED Thread= BusyThread CrawlQueues.coreCrawlJob daemon id=153 BLOCKED Thread= CrawlStacker_pool-1-thread-116 id=410 BLOCKED |
Ubuntu 24.04.1 YaCy Version: yacy_v1.940_202405250105_b8479430b6 this thread dump function can find threads that lock others, to enable this function start YaCy with 'startYACY.sh -l' THREADS WITH STATES: BLOCKED Thread= CrawlStacker_pool-1-thread-82 id=269 BLOCKED Thread= BusyThread CrawlQueues.coreCrawlJob daemon id=150 BLOCKED Thread= CrawlStacker_pool-1-thread-89 id=283 BLOCKED Thread= CrawlStacker_pool-1-thread-66 id=237 BLOCKED |
Crawler load ques lost sync when crawl rate drops very low.
Loader looses track and keeps increasing. Crawler ppm 0-3 ppm.
A restart will clear it.
crawl_for_525982_start_points
Queue Size
[Local Crawler] 841,863 Pause this queue
Limit Crawler 721,055 Pause this queue
[Remote Crawler] 0 Pause this queue
[No-Load Crawler] 0 Pause this queue
[Loader] ([20,000] 4,111
`YaCy Version: yacy_v1.930_202404051704_de941c6fe
Assigned Memory = 15728640000
Used Memory = 7949768216
Available Memory = 7778871784
this thread dump function can find threads that lock others, to enable this function start YaCy with 'startYACY.sh -l'
THREADS WITH STATES: BLOCKED
Thread= CrawlStacker_pool-1-thread-24 id=139 BLOCKED
at app//net.yacy.kelondro.index.BufferedObjectIndex.has(BufferedObjectIndex.java:182) [synchronized (this.backend) {]
at app//net.yacy.crawler.HostQueue.has(HostQueue.java:404)
at app//net.yacy.crawler.HostQueue.push(HostQueue.java:433)
at app//net.yacy.crawler.HostBalancer.push(HostBalancer.java:293)
at app//net.yacy.crawler.data.NoticedURL.push(NoticedURL.java:185)
at app//net.yacy.crawler.CrawlStacker.stackCrawl(CrawlStacker.java:406)
at app//net.yacy.crawler.CrawlStacker.process(CrawlStacker.java:145)
at app//net.yacy.crawler.CrawlStacker.process(CrawlStacker.java:64)
at app//net.yacy.kelondro.workflow.InstantBlockingThread.job(InstantBlockingThread.java:72)
at app//net.yacy.kelondro.workflow.AbstractBlockingThread.run(AbstractBlockingThread.java:82)
at java.base@17.0.11/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539)
at java.base@17.0.11/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at java.base@17.0.11/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
at java.base@17.0.11/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
at java.base@17.0.11/java.lang.Thread.run(Thread.java:840)
Thread= CrawlStacker_pool-1-thread-29 id=149 BLOCKED
at java.base@17.0.11/java.io.DataInputStream.readFully(DataInputStream.java:172)
at app//net.yacy.kelondro.table.ChunkIterator.next0(ChunkIterator.java:85)
at app//net.yacy.kelondro.table.ChunkIterator.next0(ChunkIterator.java:39)
at app//net.yacy.cora.util.LookAheadIterator.next(LookAheadIterator.java:68)
at app//net.yacy.kelondro.table.Table.(Table.java:168)
at app//net.yacy.kelondro.index.OnDemandOpenFileIndex.getIndex(OnDemandOpenFileIndex.java:61)
at app//net.yacy.kelondro.index.OnDemandOpenFileIndex.has(OnDemandOpenFileIndex.java:191)
at app//net.yacy.kelondro.index.BufferedObjectIndex.has(BufferedObjectIndex.java:183)
at app//net.yacy.crawler.HostQueue.has(HostQueue.java:404)
at app//net.yacy.crawler.HostBalancer.has(HostBalancer.java:247)
at app//net.yacy.crawler.HostBalancer.push(HostBalancer.java:287)
at app//net.yacy.crawler.data.NoticedURL.push(NoticedURL.java:185)
at app//net.yacy.crawler.CrawlStacker.stackCrawl(CrawlStacker.java:406)
at app//net.yacy.crawler.CrawlStacker.process(CrawlStacker.java:145)
at app//net.yacy.crawler.CrawlStacker.process(CrawlStacker.java:64)
at app//net.yacy.kelondro.workflow.InstantBlockingThread.job(InstantBlockingThread.java:72)
at app//net.yacy.kelondro.workflow.AbstractBlockingThread.run(AbstractBlockingThread.java:82)
at java.base@17.0.11/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539)
at java.base@17.0.11/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at java.base@17.0.11/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
at java.base@17.0.11/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
at java.base@17.0.11/java.lang.Thread.run(Thread.java:840)
Thread= CrawlStacker_pool-1-thread-26 id=143 BLOCKED
at app//net.yacy.crawler.HostQueue.push(HostQueue.java:434) [synchronized (this) {]
at app//net.yacy.crawler.HostBalancer.push(HostBalancer.java:293)
at app//net.yacy.crawler.data.NoticedURL.push(NoticedURL.java:185)
at app//net.yacy.crawler.CrawlStacker.stackCrawl(CrawlStacker.java:406)
at app//net.yacy.crawler.CrawlStacker.process(CrawlStacker.java:145)
at app//net.yacy.crawler.CrawlStacker.process(CrawlStacker.java:64)
at app//net.yacy.kelondro.workflow.InstantBlockingThread.job(InstantBlockingThread.java:72)
at app//net.yacy.kelondro.workflow.AbstractBlockingThread.run(AbstractBlockingThread.java:82)
at java.base@17.0.11/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539)
at java.base@17.0.11/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at java.base@17.0.11/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
at java.base@17.0.11/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
at java.base@17.0.11/java.lang.Thread.run(Thread.java:840)
THREADS WITH STATES: RUNNABLE`
The text was updated successfully, but these errors were encountered: