Large blacklists freeze the UI for importing a blacklist

[Avamander]

It would be really nice if larger blacklists (around 10 000 entries) were paginated in the blacklist import view. Even larger blacklists (imported malicious domain lists) cause the page to not render and crash.

[okybaca]

Is there some error message in UI or in log?

[Avamander]

At some point the page is just too large for the browser to render it properly.

[GamePlayer-8]

Is there some error message in UI or in log?

Ops!

Message: /opt/yacy_search_server/htroot/sharedBlacklist_p.html
javax.servlet.ServletException: /opt/yacy_search_server/htroot/sharedBlacklist_p.html
	at net.yacy.http.servlets.YaCyDefaultServlet.handleTemplate(YaCyDefaultServlet.java:886)
	at net.yacy.http.servlets.YaCyDefaultServlet.doGet(YaCyDefaultServlet.java:307)
	at javax.servlet.http.HttpServlet.service(HttpServlet.java:687)
	at javax.servlet.http.HttpServlet.service(HttpServlet.java:790)
	at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:799)
	at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:554)
	at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
	at org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:772)
	at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:560)
	at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)
	at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:235)
	at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1624)
	at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:233)
	at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1440)
	at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:188)
	at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:505)
	at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1594)
	at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:186)
	at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1355)
	at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
	at org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:234)
	at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)
	at net.yacy.http.CrashProtectionHandler.handle(CrashProtectionHandler.java:33)
	at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)
	at org.eclipse.jetty.server.Server.handle(Server.java:516)
	at org.eclipse.jetty.server.HttpChannel.lambda$handle$1(HttpChannel.java:487)
	at org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:732)
	at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:479)
	at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:277)
	at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:311)
	at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:105)
	at org.eclipse.jetty.io.ssl.SslConnection$DecryptedEndPoint.onFillable(SslConnection.java:555)
	at org.eclipse.jetty.io.ssl.SslConnection.onFillable(SslConnection.java:410)
	at org.eclipse.jetty.io.ssl.SslConnection$2.succeeded(SslConnection.java:164)
	at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:105)
	at org.eclipse.jetty.io.ChannelEndPoint$1.run(ChannelEndPoint.java:104)
	at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:338)
	at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:315)
	at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:173)
	at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:131)
	at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:409)
	at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:883)
	at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1034)
	at java.base/java.lang.Thread.run(Unknown Source)

[GamePlayer-8]

Example blocklist used: https://codeberg.org/api/packages/GamePlayer-8/generic/denylist-dns/rolling/domains.txt (56 MB).

[GamePlayer-8]

After the error, YaCy couldn't return a valid HTTP webpage.

In the container logs there's:

yacy-1  | I 2024/04/29 07:24:18 FILEHANDLER * Processing of multipart/form-data request failed. Early EOF
yacy-1  | W 2024/04/29 07:24:33 transferRWI * multipart/form-data; boundary=GHmAQNANlceFlzjivrsZMBSwD3dTsk29vh5B post or env is null!
yacy-1  | W 2024/04/29 07:24:33 transferRWI * multipart/form-data; boundary=GHmAQNANlceFlzjivrsZMBSwD3dTsk29vh5B post or env is null!

but I don't know if it's related to the issue as well.

[GamePlayer-8]

Current solution is to use PiHole / AdGuard as DNS for YaCy or outgoing proxy with filtering.

[Avamander]

I tried that, but it still pollutes index view unfortunately. (And there's no easy "blocklist and delete from index" button there)

[GamePlayer-8]

In the Index Administration menu there's an option to delete the url from the index and in filters you can blocklist it through regex (not always work tho).

Also DNS block wouldn't work when using P2P public network since it would only block Local Indexing.

[GamePlayer-8]

But having a little blocklist button for logged-in administrator next to the unwanted content would be nice.

[Avamander]

In the Index Administration menu there's an option to delete the url from the index and in filters you can blocklist it through regex (not always work tho).

Pretty sure it's the opposite, there's a button to blacklist a domain in index search/viewer but you have to delete using regex under index management. There's no button in the index explorer (that shows an overview of all the domains).