Optimize memory usage of hosts filters

[ameshkov]

It seems (#713 #712) people tend to load REALLY HUGE hosts lists, and with our current approach of keeping everything in memory, it leads to serious memory usage.

Here's what I suggest:

1. Serialize parsed filters to a temporary file on the disk
2. Keep only index structure in-memory (and a small LRU cache with rules which are often used)
3. Deserialize rules on-demand

I think there are some existing libraries using more or less the same approach.

Useful libraries:

• https://github.com/recoilme/pudge
• https://github.com/peterbourgon/diskv
• https://github.com/recoilme/slowpoke

[no-replies]

Just a quick Update & a Question.

Its not just large filter lists but when the yaml gets large as it has:

BLOCKLIST (Entries added here from gui block command)
e.g ||baddomain.com^

REGEX
e.g /^(.adwords)./ #

WHITELIST
e.g @@||gooddomain.com^$important

It be better if GUI had three sections (one for each) and saved in files rather than yaml.

In the meantime I assume that as Filters and hosts blocklists supports basic adblock rules and hosts files syntax that I could copy the entries from YAML, put them into files and add to Filters and hosts

[skynet01]

Since we are on topic of host files, would also be nice if you can add host aliases like in normal host files ex:
192.168.1.1 router.loc router.com

I have many local resources that have different names. Thank you all again :)

[ameshkov]

@skynet01
Yeah, this one will also be addressed by using https://github.com/AdguardTeam/urlfilter

[skynet01]

nice, is this already live? If yes how do we use it.

[ameshkov]

@skynet01 not yet ready, but it will be soon

[hesss5]

by default AdGuard Simplified Domain Names filter enable its use around 20mb
after enable this two filter( hpHosts - Ad and Tracking servers only,MalwareDomainList.com ) it goes to 42mb,imagine if add more filter!
its can not optimized with bellow link?
https://github.com/jedisct1/fpst

[ameshkov]

No, I am pretty sure this library has nothing to do with memory usage optimization.

The memory footprint of the AdGuard SDN at the moment is about 5-8MB, and for hosts-file based filter lists it's even less. The problem is that golang's GC is pretty greedy and if there's memory available, it prefers not to free it.

You can try adding debug.SetGCPercent(10) to main.go and rebuilding AGH, and see if this helps.