Expose stats via REST API

[Harnish]

Allow for viewing the stats via the http interface (make it togglable in the configuration)

[0xERR0R]

Some kind of "fancy" web client with nice tables and diagrams or just text like console output?

[Harnish]

Initially just getting it back in json format would be cool. Then using that as input into fancy graphs and whatnot.

[mihakralj]

+1 for json extract of stats that UI can poll through ajax asynch requests.

See my api enhancement idea in the discussions

[0xERR0R]

To honest, I'm not sure if the current stats implementation is really useful:

• Blocky aggregates stats and keeps all request from the last 1h in memory. After 1h aggregated values will be calculated and stored in memory for 24h. -> this approach is not precise and additional memory is needed. Not ideal for systems with huge request amount
• calculated stats overlap with prometheus metrics. Prometheus + Grafana display more detailed stats and wide time range. Only Top X domains are missing 😞 -> is not possible in Prometheus
• Some people are running multiple instances of blocky (redundancy, fallback) an bare metal or in k8s. Embedded stats are calculated only for one instance -> Prometheus solution is better and shows all stats from all instances

I'm thinking about to remove internal in-memory stats calculation. Some ideas:

• blocky stores each request/response optional (if configured) in the external database (mariaDB/Postgress). New Grafana Dashboard queries the data from the database. Top X domains should be possible
• blocky provides some kind of API like:
  • (websocket for real time-stream)
  • or pushes each request/response via web hook
  • or provides a pull-endpoint.
    => An external tool (aka blocky-webclient ❓ ) collects and aggregates stats. This tool can show the stats and provide other functionality (for example disable/enable blocking...)

What do you think? Suggestions/Ideas are welcome

[mihakralj]

For a first step there needs to be an easy API call to get to the list of recent queries that were allowed/denied - and an API call that can either whitelist denied entry or blacklist allowed entry. Stats (volume, top reqestors, top domains) is relevant, but not essential.

[mihakralj]

Worth reviewing and analyzing telnet-based API calls by pihole:

https://docs.pi-hole.net/ftldns/telnet-api/

[0xERR0R]

For a first step there needs to be an easy API call to get to the list of recent queries that were allowed/denied - and an API call that can either whitelist denied entry or blacklist allowed entry. Stats (volume, top reqestors, top domains) is relevant, but not essential.

There is currently no possibility to create a black or whitelist on-the-fly. Blocky reads white and blacklist from text files and can't write new entries (since the file can be readonly because it will be downloaded). To allow list modification within blocky, some kind of database must be introduced or this will be done in some another application

[mihakralj]

How about keeping the in-memory key-value cache of whitelists/blacklists with an option of a triggered dump (and read) of lists to/from a chosen DB provider (Redis/Elastisearch...)
This would allow Blocky to work both as a stateless (with no preservation of lists if no DB provider is specified) and stateful resolver.

The same could apply to the log stream - keep in-memory short-term buffer with a regular dump to a chosen provider (Telegraf, Logstash...). Want no logs due to privacy concerns? Configure no log provider and nothing is preserved!

Is this a deviation too far from the current direction?

[0xERR0R]

It sounds good, multiple sources for blacklists(file local, http download, database,...). But there is still no ui to create and edit list entries. Do you know some 3rd party tool which would provide some generic ui to achieve that? Otherwise some ui in blocky (or other tool) must be implemented.

[fmarcia]

Or use only local files (as already available) but add a routine to watch and reload them on change (?)

[0xERR0R]

The issue #258 (currently WIP) stores (optional) all queries (request, response, clientIP etc) in the database. This would allow to define reports (e.g. in grafana) based on actual queried domain.

[github-actions]

This issue is stale because it has been open 90 days with no activity. Remove stale label or comment or this will be closed in 5 days.

[github-actions]

This issue was closed because it has been stalled for 5 days with no activity.