Memory problem ?

[Haraade]

I have had memory problems in dnscrypt server-docker images.
Memory usage is growing rapidly.
Come across this:
tikv/rust-prometheus#424
Could prometheus v0.13.0 be the problem?
Compiled encrypted-dns with prometheus v0.12.0 and it looks like memory problem is gone.
This was done on a singleboard computer arm32v7 with 2Gbyte of LPDDR3 RAM.

[Haraade]

Do not know what does this, tried on another arm32v7 board. CPU (1-core of 4) usage also goes up to 100%

[jedisct1]

Can you double check that the issue goes away when downgrading the prometheus crate to v0.12.0?

How did you do that?

[Haraade]

The weird thing is that one device works with prometheus v0.12.0 and another does not (release 0.9.1), but both devices work well with encrypted-dns release 0.3.23 and arm32v7/ubuntu:22.04

Build dnscrypt-server-docker, FROM arm32v7/ubuntu:22.04

[Haraade]

I have also tried prometheus v0.13.1, but there are also issues.
tikv/rust-prometheus#442

[jedisct1]

Does it still happen if you don't enable the metrics feature? (disabling it removes prometheus)

[Haraade]

Does it still happen if you don't enable the metrics feature? (disabling it removes prometheus)

I have to try.

[Haraade]

disabled metrics in encrypted-dns.toml.in and -M option when initializing the docker image.
Memory and CPU usage are growing, and are running in loop.

[jedisct1]

Ok, so at least we know that this is unrelated to Prometheus.

In case this is not in a dependency, would you be able to pinpoint a specific commit that introduced this?

[Haraade]

Version 0.3.23 works.
cargo install --git https://github.com/DNSCrypt/encrypted-dns-server --rev 631ebd4

Version 0.9.0 does not work.
cargo install --git https://github.com/DNSCrypt/encrypted-dns-server --rev a178d25

So there must be something between these versions.

Will probably have to test more to find the specific commit that introduced this.

[Haraade]

I think this problem is due to jemallocator.

See what was mentioned earlier.
jemalloc/jemalloc#1497
gnzlbg/jemallocator#168
rust-lang/rust#83152

I have tried with tikv-jemallocator. Compiled fine and without memory problem, but unfortunately no response from encrypted-dns. Only gets TIMEOUT on server in dnscrypt proxy.
tikv-jemallocator supports arm32v7 ?

[jedisct1]

So, like way too many Rust crates, jemalloc is abandoned, and the maintained fork doesn't work on ARM? Great :(

As an experiment, 748b944 replaces jemalloc with mimalloc. Does in improve things on your end?

[jedisct1]

The app doesn't seem to compile at all on armv7 :(

error[E0308]: mismatched types
   --> /home/pi/.cargo/registry/src/github.com-1285ae84e5963aae/prometheus-0.13.1/src/process_collector.rs:192:13
    |
190 |     static ref CLK_TCK: i64 = {
    |                         --- expected `i64` because of return type
191 |         unsafe {
192 |             libc::sysconf(libc::_SC_CLK_TCK)
    |             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ expected `i64`, found `i32`

[Haraade]

tikv/rust-prometheus#442
I use prometheus v0.12.0
Trying mimalloc.

[jedisct1]

After removing prometheus' process feature, it works for me on armv7l (raspberry Pi with 32-bit raspbian).

[Haraade]

Update.
Has compiled with the latest version of tikv-jemallocator (0.5.0), which was released today and server works excellent.
tikv/jemallocator#26
This issue has been resolved. Thank you so much