ZFS ARC max isn't honored

[xin-hedera]

Description

In our production environments with existing large disks and large zpools, the set value of 2GB ZFS ARC max isn't honored, instead, the effective max was likely auto set to around 50% of the available system memory, e.g.,

ARC status:                                                      HEALTHY
        Memory throttle count:                                         0

ARC size (current):                                    60.5 %   28.5 GiB
        Target size (adaptive):                        61.2 %   28.9 GiB
        Min size (hard limit):                          6.2 %    2.9 GiB
        Max size (high water):                           16:1   47.1 GiB

This would then lead to much higher ARC memory usage than expected, e.g., 28.5GB as shown above.

The theory is in our init script, we do the two things in order

• zpool import existing pool
• set ZFS ARC max to 2GB

In case of a very large existing pool, ZFS will auto calculate and set ARC min/max. When the effective ARC min (here it's 2.9GB) is higher than the ARC max in subsequent attempted changes, the change is just ignored.

We also need to test that with large pool, we don't run into the issues here and here.

Steps to reproduce

check the description

Additional context

No response

Hedera network

No response

Version

v0.0.116-rc3

Operating system

None