Run arc_evict thread at higher priority

[tonynguien]

Motivation and Context

Running arc_evict thread at higher priority minimizes it getting
scheduled off CPU and can improve performance for workload with high
ARC evict activities.

On mixed read/write and sequential read workloads, I've seen between
10-20% better performance.

Description

Extending zthr_create_timer() to take a new priority_t argument and
arc_evict_zthr is created with highest priority value, maxclsyspri.

How Has This Been Tested?

Manual tests with sequential reads and mixed read/writes with larger pool size.

ZFS performance test suite also show better though smaller gains for
sequential read and sequential writes, 8% and 3%, respectively.

Test setup:

• 8 vCPU
• 32 GB memory
• 4 x 100 SSD disks

Types of changes

• Bug fix (non-breaking change which fixes an issue)
• New feature (non-breaking change which adds functionality)
• Performance enhancement (non-breaking change which improves efficiency)
• Code cleanup (non-breaking change which makes code smaller or more readable)
• Breaking change (fix or feature that would cause existing functionality to change)
• Library ABI change (libzfs, libzfs_core, libnvpair, libuutil and libzfsbootenv)
• Documentation (a change to man pages or other documentation)

Checklist:

• My code follows the OpenZFS code style requirements.
• I have updated the documentation accordingly.
• I have read the contributing document.
• I have added tests to cover my changes.
• I have run the ZFS Test Suite with this change applied.
• All commit messages are properly formatted and contain Signed-off-by.

[tonynguien]

Attaching offcpu flame graphs showing how much arc_evict thread was involuntarily context switched off CPU and that was minimized with arc_evict running at higher priority.

arc_evict.tar.gz

[adamdmoss]

Would setting CPU affinity for the eviction thread have the same effect with more consistency?
(I'm a little worried about interactive performance whenever a zfs thread with nontrivial work gets high priority - had to patch-out these cases in the past - would like to see this boosted thread priority as a last resort or a nondefault tuning.)

[amotin]

Would setting CPU affinity for the eviction thread have the same effect with more consistency?

@adamdmoss How would any affinity made thread to run faster? It would likely do opposite by limiting scheduler choices.

[tonynguien]

Would setting CPU affinity for the eviction thread have the same effect with more consistency?
(I'm a little worried about interactive performance whenever a zfs thread with nontrivial work gets high priority - had to patch-out these cases in the past - would like to see this boosted thread priority as a last resort or a nondefault tuning.)

Setting CPU affinity wouldn't help since the thread still get taken off CPU as often and may even hurt performance because setting would limit which CPU the thread can run (less run time).

When arc_evict thread is active (evicting), I/Os may be stuck waiting until arc_evict frees up enough memory so it's important that the thread gets needed CPU time to complete its work.

I'm not opposed to providing a tunable when we see a problem. I hope we can avoid an unnecessary tunable.

[tonynguien]

I need to do more testing.

Using defsysclprio, i.e. nice=0, is giving similar performance improvement and should reduce CPU starvation concerns. I'll run more tests.

[tonynguien]

I couldn't update this due to new repo. I filed #12397 and closing this.