Feedbacks on GPU2

[jennijuju]

By default, a 'gpu' option is used in the proofs library. There is now an experimental 'gpu2' feature, introduced in #1422 , which has improved performance( mainly for PC2). We would love to gather some feedback on how it performs before merging it to lotus and please leave a comment below with your hardware spec, old PC2 running time, and new PC2 running time with gpu2 enabled!

You can try this out by building the ffi package from the source and allowing building with the 'gpu2' dependency, set FFI_USE_GPU2=1.

How to build lotus-bench with a specific proof commit:

1. edit ~/lotus/Makefile and comment line [perf] Use fixed size arrays instead of vectors where possible #54 to stop it checking out the normal version of extern/filecoin-ffi #git submodule update --init --recursive
2. cd ~/lotus/extern/filecoin-ffi && git checkout d828994
3. cd ~/lotus/
   RUSTFLAGS="-C target-cpu=native -g" FFI_BUILD_FROM_SOURCE=1 FFI_USE_GPU2=1 make clean all lotus-bench lotus-shed && make install

[jennijuju]

From @hyumoon on filecoin slack,

Gpu2 is awesome. PC2 took only 9 minutes 9 seconds on EPYC 7302 & RTX 3080.

[cryptonemo]

For more information, gpu2 is a drop in replacement for gpu that reduces RAM usage and improves GPU performance since it's a pure opencl implementation of the tree building code. We intend to make gpu2 replace the existing gpu code so that it will be the default going forward in the next release in proofs. We are particularly interested in any issues that arise using it. We have tested it internally for some time and have only seen the benefits, so we're looking for more feedback.

[xinaxu]

GTX 1070 from 75 min -> 45 min. GPU Mem usage is very minimal

[TippyFlitsUK]

GPU: Tesla T4
CPU: Dual EPYC 7282

Daemon:  1.5.3-rc2+mainnet+git.9afb5ff94.dirty+api1.1.0
Local: lotus version 1.5.3-rc2+mainnet+git.9afb5ff94.dirty

50% time reduction for both PC2 (25 mins) and C2 (45 mins).

[neondragon]

GPU: RTX 3080
CPU: Ryzen 9 3900X
32GiB sector
PC2 speed improved, C2 slightly improved.

[Storage-Hive]

7232P, RTX3060 12GB,
PC2 went from 32min to 16min, 50% improvement,
C2 from 55min to 42min, 25% improvement.

[fanfu13]

1.5.3-rc2+calibnet+git.9afb5ff94.dirty
Epyc7452, 2080Ti, 64G Sector sealing
PC2 went from 51min to 24min
C2 went from 44 min to 29min

[dalingng]

lotus version 1.5.2+mainnet+git.6924e3ba7
epyc7742
RTX3090
32G Sector
PC1 15--17min
C2 29--35min

[Reiers]

HW for tested on:

Dedicated Pc1 workers
vision-epyc-worker -> No improvement, EPYC 7F32 1TB RAM
ultron-epyc-worker -> No improvement, EPYC 7F32 1TB RAM
Pc2/C2 Workers
helen-intel-worker -> Improvement, Intel Core i9-10900X, 256 GB RAM, GPU 3090
tadashi-intel-worker -> Improvement, Intel Core i9-10900X, 256 GB RAM, GPU 3090
jokasta-intel-worker -> Improvement, Intel Xeon Gold 6126, 128 GB ECC RAM + SWAP, GPU 3090
homer-intel-worker -> Improvement, Intel Core i9-10940X, 256 GB ECC RAM, GPU 3090
edith-intel-worker -> Improvement, Intel Xeon Gold 6126, 256 RAM, GPU 3090
virgil-amd-worker -> Improvement, Threadripper 3960X, 256 RAM, GPU 3090
friday-amd-worker -> Improvement, Threadripper 3970X, 256 RAM, GPU 3090
jarvis-amd-worker -> Improvement, Ryzen 9 3950X, 128 GB RAM + SWAP, GPU 3090
plato-amd-worker -> Improvement, Ryzen 9 3950X, 128 GB RAM + SWAP, GPU 3080

Pc2 times avg. before 23 minutes, after gpu2 10-12 minutes. 
C2 times avg. before 42-57 minutes, after gpu2 22-26 minutes. 

Just an FYI. All 3090s are from different vendors.
Give a response on Slack if you want me to try different GPUs.
But my guess is that its about a 50% reduction in time all over the board.

Lotus Version:
lotus version 1.5.3-rc2+mainnet+git.9afb5ff94.dirty

[f8-ptrk]

why it has so few PC1 workers? :o

[benjaminh83]

This rocks. Tested on PC2/C2 worker and Miner

Miner:

EPYC 7302P, RTX 2080TI, 256 GiB memory
WinningPoST: 3.6s -> 2.18s

Worker:

EPYC 7282, 384 GiB memory
PC2, RTX 2080TI: 25m -> 12m30s (very small memory usage on GPU)
C2, RTX 3090: 39m -> 21m (normal memory usage)
Dual C2 worker against one RTX 3090: 43m -> 23m (so total performance of 11m30s) 

This little PC2/C2 box has a sealing output of above 3.5TiB/day on Bellperson v0.13.0. WOW!

Lotus version

Daemon:  1.5.3+mainnet+git.3e6dcb578.dirty+api1.0.1
Local: lotus-miner version 1.5.3+mainnet+git.3e6dcb578.dirty

[funktafide]

AMD Ryzen 7 3800XT with 1080Ti
PC2 and C2 both reduce by 10 minutes.

[nickboot]

CPU: AMD EPYC 7542
GPU: RTX 3090
64GiB sector
P2 19 minutes

[SBudo]

CPU: EPYC 7302
GPU: Tesla T4
32GiB sectors, dedicated workers
PC2: 45mins -> 25mins
C2: 75mins -> 45mins