Proof v10 CUDA build support and pipeline optimizations #7443

jennijuju · 2021-10-04T15:10:43Z

jennijuju
Oct 4, 2021
Maintainer

An experimental 'gpu' option using CUDA can be used in the proofs library with rust-ffi-proof v10.0.0 and up.

More details on cuda support can be found here and checkout advanced gpu usage for more tweak options.

Note: CUDA is an option you can opt-in, by default OpenCL is used.

Steps

Copy Storage System and Performance and fill in the information based on your setup - then submit it as an comment!
Read Install Known Good Nvidia Drivers With Cuda 11.2 (credit to @TippyFlitsUK )
Checkout v1.13.0-rc2 and upgrade your nodes, including all miner and workers.
To enable CUDA, build with FFI_USE_CUDA=1 FFI_BUILD_FROM_SOURCE=1 ...
- Build with RUST_LOG=debug to get more details, might be useful later for troubleshooting!
Restart your miner and workers!
Seal a sector. Record the new C2 & PC2 performance in the post you created in step 1.
Finish a WindowPoSt and record the computation/verification in the post you created in step 1.
Get lucky and win a block! Record the WinningPoSt computation/verification in the post you created in step 1.
Give this post a 🚀 if you have completed this task.

⚠️ Reminder

You can always use lotus-bench to test and benchmark first before deploying to your production system.
From SpaceForce: Double check your Nvidia Driver is properly loaded and GPU is enabled upon miner/worker restart!
Create an issue if you think there is a bug, don't forget about attaching the logs!
- Save the miner logs and lotus-miner info all if you run into anything! If the log files are large, you can upload them in MinerX.3 - v1.13.0 Testing Logs folder and link it in your issue report/slack message/..

Expectations

It took SpaceForce some experiments to get lotus build and CUDA enabled correctly, document any tips you discovered along the way so we can share them with the broader community later!
SpaceForce reported a 30%-50% decrease in computation time 🚀
You may want to monitor your GPU utilization 😉

💙 to SpaceForce: @TippyFlitsUK , @Reiers, @benjaminh83, @neondragon, Angelo for early testing and guide write-ups.

TippyFlitsUK · 2021-10-04T15:13:05Z

TippyFlitsUK
Oct 4, 2021
Maintainer

Install Known Good Nvidia Drivers With Cuda 11.2

The following steps will upgrade your Nivida GPU drivers to version 460.X and install Cuda version 11.2. This installation procedure was performed with Ubuntu 18.04 and will also work with newer versions.

The 460 Nvidia driver was selected based on storage provider feedback in the following thread. It is known to be a good choice for functionality with Lotus.
Overview of Nvidia Drivers for Lotus

Please note: Although the Cuda installer itself includes an option to install a GPU driver, it is not functional. Please follow the steps below to install the driver Cuda packages separately making sure to uncheck the Driver option in step 2.3.

1. Install Nvidia Driver

1.1. Remove any existing Nvidia drivers
sudo apt-get purge nvidia*

1.2. Add Graphic Drivers PPA
sudo add-apt-repository ppa:graphics-drivers/ppa
sudo apt-get update

1.3. Search available drivers
ubuntu-drivers devices

1.4. Install the 460 Nvidia - known good Lotus version
sudo apt-get install nvidia-driver-460

1.5. Reboot your server after 460 driver installation.

2. Install Cuda 11.2

2.1. Verify that the driver is successfully running
nvidia-smi

2.2 Install CUDA 11.2 Toolkit
wget https://developer.download.nvidia.com/compute/cuda/11.2.2/local_installers/cuda_11.2.2_460.32.03_linux.run
sudo sh cuda_11.2.2_460.32.03_linux.run

2.3 UNCHECK the option of installing the driver!!!

2.4 Add the new Cuda paths to your /.bashrc or /etc/environment file.

PATH=/usr/local/cuda-11.2/bin
LD_LIBRARY_PATH=/usr/local/cuda-11.2/lib64
CUDA_HOME=/usr/local/cuda

2.5 Verify Cuda 11.2 installation
nvcc --version

12 replies

kenshyx Oct 25, 2021

Yes!

hyunmoon Oct 26, 2021

Ubuntu:

wget https://developer.download.nvidia.com/compute/cuda/11.2.2/local_installers/cuda_11.2.2_460.32.03_linux.run
chmod +x cuda_11.2.2_460.32.03_linux.run
./cuda_11.2.2_460.32.03_linux.run --silent --toolkit
echo 'export PATH=/usr/local/cuda/bin${PATH:+:${PATH}}' >> ~/.bashrc
echo "export LD_LIBRARY_PATH=/usr/local/cuda-11.2/lib64" >> ~/.bashrc
echo "export CUDA_HOME=/usr/local/cuda" >> ~/.bashrc
source ~/.bashrc
nvcc --version
rm cuda_11.2.2_460.32.03_linux.run

dayou5168 Nov 2, 2021

@TippyFlitsUK

PATH=/usr/local/cuda-11.2/bin
LD_LIBRARY_PATH=/usr/local/cuda-11.2/lib64
CUDA_HOME=/usr/local/cuda

I think here path should be PATH=$PATH:/usr/local/cuda-11.2/bin
some community members found that they can't run their other command if they just export PATH=/usr/local/cuda-11.2/bin

f8-ptrk Nov 2, 2021

@dayou5168 depends on where you put the var

f8-ptrk Dec 13, 2021

495 and 11.5.1 fail at C1

jennijuju · 2021-10-07T04:16:44Z

jennijuju
Oct 7, 2021
Maintainer Author

Ozzy's Storage System and Performance (Example)

System

lotus-miner
CPU:
GPU:
RAM:
External Storage Media:

PC1 Worker
CPU:
Sealing Media:
Storage Media:

C2 Worker 1
CPU:
GPU:
Sealing Media:
Storage Media:
Running with: BELLMAN_CUDA_NVCC_ARGS ="--fatbin --gpu-architecture=sm_75 --generate-code=arch=compute_75,code=sm_75"

C2 Worker 2
CPU:
GPU:
Sealing Media:
Storage Media:

Performance

Sealing Pipeline	OpenCl	CUDA	Perf Improvement %
PC2			50%
C2			60%
WinningPoSt Computation			70%
WinningPoSt Verification *
WindowPoSt Computation			80%
WindowPoSt Verification *

0 replies

benjaminh83 · 2021-10-07T15:24:04Z

benjaminh83
Oct 7, 2021

BenjaminH Storage System and Performance

System

lotus-miner
CPU: EPYC 7302P
GPU: RTX A4000, Driver Version: 460.91.03 CUDA Version: 11.2 (Ubuntu 20.04), Toolkit: cuda_11.2.0_460.27.04
RAM: 256 GiB ECC
Storage Media: SAS3 HDD and HBA on soft RAID6

lotus-market
CPU: EPYC 7282
GPU: none
RAM: 128 GiB ECC
Storage Media: 18TiB NVMe RAID6 deal caching

PC1/PC2/C2 Worker
CPU: EPYC 7702P
RAM: 1024 GiB ECC
Sealing Media: 8x Enterprise NVMe RAID0 (12.8TiB)
GPU: RTX 3090, Driver Version: 460.91.03 CUDA Version: 11.2 (Ubuntu 20.04), Toolkit: cuda_11.2.0_460.27.04

Performance

Sealing Pipeline	OpenCl	CUDA	Perf Improvement %
PC2	16m	8-10m	~50%
C2	22-23m	11-12m	~50%

2 replies

jennijuju Oct 7, 2021
Maintainer Author

What about wdpost?

benjaminh83 Oct 7, 2021

@jennijuju Real block win.. in <1.2s 🥇

2021-10-07T12:39:36.481 INFO filcrypto::proofs::api > generate_winning_post: start
2021-10-07T12:39:36.506 INFO filecoin_proofs::api::winning_post > generate_winning_post:start
2021-10-07T12:39:37.652 INFO filecoin_proofs::api::winning_post > generate_winning_post:finish
2021-10-07T12:39:37.652 INFO filcrypto::proofs::api > generate_winning_post: finish

And a 1936 sector 32GiB deadline done.. also very fast...

2021-10-07T05:51:32.654 INFO filecoin_proofs::api::window_post > generate_window_post:start
2021-10-07T05:51:32.655 INFO filecoin_proofs::caches > trying parameters memory cache for: Window_POST[34359738368]
2021-10-07T05:51:32.655 INFO filecoin_proofs::caches > found params in memory cache for Window_POST[34359738368]
2021-10-07T05:53:06.225 INFO storage_proofs_core::compound_proof > vanilla_proofs:start
2021-10-07T05:54:13.813 INFO storage_proofs_core::compound_proof > vanilla_proofs:finish
2021-10-07T05:54:14.405 INFO storage_proofs_core::compound_proof > snark_proof:start
2021-10-07T05:54:14.523 INFO bellperson::groth16::prover > Bellperson 0.17.0 is being used!
2021-10-07T05:54:39.247 INFO bellperson::groth16::prover > synthesis time: 24.723318962s
2021-10-07T05:54:39.247 INFO bellperson::groth16::prover > starting proof timer
2021-10-07T05:54:43.276 INFO bellperson::gpu::locks > GPU is available for FFT!
2021-10-07T05:54:43.276 INFO bellperson::gpu::program > Using kernel on CUDA.
2021-10-07T05:54:43.298 INFO bellperson::gpu::fft > FFT: 1 working device(s) selected.
2021-10-07T05:54:43.298 INFO bellperson::gpu::fft > FFT: Device 0: RTX A4000
2021-10-07T05:54:43.298 INFO bellperson::domain > GPU FFT kernel instantiated!
2021-10-07T05:55:02.473 INFO bellperson::gpu::locks > GPU is available for Multiexp!
2021-10-07T05:55:02.473 INFO bellperson::gpu::program > Using kernel on CUDA.
2021-10-07T05:55:02.475 INFO bellperson::gpu::multiexp > Multiexp: 1 working device(s) selected. (CPU utilization: 0)
2021-10-07T05:55:02.475 INFO bellperson::gpu::multiexp > Multiexp: Device 0: RTX A4000 (Chunk-size: 10572704)
2021-10-07T05:55:02.475 INFO bellperson::multiexp > GPU Multiexp kernel instantiated!
2021-10-07T05:56:55.101 INFO bellperson::groth16::prover > prover time: 135.854121828s
2021-10-07T05:56:56.450 INFO storage_proofs_core::compound_proof > snark_proof:finish
2021-10-07T05:56:56.450 INFO filecoin_proofs::api::window_post > generate_window_post:finish
2021-10-07T05:56:56.467 INFO filcrypto::proofs::api > generate_window_post: finish
2021-10-07T05:56:56.475 INFO filcrypto::proofs::api > verify_window_post: start
2021-10-07T05:56:56.476 INFO filecoin_proofs::api::window_post > verify_window_post:start
2021-10-07T05:56:56.476 INFO filecoin_proofs::caches > trying parameters memory cache for: WINDOW_POST[34359738368]-verifying-key
2021-10-07T05:56:56.476 INFO filecoin_proofs::caches > found params in memory cache for WINDOW_POST[34359738368]-verifying-key
2021-10-07T05:56:56.490 INFO filecoin_proofs::api::window_post > verify_window_post:finish
2021-10-07T05:56:56.490 INFO filcrypto::proofs::api > verify_window_post: finish

neondragon · 2021-10-08T17:38:36Z

neondragon
Oct 8, 2021

Neonix's Storage System and Performance

Miner ID: f0694396 (64GiB)

System

lotus-miner
CPU: Intel(R) Xeon(R) CPU E5-2680 v3 @ 2.50GHz
GPU: 2x NVIDIA RTX A5000
RAM: 512GiB
External Storage Media: Ceph cluster
Compilation options:

export RUSTFLAGS="-C target-cpu=native -g"
export FFI_BUILD_FROM_SOURCE=1
export CGO_CFLAGS_ALLOW="-D__BLST_PORTABLE__"
export CGO_CFLAGS="-D__BLST_PORTABLE__"
export FFI_USE_CUDA=1

lotus-miner (markets)
CPU: Intel(R) Xeon(R) CPU E5-2697 v2 @ 2.70GHz
GPU: None
RAM: 128GiB
External Storage Media: Ceph cluster

PC1 Worker 1
CPU: AMD EPYC 7551 32-Core Processor
RAM: 512GiB
Sealing Media: 2x Micron 9300 3.2TB
Storage Media: Samsung 980 PRO 500GB

PC1 Worker 2
CPU: AMD EPYC 7551 32-Core Processor
RAM: 512GiB
Sealing Media: 2x Micron 9300 3.2TB
Storage Media: Samsung 980 PRO 500GB

C2 Worker 1
CPU: AMD EPYC 7262 8-Core Processor
RAM: 512GiB
GPU: NVIDIA GeForce 3090
Sealing Media: NFS shares from PC1 workers
Storage Media: 2x Samsung SSD 980 PRO 1TB
Compilation options:

export RUSTFLAGS="-C target-cpu=native -g"
export FFI_BUILD_FROM_SOURCE=1
export FFI_USE_CUDA=1

Performance (64GiB sector size) -- TODO

Sealing Pipeline	OpenCl	CUDA	Perf Improvement %
PC2	~48m	25m38s	47%
C2	~36m	19m37s	46%
WinningPoSt Computation
WinningPoSt Verification *
WindowPoSt Computation
WindowPoSt Verification *

0 replies

Meest-NAM · 2021-10-09T00:03:09Z

Meest-NAM
Oct 9, 2021

Meest NAM Server System and Perform(f03488)

System
lotus-miner & lotus-market (one machine)
CPU: AMD Epyc 7502 Base Clock 2.5GHz, Max. Boost Clock Up to 3.35GHz, socket : 2P
GPU: NVIDIA RTX 3090 Turbo edition
RAM: 512GiB
External Storage Media: SATA Enterprise HDD / RAID6
Compilation options:

export RUSTFLAGS="-C target-cpu=native -g"
export FFI_BUILD_FROM_SOURCE=1

PC1/PC2/C Worker 1
CPU: AMD EPYC 7502 32-Core Processor
RAM: 512GiB
GPU : RTX3080 TurboEdition
Sealing Media: FireCuda 320series 2TiB * 4 / raid 0
Storage Media: Samsung 970 EVO Plus 500GB

Performance -- TODO

sealing pipe line	before	after	rate
PC2	~16m	12m18s	33%
C2	~30m	17m20s	73%
WinningPoSt Computation
WinningPoSt Verification *
WindowPoSt Computation	330s	280s
WindowPoSt Verification *

0 replies

tmyuu · 2021-10-09T05:52:03Z

tmyuu
Oct 9, 2021

Livspect Storage System and Performance

System

lotus Version

Daemon:  1.13.0-rc2+mainnet+git.530011fd0+api1.2.0
Local: lotus-miner version 1.13.0-rc2+mainnet+git.530011fd0

lotus-miner
CPU: EPYC 7402P
GPU: Nvidia T4
RAM: 192GiB (DDR4-2933 ECC RDIMM 32GBx6)
External Storage Media: ISCSI

PC1/PC2 Worker
CPU: EPYC 7402P
GPU: Nvidia T4
RAM: 512GiB (DDR4-2933 ECC RDIMM 32GB*16) + 128GiB swap
Sealing Media: 4TB (SATA3.0 SSD 1TBx4 RAID0)

AP/C2 Worker
CPU: EPYC 7402P
GPU: Nvidia T4
RAM: 192GiB (DDR4-2933 ECC RDIMM 32GB*6) + 128GiB swap
Sealing Media: 4TB (SATA3.0 SSD 1TBx4 RAID0)

Performance

Sealing Pipeline	OpenCl	CUDA	Perf Improvement
PC2	30m	20m	+33%
C2	45m	35m	+23%
WinningPoSt Computation	7s	5s	+30%
WinningPoSt Verification *	-	-	- %
WindowPoSt Computation	510s	416s	+18%
WindowPoSt Verification *	4s	4s	- %

This is my normal sealing performance in case of running parallel jobs.

0 replies

stuberman · 2021-10-12T01:49:01Z

stuberman
Oct 12, 2021

stuberman's Storage System and Performance

System

lotus-miner
CPU: Ryzen 9 3900
GPU: Nvidia 2080 Ti - Driver 460.67
RAM: 128 GB
OS media: 1TiB NVMe Sabrent Rocket 4.0
Lotus media: 2TiB NVMe Inland
External Storage Media: 120TiB ZFS pool direct SATA attach WD HC530 (12x14TiB HDDs)
Running with:

export BELLMAN_CUDA_NVCC_ARGS="--fatbin --gpu-architecture=sm_70 --generate-code=arch=compute_70,code=sm_70"
export NEPTUNE_CUDA_NVCC_ARGS="--fatbin --gpu-architecture=sm_70 --generate-code=arch=compute_70,code=sm_70"

Worker
CPU: ThreadRipper 3990x
GPU: Nvidia 3090 - Driver 460.67
RAM: 256 GB
Sealing Media: 2x 4TiB NB+VMe Gen4x4 Sabrent Rocket Q4
Storage Media: Sabrent Rocket 4.0 500GB NVMe (OS)
Swap: Sabrent Rocket 4.0 1TB
Running with:

export BELLMAN_CUDA_NVCC_ARGS="--fatbin --gpu-architecture=sm_70 --generate-code=arch=compute_70,code=sm_70"
export NEPTUNE_CUDA_NVCC_ARGS="--fatbin --gpu-architecture=sm_70 --generate-code=arch=compute_70,code=sm_70"

Market
CPU: Ryzen 5 1600
GPU: N/A
RAM: 32 GB
Lotus repo Media: 4TiB NVMe Gen3x4 Sabrent Rocket
OS Media: 60 GB SSD (OS)

Performance

Sealing Pipeline	OpenCl	CUDA	Perf Improvement %
PC2	8 min	9 min	- 10 %
C2	14 min	9 min	+ 36 %
WinningPoSt Computation	2.39 s	2.21 s	+ 8 %
WinningPoSt Verification *
WindowPoSt Computation	807 s	928 s	- 15 %
WindowPoSt Verification *

*** WinningPost has always been highly variable based on miner workload (disk/CPU/network/etc) and is comparable to OCL

0 replies

kernelogic · 2021-10-12T03:32:32Z

kernelogic
Oct 12, 2021

Fei Yan - Kernelogic storage system/performance

Miner
CPU: Threadripper 3960x
GPU: Nvidia 2080 Ti
RAM: 256 GB
SSD: 2x2TB NVME
10G LAN

Worker
CPU: Threadripper 3960x
GPU: Nvidia 3090 FE
RAM: 256 GB
SSD: 2x2TB NVME
10G LAN

Storage
Synology 8 Bay NAS with 10G LAN

Sealing Pipeline	OpenCI	CUDA	Perf Improvement %
PC2	9m	8m	11%
C2	18m	11m	38%

0 replies

William8Work · 2021-10-12T03:50:17Z

William8Work
Oct 12, 2021

WillTechMusing Storage System / Performance

Miner

CPU: AMD Epyc 7601 32 Core x 2
GPU: RTX 2080 Ti
RAM: 256 GB
10G LAN

Worker (PC1) x 3

CPU: AMD Epyc 7502P 32 Core x 2
RAM: 256 GB
10G Lan

Worker-4 (Runs two PC2 processes, and one C2 process)

CPU: AMD Threadripper 3960x
GPU: RTX 3090
RAM: 256 GB
10GB LAN

Worker-5 (Runs two PC2 processes)

CPU: Threadripper 3960x
GPU: RTX 2080 Super
RAM: 256 GB
10GB LAN

Worker-6 (Runs PC2 processes)

CPU: Threadripper 3950x
GPU: RTX 3070
RAM: 256 GB
10GB LAN

Worker-7 (C2)

CPU: Intel E5-2670 x2
GPU: RTX 1080 Ti
RAM: 256 GB
10GB LAN

Market Node

CPU: Intel E5-2670 x2
RAM: 256 GB
10GB LAN

Sealing Pipeline	OpenCL	CUDA	Perf Improvement %
Worker-4 PC2, A	74 minutes	60 minutes	18.9%
Worker-4 PC2 B	14 minutues	14 minutes	0%
Worker-5 PC2 A	55 minutes	50 minutes	<10%
Worker-6 PC2	20 minutes	17 minutes	15%
Worker-7 C2	19 minutes	13 minutes	36.8%

0 replies

rjan90 · 2021-10-12T10:19:25Z

rjan90
Oct 12, 2021
Maintainer

Phi-rjan's Storage System and Performance

lotus-miner
CPU: EPYC 7302P
GPU: Nvidia RTX 2080Ti
RAM: 256GB (4 x 64GB DDR4-2933 LRDIMM ECC)
External Storage Media: NFS-mounted ZFS-pools
Note: This machine is currently running the markets-process as well, which may have some impact on winPoSt, and wdPoSt times if deals where transferring, and or retrievals served during the monitored times. Will separate the markets-process to another machine this month, which should allow for a cleaner view of perf. improvements.

PC1/PC2/C2 Worker
CPU: EPYC 7f32
GPU: Nvidia RTX 2080Ti, Driver Version: 460.91.03 CUDA Version: 11.2 (Ubuntu 20.04), Toolkit: cuda_11.2.0_460.27.04
RAM: 512GB (8 x 64GB DDR4-2933 LRDIMM ECC)
Sealing Media: 7TB (Samsung PM1733 3.84GB * 2 in Raid0)

C2 Worker / Tester Machine (only used sometimes)
CPU: Ryzen 9 5950X
GPU: Nvidia RTX 3090
RAM: 128GB + 256GB Swap
Sealing Media: 2TB (SATA3.0 SSD)

Sealing Pipeline	OpenCL	CUDA	Perf Improvement %
PC2	~12 minutes	~9 minutes	25%
C2	~21 minutes	~15 minutes	~29%
winningPoSt computation	3.9 seconds	2.1 seconds	46.1%
windowPoSt computation	844 seconds	481 seconds	43%

**Note that the performance times for the PC2 / C2 where done with only one sectors sealing with the PC1/PC2/C2-worker, under a full workload where you can have up to multiple PC1's and C2 at a time taking up available resources, those exact numbers might not be achievable, but it's a good ballpark measure

0 replies

kenshyx · 2021-10-12T11:45:54Z

kenshyx
Oct 12, 2021

Marius - Storage System and Performance

Miner/Market:

CPU: Intel(R) Xeon(R) Silver 4216
GPU: GeForce RTX 2080
RAM: 512GiB ECC RAM
Storage Media: Exos X18 raid-z3

PC2/C2 Worker:

CPU: AMD Ryzen Threadripper 3990X
GPU: Tesla V100S
RAM: 256GiB ECC RAM
Storage Media: 8TiB nvme

NVIDIA driver version: 470.57.02
CUDA compilation tools: V11.4.120

I had to add ~~FIL_PROOFS_CUDA_NVCC_ARGS="--fatbin --gpu-architecture=sm_70 --generate-code=arch=compute_70,code=sm_70"~~ BELLMAN_CUDA_NVCC_ARGS="--fatbin --gpu-architecture=sm_70 --generate-code=arch=compute_70,code=sm_70" NEPTUNE_CUDA_NVCC_ARGS="--fatbin --gpu-architecture=sm_70 --generate-code=arch=compute_70,code=sm_70" to target Tesla V100S

Sealing Pipeline	OpenCL	CUDA	Perf Improvement %
PC2	15m	11m	26.7%
C2	25m	15m	40%
WinningPoSt Computation	2.1s	1.5s	28.5%
WindowPoSt Computation	860s	458s	46.7%

0 replies

Reiers · 2021-10-12T11:57:27Z

Reiers
Oct 12, 2021

Reiers Storage servers, worker and Performance (OpenCL vs CUDA)

Lotus version:

the-magikal-branch
Daemon:  1.13.0-rc2+mainnet+git.82e1e474b+api1.4.0
Local: lotus version 1.13.0-rc2+mainnet+git.82e1e474b

Miner+Daemon:
CPU - AMD® Ryzen Threadripper Pro 3995wx
GPU - NVIDIA 3090 FE
RAM - 512GB 3200MHz DDR4 ECC
Markets node:
CPU - AMD® Ryzen Threadripper 3970X
GPU - NVIDIA 3090 FE
RAM - 256GB 3200MHz DDR4
Storage Servers:
Storinator Q30 Turbo+Storinator S45, total units = 5
Seagate Exos 18TB Enterprise HDD, total disks = 180
CPU - Dual Intel Xeon Silver 4210
RAM - 256GB
Networking - 10G SFP+ EdgeRouter Infinity

Lotus Workers:

PC1 Workers (x2 machines):
PowerEdge R7515 Server
CPU - AMD EPYC 7F32
RAM - 1024GB 3200MHz ECC
Sealing Media - Samsung 860 PRO 2TB RAID 0, total disk per unit = 8

PC1/PC2 Workers (x9 machines):
CPU - Intel Core i9-10900X, AMD Ryzen 5950x and AMD® Ryzen Threadripper Pro 3995wx
GPU - NVIDIA RTX 3090
RAM - 128-256 GB 3200MHz + Swap 512GB
Sealing Media - WD BLACK 4TB NVMe

C2 Workers (x9 machines):
CPU - Intel Core i9-10940X or similar
GPU - NVIDIA RTX 3080/90
RAM - 256 GB 3200MHz
Sealing Media - WD BLACK 4TB NVMe
Networking - NETGEAR 10G Ethernet Plus Switch (XS708E)

Sealing Pipeline	OpenCL	CUDA	Perf Improvement %
PC2	~12 minutes	~10 minutes	16.6%
C2	~24 minutes	~12 minutes	50%
winningPoSt computation	~3.2 seconds	~1.2 seconds	62.5%
windowPoSt computation	~844 seconds	583 seconds	30.92%

All times is an average over all the workers and deadlines before and after CUDA.
windowPosts and deadlines are spread out over different Storage Unit, some windowPost will be faster than others.
Some GPUs perform better than other even if its same GPU (different providers)

Been running CUDA stable for over a week.

0 replies

TippyFlitsUK · 2021-10-12T16:31:28Z

TippyFlitsUK
Oct 12, 2021
Maintainer

TippyFlits - Storage System and Performance

Daemon:  1.13.0-rc2+mainnet+git.a1a7ea66c+api1.2.0
Local: lotus-miner version 1.13.0-rc2+mainnet+git.a1a7ea66c

Miner/Markets/Workers X 2/PC1/PC2/C2
CPU: Dual EPYC 7282
GPU: Dual Nvidia Tesla T4 - Driver Version: 460.91.03 - CUDA Version: 11.2
RAM: 1TiB DDR4 3200 MHz ECC
Sealing Media: Dual 6.4TB Samsung NVMe
Storage Media: Seagate Exos 16TB Enterprise HDD - ZFS pools

Sealing Pipeline	OpenCL	CUDA	Perf Improvement %
PC2	29 minutes	19 minutes	35%
C2	49 minutes	26 minutes	45%
winningPoSt computation	2.9 seconds	1.7 seconds	42%
windowPoSt computation	~960 seconds	~800 seconds	17%

0 replies

f8-ptrk · 2021-10-13T02:23:18Z

f8-ptrk
Oct 13, 2021

System

lotus-miner
CPU:3960X GPU:3090 RAM:256GB External Storage Media:HDDs

PC2/C2 Worker 1
CPU:3990X GPU:3090 RAM:256GB Sealing Media:nvme

Performance

Sealing Pipeline	OpenCl	CUDA	Perf Improvement %
PC2	8m	6.5m	~20%
C2	14m	9m	35%
WinningPoSt Computation	1s	650ms	35%
WindowPoSt Computation	114-120s	76-78s	~35%

pure gpu times for wd and winning post.

the pc2/c2 times are not under full load of the gpu but single sector... it will need weeks of testing to really grasp the full impact of the cuda implementation - especially when utilizing the gpu with more than one worker at the same time

0 replies

ksidles · 2021-10-13T16:48:02Z

ksidles
Oct 13, 2021

lotus-miner Miner/daemon/Commit
3900X, 128GB NON-ECC, 1080TI, 10Gb, 1 1TB & 1 2TB NVME

PC1 Worker
Threadripper 3960X
128GB NON-ECC, 10Gb 2x 2TB NVME (Motherboard has a RAM slot issue)

PC1 Worker:
Epyc 7351p, 512GB ECC, 10Gb, 2x 2TB NVME

PC2 Worker
Dual Intel(R) Xeon(R) CPU E5-2603 v4 @ 1.70GHz, 256Gb ECC, 10Gb, 3 1060 GPU, 7.68TB NVME

Commit Worker
3900X, 128GB NON-ECC, 1080TI, 10Gb, 1 1TB & 1 2TB NVME

Performance 1080TI

Sealing Pipeline	OpenCl	CUDA	Perf Improvement %
PC2	TBD	TBD	TBD
C2	71m	102m	-43%
WinningPoSt Computation	3.6s	5.6s	-55%
WindowPoSt Computation	961	1087	-13%

Performance A4000

Sealing Pipeline	OpenCl	CUDA	Perf Improvement %
PC2	TBD	18m	TBD
C2	TBD	25m	TBD
WinningPoSt Computation	TBD	2.1s	TBD
WindowPoSt Computation	TBD	TBD	-TBD

0 replies

marco-storswift · 2021-10-14T09:37:10Z

marco-storswift
Oct 14, 2021

Miner/Market:
CPU: AMD EPYC 7532 32-Core Processor
GPU: GeForce RTX 2080 Ti
RAM: 512GiB ECC RAM
Storage Media:

PC2/C2 Worker:
CPU: AMD EPYC 7532 32-Core Processor
GPU: GeForce RTX 2080 Ti
RAM: 1TGiB ECC RAM
Storage Media: 8TiB nvme

NVIDIA driver version: 460.39
CUDA compilation tools: 11.2

Sealing Pipeline	OpenCl	CUDA	Perf Improvement %
PC2	12m	9m	25%
C2	20m	15m	25%

0 replies

donkabat · 2021-11-02T17:56:04Z

donkabat
Nov 2, 2021

BOX1
PC2/C2 Worker:
CPU: AMD TR 3960x
GPU: GeForce RTX 3090
RAM: 256 GB 3200 MHz
Storage Media: 3 x 2 TB Samsung 980 PRO

NVIDIA driver version: 460.39
CUDA compilation tools: 11.2

Sealing Pipeline	OpenCl	CUDA	Perf Improvement %
PC2 BELLMAN_CPU_UTILIZATION=0.5	12m	10m	-17%
C2 BELLMAN_CPU_UTILIZATION=0	15m	10m	-33%

BOX2
PC2/C2 Worker:
CPU: 2x Intel Xeon 4215R
GPU1: GeForce RTX 3090
GPU2: GeForce RTX 3060
RAM: 384 GB 3200 MHz
Storage Media: 4 x 2 TB Corsair MP510

NVIDIA driver version: 460.91.03
CUDA compilation tools: 11.2

Sealing Pipeline	OpenCl	CUDA	Perf Improvement %
GPU1: PC2 BELLMAN_CPU_UTILIZATION=0	18m	12m	-33%
GPU1: C2 BELLMAN_CPU_UTILIZATION=0	24m	15m	-37%
GPU2: PC2 BELLMAN_CPU_UTILIZATION=0	-	15m	-
GPU2: C2 BELLMAN_CPU_UTILIZATION=0	28m	16m	-43%

0 replies

Proof v10 CUDA build support and pipeline optimizations #7443

jennijuju Oct 4, 2021 Maintainer

Steps

⚠️ Reminder

Expectations

Replies: 17 comments · 14 replies

TippyFlitsUK Oct 4, 2021 Maintainer

Install Known Good Nvidia Drivers With Cuda 11.2

1. Install Nvidia Driver

2. Install Cuda 11.2

jennijuju Oct 7, 2021 Maintainer Author

Ozzy's Storage System and Performance (Example)

System

Performance

BenjaminH Storage System and Performance

System

jennijuju Oct 7, 2021 Maintainer Author

Neonix's Storage System and Performance

Miner ID: f0694396 (64GiB)

System

Performance (64GiB sector size) -- TODO

Meest NAM Server System and Perform(f03488)

Livspect Storage System and Performance

System

Performance

stuberman's Storage System and Performance

System

Performance

Fei Yan - Kernelogic storage system/performance

WillTechMusing Storage System / Performance

Miner

Worker (PC1) x 3

Worker-4 (Runs two PC2 processes, and one C2 process)

Worker-5 (Runs two PC2 processes)

Worker-6 (Runs PC2 processes)

Worker-7 (C2)

Market Node

rjan90 Oct 12, 2021 Maintainer

Phi-rjan's Storage System and Performance

Marius - Storage System and Performance

Miner/Market:

PC2/C2 Worker:

Reiers Storage servers, worker and Performance (OpenCL vs CUDA)

Lotus Workers:

TippyFlitsUK Oct 12, 2021 Maintainer

TippyFlits - Storage System and Performance

System

Performance

Performance 1080TI

Performance A4000

jennijuju
Oct 4, 2021
Maintainer

Replies: 17 comments 14 replies

TippyFlitsUK
Oct 4, 2021
Maintainer

jennijuju
Oct 7, 2021
Maintainer Author

jennijuju Oct 7, 2021
Maintainer Author

rjan90
Oct 12, 2021
Maintainer

TippyFlitsUK
Oct 12, 2021
Maintainer