added gpu benchmarking script

[jcaip]

Add combined GPU sparsity benchmarking script.

This is really a combination of two scripts - https://gist.github.com/cpuhrsch/7fec60079cbe2daeff59c0577f933320 for BSR benchmarking and https://github.com/pytorch/pytorch/blob/8db72a430d0c3a7d3388749d5d438fb805f53407/benchmarks/sparse/benchmark_semi_structured_sparsity.py for semi-structured sparse benchmarking

We're planning on releasing superblock soon, so I want to point the benchmarks to here, with idea being we can farm out consumer card benchmarks for block sparse like we did with #174

For superblock benchmarks run:

python benchmarks/benchmark_gpu_sparsity.py --mode sam-vitb-shapes --sparsity block-sparse --sparsity-level 0.8 --block-size 64 --dtype fp32
python benchmarks/benchmark_gpu_sparsity.py --mode sam-vitb-shapes --sparsity block-sparse --sparsity-level 0.9 --block-size 64 --dtype fp32
python benchmarks/benchmark_gpu_sparsity.py --mode sam-vitb-shapes --sparsity block-sparse --sparsity-level 0.8 --block-size 32 --dtype fp32
python benchmarks/benchmark_gpu_sparsity.py --mode sam-vitb-shapes --sparsity block-sparse --sparsity-level 0.9 --block-size 32 --dtype fp32

[msaroufim]

wanna also recommend people post their results on a central issue here?

[msaroufim]

does this require nightlies?

[jcaip]

We should probably log torch.__version__, but this doesn't require nightlies. Is there a way we can track torchao version as well?

[cpuhrsch]

There's a couple caveats to this function. I somewhat trust it less than using cuda synchronize and a for loop. Also I'd report the standard deviation as well. If you have 5us but it's +/- 20us something went wrong. blocked autorange is supposed to help with that, but better to verify and print it.

[jcaip]

Yeah I think blocked autorange is not great - Actually for the benchmarks, a lot of the time it's just running once, likey as @HDCharles highlighted here.

I think we can use adaptive autorange instead, wrapped in torch.synchronize() to minimze the variability.

[cpuhrsch]

Since we don't care about accuracy here I assume, subnormal number performance aside, you could also try torch.empty(n, k, dtype=dtype, device='cuda') which might be faster to allocate and doesn't require calling randn. Especially if you run a lot of benchmarks in a row this can become annoying to wait for.

[jcaip]

I think we want to avoid this because this will bias numbers https://www.thonking.ai/p/strangely-matrix-multiplications

[cpuhrsch]

Yes, zeros will be an issue. Also subnormal numbers or such. But yes, we can't rely on empty to not give us just all zeros.

Hm, I guess if this does ever really become a bottleneck we can write a simpler random number generator (like arange + some mod with a prime number etc.).

[cpuhrsch]

Alright, so we do care about correctness. It seems like maybe something to turn on/off. Morally this should be covered by unit tests, but I also never mind more sources of verification.

[cpuhrsch]

You can use torch.testing.assert_allclose to have it raise an exception with location and error.

[jcaip]

I think when I first wrote this script, we didn't have tests. Let's just remove this correctness checking - since we have better testing now.

[cpuhrsch]

There's pros/cons to creating new processes for each benchmark, but it seems like in general this script will need a default setting to run all relevant or interesting configurations. If someone runs python benchmark_gpu_sparsity.py and then posts the result, is that enough to be useful?

[pytorch-bot]

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/ao/192

• 📄 Preview Python docs built from this PR

Note: Links to docs will display an error until the docs builds have been completed.

✅ No Failures

As of commit 7f8b773 with merge base f2c908b ():
💚 Looks good so far! There are no failures yet. 💚

This comment was automatically generated by Dr. CI and updates every 15 minutes.