sparse benchmarking numbers #303
Conversation
jcaip
commented
Jun 3, 2024
jcaip
commented
- Updated benchmark script for standalone sparse numbers.
- Switched from segment-anything to segment-anything-fast
- Updated README with results for segment-anything and BERT
🔗 Helpful Links🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/ao/303
Note: Links to docs will display an error until the docs builds have been completed. ✅ No FailuresAs of commit 1fbea59 with merge base 8a4e693 ( This comment was automatically generated by Dr. CI and updates every 15 minutes. |
facebook-github-bot
added
the
CLA Signed
| import pandas as pd | ||
| from segment_anything_fast import sam_model_registry |
There was a problem hiding this comment.
I missed that sam fast isn't included in our benchmarks - suggestion is maybe to put a sam folder under benchmarjs with a README on custom dependencies and how to install them or just add a comment above this line as to how people can install sam fast
torchao/sparsity/README.md
Outdated
|
|
||
| #### BERT | ||
|
|
||
| We were able to accelerate BERT 1.23x with a negligible accuracy drop on SQuAD. |
torchao/sparsity/README.md
Outdated
|
|
||
| #### segment-anything | ||
| We applied 2:4 sparsity to accelerate segment-anything, as part of [segment-anything-fast](https://github.com/pytorch-labs/segment-anything-fast). | ||
| The results mentioned in the REAADME of the repo compose sparsity with a suite of other inference acceleration techniques. |
There was a problem hiding this comment.
| The results mentioned in the REAADME of the repo compose sparsity with a suite of other inference acceleration techniques. | |
| The results mentioned in the README of the repo compose sparsity with a suite of other inference acceleration techniques. |
| From our benchmarking, we see a 1.1x speedup when running with SEGMENT_ANYTHING_FAST_USE_FLASH_4 enabled. | ||
|
|
||
| ``` | ||
| python benchmarks/benchmark_sam.py |
There was a problem hiding this comment.
Put a direct link to the benchmarks script
torchao/sparsity/README.md
Outdated
| ``` | ||
| python benchmarks/benchmark_sam.py | ||
|
|
||
| block_only batchsize dtype compile qkv proj lin1 lin2 time memory img/s |
There was a problem hiding this comment.
Can you format this as a table it's a bit hard to read because I have to scroll all the way to the right? Also what does lin1 and lin2 mean? Presumably memory is in GB?
Also I feel like a quick sentence before this table along the lines of we have support for SparseSemiStructuredTensor we can apply it to each of the qkv of attention or just the proj and here's how to apply it would be helpful
- Updated benchmark script for standalone sparse numbers. - Switched from segment-anything to segment-anything-fast - Updated README with results for segment-anything and BERT
