Scaling Test up to 8x A100
Results (NPS):
| Num A100 | T60-30B | 40B | 40B+PR1661 |
|---|---|---|---|
| 1 | 70554 | 21764 | 32997 |
| 2 | 124481 | 47962 | 68929 |
| 3 | 154404 | 72639 | 99074 |
| 4 | 168115 | 90859 | 122476 |
| 6 | 192153 | 130750 | 155202 |
| 8 | 192340 | 149522 | 172692 |
Graph:
Details:
T60-30B: 610670
40B: mg-40b-swa-1670000
40B+PR1661: mg-40b-swa-2000000
All benches ran:
https://pastebin.com/RbmUaafk
https://pastebin.com/kfgryiK5