Update: Analyze api, add DataStructures for summary #288

rahul-tuli · 2023-03-31T13:21:39Z

This PR updates the analyze api to conform with the PRD requirements; additionally it provides more structure to the summary by adding a ModelAnalysisSummary class; Additionally this class includes a __sub__ function allowing us to do something like summary_a - summary_b out of the box, which means all summaries can be compared against each other

Test Command: sparsezoo.analyze ~/models/resnet50-quac.onnx

Output:

sparsezoo.analyze ~/models/resnet50-quac.onnx

INFO:root:Starting Analysis ...
INFO:root:Analysis complete, collating results...
Params:
                                   MODEL       SPARSITY       QUANTIZED           COUNT            SIZE
   /home/rahul/models/resnet50-quac.onnx          94.90           91.97        25530472       253419776

Ops:
                                   MODEL       SPARSITY       QUANTIZED           COUNT            SIZE
   /home/rahul/models/resnet50-quac.onnx          94.90           91.97        25530559       253422560

Overall:
                                   MODEL       SPARSITY       QUANTIZED
   /home/rahul/models/resnet50-quac.onnx          94.90           91.97

Add by_types info to analyze api #292 Merged
This PR Adds compare functionality to analyze api #299 Merged
Add by-layers support to analyze api(s) #301 Merged
Add support to compare by types and layers #302
Add: Multiline row printing #303
Changes needed for Deepsparse.Analyze #304 This can land on Main directly but putting it here for tracking, (still in draft mode)

After Merging all PRs in the updated output is as follows:

 sparsezoo sparsezoo.analyze \
  "zoo:cv/classification/resnet_v1-50/pytorch/sparseml/imagenet/pruned95_uniform_quant-none" \
  --by-layers True \
  --by-types True \
  --compare "zoo:cv/classification/resnet_v1-50/pytorch/sparseml/imagenet/pruned95_quant-none"
INFO:sparsezoo.analyze_cli:Starting Analysis ...
INFO:sparsezoo.analyze_cli:Analysis complete, collating results...
INFO:root:Running analysis `by_layers`
INFO:root:Running analysis `by_types`
Analysis by Layers:
 NAME                           TOTAL                          SIZE                           SPARSITY                       QUANTIZED
 Conv_13_quant                  9472.00                        58679                          24.10                          99.32
 Conv_37_quant                  4160.00                        2252                           93.53                          98.46
 Conv_105_quant                 16640.00                       8982                           93.55                          98.46
 Conv_60_quant                  36928.00                       15234                          94.87                          99.83
 Conv_83_quant                  16640.00                       8912                           93.60                          98.46
 Conv_129_quant                 16448.00                       7148                           94.63                          99.61
 Conv_152_quant                 36928.00                       15293                          94.85                          99.83
 Conv_175_quant                 16640.00                       8968                           93.56                          98.46
 Conv_199_quant                 16448.00                       7135                           94.64                          99.61
 Conv_222_quant                 36928.00                       15293                          94.85                          99.83
 Conv_245_quant                 16640.00                       8898                           93.61                          98.46
 Conv_269_quant                 32896.00                       14270                          94.64                          99.61
 Conv_337_quant                 131584.00                      54951                          94.84                          99.61
 Conv_292_quant                 147584.00                      60015                          94.93                          99.91
 Conv_315_quant                 66048.00                       30872                          94.29                          99.22
 Conv_361_quant                 65664.00                       26789                          94.93                          99.81
 Conv_384_quant                 147584.00                      59660                          94.96                          99.91
 Conv_407_quant                 66048.00                       30764                          94.31                          99.22
 Conv_431_quant                 65664.00                       27264                          94.84                          99.81
 Conv_454_quant                 147584.00                      59779                          94.95                          99.91
 Conv_477_quant                 66048.00                       30926                          94.28                          99.22
 Conv_501_quant                 65664.00                       27264                          94.84                          99.81
 Conv_524_quant                 147584.00                      60015                          94.93                          99.91
 Conv_547_quant                 66048.00                       30926                          94.28                          99.22
 Conv_571_quant                 131328.00                      54740                          94.82                          99.81
 Conv_639_quant                 525312.00                      216426                         94.88                          99.81
 Conv_594_quant                 590080.00                      236811                         94.99                          99.96
 Conv_617_quant                 263168.00                      114163                         94.64                          99.61
 Conv_663_quant                 262400.00                      105477                         94.99                          99.90
 Conv_686_quant                 590080.00                      236339                         95.00                          99.96
 Conv_709_quant                 263168.00                      113737                         94.66                          99.61
 Conv_733_quant                 262400.00                      105267                         95.00                          99.90
 Conv_756_quant                 590080.00                      237284                         94.98                          99.96
 Conv_779_quant                 263168.00                      113524                         94.67                          99.61
 Conv_803_quant                 262400.00                      106319                         94.95                          99.90
 Conv_826_quant                 590080.00                      237284                         94.98                          99.96
 Conv_849_quant                 263168.00                      113737                         94.66                          99.61
 Conv_873_quant                 262400.00                      106319                         94.95                          99.90
 Conv_896_quant                 590080.00                      236811                         94.99                          99.96
 Conv_919_quant                 263168.00                      113737                         94.66                          99.61
 Conv_943_quant                 262400.00                      106530                         94.94                          99.90
 Conv_966_quant                 590080.00                      237284                         94.98                          99.96
 Conv_989_quant                 263168.00                      113737                         94.66                          99.61
 Conv_1013_quant                524800.00                      214324                         94.91                          99.90
 Conv_1081_quant                2099200.00                     838769                         95.02                          99.90
 Conv_1036_quant                2359808.00                     940759                         95.02                          99.98
 Conv_1059_quant                1050624.00                     436233                         94.84                          99.81
 Conv_1105_quant                1049088.00                     421930                         94.98                          99.95
 Conv_1128_quant                2359808.00                     942648                         95.01                          99.98
 Conv_1151_quant                1050624.00                     434543                         94.86                          99.81
 Conv_1175_quant                1049088.00                     422771                         94.97                          99.95
 Conv_1198_quant                2359808.00                     946426                         94.99                          99.98
 Conv_1221_quant                1050624.00                     436233                         94.84                          99.81
 Gemm_1239                      2049000.00                     65568000                       0.00                           0.00

Parameters by types:
 TYPE                           SIZE                           SPARSITY                       QUANTIZED
 Weight                         31950722                       87.38                          91.97
 Bias                           881920                         0.00                           0.00
 Total                          32316077                       87.28                          91.87

Ops by types:
 TYPE                           SIZE                           SPARSITY                       QUANTIZED
 QLinearConv                    9580440                        94.90                          100.00
 Gemm                           65568000                       0.00                           0.00
 Add                            512                            0.00                           0.00
 MaxPool                        32                             0.00                           0.00
 Softmax                        32                             0.00                           0.00
 Reshape                        32                             0.00                           0.00
 Shape                          32                             0.00                           0.00
 Relu                           416                            0.00                           0.00
 DequantizeLinear               1056                           0.00                           0.00
 Unsqueeze                      32                             0.00                           0.00
 Gather                         32                             0.00                           0.00
 Concat                         32                             0.00                           0.00
 GlobalAveragePool              32                             0.00                           0.00
 QuantizeLinear                 544                            0.00                           0.00
 Total                          32235349                       87.28                          91.97

Params:
 MODEL                          SPARSITY                       QUANTIZED                      COUNT                          SIZE
 zoo:cv/classification/resnet_v 87.28                          91.87                          25530472                       32316077
 1-50/pytorch/sparseml/imagenet
 /pruned95_uniform_quant-none

Ops:
 MODEL                          SPARSITY                       QUANTIZED                      COUNT                          SIZE
 zoo:cv/classification/resnet_v 87.28                          91.97                          25530559                       32235349
 1-50/pytorch/sparseml/imagenet
 /pruned95_uniform_quant-none

Overall:
 MODEL                          SPARSITY                       QUANTIZED
 zoo:cv/classification/resnet_v 87.28                          91.92
 1-50/pytorch/sparseml/imagenet
 /pruned95_uniform_quant-none

Comparison Analysis!!!
INFO:root:Running analysis `by_layers`
INFO:root:Running analysis `by_types`
Comparing zoo:cv/classification/resnet_v1-50/pytorch/sparseml/imagenet/pruned95_uniform_quant-none with zoo:cv/classification/resnet_v1-50/pytorch/sparseml/imagenet/pruned95_quant-none
Note: comparison analysis displays differences b/w models
Analysis by Layers:
 NAME                           TOTAL                          SIZE                           SPARSITY                       QUANTIZED
 Conv_13_quant                  0.00                           17797                          -23.02                         0.00
 Conv_37_quant                  0.00                           -12395                         35.60                          0.00
 Conv_105_quant                 0.00                           -27783                         19.95                          0.00
 Conv_60_quant                  0.00                           -35754                         12.04                          0.00
 Conv_83_quant                  0.00                           -24483                         17.58                          0.00
 Conv_129_quant                 0.00                           -24321                         18.27                          0.00
 Conv_152_quant                 0.00                           -44395                         14.95                          0.00
 Conv_175_quant                 0.00                           -22073                         15.85                          0.00
 Conv_199_quant                 0.00                           -28740                         21.59                          0.00
 Conv_222_quant                 0.00                           -62094                         20.91                          0.00
 Conv_245_quant                 0.00                           -24567                         17.64                          0.00
 Conv_269_quant                 0.00                           -68264                         25.64                          0.00
 Conv_337_quant                 0.00                           -36103                         3.39                           0.00
 Conv_292_quant                 0.00                           -100855                        8.52                           0.00
 Conv_315_quant                 0.00                           -59474                         11.00                          0.00
 Conv_361_quant                 0.00                           -6340                          1.20                           0.00
 Conv_384_quant                 0.00                           -15507                         1.31                           0.00
 Conv_407_quant                 0.00                           -16490                         3.05                           0.00
 Conv_431_quant                 0.00                           -42059                         7.96                           0.00
 Conv_454_quant                 0.00                           -60726                         5.13                           0.00
 Conv_477_quant                 0.00                           -53472                         9.89                           0.00
 Conv_501_quant                 0.00                           -51095                         9.67                           0.00
 Conv_524_quant                 0.00                           -96002                         8.11                           0.00
 Conv_547_quant                 0.00                           -42929                         7.94                           0.00
 Conv_571_quant                 0.00                           -201843                        19.10                          0.00
 Conv_639_quant                 0.00                           16063                          -0.38                          0.00
 Conv_594_quant                 0.00                           -29779                         0.63                           0.00
 Conv_617_quant                 0.00                           -184451                        8.66                           0.00
 Conv_663_quant                 0.00                           42949                          -2.04                          0.00
 Conv_686_quant                 0.00                           106826                         -2.26                          0.00
 Conv_709_quant                 0.00                           -26411                         1.24                           0.00
 Conv_733_quant                 0.00                           12843                          -0.61                          0.00
 Conv_756_quant                 0.00                           85555                          -1.81                          0.00
 Conv_779_quant                 0.00                           -54739                         2.57                           0.00
 Conv_803_quant                 0.00                           -17264                         0.82                           0.00
 Conv_826_quant                 0.00                           65702                          -1.39                          0.00
 Conv_849_quant                 0.00                           -37274                         1.75                           0.00
 Conv_873_quant                 0.00                           -24843                         1.18                           0.00
 Conv_896_quant                 0.00                           108716                         -2.30                          0.00
 Conv_919_quant                 0.00                           -3621                          0.17                           0.00
 Conv_943_quant                 0.00                           -51370                         2.44                           0.00
 Conv_966_quant                 0.00                           93118                          -1.97                          0.00
 Conv_989_quant                 0.00                           -20021                         0.94                           0.00
 Conv_1013_quant                0.00                           -585285                        13.90                          0.00
 Conv_1081_quant                0.00                           517073                         -3.07                          0.00
 Conv_1036_quant                0.00                           817970                         -4.33                          0.00
 Conv_1059_quant                0.00                           -135267                        1.60                           0.00
 Conv_1105_quant                0.00                           246266                         -2.93                          0.00
 Conv_1128_quant                0.00                           680067                         -3.60                          0.00
 Conv_1151_quant                0.00                           149639                         -1.77                          0.00
 Conv_1175_quant                0.00                           -237861                        2.83                           0.00
 Conv_1198_quant                0.00                           853862                         -4.52                          0.00
 Conv_1221_quant                0.00                           -45653                         0.54                           0.00
 Gemm_1239                      0.00                           57319546                       -87.42                         0.00

Parameters by types:
 TYPE                           SIZE                           SPARSITY                       QUANTIZED
 Weight                         19291958                       -7.62                          0.00
 Bias                           0                              0.00                           0.00
 Total                          19359159                       -7.62                          0.00

Ops by types:
 TYPE                           SIZE                           SPARSITY                       QUANTIZED
 QLinearConv                    1221036                        -0.65                          0.00
 Gemm                           57319546                       -87.42                         0.00
 Add                            0                              0.00                           0.00
 MaxPool                        0                              0.00                           0.00
 Softmax                        0                              0.00                           0.00
 Reshape                        0                              0.00                           0.00
 Shape                          0                              0.00                           0.00
 Relu                           0                              0.00                           0.00
 DequantizeLinear               0                              0.00                           0.00
 Unsqueeze                      0                              0.00                           0.00
 Gather                         0                              0.00                           0.00
 Concat                         0                              0.00                           0.00
 GlobalAveragePool              0                              0.00                           0.00
 QuantizeLinear                 0                              0.00                           0.00
 Total                          19310799                       -7.62                          0.00

Params:
 MODEL                          SPARSITY                       QUANTIZED                      COUNT                          SIZE
 zoo:cv/classification/resnet_v -7.62                          0.00                           0                              19359159
 1-50/pytorch/sparseml/imagenet
 /pruned95_uniform_quant-none -
 zoo:cv/classification/resnet_v
 1-50/pytorch/sparseml/imagenet
 /pruned95_quant-none

Ops:
 MODEL                          SPARSITY                       QUANTIZED                      COUNT                          SIZE
 zoo:cv/classification/resnet_v -7.62                          0.00                           0                              19310799
 1-50/pytorch/sparseml/imagenet
 /pruned95_uniform_quant-none -
 zoo:cv/classification/resnet_v
 1-50/pytorch/sparseml/imagenet
 /pruned95_quant-none

Overall:
 MODEL                          SPARSITY                       QUANTIZED
 zoo:cv/classification/resnet_v -7.62                          0.00
 1-50/pytorch/sparseml/imagenet
 /pruned95_uniform_quant-none -
 zoo:cv/classification/resnet_v
 1-50/pytorch/sparseml/imagenet
 /pruned95_quant-none

src/sparsezoo/analyze/analysis.py

src/sparsezoo/analyze/utils/models.py

dbogunowicz

Left comments, nice job overall, the framework looks very flexible

KSGulin

LGTM, the subtraction logic looks very solid. I think either more thorough testing of the subtraction functionality or inclusion of a printed example in the PR description would assure more coverage for this functionality, but will leave it to your discretion

…aries from ModelAnalysis Add: Tests for yaml serialization of the same

Move ModelAnalysisSummary to analysis.py Moved analysis tests to a separate dir Remove extraneous test Simplify logic Move Summary generation to a separate class Delete old code

* Add by types * Remove unintended changes

Move pandas import within a pretty_print_summary(...) function that will be deprecated in a future version

* Remove Not Implemented Error * Add: `by-layers` analysis Add: total property to `ZeroNonZeroParams` * Propagate `by-layers` to comparison summary * Rename: LINEAR_OP_TYPES --> TARGETED_LINEAR_OP_TYPES * BugFix: int32 was ignored from dense ops during analysis by types

* Feature: Add compare across types and layers * Renames: * sparsity --> sparsity_percent * quantized --> quantized_percent Update: Size calculation to include sparsity Add: __name__ to avoid using root logger Added: slight improvements to print messaging, and logs Style Updates * Rename: * _get_entries_to_compare --> _get_comparable_entries * Add: docstrings

* Feature: Add compare across types and layers * Renames: * sparsity --> sparsity_percent * quantized --> quantized_percent Update: Size calculation to include sparsity Add: __name__ to avoid using root logger Added: slight improvements to print messaging, and logs Style Updates * Rename: * _get_entries_to_compare --> _get_comparable_entries * Add: docstrings * Add: Support to print multiline rows

* Feature: Add compare across types and layers * Renames: * sparsity --> sparsity_percent * quantized --> quantized_percent Update: Size calculation to include sparsity Add: __name__ to avoid using root logger Added: slight improvements to print messaging, and logs Style Updates * Rename: * _get_entries_to_compare --> _get_comparable_entries * Add: docstrings * Add: Support to print multiline rows * Connect deepsparse.analyze to sparsezoo.analyze Add: PerformanceEntry to entry types, else it will be converted to a ModelEntry Fix: model_name while instantiating PerformanceEntry * Add: supported_graph_percentage to BenchmarkResult * Add node level timings

rahul-tuli · 2023-04-18T14:22:00Z

LGTM, the subtraction logic looks very solid. I think either more thorough testing of the subtraction functionality or inclusion of a printed example in the PR description would assure more coverage for this functionality, but will leave it to your discretion

The Description has been updated with latest output after all PRs were merged into this feature branch; the compare functionality here invokes the __sub__ method

KSGulin

Thanks for updating the description. It was really interesting looking through the difference printout between the two resnet models - even more excited about this feature now

rahul-tuli mentioned this pull request Apr 5, 2023

Analyze feature complete neuralmagic/deepsparse#987

Merged

3 tasks

rahul-tuli self-assigned this Apr 5, 2023

rahul-tuli added the mle-team label Apr 5, 2023

rahul-tuli requested review from KSGulin, bfineran, dbogunowicz and corey-nm April 5, 2023 15:10

rahul-tuli marked this pull request as ready for review April 11, 2023 13:43