Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Changes needed for Deepsparse.Analyze #304

Merged
merged 9 commits into from
Apr 18, 2023

Conversation

rahul-tuli
Copy link
Member

@rahul-tuli rahul-tuli commented Apr 13, 2023

This PR will represent all the changes needed to connect
deepsparse.analyze with sparsezoo.analyze

Some Notable Changes include:

  • Add: PerformanceEntry to entry types, else it will be converted to a ModelEntry
  • Fix: model_name while instantiating PerformanceEntry
  • Add: Node level timings
deepsparse.analyze \
  "zoo:cv/classification/resnet_v1-50/pytorch/sparseml/imagenet/pruned95_uniform_quant-none" \
  --by-layers True \
  --by-types True \
  --compare "zoo:cv/classification/resnet_v1-50/pytorch/sparseml/imagenet/pruned95_quant-none" 
(deepsparse3.8) 🥃 deepsparse deepsparse.analyze \
  "zoo:cv/classification/resnet_v1-50/pytorch/sparseml/imagenet/pruned95_uniform_quant-none" \
  --by-layers True \
  --by-types True \
  --compare "zoo:cv/classification/resnet_v1-50/pytorch/sparseml/imagenet/pruned95_quant-none"
2023-04-14 08:42:02 deepsparse.analyze INFO     Starting Analysis ...
INFO:deepsparse.analyze:Starting Analysis ...
2023-04-14 08:42:03 deepsparse.analyze INFO     Analysis complete, collating results...
INFO:deepsparse.analyze:Analysis complete, collating results...
DeepSparse, Copyright 2021-present / Neuralmagic, Inc. version: 1.5.0.20230303 COMMUNITY | () (release) (optimized) (system=avx512, binary=avx512)
[7fe5b6ffd700 >WARN<  operator() ./src/include/wand/utility/warnings.hpp:14] Generating emulated code for quantized (INT8) operations since no VNNI instructions were detected. Set NM_FAST_VNNI_EMULATION=1 to increase performance at the expense of accuracy.
INFO:root:Running analysis `by_layers`
INFO:root:Running analysis `by_types`
Analysis by Layers:
 NAME                           TOTAL                          SIZE                           SPARSITY                       QUANTIZED
 Conv_13_quant                  9472.00                        58679                          24.10                          99.32
 Conv_37_quant                  4160.00                        2252                           93.53                          98.46
 Conv_105_quant                 16640.00                       8982                           93.55                          98.46
 Conv_60_quant                  36928.00                       15234                          94.87                          99.83
 Conv_83_quant                  16640.00                       8912                           93.60                          98.46
 Conv_129_quant                 16448.00                       7148                           94.63                          99.61
 Conv_152_quant                 36928.00                       15293                          94.85                          99.83
 Conv_175_quant                 16640.00                       8968                           93.56                          98.46
 Conv_199_quant                 16448.00                       7135                           94.64                          99.61
 Conv_222_quant                 36928.00                       15293                          94.85                          99.83
 Conv_245_quant                 16640.00                       8898                           93.61                          98.46
 Conv_269_quant                 32896.00                       14270                          94.64                          99.61
 Conv_337_quant                 131584.00                      54951                          94.84                          99.61
 Conv_292_quant                 147584.00                      60015                          94.93                          99.91
 Conv_315_quant                 66048.00                       30872                          94.29                          99.22
 Conv_361_quant                 65664.00                       26789                          94.93                          99.81
 Conv_384_quant                 147584.00                      59660                          94.96                          99.91
 Conv_407_quant                 66048.00                       30764                          94.31                          99.22
 Conv_431_quant                 65664.00                       27264                          94.84                          99.81
 Conv_454_quant                 147584.00                      59779                          94.95                          99.91
 Conv_477_quant                 66048.00                       30926                          94.28                          99.22
 Conv_501_quant                 65664.00                       27264                          94.84                          99.81
 Conv_524_quant                 147584.00                      60015                          94.93                          99.91
 Conv_547_quant                 66048.00                       30926                          94.28                          99.22
 Conv_571_quant                 131328.00                      54740                          94.82                          99.81
 Conv_639_quant                 525312.00                      216426                         94.88                          99.81
 Conv_594_quant                 590080.00                      236811                         94.99                          99.96
 Conv_617_quant                 263168.00                      114163                         94.64                          99.61
 Conv_663_quant                 262400.00                      105477                         94.99                          99.90
 Conv_686_quant                 590080.00                      236339                         95.00                          99.96
 Conv_709_quant                 263168.00                      113737                         94.66                          99.61
 Conv_733_quant                 262400.00                      105267                         95.00                          99.90
 Conv_756_quant                 590080.00                      237284                         94.98                          99.96
 Conv_779_quant                 263168.00                      113524                         94.67                          99.61
 Conv_803_quant                 262400.00                      106319                         94.95                          99.90
 Conv_826_quant                 590080.00                      237284                         94.98                          99.96
 Conv_849_quant                 263168.00                      113737                         94.66                          99.61
 Conv_873_quant                 262400.00                      106319                         94.95                          99.90
 Conv_896_quant                 590080.00                      236811                         94.99                          99.96
 Conv_919_quant                 263168.00                      113737                         94.66                          99.61
 Conv_943_quant                 262400.00                      106530                         94.94                          99.90
 Conv_966_quant                 590080.00                      237284                         94.98                          99.96
 Conv_989_quant                 263168.00                      113737                         94.66                          99.61
 Conv_1013_quant                524800.00                      214324                         94.91                          99.90
 Conv_1081_quant                2099200.00                     838769                         95.02                          99.90
 Conv_1036_quant                2359808.00                     940759                         95.02                          99.98
 Conv_1059_quant                1050624.00                     436233                         94.84                          99.81
 Conv_1105_quant                1049088.00                     421930                         94.98                          99.95
 Conv_1128_quant                2359808.00                     942648                         95.01                          99.98
 Conv_1151_quant                1050624.00                     434543                         94.86                          99.81
 Conv_1175_quant                1049088.00                     422771                         94.97                          99.95
 Conv_1198_quant                2359808.00                     946426                         94.99                          99.98
 Conv_1221_quant                1050624.00                     436233                         94.84                          99.81
 Gemm_1239                      2049000.00                     65568000                       0.00                           0.00

Parameters by types:
 TYPE                           SIZE                           SPARSITY                       QUANTIZED
 Weight                         31950722                       87.38                          91.97
 Bias                           881920                         0.00                           0.00
 Total                          32316077                       87.28                          91.87

Ops by types:
 TYPE                           SIZE                           SPARSITY                       QUANTIZED
 QLinearConv                    9580440                        94.90                          100.00
 Gemm                           65568000                       0.00                           0.00
 Relu                           416                            0.00                           0.00
 GlobalAveragePool              32                             0.00                           0.00
 Gather                         32                             0.00                           0.00
 Unsqueeze                      32                             0.00                           0.00
 Reshape                        32                             0.00                           0.00
 Add                            512                            0.00                           0.00
 QuantizeLinear                 544                            0.00                           0.00
 Shape                          32                             0.00                           0.00
 Concat                         32                             0.00                           0.00
 MaxPool                        32                             0.00                           0.00
 Softmax                        32                             0.00                           0.00
 DequantizeLinear               1056                           0.00                           0.00
 Total                          32235349                       87.28                          91.97

Node Timings for Benchmark # 1:
 NODE_NAME                      AVG_RUNTIME
 Conv_13_quant                  0.24
 MaxPool_23                     0.06
 Conv_37_quant                  0.01
 Conv_60_quant                  0.02
 Conv_83_quant                  0.02
 Conv_105_quant                 0.04
 Conv_129_quant                 0.01
 Conv_152_quant                 0.02
 Conv_175_quant                 0.04
 Conv_199_quant                 0.01
 Conv_222_quant                 0.02
 Conv_245_quant                 0.04
 Conv_269_quant                 0.02
 Conv_292_quant                 0.04
 Conv_315_quant                 0.02
 Conv_337_quant                 0.05
 Conv_361_quant                 0.01
 Conv_384_quant                 0.03
 Conv_407_quant                 0.04
 Conv_431_quant                 0.01
 Conv_454_quant                 0.03
 Conv_477_quant                 0.04
 Conv_501_quant                 0.01
 Conv_524_quant                 0.03
 Conv_547_quant                 0.04
 Conv_571_quant                 0.02
 Conv_594_quant                 0.06
 Conv_617_quant                 0.02
 Conv_639_quant                 0.04
 Conv_663_quant                 0.02
 Conv_686_quant                 0.03
 Conv_709_quant                 0.02
 Conv_733_quant                 0.02
 Conv_756_quant                 0.03
 Conv_779_quant                 0.03
 Conv_803_quant                 0.02
 Conv_826_quant                 0.03
 Conv_849_quant                 0.02
 Conv_873_quant                 0.02
 Conv_896_quant                 0.03
 Conv_919_quant                 0.02
 Conv_943_quant                 0.02
 Conv_966_quant                 0.03
 Conv_989_quant                 0.02
 Conv_1013_quant                0.02
 Conv_1036_quant                0.08
 Conv_1059_quant                0.02
 Conv_1081_quant                0.04
 Conv_1105_quant                0.02
 Conv_1128_quant                0.03
 Conv_1151_quant                0.02
 Conv_1175_quant                0.02
 Conv_1198_quant                0.03
 Conv_1221_quant                0.02
 Add_1230                       0.02
 GlobalAveragePool_1232         0.01
 Gemm_1239                      0.13
 Softmax_1240                   0.00

Params:
 MODEL                          SPARSITY                       QUANTIZED                      COUNT                          SIZE
 zoo:cv/classification/resnet_v 87.28                          91.87                          25530472                       32316077
 1-50/pytorch/sparseml/imagenet
 /pruned95_uniform_quant-none

Ops:
 MODEL                          SPARSITY                       QUANTIZED                      COUNT                          SIZE
 zoo:cv/classification/resnet_v 87.28                          91.97                          25530559                       32235349
 1-50/pytorch/sparseml/imagenet
 /pruned95_uniform_quant-none

Overall:
 MODEL                          LATENCY                        THROUGHPUT                     SUPPORTED_GRAPH                SPARSITY                       QUANTIZED
 zoo:cv/classification/resnet_v 2.07                           482.99                         1.00                           87.28                          91.92
 1-50/pytorch/sparseml/imagenet
 /pruned95_uniform_quant-none

Comparison Analysis!!!
INFO:root:Running analysis `by_layers`
INFO:root:Running analysis `by_types`
Comparing zoo:cv/classification/resnet_v1-50/pytorch/sparseml/imagenet/pruned95_uniform_quant-none with zoo:cv/classification/resnet_v1-50/pytorch/sparseml/imagenet/pruned95_quant-none
Note: comparison analysis displays differences b/w models
Analysis by Layers:
 NAME                           TOTAL                          SIZE                           SPARSITY                       QUANTIZED
 Conv_13_quant                  0.00                           17797                          -23.02                         0.00
 Conv_37_quant                  0.00                           -12395                         35.60                          0.00
 Conv_105_quant                 0.00                           -27783                         19.95                          0.00
 Conv_60_quant                  0.00                           -35754                         12.04                          0.00
 Conv_83_quant                  0.00                           -24483                         17.58                          0.00
 Conv_129_quant                 0.00                           -24321                         18.27                          0.00
 Conv_152_quant                 0.00                           -44395                         14.95                          0.00
 Conv_175_quant                 0.00                           -22073                         15.85                          0.00
 Conv_199_quant                 0.00                           -28740                         21.59                          0.00
 Conv_222_quant                 0.00                           -62094                         20.91                          0.00
 Conv_245_quant                 0.00                           -24567                         17.64                          0.00
 Conv_269_quant                 0.00                           -68264                         25.64                          0.00
 Conv_337_quant                 0.00                           -36103                         3.39                           0.00
 Conv_292_quant                 0.00                           -100855                        8.52                           0.00
 Conv_315_quant                 0.00                           -59474                         11.00                          0.00
 Conv_361_quant                 0.00                           -6340                          1.20                           0.00
 Conv_384_quant                 0.00                           -15507                         1.31                           0.00
 Conv_407_quant                 0.00                           -16490                         3.05                           0.00
 Conv_431_quant                 0.00                           -42059                         7.96                           0.00
 Conv_454_quant                 0.00                           -60726                         5.13                           0.00
 Conv_477_quant                 0.00                           -53472                         9.89                           0.00
 Conv_501_quant                 0.00                           -51095                         9.67                           0.00
 Conv_524_quant                 0.00                           -96002                         8.11                           0.00
 Conv_547_quant                 0.00                           -42929                         7.94                           0.00
 Conv_571_quant                 0.00                           -201843                        19.10                          0.00
 Conv_639_quant                 0.00                           16063                          -0.38                          0.00
 Conv_594_quant                 0.00                           -29779                         0.63                           0.00
 Conv_617_quant                 0.00                           -184451                        8.66                           0.00
 Conv_663_quant                 0.00                           42949                          -2.04                          0.00
 Conv_686_quant                 0.00                           106826                         -2.26                          0.00
 Conv_709_quant                 0.00                           -26411                         1.24                           0.00
 Conv_733_quant                 0.00                           12843                          -0.61                          0.00
 Conv_756_quant                 0.00                           85555                          -1.81                          0.00
 Conv_779_quant                 0.00                           -54739                         2.57                           0.00
 Conv_803_quant                 0.00                           -17264                         0.82                           0.00
 Conv_826_quant                 0.00                           65702                          -1.39                          0.00
 Conv_849_quant                 0.00                           -37274                         1.75                           0.00
 Conv_873_quant                 0.00                           -24843                         1.18                           0.00
 Conv_896_quant                 0.00                           108716                         -2.30                          0.00
 Conv_919_quant                 0.00                           -3621                          0.17                           0.00
 Conv_943_quant                 0.00                           -51370                         2.44                           0.00
 Conv_966_quant                 0.00                           93118                          -1.97                          0.00
 Conv_989_quant                 0.00                           -20021                         0.94                           0.00
 Conv_1013_quant                0.00                           -585285                        13.90                          0.00
 Conv_1081_quant                0.00                           517073                         -3.07                          0.00
 Conv_1036_quant                0.00                           817970                         -4.33                          0.00
 Conv_1059_quant                0.00                           -135267                        1.60                           0.00
 Conv_1105_quant                0.00                           246266                         -2.93                          0.00
 Conv_1128_quant                0.00                           680067                         -3.60                          0.00
 Conv_1151_quant                0.00                           149639                         -1.77                          0.00
 Conv_1175_quant                0.00                           -237861                        2.83                           0.00
 Conv_1198_quant                0.00                           853862                         -4.52                          0.00
 Conv_1221_quant                0.00                           -45653                         0.54                           0.00
 Gemm_1239                      0.00                           57319546                       -87.42                         0.00

Parameters by types:
 TYPE                           SIZE                           SPARSITY                       QUANTIZED
 Weight                         19291958                       -7.62                          0.00
 Bias                           0                              0.00                           0.00
 Total                          19359159                       -7.62                          0.00

Ops by types:
 TYPE                           SIZE                           SPARSITY                       QUANTIZED
 QLinearConv                    1221036                        -0.65                          0.00
 Gemm                           57319546                       -87.42                         0.00
 Relu                           0                              0.00                           0.00
 GlobalAveragePool              0                              0.00                           0.00
 Gather                         0                              0.00                           0.00
 Unsqueeze                      0                              0.00                           0.00
 Reshape                        0                              0.00                           0.00
 Add                            0                              0.00                           0.00
 QuantizeLinear                 0                              0.00                           0.00
 Shape                          0                              0.00                           0.00
 Concat                         0                              0.00                           0.00
 MaxPool                        0                              0.00                           0.00
 Softmax                        0                              0.00                           0.00
 DequantizeLinear               0                              0.00                           0.00
 Total                          19310799                       -7.62                          0.00

Node Timings for Benchmark # 1:
 NODE_NAME                      AVG_RUNTIME
 Conv_13_quant                  -0.00
 MaxPool_23                     0.00
 Conv_37_quant                  -0.01
 Conv_60_quant                  -0.03
 Conv_83_quant                  -0.02
 Conv_105_quant                 -0.02
 Conv_129_quant                 -0.02
 Conv_152_quant                 -0.04
 Conv_175_quant                 -0.01
 Conv_199_quant                 -0.02
 Conv_222_quant                 -0.05
 Conv_245_quant                 -0.01
 Conv_269_quant                 -0.04
 Conv_292_quant                 -0.03
 Conv_315_quant                 -0.01
 Conv_337_quant                 -0.00
 Conv_361_quant                 -0.00
 Conv_384_quant                 -0.01
 Conv_407_quant                 0.01
 Conv_431_quant                 -0.01
 Conv_454_quant                 -0.02
 Conv_477_quant                 0.00
 Conv_501_quant                 -0.01
 Conv_524_quant                 -0.03
 Conv_547_quant                 0.00
 Conv_571_quant                 -0.05
 Conv_594_quant                 -0.00
 Conv_617_quant                 -0.01
 Conv_639_quant                 -0.00
 Conv_663_quant                 0.00
 Conv_686_quant                 0.01
 Conv_709_quant                 -0.01
 Conv_733_quant                 0.00
 Conv_756_quant                 0.01
 Conv_779_quant                 0.00
 Conv_803_quant                 -0.00
 Conv_826_quant                 0.01
 Conv_849_quant                 0.00
 Conv_873_quant                 -0.00
 Conv_896_quant                 0.01
 Conv_919_quant                 0.00
 Conv_943_quant                 -0.00
 Conv_966_quant                 0.01
 Conv_989_quant                 -0.00
 Conv_1013_quant                -0.04
 Conv_1036_quant                0.02
 Conv_1059_quant                -0.00
 Conv_1081_quant                0.01
 Conv_1105_quant                0.01
 Conv_1128_quant                0.02
 Conv_1151_quant                0.00
 Conv_1175_quant                -0.00
 Conv_1198_quant                0.02
 Conv_1221_quant                -0.00
 Add_1230                       0.01
 GlobalAveragePool_1232         -0.00
 Gemm_1239                      0.06
 Softmax_1240                   -0.00

Params:
 MODEL                          SPARSITY                       QUANTIZED                      COUNT                          SIZE
 zoo:cv/classification/resnet_v -7.62                          0.00                           0                              19359159
 1-50/pytorch/sparseml/imagenet
 /pruned95_uniform_quant-none -
 zoo:cv/classification/resnet_v
 1-50/pytorch/sparseml/imagenet
 /pruned95_quant-none

Ops:
 MODEL                          SPARSITY                       QUANTIZED                      COUNT                          SIZE
 zoo:cv/classification/resnet_v -7.62                          0.00                           0                              19310799
 1-50/pytorch/sparseml/imagenet
 /pruned95_uniform_quant-none -
 zoo:cv/classification/resnet_v
 1-50/pytorch/sparseml/imagenet
 /pruned95_quant-none

Overall:
 MODEL                          LATENCY                        THROUGHPUT                     SUPPORTED_GRAPH                SPARSITY                       QUANTIZED
 zoo:cv/classification/resnet_v -0.38                          74.40                          0.00                           -7.62                          0.00
 1-50/pytorch/sparseml/imagenet
 /pruned95_uniform_quant-none -
 zoo:cv/classification/resnet_v
 1-50/pytorch/sparseml/imagenet
 /pruned95_quant-none

 * sparsity --> sparsity_percent
 * quantized --> quantized_percent

 Update: Size calculation to include sparsity
 Add: __name__ to avoid using root logger
 Added: slight improvements to print messaging, and logs
 Style Updates
* _get_entries_to_compare --> _get_comparable_entries
Add: PerformanceEntry to entry types, else it will be converted to a ModelEntry
Fix: model_name while instantiating PerformanceEntry
Copy link
Contributor

@bfineran bfineran left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM - let's update the PR title

src/sparsezoo/analyze/analysis.py Outdated Show resolved Hide resolved
src/sparsezoo/analyze/utils/models.py Show resolved Hide resolved
@rahul-tuli rahul-tuli marked this pull request as ready for review April 14, 2023 14:03
Base automatically changed from multiline-pretty-print to update-summary-generation April 18, 2023 13:25
@rahul-tuli rahul-tuli merged commit 36ec235 into update-summary-generation Apr 18, 2023
@rahul-tuli rahul-tuli deleted the deepsparse-analyze-connection branch April 18, 2023 13:27
@rahul-tuli rahul-tuli changed the title Connect Deepsparse.Analyze Changes needed for Deepsparse.Analyze Apr 18, 2023
rahul-tuli added a commit that referenced this pull request Apr 18, 2023
* Feature: Add compare across types and layers

* Renames:
 * sparsity --> sparsity_percent
 * quantized --> quantized_percent

 Update: Size calculation to include sparsity
 Add: __name__ to avoid using root logger
 Added: slight improvements to print messaging, and logs
 Style Updates

* Rename:
* _get_entries_to_compare --> _get_comparable_entries

* Add: docstrings

* Add: Support to print multiline rows

* Connect deepsparse.analyze to sparsezoo.analyze
Add: PerformanceEntry to entry types, else it will be converted to a ModelEntry
Fix: model_name while instantiating PerformanceEntry

* Add: supported_graph_percentage to BenchmarkResult

* Add node level timings
rahul-tuli added a commit that referenced this pull request Apr 19, 2023
* Add:ModuleAnalysisSummary schema for easily displaying comparing summaries from ModelAnalysis
Add: Tests for yaml serialization of the same

* Move: yaml de-serialization methods to YAMLSerializableBaseModel

* Style

* Add: Tests for serialization, pretty printing, and subtraction

* Get path programmatically

* Add from_analysis method

* Move relevant schemas to models.py
Move ModelAnalysisSummary to analysis.py
Moved analysis tests to a separate dir
Remove extraneous test
Simplify logic
Move Summary generation to a separate class
Delete old code

* Fix failing tests

* Update CLI

* Update CLI

* Add by_types info to analyze api (#292)

* Add by types

* Remove unintended changes

* Style

* Add kwargs

* Fix failing tests

* Fix failing test + `by-layers`

* Add compare functionality

* Add deprecation warning for `ModelAnalysis.pretty_print_summary(...)`
Move pandas import within a pretty_print_summary(...) function that will be deprecated in a future version

* Add `by-layers` support to analyze api(s) (#301)

* Remove Not Implemented Error

* Add: `by-layers` analysis
Add: total property to `ZeroNonZeroParams`

* Propagate `by-layers` to comparison summary

* Rename: LINEAR_OP_TYPES --> TARGETED_LINEAR_OP_TYPES

* BugFix: int32 was ignored from dense ops during analysis by types

* Add support to compare by types and layers (#302)

* Feature: Add compare across types and layers

* Renames:
 * sparsity --> sparsity_percent
 * quantized --> quantized_percent

 Update: Size calculation to include sparsity
 Add: __name__ to avoid using root logger
 Added: slight improvements to print messaging, and logs
 Style Updates

* Rename:
* _get_entries_to_compare --> _get_comparable_entries

* Add: docstrings

* Add: Multiline row printing (#303)

* Feature: Add compare across types and layers

* Renames:
 * sparsity --> sparsity_percent
 * quantized --> quantized_percent

 Update: Size calculation to include sparsity
 Add: __name__ to avoid using root logger
 Added: slight improvements to print messaging, and logs
 Style Updates

* Rename:
* _get_entries_to_compare --> _get_comparable_entries

* Add: docstrings

* Add: Support to print multiline rows

* Connect Deepsparse.Analyze (#304)

* Feature: Add compare across types and layers

* Renames:
 * sparsity --> sparsity_percent
 * quantized --> quantized_percent

 Update: Size calculation to include sparsity
 Add: __name__ to avoid using root logger
 Added: slight improvements to print messaging, and logs
 Style Updates

* Rename:
* _get_entries_to_compare --> _get_comparable_entries

* Add: docstrings

* Add: Support to print multiline rows

* Connect deepsparse.analyze to sparsezoo.analyze
Add: PerformanceEntry to entry types, else it will be converted to a ModelEntry
Fix: model_name while instantiating PerformanceEntry

* Add: supported_graph_percentage to BenchmarkResult

* Add node level timings

* Style
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants