More control over external data size #412

dbogunowicz · 2024-01-22T10:37:01Z

As discussed recently:

let's fold the splitting of big internal files into onnx_save
split the model.onnx into graph and constant tensors once the model.onnx is larger then DUMP_EXTERNAL_DATA_THRESHOLD (500MB)
let's allow the user to control the size of created external tensor files
For usage see: [Export Refactor] Support for external data chunks sparseml#1983

mgoin · 2024-01-22T14:44:21Z

@bfineran @markurtz this seems quite small, i would recommend at least a few gigabytes, if not the relatively standard 10GB used in HF

rahul-tuli · 2024-01-22T14:49:15Z

I thought we were splitting the model into data files if it was >400mb and not restricting the model.data file(s) sizes? The only restriction on model.data file sizes is for easier download/upload from/to sparsezoo for which 400mb seems small? Thoughts?

Did not see @mgoin comment, can ignore this

bfineran

noting we shouldn't be updating chunk size here as discussed via slack - @dbogunowicz is aware of changes

dbogunowicz · 2024-01-23T12:47:51Z

@alexm-nm just giving you the heads up, that splitting the external data is now part of the save_onnx function

bfineran · 2024-01-24T15:13:57Z

src/sparsezoo/utils/onnx/external_data.py

        external_data_file = external_data_file or EXTERNAL_ONNX_DATA_NAME
        _LOGGER.warning(
            "The ONNX model is too large to be saved as a single protobuf. "
-            f"Saving with external data: {external_data_file}"
+            "Saving with external data, with file chunks of maximum size "
+            f"{DUMP_EXTERNAL_DATA_THRESHOLD / 1e9} GB"


wrong threshold - also let's downgrade from warning - probably a bit to scary for an expected, default flow

src/sparsezoo/utils/onnx/external_data.py

* `RegistryMixin` improved alias management (#404) * initial commit * add docstrings * simplify * hardening * refactor * format registry lookup strings to be lowercases * standardise aliases * Move evaluator registry (#411) * More control over external data size (#412) --------- Co-authored-by: Rahul Tuli <rahul@neuralmagic.com>

* `RegistryMixin` improved alias management (#404) * initial commit * add docstrings * simplify * hardening * refactor * format registry lookup strings to be lowercases * standardise aliases * Move evaluator registry (#411) * More control over external data size (#412) * When splitting external data, avoid renaming `model.data` to `model.data.1` if only one external data file gets eventually saved (#414) * [model.download] fix function returning nothing (#420) * [BugFix] Path not expanded (#418) * [Fix] Allow for processing Path in the sparsezoo analysis (#417) * Raise TypeError instead of ValueError (#426) * Fix misleading docstring (#416) Add test * add support for benchmark.yaml (#415) * add support for benchmark.yaml recent zoo models use `benchmark.yaml` instead of `benchmarks.yaml`. adding this additional pathway so `benchmark.yaml` is downloaded in the bulk model download * update files filter * fix tests --------- Co-authored-by: dbogunowicz <damian@neuralmagic.com> * [BugFix] Add analyze to init (#421) * Add analyze to init * Move onnxruntime to deps * Print model analysis (#423) * [model.download] fix function returning nothing (#420) * [BugFix] Path not expanded (#418) * print model-analysis * [Fix] Allow for processing Path in the sparsezoo analysis (#417) * add print statement at the end of cli run --------- Co-authored-by: Dipika Sikka <dipikasikka1@gmail.com> Co-authored-by: Rahul Tuli <rahul@neuralmagic.com> Co-authored-by: dbogunowicz <97082108+dbogunowicz@users.noreply.github.com> * Omit scalar weight (#424) * ommit scalar weights: * remove unwanted files * comment * Update src/sparsezoo/utils/onnx/analysis.py Co-authored-by: Benjamin Fineran <bfineran@users.noreply.github.com> --------- Co-authored-by: Benjamin Fineran <bfineran@users.noreply.github.com> --------- Co-authored-by: George <george@neuralmagic.com> Co-authored-by: Dipika Sikka <dipikasikka1@gmail.com> Co-authored-by: dbogunowicz <97082108+dbogunowicz@users.noreply.github.com> Co-authored-by: Benjamin Fineran <bfineran@users.noreply.github.com> * update analyze help message for correctness (#432) * initial commit (#430) * [sparsezoo.analyze] Fix pathway such that it works for larger models (#437) * fix analyze to work with larger models * update for failing tests; add comments * Update src/sparsezoo/utils/onnx/external_data.py Co-authored-by: dbogunowicz <97082108+dbogunowicz@users.noreply.github.com> --------- Co-authored-by: Dipika Sikka <dipikasikka1@gmail.coom> Co-authored-by: dbogunowicz <97082108+dbogunowicz@users.noreply.github.com> * Delete hehe.py (#439) * Download deployment dir for llms (#435) * Download deployment dir for llms * Use path instead of download * only set save_as_external_data to true if the model originally had external data (#442) * Add Channel Wise Quantization Support (#441) * Chunk download (#429) * chunk download, break down into 10 * lint * threads download * draft * chunk download draft * job based download and combining/deleteing chunks * delete old code * lint * fix num jobs if file_size is less than the chunk size * doc string and return types * test * lint * fix type hints (#445) * fix bug if the value is a dict (#447) * [deepsparse.analyze] Fix v1 functionality to work with llms (#451) * fix equivalent changes made to analyze_v2 such that inference session works for llms; update wanrings to be debug printouts * typo * overwrite file (#450) Co-authored-by: 21 <a21@21s-MacBook-Pro.local> * Adds a `numpy_array_representer` to yaml (#454) on runtime, to avoid serialization issues * Avoid division by zero (#457) Avoid log of zero * op analysis total counts had double sparse counts (#461) * Rename legacy analyze to analyze_v1 (#459) * Fixing Quant % Calcuation (#462) * initial fix * style * Include Sparsity in Size Calculation (#463) * initial fix * style * incorporate sparsity into size calculation * quality * op analysis total counts had double sparse counts (#461) * Fixing Quant % Calcuation (#462) * initial fix * style * Include Sparsity in Size Calculation (#463) * initial fix * style * incorporate sparsity into size calculation * quality * Revert "Merge branch 'main' into analyze_cherry_picks" This reverts commit 509fa1a, reversing changes made to 08f94c4. --------- Co-authored-by: dbogunowicz <97082108+dbogunowicz@users.noreply.github.com> Co-authored-by: Rahul Tuli <rahul@neuralmagic.com> Co-authored-by: Dipika Sikka <dipikasikka1@gmail.com> Co-authored-by: Benjamin Fineran <bfineran@users.noreply.github.com> Co-authored-by: dbogunowicz <damian@neuralmagic.com> Co-authored-by: George <george@neuralmagic.com> Co-authored-by: Dipika Sikka <dipikasikka1@gmail.coom> Co-authored-by: 21 <a21@21s-MacBook-Pro.local>

initial commit

1258089

dbogunowicz mentioned this pull request Jan 22, 2024

[Export Refactor] Support for external data chunks neuralmagic/sparseml#1983

Merged

more verbose logging

0962896

dbogunowicz requested a review from bfineran January 22, 2024 12:15

bfineran suggested changes Jan 22, 2024

View reviewed changes

fix the tresholds

4e698b9

dbogunowicz changed the title ~~Set model.data chunk size limit to 400mb~~ More control over external data size Jan 23, 2024

dbogunowicz requested review from bfineran, anmarques and mgoin January 23, 2024 12:47

Merge branch 'main' into feature/damian/set_400_mb_limit

807898f

bfineran reviewed Jan 24, 2024

View reviewed changes

dbogunowicz commented Jan 24, 2024

View reviewed changes

src/sparsezoo/utils/onnx/external_data.py Outdated Show resolved Hide resolved

dbogunowicz commented Jan 24, 2024

View reviewed changes

src/sparsezoo/utils/onnx/external_data.py Outdated Show resolved Hide resolved

Apply suggestions from code review

c45125a

dbogunowicz requested a review from bfineran January 24, 2024 15:22

bfineran approved these changes Jan 24, 2024

View reviewed changes

bfineran merged commit e725fa6 into main Jan 24, 2024
4 checks passed

bfineran deleted the feature/damian/set_400_mb_limit branch January 24, 2024 15:57

dbogunowicz mentioned this pull request Jan 24, 2024

[Cherry-Picks] for 1.7 as of Wed, Jan 24th, 2024 11:01PM #413

Merged

3 tasks

dbogunowicz added a commit that referenced this pull request Jan 24, 2024

More control over external data size (#412)

6c8f5fe

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

More control over external data size #412

More control over external data size #412

dbogunowicz commented Jan 22, 2024 •

edited

Loading

mgoin commented Jan 22, 2024

rahul-tuli commented Jan 22, 2024 •

edited

Loading

bfineran left a comment

dbogunowicz commented Jan 23, 2024

bfineran Jan 24, 2024

More control over external data size #412

More control over external data size #412

Conversation

dbogunowicz commented Jan 22, 2024 • edited Loading

mgoin commented Jan 22, 2024

rahul-tuli commented Jan 22, 2024 • edited Loading

bfineran left a comment

Choose a reason for hiding this comment

dbogunowicz commented Jan 23, 2024

bfineran Jan 24, 2024

Choose a reason for hiding this comment

dbogunowicz commented Jan 22, 2024 •

edited

Loading

rahul-tuli commented Jan 22, 2024 •

edited

Loading