Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Observer Restructure]: Remove MemoryLess Observer; use helper function for dynamic quantization #187

Merged
merged 7 commits into from
Oct 11, 2024

Conversation

dsikka
Copy link
Contributor

@dsikka dsikka commented Oct 8, 2024

Summary

  • As part of the observer restructure, this is the first of the PRs to move observers out of compressed-tensors
  • Removes the MemoryLessObserver and replaces it with a simple helper function which can determine the zp/scales dynamically during the forward pass
  • Will now return None for observers when running dynamic quantization and update the config such that null shows up in the config

Testing

  • Tested llm-compressor examples for int8 and fp8, w8a8 for both; will no longer attach observers and the config will look like the example below:
"quantization_config": {
    "config_groups": {
      "group_0": {
        "input_activations": {
          "actorder": null,
          "block_structure": null,
          "dynamic": true,
          "group_size": null,
          "num_bits": 8,
          "observer": null,
          "observer_kwargs": {},
          "strategy": "token",
          "symmetric": true,
          "type": "float"
        },
        "output_activations": null,
        "targets": [
          "Linear"
        ],
        "weights": {
          "actorder": null,
          "block_structure": null,
          "dynamic": false,
          "group_size": null,
          "num_bits": 8,
          "observer": "minmax",
          "observer_kwargs": {},
          "strategy": "channel",
          "symmetric": true,
          "type": "float"
        }
      }
    },
  • Tested compatibility with existing models (can still be loaded and run as expected)

@dsikka dsikka changed the title Observer Restructure: Reemove MemoryLess Observer; use helper function for dynamic quantization Observer Restructure: Remove MemoryLess Observer; use helper function for dynamic quantization Oct 8, 2024
@dsikka dsikka changed the title Observer Restructure: Remove MemoryLess Observer; use helper function for dynamic quantization [Observer Restructure]: Remove MemoryLess Observer; use helper function for dynamic quantization Oct 8, 2024
Copy link
Member

@mgoin mgoin left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Need to update observer: str = Field( to observer: Optional[str] = Field( in src/compressed_tensors/quantization/quant_args.py

@dsikka dsikka marked this pull request as ready for review October 9, 2024 15:24
rahul-tuli
rahul-tuli previously approved these changes Oct 10, 2024
@dsikka dsikka requested a review from mgoin October 10, 2024 20:41
Copy link
Contributor

@kylesayrs kylesayrs left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lgtm, would like to see more validation/error catching happen earlier in the config

@dsikka dsikka requested a review from kylesayrs October 11, 2024 13:39
Copy link
Member

@rahul-tuli rahul-tuli left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@dsikka dsikka merged commit b2abe72 into main Oct 11, 2024
1 check passed
@dsikka dsikka deleted the update-observers branch October 11, 2024 18:14
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants