Add support for quantization with bitsandbytes #490
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
How's the PR going? need any help? |
|
Not sure if I need any help, since we're mostly waiting for the new bitsandbytes release |
Codecov Report
@@ Coverage Diff @@
## master #490 +/- ##
==========================================
- Coverage 86.31% 86.27% -0.05%
==========================================
Files 81 81
Lines 7887 7919 +32
==========================================
+ Hits 6808 6832 +24
- Misses 1079 1087 +8
|
mryab
changed the title
borzunov
reviewed
Aug 23, 2022
README.md
Outdated
| @@ -53,6 +53,11 @@ If your versions of Python and PyTorch match the requirements, you can install h | |||
| pip install hivemind | |||
| ``` | |||
|
|
|||
| Also, if you want to use blockwise 8-bit compression from [bitsandbytes](https://github.com/TimDettmers/bitsandbytes) | |||
| during data transfer, you can [build it from source](https://github.com/TimDettmers/bitsandbytes#compile-from-source) | |||
There was a problem hiding this comment.
Why suggest building it from source?
Instead, we can just suggest to run pip install bitsandbytes==0.32.1 (if you have GPU) or run pip install git+https://github.com/TimDettmers/bitsandbytes.git@4cd7ea6 (for CPU-only builds).
borzunov
reviewed
Aug 23, 2022
borzunov
reviewed
Aug 23, 2022
tests/test_start_server.py
Outdated
| @@ -32,10 +31,15 @@ def test_cli_run_server_identity_path(): | |||
| encoding="utf-8", | |||
| ) | |||
|
|
|||
| # Skip line "UserWarning: The installed version of bitsandbytes was compiled without GPU support. <...>" | |||
There was a problem hiding this comment.
Can we assert that the skipped line contains this?
Then we'll be safe if smth changes, and the code will be self-descriptive (no need for the comment anymore).
Co-authored-by: Alexander Borzunov <borzunov.alexander@gmail.com>
(to make the test less flaky)
This ensures that the server can actually launch in a GPU-enabled environment: otherwise initializing the CUDA context in a parent process prevents it
mryab
force-pushed
the
bnb_integration
branch
from
c661d82 to
4c0fef6
Compare
justheuristic
approved these changes
Sep 9, 2022
|
LGTM, please merge at will |
mryab
added a commit
that referenced
this pull request
Sep 13, 2022
* Add support for quantization with bitsandbytes * Extend the compression benchmark * Add a test for blockwise compression * Add a note to README about bitsandbytes * Install bitsandbytes in tests as well * Verify outputs consistently in test_moe.py (to make the test less flaky) * Pass device="cpu" in test_background_server_identity_path This ensures that the server can actually launch in a GPU-enabled environment: otherwise initializing the CUDA context in a parent process prevents it * Filter bitsandbytes warnings (cherry picked from commit 131f82c)
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This PR integrates blockwise quantization from bitsandbytes as a new compression mechanism of Hivemind. The important part is that it is an optional compression protocol: the user should only install an external library if they are going to need it, and hence the "conditional import"/"extra dependency" parts.
The code on the Hivemind side is pretty simple, but it'd be cool to have a way to include a CPU-only build of bitsandbytes as a dependency, so that we'll be able to both include it without checking for a CUDA version and to test the integration in GHA. @TimDettmers has granted me access to the bitsandbytes repo, so I'm going to work on that first before making this PR as ready to merge.