Skip to content

add bfloat16 to inference support dtypes #6528

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
tohtana merged 6 commits into from
Sep 27, 2024
Merged

add bfloat16 to inference support dtypes #6528

Changes from all commits
Commits
Show all changes
6 commits
Select commit Hold shift + click to select a range
ed35e07
Extend inference engine dtype validation
Sep 12, 2024
93ba91f
Merge branch 'master' into add_bf16_to_inference_engine
tjruwase Sep 20, 2024
3bd5255
Merge branch 'master' into add_bf16_to_inference_engine
loadams Sep 26, 2024
79b5759
Fix check on supported_dtypes
loadams Sep 26, 2024
18d2f6d
Formatting fixes
loadams Sep 26, 2024
104e889
Merge branch 'master' into add_bf16_to_inference_engine
loadams Sep 26, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
7 changes: 4 additions & 3 deletions deepspeed/inference/engine.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -73,8 +73,9 @@ def __init__(self, model, config):
if hasattr(self.module, "config"):
TransformerPolicy.hf_model_config = self.module.config

if config.dtype == torch.half and not get_accelerator().is_fp16_supported():
raise ValueError("Type fp16 is not supported.")
if config.dtype not in get_accelerator().supported_dtypes():
raise ValueError(
f"Data type {config.dtype} is not supported by {get_accelerator().device_name()} accelerator")

# todo: keep this self.injection_dict because we don't use to change config.injection_policy API
# todo: this will get changed when Molly's PR on auto injection dict is merged
Expand Down Expand Up @@ -324,7 +325,7 @@ def _validate_args(self, mpu, replace_with_kernel_inject):
if self._config.checkpoint is not None and not isinstance(self._config.checkpoint, (str, dict)):
raise ValueError(f"checkpoint must be None, str or dict, got {type(self._config.checkpoint)}")

supported_dtypes = [None, torch.half, torch.int8, torch.float]
supported_dtypes = [None, torch.half, torch.int8, torch.float, torch.bfloat16]
if self._config.dtype not in supported_dtypes:
raise ValueError(f"{self._config.dtype} not supported, valid dtype: {supported_dtypes}")

Expand Down