Accelerate refuse to work on balanced_low_0 when GPU 0 is not filled.

[xkszltl]

System Info

- `Accelerate` version: 0.26.1
- Platform: Linux-3.10.0-1160.105.1.el7.x86_64-x86_64-with-glibc2.35
- Python version: 3.10.12
- Numpy version: 1.26.3
- PyTorch version (GPU?): 2.2.0+cu121 (True)
- GPU type: NVIDIA TITAN V
- `Accelerate` default config:
        Not found

accelerate/src/accelerate/accelerator.py

Lines 1324 to 1333 in 6f05bbd

	current_device = list(model_devices)[0]
	current_device_index = current_device.index if isinstance(current_device, torch.device) else current_device
	if torch.device(current_device_index) != self.device:
	# if on the first device (GPU 0) we don't care
	if (self.device.index is not None) or (current_device_index != 0):
	raise ValueError(
	"You can't train a model that has been loaded in 8-bit precision on a different device than the one "
	"you're training on. Make sure you loaded the model on the correct device using for example `device_map={'':torch.cuda.current_device() or device_map={'':torch.xpu.current_device()}"
	)

This part would throw if model is loaded by transformers.AutoModelForCausalLM.from_pretrained(..., device_map="balanced_low_0", ...), because GPU 0 may be left completely unused.
This doesn't seem to be a good behavior as there's no way to tell what's the "first device" without computing the device map first.

[SunMarc]

Hi @xkszltl , could you tell me more about your specific use case and the issue that you are facing ? I don't understand the part about the device map since we do compute it before and we do model_devices = model_devices = set(model.hf_device_map.values())
Note that the related PR to this section is this one.

[xkszltl]

We are loading LLAMA2 7B and Mistral for finetuning in a single node 8 GPU setup.
It works with balanced but not with balanced_low_0 because of the exception thrown here.
I assume that means model device is not GPU0 probably to make room for low_0 requirement?

[xkszltl]

The part I don’t understand is what make GPU 0 so special and worth asserting here?

[SunMarc]

Hi @xkszltl, this check might be outdated since we added the possibility to fine-tune BNB model with naive PP . I'll let @younesbelkada confirms this point !

[github-actions]

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the contributing guidelines are likely to be ignored.

[xkszltl]

Not stale

[xkszltl]

@SunMarc @younesbelkada
Any update on this?

[SunMarc]

Hi @xkszltl, sorry for the delay. Would you like to submit a PR to fix this and check that the tests are passing on transformers and accelerate ? Thanks !

[github-actions]

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the contributing guidelines are likely to be ignored.

[xkszltl]

Not stale

[github-actions]

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the contributing guidelines are likely to be ignored.

[SunMarc]

Closing this since this issue should be solved by this PR