[tests] enable test_mixed_adapter_batches_lora_opt_timing on XPU

[faaany]

After Fix:

====================================== short test summary info ======================================
PASSED tests/test_custom_models.py::TestMixedAdapterBatches::test_mixed_adapter_batches_lora_opt_timing
=================================== 1 passed, 1 warning in 7.59s ====================================

Just like other tests in this file, this function should not only apply to NV GPU. We can actually remove this test marker.

[faaany]

@BenjaminBossan

[BenjaminBossan]

The issue with this change is that this would mean the test also runs on CPU. As the comment further below indicates, we want to avoid this to prevent flakiness:

peft/tests/test_custom_models.py

Lines 3340 to 3341 in eb5eb6e

	# Measure timing of running base and adapter separately vs using a mixed batch. Note that on CPU, the
	# differences are quite small, so this test requires GPU to avoid flakiness.

I tried the test again just now on CPU and the time differences are indeed much smaller (~25%) compared to GPU (~150%), so this is still true. One solution would be to check if either GPU or XPU is being used.

[faaany]

The issue with this change is that this would mean the test also runs on CPU. As the comment further below indicates, we want to avoid this to prevent flakiness:

peft/tests/test_custom_models.py

Lines 3340 to 3341 in eb5eb6e

	# Measure timing of running base and adapter separately vs using a mixed batch. Note that on CPU, the
	# differences are quite small, so this test requires GPU to avoid flakiness.

I tried the test again just now on CPU and the time differences are indeed much smaller (~25%) compared to GPU (~150%), so this is still true. One solution would be to check if either GPU or XPU is being used.

Sure, should I add a marker called "require_torch_accelerator" just like in accelerate?

[BenjaminBossan]

Sure, should I add a marker called "require_torch_accelerator" just like in accelerate?

Hmm, I think let's not go that far yet, a simple check at the start of the test function for the device is more explicit, we can add an extra decorator if we need the same check in many places.

[faaany]

Sure, should I add a marker called "require_torch_accelerator" just like in accelerate?

Hmm, I think let's not go that far yet, a simple check at the start of the test function for the device is more explicit, we can add an extra decorator if we need the same check in many places.

Sure, let me update!

[faaany]

@BenjaminBossan how about the require_non_cpu solution I introduced in this PR #2026 ?

[BenjaminBossan]

how about the require_non_cpu solution I introduced in this PR #2026 ?

Sounds good, I merged that PR.

[faaany]

rebase done.

[HuggingFaceDocBuilderDev]

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

[BenjaminBossan]

Thanks for extending this test, LGTM.

Btw. is there some public place where I can check the XPU tests?

[faaany]

Thanks for extending this test, LGTM.

Btw. is there some public place where I can check the XPU tests?

do you mean the test summary on XPU?

[BenjaminBossan]

do you mean the test summary on XPU?

Yes, so that me and others can check the current state.

[faaany]

do you mean the test summary on XPU?

Yes, so that me and others can check the current state.

Sure, currently we don't upload the test results to a public repo. But let me come out with a solution. Talk to you next Monday.

[BenjaminBossan]

Sure, currently we don't upload the test results to a public repo. But let me come out with a solution. Talk to you next Monday.

Thanks. It's not super high priority, but right now if we ever break something in PEFT for XPU, we won't know until someone comes to us to report it.

[yao-matrix]

Sure, currently we don't upload the test results to a public repo. But let me come out with a solution. Talk to you next Monday.

Thanks. It's not super high priority, but right now if we ever break something in PEFT for XPU, we won't know until someone comes to us to report it.

We have a 2-step plan:

1. screen HF libraries(transformers, accelerate, peft, diffusers, trl) UTs, run in XPU, and fix or extend if we target them run on XPU
2. work w/ Hugging Face to provide CI machines and integrate into CI flow
   We are in 1st now, and progress is transformers & accelerate & peft are done, diffusers & trl are undergoing. Once this step is done, we are glad to work w/ you guys to enable CI, :).

[BenjaminBossan]

Thanks for the update on the plan.