[NPU] Add NPU support for unit test

[RUAN-ZX]

Unit tests would fail or skip when device=npu, and we definitely want to test all these wonderful features by official unit tests.
Here comes the commit to add NPU support for unit test. P.S. see what we have already done #4567.

What I do in this commit

1. Just add npu logic branch
   feat: Add npu support for skip_on_arch in tests/unit/util.py
   feat: Add npu support for skip_on_cuda in tests/unit/util.py
   feat: Add npu support for tests/unit/common.py

2. Set_device of accelerator before deepspeed.init_distributed in tests/unit/common.py
   It would be friendlier and easier for other device like npu, if we can set_device of accelerator before init_distributed. Plus, setting device param before init sounds more reasonable.

3. Solve the problem of calling get_accelerator().random().fork_rng with non-cuda device
   Function train_cifar() in tests/unit/alexnet_model.py calls get_accelerator().random().fork_rng without passing device_type explicitly. Unfortunately, torch.random.fork_rng() has default value setting device_type=cuda and non-cuda devices would fail to run. So my solution is explicitly passing device_type=get_accelerator().device_name(), and either cuda or non-cuda devices would perform correctly.

[RUAN-ZX]

@microsoft-github-policy-service agree

[hipudding]

Please add cuda check in "check_environment" to avoid warning meesage when using NPU as the backend.
Please squash these commits into one(use git rebase -i), and describe all the changes.

[RUAN-ZX]

@tjruwase Would you be so kind to review this commit, since we have some other commits based on this? Or maybe you can invite other reviewers to do the job? Thank you.

[RUAN-ZX]

Please add cuda check in "check_environment" to avoid warning meesage when using NPU as the backend. Please squash these commits into one(use git rebase -i), and describe all the changes.

Npu support for check_environment will be done in another PR :)
For commits squashing, I think these commits should be seperated for clarity :)
If you have more suggestions, please let me know, thank you for your advice.

[RUAN-ZX]

@tjruwase It seems that the CI problem might be fixed by loadams in #4590? If so, would you please launch CI for me again?
Several PRs (#4588, #4585, #4578, etc) have met the same problem as below:
unit/inference/test_inference.py::TestMPSize::test[fp16-bloom] FAILED [ 91%]

P.S. I have push new code for solving problems, you can see that in no.3 description in the first comment. Thanks :)

[RUAN-ZX]

@tjruwase Perhaps #4591 has solved the problem of CI? Cause I see the latest commit #4598 manages to pass. Would you launch CI for me again?

[RUAN-ZX]

@tjruwase Perhaps #4591 has solved the problem of CI? Cause I see the latest commit #4598 manages to pass. Would you launch CI for me again?

@tjruwase Could you launch CI for me again? The last two commits have already pass.

[RUAN-ZX]

@tjruwase Would you please launch CI for this PR? I have been waiting for a long time. Or maybe is there something that I need to improve? In that case I really hope you can point that out for me, so that I can fix it ASAP and do better next time. Thanks :)

[tjruwase]

@RUAN-ZX, apologies for the delay in merging this PR. We had to push out a scheduled release last week. Unfortunately, I notice a new CI failure https://github.com/microsoft/DeepSpeed/actions/runs/6785863907/job/18445092642?pr=4569.

Can you please take a look? It seems to have been caused by a recent merge. I will play close attention to ensure this PR is merged this week. Thanks for your patience.

[RUAN-ZX]

@RUAN-ZX, apologies for the delay in merging this PR. We had to push out a scheduled release last week. Unfortunately, I notice a new CI failure https://github.com/microsoft/DeepSpeed/actions/runs/6785863907/job/18445092642?pr=4569.

Can you please take a look? It seems to have been caused by a recent merge. I will play close attention to ensure this PR is merged this week. Thanks for your patience.

Thank you! @tjruwase About the failure, I found an assert error: CUDA_HOME does not exist, unable to compile CUDA op(s) from op_builder/builder.py, and eventually it uses cuda_home = os.environ.get('CUDA_HOME') or os.environ.get('CUDA_PATH') to get CUDA_HOME in torch\utils\cpp_extension.py.
There is no code that may change this env param and I don't understand why CUDA_HOME is None :)

[RUAN-ZX]

@tjruwase I have solved problems raised by pre-commit hooks, please launch CI again :)