(Do not merge) (CPU) aggregation of few recent fixes/optimizations

[delock]

This PR is aggregation of a few recent fixes inorder to support customer. This PR contains the following PRs with some other minor fixes:

• Fix for moe Fix RuntimeError for moe on XPU: tensors found at least two devices #5519
• Add compressed ops for 1bit adam Add Compressedbackend for Onebit optimizers #5473
• Enable Yuan AutoTP enable yuan autotp & add conv tp #5428
• Add more tests for XPU Add more tests into XPU CI #5427
• Support SHM inference_all_reduce in TorchBackend [CPU] Support SHM based inference_all_reduce in TorchBackend #5391
• Skip tests if certain OpBuilder not implemented Skip the UT cases that use unimplemented op builders. #5372
• Enable XPU CI Add CI for Intel XPU/Max1100 #5376
• Enable AutoTP for Mixtral-8bx7 Enable mixtral 8x7b autotp #5257
• Fix fused_qkv model accuracy issue fix fused_qkv model accuracy issue #5217
• Enable load from meta for T5 and Mistral Fix T5 and mistral model meta data error #4958
• Enable Falcon model load from_config fix falcon model load from_config meta_data error #4783
• Enable AutoTP for Baichuan model support baichuan model: #4721
• Enable AutoTP for Falcon-40b with odd number of heads fix num_kv_heads sharding in uneven autoTP for Falcon-40b #4712
• More balanced sharding for MLP layers with odd number of heads fix uneven issue & add balance autotp #4697
• Fix replace lmhead issue when ckpt not load fix replace lmhead issue when ckpt not load #4522
• Fix CPU inference workflow [Fix] Fix cpu inference UT failure #4430
• Support StarCoder enable starcode((kv_head=1)) autotp #4896
• Fix Falcon-40B accuracy issue fix falcon-40b accuracy issue #4895

Besides, we have these PRs under track (not in this PR branch but we hope they be merged):

• Needed by CPU training Support cpu tensors without direct device invocation #3842
• short kernel sequence to graph support Capture short kernel sequences to graph #4318
• Larger scale support with MPICH launcher deepspeed/launcher: add launcher_helper as each rank's start portal #4699
• XPU upstream [XPU] XPU accelerator support for Intel GPU device #4547
• WOQ support for autotp support autoTP with weight only quantization in DS inference path #4750
• Support model with .safetensor model file only add sharded loading for safetensors in AutoTP #4854
• (new) Update list of supported AutoTP models [docs] Add new autotp supported model in tutorial #4960

[loadams]

This looks to have been merged already so we can close this PR?

[delock]

Hi @loadams This PR has some new changes that is working on merge into master, I have updated PR description. Can you help reopen this PR with draft mode? Thanks!

We get AutoTP support request for new model from time to time or get bug reports, so sometimes we need to submit new PRs to DeepSpeed for supporting, and add these changes to this PR for early customer access before these changes goes to master. Hope this helps.

[loadams]

Apologies, yes happy to re-open.

[delock]

Close as most pending PRs in this list is merged. We will create an issue as to track current open PRs.