2d sbp data tensor parallel training bug

[L1aoXingyu]

Summary

Training model with 2d sbp data&tensor parallel will raise errors when graph building.

File "/workspace/oneflow/python/oneflow/nn/graph/graph.py", line 576, in _build_graph
    oneflow._oneflow_internal.CurJobBuildAndInferCtx_Complete()
IndexError: vector::_M_range_check: __n (which is 1) >= this->size() (which is 1)

• 120ecad -> error
• f278606 -> ok

Code to reproduce bug

Use this repo and branch https://github.com/Oneflow-Inc/libai/tree/dev_lxy_bert_profile can reproduce the errors.

System Information

version: 0.6.0+cu111.git.f27860650
git_commit: 120ecadf3
cmake_build_type: Release
rdma: False
mlir: False

[strint]

#7032

引入了一个bug，辛苦 @jackalcooper 看下

[jackalcooper]

Code to reproduce bug

Use this repo and branch https://github.com/Oneflow-Inc/libai/tree/dev_lxy_bert_profile can reproduce the errors.

• Could you provide a minimum python script to reproduce the bug?
• Or use gdb to run and get the stacktrace

[leaves-zwx]

I have found the reason for this bug located at

oneflow/oneflow/core/operator/operator.cpp

Line 758 in 89547d0

got_input_sbp_ss << SbpToString(nd_sbp.sbp_parallel(i));

and introduced by #7077.

The reason is that the number of dimensions of nd_sbp is 1 and the number of dimensions of parallel_hierarchy is 2.

I'll submit a pull request to fix this problem.