[BUG FIX] place multi-processing init to main method

[lanking520]

If spin a process and run with HuggingFace download at the same time, the error is appearing:

[WARN ] PyProcess - Downloading (?)"pytorch_model.bin";: 100%|??????????| 548M/548M [00:01<00:00, 358MB/s]
[WARN ] PyProcess - [1,0]<stderr>:Traceback (most recent call last):
[WARN ] PyProcess - [1,0]<stderr>:  File "/usr/local/lib/python3.9/dist-packages/fastertransformer/examples/gpt/huggingface_gpt_convert.py", line 203, in <module>
[WARN ] PyProcess - [1,0]<stderr>:    split_and_convert(args)
[WARN ] PyProcess - [1,0]<stderr>:  File "/usr/local/lib/python3.9/dist-packages/fastertransformer/examples/gpt/huggingface_gpt_convert.py", line 161, in split_and_convert
[WARN ] PyProcess - [1,0]<stderr>:    torch.multiprocessing.set_start_method("spawn")
[WARN ] PyProcess - [1,0]<stderr>:  File "/usr/lib/python3.9/multiprocessing/context.py", line 243, in set_start_method
[WARN ] PyProcess - [1,0]<stderr>:    raise RuntimeError('context has already been set')
[WARN ] PyProcess - [1,0]<stderr>:RuntimeError: context has already been set
[INFO ] PyProcess - [1,0]<stdout>:Failed invoke service.invoke_handler()
[INFO ] PyProcess - [1,0]<stdout>:  File "/usr/local/lib/python3.9/dist-packages/fastertransformer/utils/common_utils.py", line 20, in execute_command
[INFO ] PyProcess - [1,0]<stdout>:    subprocess.check_call(command, shell=True)
[INFO ] PyProcess - [1,0]<stdout>:  File "/usr/lib/python3.9/subprocess.py", line 373, in check_call
[INFO ] PyProcess - [1,0]<stdout>:    raise CalledProcessError(retcode, cmd)
[INFO ] PyProcess - [1,0]<stdout>:subprocess.CalledProcessError: Command 'python /usr/local/lib/python3.9/dist-packages/fastertransformer/examples/gpt/huggingface_gpt_convert.py -i gpt2 -o /opt/ml/model/test/ft_gpt_model/ -i_g 1 -weight_data_type fp32' returned non-zero exit status 1.

This is a fix meant to solve the huggingface download and pool spinning

[byshiue]

Can you explain how to reproduce this error?

[lanking520]

Run the last line in the error message

[lanking520]

huggingface_gpt_convert.py -i gpt2 -o /opt/ml/model/test/ft_gpt_model/ -i_g 1 -weight_data_type fp32

[byshiue]

Thank you. We have verified this issue and your solution.