fix: multi gpu gpt not stop generate when end_id

[xbugliu]

With the Multi GPU GPT model, the generate is not stopped immediately when the end_id is hit, and it does not stop until output_seq_len.

[byshiue]

Hi, xbugliu. Thank you for the feedback. We are fixing the issue now and the reason is not only here. We will update the fixing ASAP.

[hongqing1986]

t

With One Gpu, GPT model also not stopped immediately when the end_id is hit
Merging is one the way, good job!

[byshiue]

Hi. This fix would lead to hang on on pipeline parallelism, so we cannot merge it into main branch.
Temporarily fixing this issue in this branch https://github.com/NVIDIA/FasterTransformer/tree/tmp/fix_gpt_earlystop.

[xbugliu]

ok

[Lzhang-hub]

t

With One Gpu, GPT model also not stopped immediately when the end_id is hit Merging is one the way, good job!

@hongqing1986 I deploy a BLOOM model with one GPU, it not stopped immediately when the end_id is hit, and I build the image triton_with_ft with https://github.com/xbugliu/FasterTransformer.git, but it still not work well,
I would really appreciate it if you could describe how you use.

[byshiue]

Do you use the branch https://github.com/NVIDIA/FasterTransformer/tree/tmp/fix_gpt_earlystop?

[Lzhang-hub]

Yes, I also tried this branch.

I changed the Fastertransformer rep in the file CMakelists.txt in https://github.com/triton-inference-server/fastertransformer_backend rep, like :

and build the triton_with_ft image whith

docker build --rm   \
    --build-arg TRITON_VERSION=${CONTAINER_VERSION}   \
    -t ${TRITON_DOCKER_IMAGE} \
    -f docker/Dockerfile \
    .

[byshiue]

Yes, I also tried this branch.

I changed the Fastertransformer rep in the file CMakelists.txt in https://github.com/triton-inference-server/fastertransformer_backend rep, like :

and build the triton_with_ft image whith

docker build --rm   \
    --build-arg TRITON_VERSION=${CONTAINER_VERSION}   \
    -t ${TRITON_DOCKER_IMAGE} \
    -f docker/Dockerfile \
    .

Let's focus on FT's c example first. Can you share how to reproduce the issue on FT c example? Or you don't encounter such issue on FT, but only encounter on backend?

[Lzhang-hub]

Thank you very much, at present, I encountered in the issue of using the backend, I have not tried FT c example, I will try to the FT c examplenext, and provide feedback.

[byshiue]

The issue is fixed in the MR #584 and merge in main branch. Sorry for the late fixing.