fix dataloader exit terminate error

[heavengate]

PR types

Bug fixes

PR changes

APIs

Describe

fix dataloader exit terminate error
fix terminate called without an active exception
if for loop break and program exit immediately(with no model layers processing) after iterate the first few data in distributed lauch mode, distributed launch will call terminate() to kill main process on each devices, but thread is still iterating to fullfill blocking queue caches, which may cause thread error terminate called without an active exception for terminate is a strong singal and __del__ of DataLoader may not be called, so we add a global link to the last DataLoader instance to call __del__ to clean up resources

testing script
https://github.com/PaddlePaddle/PaddleNLP/blob/develop/examples/simultaneous_translation/stacl/train.py#L121

original error

terminate called without an active exception


--------------------------------------
C++ Traceback (most recent call last):
--------------------------------------
0   paddle::framework::SignalHandle(char const*, int)
1   paddle::platform::GetCurrentTraceBackString[abi:cxx11]()

----------------------
Error Message Summary:
----------------------
FatalError: `Process abort signal` is detected by the operating system.
  [TimeInfo: *** Aborted at 1627618860 (unix time) try "date -d @1627618860" if you are using GNU date ***]
  [SignalInfo: *** SIGABRT (@0x77e) received by PID 1918 (TID 0x7f6e7bfff700) from PID 1918 ***]

INFO 2021-07-30 04:21:09,967 launch_utils.py:327] terminate all the procs
ERROR 2021-07-30 04:21:09,968 launch_utils.py:584] ABORT!!! Out of all 2 trainers, the trainer process with rank=[0] was aborted. Please check its log.
INFO 2021-07-30 04:21:12,971 launch_utils.py:327] terminate all the procs

[paddle-bot-old]

Thanks for your contribution!
Please wait for the result of CI firstly. See Paddle CI Manual for details.

[chenwhql]

LGTM