add timeout thread to avoid rabit hang forever

[chenqin]

We observed some rabit agent hang forever in some production jobs, this is due to fault tolerant underlying assumption "failed worker will retry and catch up".

In some cases where scheduler has limited resources or configured not launching retry task, rest of fleet will hang forever.

Some options

• add heatbeat to tracker and shutdown worker after timeout
• use socket OOB message to propagate and init count down thread, shutdown agent after timeout

heartbeat put periodical weight on tracker which already shows performance issue given large scale cluster. This might not be best approach moving forward.

Socket OOB seems more promising. The idea is when allreduce/broadcast/checkpoint operations return error due to connected peers giving out socket error. CheckAndRecover implementation checks return type which already false and reset links. we might be able to have a singleton thread which sleeps for configurable time before exit program. Only when tracker signal worker with recover signal, we might terminate that singleton timeout thread

  /*!
   * \brief if err_type indicates an error
   *         recover links according to the error type reported
   *        if there is no error, return true
   * \param err_type the type of error happening in the system
   * \return true if err_type is kSuccess, false otherwise
   */
  bool CheckAndRecover(ReturnType err_type);

[chenqin]

merged to master