Feature request: put the fit time in evaluated_individuals_

[louisabraham]

It would be handy. GridSearchCV does it for example.
I also think I encountered some strange pipelines that did not stop after max_eval_time_mins, and this would help me to reproduce the issue.

[weixuanfu]

This is a old known issue for python.

The way CPython supports threading and asynchronous features has impacts on the accuracy of the timeout. For more background about this issue - that cannot be fixed - Please read Python gurus thoughts about Python threading, the GIL and context switching like these ones:

http://pymotw.com/2/threading/
https://wiki.python.org/moin/GlobalInterpreterLock

But I think it is a good idea to add this fit time into pipeline statistics.

[louisabraham]

I think the relevant code is there

tpot/tpot/base.py

Lines 1236 to 1239 in 507b45d

	parallel = Parallel(n_jobs=self._n_jobs, verbose=0, pre_dispatch='2*n_jobs')
	tmp_result_scores = parallel(
	delayed(partial_wrapped_cross_val_score)(sklearn_pipeline=sklearn_pipeline)
	for sklearn_pipeline in sklearn_pipeline_list[chunk_idx:chunk_idx + chunk_size])

It seems you used the threading_timeoutable from stopit to handle the timeout. Why didn't you use instead the timeout parameter of joblib.Parallel?

[louisabraham]

Oh, the timeout parameter of joblib.Parallel raises a timeout if any task lasts to long.

Would joblib/joblib#366 allow for a more precise time control?

[weixuanfu]

Maybe, I will look into it. But two issues need attentions when using timeout in joblib:

1. TPOT uses joblib in sklearn to avoid adding one more dependency, so we need watch if scikit-learn updates the built-in joblib.
2. this timeout in joblib only works when n_jobs !=1. We need a workaround for this.

[louisabraham]

1. Maybe we should just integrate some custom joblib code?
2. I'm not sure about what causes the issues with threads in the first place, but wouldn't multiprocessing.Process provide a timeout ability as well?