Refactor TPOT to work directly with numpy matrix instead of pandas DataFrames

[rhiever]

Since we don't maintain the column names any more, it seems that we could replace the pandas DataFrames in our pipeline structure with numpy matrices. We're always changing the data into numpy matrices anyway when passing them to the sklearn operations, so I'm not seeing the point of using pandas DataFrames any more.

This might make TPOT more memory efficient, as we won't introduce DataFrame overhead either.

To make this happen, we would need to:

• Store the train/test indices as internal self variables (in place of having a group column)
• Ensure that the class column is always the last entry in the matrix (in place of having a class column)
• Ensure that the latest guess column is always the second-to-last entry in the matrix (in place of having a guess column)

I believe that this would also make #29 much easier to implement.

Any downsides to this change that we can think of?

[rasbt]

We're always changing the data into numpy matrices anyway when passing them to the sklearn operations, so I'm not seeing the point of using pandas DataFrames any more.

Yes, I agree 100%

Ensure that the class column is always the last entry in the matrix (in place of having a class column)

I'd suggest using 2 arrays ("matrices") instead of 1. The reason is that you may want to have separate Numpy dtypes: The feature array (e.g. X) as float array and the class label array (e.g., y) as integer array. Here, you can simply operate via a meta-array, an index array, that is created at the very beginning of the TPOT tree: something like idx = np.asarray(range(data.shape[0]). Operating via the idx variable then (X[idx], y[idx]) could also help you to avoid creating unnecessary copies of the arrays internally (plus, having "shared memory" for parallelization at some point is probably a good ideas as well)

[dipanjanS]

Really looking forward to this!

[tonyfast]

Perhaps it might be best to use pandas under the hood? It is a supercharged numpy object. Much of tpot relies on pandas machinery and it might make extra work to manage state with it.

[rhiever]

pandas comes with quite a bit of overhead. sklearn doesn't use pandas; I don't think it's necessary for TPOT to use it either.