Simplify InputValidator: Allows pandas frame to directly reach the pipeline

[franchuterivera]

• Moves the encoder that translates pandas dataframes to numpy into the pipeline
• Enhances Auto-Sklearn to work with pandas internally, rather than numpy
• Feature type list is internally translated to a dictionary of column->data type to be robust against different pandas column ordering
• Adds extra check to make sure a pandas frame can produce a pipeline (in order words, this new set of checks make sure that A- a pandas frame reaches the base pipeline without being translated to numpy, and that - we can fit the pipeline with a pandas frame)

[codecov]

Codecov Report

Merging #1135 (3a31f01) into development (0982410) will increase coverage by 0.13%.
The diff coverage is 97.60%.

@@               Coverage Diff               @@
##           development    #1135      +/-   ##
===============================================
+ Coverage        85.83%   85.96%   +0.13%     
===============================================
  Files              137      138       +1     
  Lines            10625    10703      +78     
===============================================
+ Hits              9120     9201      +81     
+ Misses            1505     1502       -3     

Impacted Files	Coverage Δ
autosklearn/data/validation.py	97.14% <ø> (ø)
autosklearn/estimators.py	93.47% <ø> (ø)
autosklearn/data/xy_data_manager.py	84.84% <83.33%> (+1.51%)	⬆️
...onents/data_preprocessing/rescaling/standardize.py	95.23% <88.88%> (-4.77%)	⬇️
...ents/data_preprocessing/rescaling/robust_scaler.py	96.87% <90.90%> (-3.13%)	⬇️
...data_preprocessing/rescaling/abstract_rescaling.py	91.30% <92.30%> (-1.01%)	⬇️
...omponents/data_preprocessing/data_preprocessing.py	90.09% <94.28%> (-0.23%)	⬇️
autosklearn/evaluation/train_evaluator.py	73.58% <94.73%> (+0.11%)	⬆️
...ata_preprocessing/categorical_encoding/encoding.py	96.42% <96.42%> (ø)
autosklearn/data/feature_validator.py	97.50% <98.38%> (+1.14%)	⬆️
... and 37 more

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 0982410...3a31f01. Read the comment docs.

[mfeurer]

Sorry, just a few high-level comments so far. I'll do my best to give more comments in a timely manner.

[mfeurer]

I can't add anything to the comment about the OrdinalEncoder, but according to the docs it can handle NaN: https://scikit-learn.org/stable/modules/generated/sklearn.preprocessing.OrdinalEncoder.html

[mfeurer]

Alright, just checked everything that's not a test.

[mfeurer]

And the last part.

Could you please also check whether we need to unit test the meta-feature calculation (i.e. add a new set of checks for pandas in addition to the current tests which check for sparse and ndarray)?

[mfeurer]

Finished the review of the changes. I'll now think a bit more whether we can include more tests for obscure datasets.

[mfeurer]

Hey, I just took the liberty to debug and change the metafeature calculation, I hope that's okay.

I do have two minor questions left :)

[franchuterivera]

Thanks a lot for the help, that makes a lot of sense. Since yesterday I was thinking why KNN will care about the order of the columns and it was just a dumb error. Sorry about that.

[mfeurer]

Thanks a lot for the help, that makes a lot of sense. Since yesterday I was thinking why KNN will care about the order of the columns and it was just a dumb error. Sorry about that.

No worries, I was looking at DT for which it also makes sense that it depends at the order; but then I realized that the numbers are crazy different so I thought there must be something else to it.