Feed model engine

[davitbzh]

Work is not ready yet but will give you good idea what FeedModel engine looks looks like:

train_dataset = fs.get_training_dataset("train_dataset", 1)
train_tfdataset = train_dataset.feed(label_name='target', batch_size=32, num_epochs=1).TFRecordDataset()

At the moment only tfrecords reader is implemented (can't yet parse array data types).

[moritzmeister]

A general comment upfront. So this is a lot of logic to convert between Spark Types to the Tensorflow Types. This logic is currently duplicated in java. I think we should find a solution that does not require this massive if ... else ... construct. The TFrecord spark connector is somehow converting between the types, so we should make use of that and call that to infer the TF schema, so we are consistent.

[davitbzh]

@moritzmeister now it can be directly inferred from ftrecord files. no spark and no extra information is required to read. I tested both on tf1 and tf2. However, didn't manage to install hsfs to properly test it. Getting certificate error when I try to connect to fs. Can you please look at it if in general it looks fine?

API now looks like this

anomaly_data = fs.get_training_dataset("anomaly_data", 1)
train = anomaly_data.read("train")
train_input = anomaly_data.feed(target_names=['target'],split='train').tf_record_dataset()