Fundamental Data Reading in C++ #8009
Conversation
JiayiFeng
changed the title
paddle/framework/reader.h
Outdated
| // file readers | ||
|
|
||
| template <typename T> | ||
| class RandomReader : public FileReader { |
There was a problem hiding this comment.
This reader is used by unittests.
Maybe we should use `RandomDataGenerator` instead of `RandomReader`, because
1. It does not read anything.
2. It is confusing between `ShuffleReader` and `RandomReader`.
| ReaderBase* Get() const { return reader_.get(); } | ||
|
|
||
| void ReadNext(std::vector<LoDTensor>* out) { reader_->ReadNext(out); } | ||
| bool HasNext() const { return reader_->HasNext(); } |
There was a problem hiding this comment.
For reader, maybe we should add a Reset() method to read from beginning?
There was a problem hiding this comment.
Yes. And the further discussion is needed about what shall we exactly do when a reader finishes one pass reading.
And Reset has been used by ReaderHolder to make the API consistent with unique_ptr. So I will add readers a ReInit method instead.
paddle/operators/create_reader_op.cc
Outdated
| namespace paddle { | ||
| namespace operators { | ||
|
|
||
| std::vector<framework::DDim> RestoreShapes(const std::vector<int>& shape_concat, |
There was a problem hiding this comment.
this method should be static.
paddle/framework/reader.cc
Outdated
| break; | ||
| } | ||
| } | ||
| std::random_shuffle(buffer_.begin(), buffer_.end()); |
There was a problem hiding this comment.
This implementation could be very slow. However, we can optimize it later.
There was a problem hiding this comment.
Thanks, I didn't know that before. I will add a TOOD here.
| batch_shape[0] += ins_shape[0]; | ||
| } | ||
|
|
||
| LoDTensor out_tensor; |
There was a problem hiding this comment.
Maybe we can invoke MergeTensorOp here.
There was a problem hiding this comment.
The MergeTensorOp can only merge two LoDTensor(true branch out and false branch out). However, in BatchReader we need to merge far more than two LoDTensor.
|
@JiayiFeng Maybe we can move the PR description to design doc? |
|
Sure. I will move them to a new design doc in some later PR. |
Until now, Paddle Fluid's data feeding still wholly dependents on Python code. To get rid of the Python environment and achieve the goal of "wrapping the whole training by a while loop op", a C++ data feeding mechanism is required.
In this PR we show a fundamental C++ data feeding process, which implements the data reading(simulated by random data generator), shuffling and batching. We can regard this PR as the foundation of further developing for #7646.
Following concepts are introduced in this PR:
ReaderBase(code)ReaderBaseis the abstract base class of all Readers.FileReader(code) andDecoratedReader(code)These two classes are derived from the
ReaderBaseand will further be derived by respective specific readers. That is to say, in our design, there are two kinds of readers: file readers and decorated readers. A file reader reads from a file of some specific format, and yield only one instance of data at a time. e.g. RecordIO reader, jpg reader, .... A decorated reader takes another reader(both file reader and decorated reader are OK) as its 'underlying reader'. It gets data from its underlying reader, does some process on them(shuffling, or batching), then yields processed data. The output data of a decorated reader can be a single instance or a batch.ShuffleReader(code) andBatchReader(code) are both decorated readers.All the readers share exactly the same interfaces defined in
ReaderBase. So they can be decorated for more than one time: We can shuffle a reader's outputs and then batch the shuffle outputs. The interface consistency also allows related ops use readers without knowing what they are exactly.ReaderHolder(code)Different readers belong to different class types. It leads to a problem: How can we drop them into
Variables and fetch them out by a unified method? For example, if a Variable holds aBatchReader, we can not get it by the following code:var->Get<ReaderBase>("batch_reader");we have to write:
var->Get<BatchReader>("batch_reader");This requires each time getting a reader from a variable we must know the reader's type exactly. It is nearly impossible.
To solve this problem, we introduce
ReaderHolderas a wrapper. It acts as an empty decorator ofReaderBase, which erasing reader's type. WithReaderHolderwe are able to fetch all types of readers byvar->Get<ReaderHolder>("...")and regard the obtained object as a reader.To create and invoke readers, some now ops are introduced:
CreateReaderOpEach reader has its creating op. File readers' creating ops have no input and yield the created file reader as its output. Decorated readers' creating ops take the underlying readers as inputs and then yield new decorated readers.
ReadOp(code)A reader is only a Variable. It cannot trigger the reading process by itself. So we add the
ReadOpto execute it. AReadOptakes a reader Variable as input. Each time it runs, it invokes the reader‘sReadNext()function and gets a new batch of data(or only one instance of data, if we use file reader directly). The output data of a reader are in the form ofstd::vector<LoDTenosr>, so theReadOpalso needs to split the vector and move LoDTensors to their respective output Variables.