Model format proposal.

[kavyasrinet]

As of now, we store different files (one for each parameter) when we save the model in fluid, I want to look further into how we can structure this better.

[kavyasrinet]

The current approach in Fluid is as follows:

save_program = Program()
save_block = save_program.global_block()

for each_var in vars: 
     new_var = _clone_var_in_block_(save_block, each_var)
     save_block.append_op(
               type='save',
                inputs={'X': [new_var]},
                outputs={},
                attrs={'file_path': os.path.join(dirname, new_var.name)})

This is an excerpt taken from the save_vars function.

Here, we iterate over all the variables/parameters one by one and create a separate file one for each variable in the parent directory.

I am looking at different ways of efficiently achieving this, without having to create a new file for each variable.

Proposal 1

Inside each op we write to the filename , which is an attribute of the operator.
One proposal is to append to a model file instead of writing to the var_name file. So instead of opening the file in write mode here
we can open it in append mode, i.e. do something similar to the following:

std::ofstream outfile;
if (append)
    outfile.open(name, std::ios_base::app);
else
    outfile.open(name);

Overall, we have one model file that contains a list of operators, one after the other all appended to the same file and the final file is a binary file.

Proposal 2

This proposal is similar to how the most recent version of TensorFlow stores the model. The TensorFlow save method saves three different kinds of files because it stores the graph structure separately from the variable values. The .meta file describes the saved graph structure, so we need to import it before restoring the checkpoint (otherwise it doesn't know what variables the saved checkpoint values correspond to).

In TensorFlow, we have the following files every time a checkpoint is saved:

1. The meta file: This file describes the saved graph structure, includes GraphDef, SaverDef, and so on; then using: tf.train.import_meta_graph('/tmp/model.ckpt.meta'), will restore Saver and Graph.
2. The index file: This file is a string-string immutable table(tensorflow::table::Table). Each key is a name of a tensor and the value is a serialized BundleEntryProto. Each BundleEntryProto describes the metadata of a tensor: i.e. which of the "data" files contain the content of a tensor, the offset into that file, checksum, some auxiliary data, etc.
3. The data file: This file is TensorBundle collection, and has the values of all variables saved in here.

When restoring the model, the following block of code first restores the graph structure from the meta file and then restores all the values of the variables:

with tf.Session() as sess:
    saver = tf.train.import_meta_graph('/tmp/model.ckpt.meta')
    saver.restore(sess, "/tmp/model.ckpt")

There is no file specifically named model.ckpt but we still refer to the saved checkpoint by that name when restoring it, since the Saver source code picks it up from that prefix. In our case, the meta file will be the ProgramDesc.

[kexinzhao]

@kavyasrinet In your Proposal 1, would there be two files (one for model and one for parameters) or one file (for both model and parameters)?

[Xreki]

I think the design in @kavyasrinet 's Proposal 1, is to save the protobuf message and parameters into one file, a single binary file. In this design, the attribute file_path of save_op will be set to the same file, and we need another attribute offset to record the start offset in the saving file.

[abhinavarora]

I prefer storing the Parameters and the Protobuf message separately because it is possible that users might like to explore different Parameters for the same Inference Model. For easy distribution of the model, we can always write a small wrapper that can zip the model and the parameter files together.

[kavyasrinet]

@Xreki Thanks for clarifying, yes that's the approach I was proposing.
@abhinavarora I have thought about that approach but the design for that will need to be well thought out before proposing it, since there will be a bunch of additional components that will need to be written to make that design work with PaddlePaddle. The first approach I have proposed above is the fastest to implement. I will write more about other approaches today/tomorrow.

[kavyasrinet]

Hi @Xreki and @kexinzhao , I have added another proposal in the discussion above. Working on understanding a couple more.

[sidgoyal78]

Even in proposal 1, it is a bit more involved than simply appending. Since we need to retrieve the merged files into separate parameter files (and that seems non-trivial because different weights can have different number of parameters, so the length won't be fixed). Hence, some extra information has to be kept somewhere to successfully retrieve separate params.