Support inference value2disk #119
Conversation
| @@ -361,6 +365,13 @@ void ModuleInterpreter::invoke(bool express_type) { | |||
| case mem_mode_t::PART_TENSOR_IN_MEM: | |||
| invoke_part_in_mem(express_type); | |||
| break; | |||
| case mem_mode_t::ALL_TENSOR_IN_DISK: | |||
There was a problem hiding this comment.
line 37-47 can not set mem_mode to ALL_TENSOR_IN_DISK, so this branch cannot be executed?
There was a problem hiding this comment.
Yes, there is no corresponding case here. I modified the mem_ mode=mem_ mode_t: : ALL_ TENSOR_ IN_ DISK for testing, perhaps a threshold greater than 16GB is needed to activate the case. I don't know what the threshold is, but I hope you can set it. In the above test, it was passed.
There was a problem hiding this comment.
Please format this change with the clang-format. If you use vscode, it is easy to achieve by this plugin clang-format.
lib/Support/ModuleInterpreter.cpp
Outdated
| fclose(file); | ||
| std::remove("./value2disk.npz"); | ||
| } | ||
| invoke_to_disk("./value2disk.npz",express_type); |
There was a problem hiding this comment.
This hard coding file name "value2disk.npz" could be improved using llvm TempFile or a related strategy.
There was a problem hiding this comment.
In the new commit, the llvm TempFile class was used, which seems to be more complex. In order to obtain the file on the Python side, an additional pybind interface was added because the file name was randomly generated.
| @@ -82,6 +82,7 @@ class ModuleInterpreter { | |||
| std::map<std::string, Value> value_map; | |||
| std::map<std::string, std::shared_ptr<InferenceParameter>> inference_map; | |||
| std::map<std::string, std::shared_ptr<std::vector<float>>> mem_map; | |||
| std::vector<size_t> store_disk_shape; | |||
There was a problem hiding this comment.
Do we need this variable?
During the inference process, the intermediate tensor is saved to disk. The current work is not automatically enabled during the compilation process, and a threshold can be set to enable it. After testing, there were issues with the API. Therefore, the current save API was selected and a new weight file was added for saving. In order to pass the final data comparison test, the disk file was read from the Python end for data comparison.