Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[don't review] OV file path migration to std lib interface #26595

Open
wants to merge 67 commits into
base: master
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from 10 commits
Commits
Show all changes
67 commits
Select commit Hold shift + click to select a range
5d734d2
path_join migration sdt
barnasm1 Sep 13, 2024
8ade84b
test LoadModelMemoryToCore use file descriptor wrapper
barnasm1 Sep 13, 2024
4a8e27e
file_util migrate: path_to_string, get_file_name, get_file_ext, itera…
barnasm1 Sep 13, 2024
3d6e049
imporove tmp_path_item initialization
barnasm1 Sep 13, 2024
3debdc0
CheckpointV1Reader migrate std::path
barnasm1 Sep 13, 2024
9b6be8d
plugin_loader: get_name_from_file, find_plugins migration to std path
barnasm1 Sep 13, 2024
03d02b2
get_filelist_recursive, read_lst_file migration to std path
barnasm1 Sep 13, 2024
0198ed7
cache, models fix
barnasm1 Sep 13, 2024
9747433
directory_exists migration
barnasm1 Sep 13, 2024
44a00be
revert conformance_ir_paths migration
barnasm1 Sep 13, 2024
6ba6ba0
code format
barnasm1 Sep 13, 2024
c8fd872
fix filesystem lib include
barnasm1 Sep 16, 2024
5ebddf6
define include variables
barnasm1 Sep 16, 2024
17e7327
File factory as a function
barnasm1 Sep 16, 2024
771ff7a
update ov_filesystem include
barnasm1 Sep 17, 2024
ac9ad16
revert ov::util::File
barnasm1 Sep 17, 2024
2efd421
Merge branch 'master' into ov_file_path
barnasm1 Sep 18, 2024
9abb3f2
advanced experimental include
barnasm1 Sep 19, 2024
3f3cd8b
clang foramt
barnasm1 Sep 19, 2024
01feeba
Merge branch 'master' into ov_file_path
barnasm1 Sep 19, 2024
3683d4a
Merge branch 'master' into ov_file_path
barnasm1 Sep 20, 2024
b02d931
Merge branch 'master' into ov_file_path
barnasm1 Sep 20, 2024
c9eea67
link libs
barnasm1 Sep 20, 2024
dd78713
fix tests
barnasm1 Sep 20, 2024
b95435f
remove unnecessary dependency
barnasm1 Sep 20, 2024
d22c260
Merge branch 'master' into ov_file_path
barnasm1 Sep 24, 2024
fe43c6e
just for CI test
barnasm1 Sep 24, 2024
d89d6ab
add cpp14 case for windows build
barnasm1 Sep 25, 2024
640065a
fix windows build
barnasm1 Sep 25, 2024
c980624
fix windows compilation errors
barnasm1 Sep 25, 2024
84be552
get_compiled_plugin_path return wstring
barnasm1 Sep 25, 2024
c8be1df
fix windows build errors
barnasm1 Sep 25, 2024
05a581d
missing to string convertion
barnasm1 Sep 25, 2024
82fab71
fix wondows op_conformance_utils
barnasm1 Sep 25, 2024
5c4f8a9
fix lib link for ov_integration_snippet
barnasm1 Sep 25, 2024
a240b5c
fix windows ov_core_unit_tests build
barnasm1 Sep 25, 2024
63de580
Merge branch 'master' into ov_file_path
barnasm1 Sep 25, 2024
8b4ab44
fix for openvino_onnx_frontend build
barnasm1 Sep 25, 2024
6558264
Merge branch 'master' into ov_file_path
barnasm1 Sep 25, 2024
86b4a74
add path to string casts
barnasm1 Sep 26, 2024
a8e943c
Merge branch 'master' into ov_file_path
barnasm1 Sep 26, 2024
ea0e39b
add path to string casts part2
barnasm1 Sep 26, 2024
971b81e
add path to string casts part3
barnasm1 Sep 26, 2024
fa4363b
add path to string casts part4
barnasm1 Sep 26, 2024
0070196
add path to string casts part5
barnasm1 Sep 26, 2024
e366b03
use const + format
barnasm1 Sep 26, 2024
57ece56
rewrite trim_file_name
barnasm1 Sep 26, 2024
939c5a2
turn off filesystem warning
barnasm1 Sep 26, 2024
d73ba01
avoid quoted paths at exception messages
barnasm1 Sep 27, 2024
6817ec6
fix trim_file_name for windows build
barnasm1 Sep 27, 2024
c1d5d3d
cast trim_file_name to string on exception
barnasm1 Sep 27, 2024
dadeb93
revert test update, apply origin logic to new join_paths impl
barnasm1 Sep 27, 2024
1437ed7
Merge branch 'master' into ov_file_path
barnasm1 Sep 30, 2024
3e4aac4
clang code format
barnasm1 Sep 30, 2024
9ad7dad
unnecessary include
barnasm1 Sep 30, 2024
82ac718
Merge branch 'master' into ov_file_path
barnasm1 Oct 1, 2024
7069710
fix android experimental filesystem include
barnasm1 Oct 2, 2024
8d6fd90
fix android experimental filesystem include2
barnasm1 Oct 2, 2024
71108d3
link stdc++fs for STD_FS_NO_LIB_NEEDED
barnasm1 Oct 2, 2024
b820345
revert: link stdc++fs for STD_FS_NO_LIB_NEEDED
barnasm1 Oct 2, 2024
a4d5a38
remove unnecessary checks
barnasm1 Oct 2, 2024
6162145
add cmake fs debug message
barnasm1 Oct 2, 2024
2130b23
Merge branch 'master' into ov_file_path
barnasm1 Oct 3, 2024
6fd43fd
Merge branch 'master' into ov_file_path
barnasm1 Oct 7, 2024
f5cf5fa
Merge branch 'master' into ov_file_path
barnasm1 Oct 8, 2024
8905ff3
fix android linker error
barnasm1 Oct 8, 2024
068afdd
Merge branch 'master' into ov_file_path
barnasm1 Oct 11, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 3 additions & 1 deletion src/common/util/CMakeLists.txt
Original file line number Diff line number Diff line change
Expand Up @@ -37,7 +37,9 @@ add_library(${TARGET_NAME} STATIC ${LIBRARY_SRC} ${PUBLIC_HEADERS})
add_library(openvino::util ALIAS ${TARGET_NAME})
set_target_properties(${TARGET_NAME} PROPERTIES EXPORT_NAME util)

target_link_libraries(${TARGET_NAME} PRIVATE ${CMAKE_DL_LIBS} PUBLIC openvino::pugixml)
# TODO: make lib linkage portable e.g.: openvino/src/bindings/python/thirdparty/pybind11/tests/CMakeLists.txt
target_link_libraries(${TARGET_NAME} PRIVATE ${CMAKE_DL_LIBS} PUBLIC openvino::pugixml stdc++fs )
mlukasze marked this conversation as resolved.
Show resolved Hide resolved
set(STD_FS_LIB stdc++fs)
if (WIN32)
target_link_libraries(${TARGET_NAME} PRIVATE Shlwapi)
endif()
Expand Down
23 changes: 23 additions & 0 deletions src/common/util/include/openvino/util/file_path.hpp
Original file line number Diff line number Diff line change
@@ -0,0 +1,23 @@
// Copyright (C) 2018-2024 Intel Corporation
// SPDX-License-Identifier: Apache-2.0
//

#pragma once

#include <memory>
#include <experimental/filesystem>

namespace ov {
namespace util {

namespace fs = std::experimental::filesystem;
using Path = fs::path;


auto File = [](std::FILE* file) {
auto deleter = [](std::FILE* file){ std::fclose(file); };
return std::unique_ptr<std::FILE, decltype(deleter)> { file, deleter };
};

} // namespace util
} // namespace ov
23 changes: 17 additions & 6 deletions src/common/util/include/openvino/util/file_util.hpp
Original file line number Diff line number Diff line change
Expand Up @@ -11,6 +11,8 @@
#include <vector>

#include "openvino/util/util.hpp"
#include "openvino/util/file_path.hpp"


namespace ov {
namespace util {
Expand Down Expand Up @@ -87,6 +89,11 @@ template <class Path,
const std::string& path_to_string(const Path& path) {
return path;
}
template <class Path,
typename std::enable_if<std::is_same<typename std::decay<Path>::type, ov::util::Path>::value>::type* = nullptr>
std::string path_to_string(const Path& path) {
return path.string();
}

#ifdef OPENVINO_ENABLE_UNICODE_PATH_SUPPORT
/**
Expand Down Expand Up @@ -122,7 +129,7 @@ std::string sanitize_path(const std::string& path);

/// \brief Returns the name with extension for a given path
/// \param path The path to the output file
std::string get_file_name(const std::string& path);
ov::util::Path get_file_name(const ov::util::Path& path);

/**
* @brief Interface function to get absolute path of file
Expand Down Expand Up @@ -161,7 +168,7 @@ void create_directory_recursive(const std::wstring& path);
* @param path - path to directory
* @return true if directory exists, false otherwise
*/
bool directory_exists(const std::string& path);
bool directory_exists(const ov::util::Path& path);

#ifdef OPENVINO_ENABLE_UNICODE_PATH_SUPPORT
/**
Expand Down Expand Up @@ -255,12 +262,16 @@ inline bool file_exists(const std::string& path) {
return file_exists(path.c_str());
}

std::string get_file_ext(const std::string& path);
inline bool file_exists(const ov::util::Path& path) {
return ov::util::fs::exists(path);
}

ov::util::Path get_file_ext(const ov::util::Path& path);
std::string get_directory(const std::string& path);
std::string path_join(const std::vector<std::string>& paths);
ov::util::Path path_join(const std::vector<ov::util::Path>& paths);

void iterate_files(const std::string& path,
const std::function<void(const std::string& file, bool is_dir)>& func,
void iterate_files(const ov::util::Path& path,
const std::function<void(const ov::util::Path& file, bool is_dir)>& func,
bool recurse = false,
bool include_links = false);

Expand Down
107 changes: 18 additions & 89 deletions src/common/util/src/file_util.cpp
Original file line number Diff line number Diff line change
Expand Up @@ -64,31 +64,12 @@
# endif
#endif

std::string ov::util::get_file_name(const std::string& s) {
std::string rc = s;
// Linux-style separator
auto pos = s.find_last_of('/');
if (pos != std::string::npos) {
rc = s.substr(pos + 1);
return rc;
}
// Windows-style separator
pos = s.find_last_of('\\');
if (pos != std::string::npos) {
rc = s.substr(pos + 1);
}
return rc;
ov::util::Path ov::util::get_file_name(const ov::util::Path& s) {
return s.filename();
}

std::string ov::util::get_file_ext(const std::string& s) {
std::string rc = get_file_name(s);
auto pos = rc.find_last_of('.');
if (pos != std::string::npos) {
rc = rc.substr(pos);
} else {
rc = "";
}
return rc;
ov::util::Path ov::util::get_file_ext(const ov::util::Path& s) {
return s.extension();
}

std::string ov::util::get_directory(const std::string& s) {
Expand Down Expand Up @@ -122,60 +103,13 @@ std::wstring ov::util::get_directory(const std::wstring& s) {
#endif

namespace {

std::string join_paths(const std::string& s1, const std::string& s2) {
std::string rc;
if (s2.size() > 0) {
if (s2[0] == '/') {
rc = s2;
} else if (s1.size() > 0) {
rc = s1;
if (rc[rc.size() - 1] != '/') {
#ifndef _WIN32
rc += '/';
#else
rc += '\\';
#endif
}
rc += s2;
} else {
rc = s2;
}
} else {
rc = s1;
}
return rc;
ov::util::Path join_paths(const ov::util::Path& s1, const ov::util::Path& s2) {
return s1 / s2;
}

#ifdef OPENVINO_ENABLE_UNICODE_PATH_SUPPORT
std::wstring join_paths(const std::wstring& s1, const std::wstring& s2) {
std::wstring rc;
if (s2.size() > 0) {
if (s2[0] == '/') {
rc = s2;
} else if (s1.size() > 0) {
rc = s1;
if (rc[rc.size() - 1] != '/') {
# ifndef _WIN32
rc += '/';
# else
rc += '\\';
# endif
}
rc += s2;
} else {
rc = s2;
}
} else {
rc = s1;
}
return rc;
}
#endif
} // namespace

std::string ov::util::path_join(const std::vector<std::string>& paths) {
std::string result;
ov::util::Path ov::util::path_join(const std::vector<ov::util::Path>& paths) {
ov::util::Path result{};
if (paths.empty()) {
return result;
}
Expand All @@ -194,7 +128,7 @@ std::wstring ov::util::path_join_w(const std::vector<std::wstring>& paths) {
}
result = paths[0];
for (size_t i = 1; i < paths.size(); i++) {
result = join_paths(result, paths[i]);
result = join_paths(result, paths[i]).wstring();
}
return result;
}
Expand Down Expand Up @@ -249,12 +183,12 @@ static void iterate_files_worker(const std::string& path,
}
#endif

void ov::util::iterate_files(const std::string& path,
const std::function<void(const std::string& file, bool is_dir)>& func,
void ov::util::iterate_files(const ov::util::Path& path,
const std::function<void(const ov::util::Path& file, bool is_dir)>& func,
bool recurse,
bool include_links) {
std::vector<std::string> files;
std::vector<std::string> dirs;
std::vector<ov::util::Path> files;
std::vector<ov::util::Path> dirs;
#ifdef _WIN32
# ifdef OPENVINO_ENABLE_UNICODE_PATH_SUPPORT
std::wstring pathw = string_to_wstring(path);
Expand Down Expand Up @@ -307,7 +241,7 @@ void ov::util::iterate_files(const std::string& path,
#else
iterate_files_worker(
path,
[&files, &dirs](const std::string& file, bool is_dir) {
[&files, &dirs](const ov::util::Path& file, bool is_dir) {
if (is_dir) {
dirs.push_back(file);
} else {
Expand All @@ -319,10 +253,10 @@ void ov::util::iterate_files(const std::string& path,
#endif

for (const auto& f : files) {
func(f, false);
func(f.native(), false);
}
for (const auto& f : dirs) {
func(f, true);
func(f.native(), true);
}
}

Expand Down Expand Up @@ -443,13 +377,8 @@ void ov::util::create_directory_recursive(const std::string& path) {
}
}

bool ov::util::directory_exists(const std::string& path) {
struct stat sb;

if (stat(path.c_str(), &sb) == 0 && S_ISDIR(sb.st_mode)) {
return true;
}
return false;
bool ov::util::directory_exists(const ov::util::Path& path) {
return ov::util::fs::is_directory(path);
}

#ifdef OPENVINO_ENABLE_UNICODE_PATH_SUPPORT
Expand Down
2 changes: 1 addition & 1 deletion src/core/tests/check.cpp
Original file line number Diff line number Diff line change
Expand Up @@ -55,7 +55,7 @@ TEST(check, ov_throw_exception_check_relative_path_to_source) {
}
using namespace testing;
const auto path = ov::util::path_join({"src", "core", "tests", "check.cpp"});
const auto exp_native_slash = "Exception from " + path + ":";
const auto exp_native_slash = "Exception from " + path.native() + ":";
const auto exp_fwd_slash = "Exception from src/core/tests/check.cpp:";
OV_EXPECT_THROW(OPENVINO_THROW("Test message"),
ov::Exception,
Expand Down
34 changes: 17 additions & 17 deletions src/frontends/common/src/plugin_loader.cpp
Original file line number Diff line number Diff line change
Expand Up @@ -76,19 +76,18 @@ void load_static_plugins(std::vector<PluginInfo>& res) {

#endif // OPENVINO_STATIC_LIBRARY

// TODO: change to std::filesystem for C++17
static std::vector<std::string> list_files(const std::string& path) {
std::vector<std::string> res;
static std::vector<ov::util::Path> list_files(const ov::util::Path& path) {
std::vector<ov::util::Path> res{};
try {
const auto prefix = std::string(FRONTEND_LIB_PREFIX);
const auto suffix = std::string(FRONTEND_LIB_SUFFIX);
const auto prefix = ov::util::Path{FRONTEND_LIB_PREFIX};
const auto suffix = ov::util::Path{FRONTEND_LIB_SUFFIX};
ov::util::iterate_files(
path,
[&res, &prefix, &suffix](const std::string& file_path, bool is_dir) {
[&res, &prefix, &suffix](const ov::util::Path& file_path, bool is_dir) {
auto file = ov::util::get_file_name(file_path);
if (!is_dir && (prefix.empty() || file.compare(0, prefix.length(), prefix) == 0) &&
file.length() > suffix.length() &&
file.rfind(suffix) == (file.length() - std::string(suffix).length())) {
if (!is_dir && (prefix.empty() || file.native().compare(0, prefix.native().length(), prefix) == 0) &&
file.native().length() > suffix.native().length() &&
file.native().rfind(suffix) == (file.native().length() - suffix.native().length())) {
res.push_back(file_path);
}
},
Expand All @@ -100,7 +99,7 @@ static std::vector<std::string> list_files(const std::string& path) {
return res;
}

void ov::frontend::find_plugins(const std::string& dir_name, std::vector<PluginInfo>& res) {
void ov::frontend::find_plugins(const ov::util::Path& dir_name, std::vector<PluginInfo>& res) {
#ifdef OPENVINO_STATIC_LIBRARY
load_static_plugins(res);
#endif // OPENVINO_STATIC_LIBRARY
Expand All @@ -120,13 +119,14 @@ void ov::frontend::find_plugins(const std::string& dir_name, std::vector<PluginI
}

/////////////////////////
std::string PluginInfo::get_name_from_file() const {
const auto prefix = std::string(FRONTEND_LIB_PREFIX);
const auto suffix = std::string(FRONTEND_LIB_SUFFIX);
auto prefix_pos = m_file_name.find(prefix);
auto suffix_pos = m_file_name.rfind(suffix);
if (prefix_pos == 0 && suffix_pos + suffix.length() == m_file_name.length()) {
return m_file_name.substr(prefix_pos + prefix.length(), suffix_pos - prefix_pos - prefix.length());
ov::util::Path PluginInfo::get_name_from_file() const {
const ov::util::Path prefix {FRONTEND_LIB_PREFIX};
const ov::util::Path suffix {FRONTEND_LIB_SUFFIX};
auto prefix_pos = m_file_name.native().find(prefix.native());
auto suffix_pos = m_file_name.native().rfind(suffix.native());
const auto prefix_length = prefix.native().length();
if (prefix_pos == 0 && suffix_pos + suffix.native().length() == m_file_name.native().length()) {
return m_file_name.native().substr(prefix_pos + prefix_length, suffix_pos - prefix_pos - prefix_length);
}
return m_file_name;
}
Expand Down
8 changes: 4 additions & 4 deletions src/frontends/common/src/plugin_loader.hpp
Original file line number Diff line number Diff line change
Expand Up @@ -44,8 +44,8 @@ class PluginInfo {
bool load_internal();

public:
std::string m_file_name; // Plugin file name, e.g. "libopenvino_ir_frontend.so"
std::string m_file_path; // Plugin file full path
ov::util::Path m_file_name; // Plugin file name, e.g. "libopenvino_ir_frontend.so"
ov::util::Path m_file_path; // Plugin file full path

PluginInfo() = default;

Expand All @@ -68,7 +68,7 @@ class PluginInfo {
return m_loaded;
}

std::string get_name_from_file() const;
ov::util::Path get_name_from_file() const;

bool is_file_name_match(const std::string& name) const;

Expand All @@ -77,7 +77,7 @@ class PluginInfo {

// Searches for available plugins in a specified directory
// Appends found plugins to existing list
void find_plugins(const std::string& dir_name, std::vector<PluginInfo>& res);
void find_plugins(const ov::util::Path& dir_name, std::vector<PluginInfo>& res);

} // namespace frontend
} // namespace ov
13 changes: 6 additions & 7 deletions src/frontends/paddle/tests/read_paddle_model_test.cpp
Original file line number Diff line number Diff line change
Expand Up @@ -24,17 +24,16 @@ TEST(Paddle_Reader_Tests, LoadModelMemoryToCore) {
"conv2d_relu/conv2d_relu.pdiparams");

ov::Core core;
auto read_file = [&](const std::string& file_name, size_t& size) {
FILE* sFile = fopen(file_name.c_str(), "r");
fseek(sFile, 0, SEEK_END);
size = ftell(sFile);
auto read_file = [&](const ov::util::Path& file_name, size_t& size) {
auto sFile = ov::util::File(fopen(file_name.c_str(), "r"));
fseek(sFile.get(), 0, SEEK_END);
size = ftell(sFile.get());
uint8_t* ss = (uint8_t*)malloc(size);
rewind(sFile);
const size_t length = fread(&ss[0], 1, size, sFile);
rewind(sFile.get());
const size_t length = fread(&ss[0], 1, size, sFile.get());
if (size != length) {
std::cerr << "file size is not correct\n";
}
fclose(sFile);
return ss;
};

Expand Down
Loading
Loading