Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Various performance improvements #224

Draft
wants to merge 41 commits into
base: main
Choose a base branch
from
Draft
Changes from 1 commit
Commits
Show all changes
41 commits
Select commit Hold shift + click to select a range
0962fe3
Add performance checks to clang-tidy config
hikinggrass Dec 10, 2024
3f76a88
Log module startup time for JavaScript and Python modules as well
hikinggrass Dec 10, 2024
2b17dfb
Use const ref for a few more variables and parameters
hikinggrass Dec 10, 2024
f5d4333
Remove serialize function from config
hikinggrass Dec 10, 2024
9d4da56
Refactor schema loading and validation
hikinggrass Dec 10, 2024
9d5496c
Ad some moves that clang-tidy suggested
hikinggrass Dec 10, 2024
a93bdfa
Directly pass the active_modules part of a config to the parse function
hikinggrass Dec 10, 2024
c74055b
Improve yaml file loading performance by reserving the appropriate am…
hikinggrass Dec 10, 2024
b8385de
Log the number of modules started
hikinggrass Dec 10, 2024
0bc601b
Fix narrowing conversion warning in yaml error handler by claming max…
hikinggrass Dec 10, 2024
bb1c2de
Fix widening conversion warning by multiplying two std::size_t already
hikinggrass Dec 10, 2024
0acb24f
Bump version to 0.19.2
hikinggrass Dec 10, 2024
9ddb9ce
Only publish module_names (a mapping of module type to id) once and r…
hikinggrass Dec 11, 2024
d40f7c3
Publish manifest individually
hikinggrass Dec 11, 2024
dbd4dc2
async get() function for MQTT used in get_module_config
hikinggrass Dec 12, 2024
1077b00
Fix everestjs config parsing with new config entry format
hikinggrass Dec 16, 2024
250ee4a
Capturing more vars as refs
hikinggrass Dec 16, 2024
5551fed
Log which topic cause a timeout exception in get()
hikinggrass Dec 16, 2024
c2f2f60
Use get_with_timeout in get_module_config
hikinggrass Dec 18, 2024
b96933b
Fix usage of MQTTSettings uses socket in manager
hikinggrass Dec 18, 2024
e6bf11c
Use get_module_name instead of get_module_info if only the module nam…
hikinggrass Dec 18, 2024
5d63f80
More const ref usage
hikinggrass Dec 18, 2024
04ec269
Re-order config handler in manager
hikinggrass Dec 21, 2024
4b6d16e
everestpy: initialize logging in module ctor not RuntimeSession
hikinggrass Dec 21, 2024
ed5152e
everestpy: add new RuntimeSession ctor that accepts MQTTSettings and …
hikinggrass Dec 21, 2024
30cf7e2
everestpy: add short documentation comments and deprecated RuntimeSes…
hikinggrass Dec 21, 2024
9a90570
More constref and move usage
hikinggrass Dec 21, 2024
c4737e3
Add --retain-topics flag to manager to keep retained topics after sta…
hikinggrass Dec 21, 2024
61603c9
Remove old cleanup_retained_topics functions since this is now provid…
hikinggrass Dec 21, 2024
6c19fb4
clang-format
hikinggrass Dec 21, 2024
6405e8e
Erase MessageHandler for topics without any registered handlers
hikinggrass Jan 2, 2025
7c539ac
Use .at() instead of [] for map access
hikinggrass Jan 2, 2025
1e4f79e
Exit early with EXIT_FAILURE if there are no modules to start
hikinggrass Jan 3, 2025
64e9ae2
Merge remote-tracking branch 'origin/main' into feature/performance-i…
hikinggrass Jan 3, 2025
9b68f6d
Make logged duration of get_module_config less verbose
hikinggrass Jan 6, 2025
abad2b8
Remove redundant unique_ptrs in Validators struct
hikinggrass Jan 7, 2025
3814cdd
Turn load_schema and load_schemas into free functions
hikinggrass Jan 7, 2025
ad1ecda
Move get and get_async from MQTTAbstraction to module config since it…
hikinggrass Jan 7, 2025
96835c0
load_schema and load_schemas: clang-format
hikinggrass Jan 7, 2025
b42680a
Revert to old file loading code for redability reasons
hikinggrass Jan 7, 2025
560c5ab
Merge remote-tracking branch 'origin/main' into feature/performance-i…
hikinggrass Jan 7, 2025
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
16 changes: 12 additions & 4 deletions lib/yaml_loader.cpp
Original file line number Diff line number Diff line change
Expand Up @@ -78,6 +78,16 @@ static nlohmann::ordered_json ryml_to_nlohmann_json(const c4::yml::NodeRef& ryml
}
}

static std::string load_file_content(const std::filesystem::path& path) {
std::ifstream ifs(path.string());
ifs.seekg(0, std::ios::end);
std::string content;
content.reserve(ifs.tellg());
ifs.seekg(0);
content.assign(std::istreambuf_iterator<char>(ifs), std::istreambuf_iterator<char>());
return content;
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is probably too much optimization. The previous version was easier to read - but having a separate function is completely valid.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Agree, I did some quick comparisons between the two and didn't really see much different one way or the other so I reverted this back to the original code

}

static std::string load_yaml_content(std::filesystem::path path) {
namespace fs = std::filesystem;

Expand All @@ -92,16 +102,14 @@ static std::string load_yaml_content(std::filesystem::path path) {

// first check for yaml, if not found try fall back to json and evlog debug deprecated
if (fs::exists(path)) {
std::ifstream ifs(path.string());
return std::string(std::istreambuf_iterator<char>(ifs), std::istreambuf_iterator<char>());
return load_file_content(path);
}

path.replace_extension(".json");

if (fs::exists(path)) {
EVLOG_info << "Deprecated: loaded file in json format";
std::ifstream ifs(path.string());
return std::string(std::istreambuf_iterator<char>(ifs), std::istreambuf_iterator<char>());
return load_file_content(path);
}

// failed to find yaml and json
Expand Down