Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Various performance improvements #224

Draft
wants to merge 41 commits into
base: main
Choose a base branch
from
Draft
Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
41 commits
Select commit Hold shift + click to select a range
0962fe3
Add performance checks to clang-tidy config
hikinggrass Dec 10, 2024
3f76a88
Log module startup time for JavaScript and Python modules as well
hikinggrass Dec 10, 2024
2b17dfb
Use const ref for a few more variables and parameters
hikinggrass Dec 10, 2024
f5d4333
Remove serialize function from config
hikinggrass Dec 10, 2024
9d4da56
Refactor schema loading and validation
hikinggrass Dec 10, 2024
9d5496c
Ad some moves that clang-tidy suggested
hikinggrass Dec 10, 2024
a93bdfa
Directly pass the active_modules part of a config to the parse function
hikinggrass Dec 10, 2024
c74055b
Improve yaml file loading performance by reserving the appropriate am…
hikinggrass Dec 10, 2024
b8385de
Log the number of modules started
hikinggrass Dec 10, 2024
0bc601b
Fix narrowing conversion warning in yaml error handler by claming max…
hikinggrass Dec 10, 2024
bb1c2de
Fix widening conversion warning by multiplying two std::size_t already
hikinggrass Dec 10, 2024
0acb24f
Bump version to 0.19.2
hikinggrass Dec 10, 2024
9ddb9ce
Only publish module_names (a mapping of module type to id) once and r…
hikinggrass Dec 11, 2024
d40f7c3
Publish manifest individually
hikinggrass Dec 11, 2024
dbd4dc2
async get() function for MQTT used in get_module_config
hikinggrass Dec 12, 2024
1077b00
Fix everestjs config parsing with new config entry format
hikinggrass Dec 16, 2024
250ee4a
Capturing more vars as refs
hikinggrass Dec 16, 2024
5551fed
Log which topic cause a timeout exception in get()
hikinggrass Dec 16, 2024
c2f2f60
Use get_with_timeout in get_module_config
hikinggrass Dec 18, 2024
b96933b
Fix usage of MQTTSettings uses socket in manager
hikinggrass Dec 18, 2024
e6bf11c
Use get_module_name instead of get_module_info if only the module nam…
hikinggrass Dec 18, 2024
5d63f80
More const ref usage
hikinggrass Dec 18, 2024
04ec269
Re-order config handler in manager
hikinggrass Dec 21, 2024
4b6d16e
everestpy: initialize logging in module ctor not RuntimeSession
hikinggrass Dec 21, 2024
ed5152e
everestpy: add new RuntimeSession ctor that accepts MQTTSettings and …
hikinggrass Dec 21, 2024
30cf7e2
everestpy: add short documentation comments and deprecated RuntimeSes…
hikinggrass Dec 21, 2024
9a90570
More constref and move usage
hikinggrass Dec 21, 2024
c4737e3
Add --retain-topics flag to manager to keep retained topics after sta…
hikinggrass Dec 21, 2024
61603c9
Remove old cleanup_retained_topics functions since this is now provid…
hikinggrass Dec 21, 2024
6c19fb4
clang-format
hikinggrass Dec 21, 2024
6405e8e
Erase MessageHandler for topics without any registered handlers
hikinggrass Jan 2, 2025
7c539ac
Use .at() instead of [] for map access
hikinggrass Jan 2, 2025
1e4f79e
Exit early with EXIT_FAILURE if there are no modules to start
hikinggrass Jan 3, 2025
64e9ae2
Merge remote-tracking branch 'origin/main' into feature/performance-i…
hikinggrass Jan 3, 2025
9b68f6d
Make logged duration of get_module_config less verbose
hikinggrass Jan 6, 2025
abad2b8
Remove redundant unique_ptrs in Validators struct
hikinggrass Jan 7, 2025
3814cdd
Turn load_schema and load_schemas into free functions
hikinggrass Jan 7, 2025
ad1ecda
Move get and get_async from MQTTAbstraction to module config since it…
hikinggrass Jan 7, 2025
96835c0
load_schema and load_schemas: clang-format
hikinggrass Jan 7, 2025
b42680a
Revert to old file loading code for redability reasons
hikinggrass Jan 7, 2025
560c5ab
Merge remote-tracking branch 'origin/main' into feature/performance-i…
hikinggrass Jan 7, 2025
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
56 changes: 36 additions & 20 deletions include/utils/config.hpp
Original file line number Diff line number Diff line change
Expand Up @@ -30,14 +30,27 @@ struct RuntimeSettings;
///
/// \brief A structure that contains all available schemas
///
struct schemas {
struct Schemas {
nlohmann::json config; ///< The config schema
nlohmann::json manifest; ///< The manifest scheme
nlohmann::json interface; ///< The interface schema
nlohmann::json type; ///< The type schema
nlohmann::json error_declaration_list; ///< The error-declaration-list schema
};

struct Validators {
std::unique_ptr<nlohmann::json_schema::json_validator> config;
std::unique_ptr<nlohmann::json_schema::json_validator> manifest;
std::unique_ptr<nlohmann::json_schema::json_validator> type;
std::unique_ptr<nlohmann::json_schema::json_validator> interface;
std::unique_ptr<nlohmann::json_schema::json_validator> error_declaration_list;
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looking a the class json_schema::json_validator, it already only contains a unique_ptr. So you don't need to put it into a unique_ptr again. Just use it as a regular value type and std::move it.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

done

};

struct SchemaValidation {
Schemas schemas;
Validators validators;
};

///
/// \brief Allowed format of a type URI, which are of a format like this /type_file_name#/TypeName
///
Expand All @@ -50,6 +63,17 @@ struct ImplementationInfo {
std::string impl_intf;
};

///
/// \brief A simple json schema loader that uses the builtin draft7 schema of
/// the json schema validator when it encounters it, throws an exception
/// otherwise
void loader(const nlohmann::json_uri& uri, nlohmann::json& schema);

///
/// \brief An extension to the default format checker of the json schema
/// validator supporting uris
void format_checker(const std::string& format, const std::string& value);

///
/// \brief Base class for configs
///
Expand All @@ -62,7 +86,7 @@ class ConfigBase {
nlohmann::json interfaces;
nlohmann::json interface_definitions;
nlohmann::json types;
schemas _schemas;
Schemas schemas;

std::unordered_map<std::string, ModuleTierMappings> tier_mappings;
// experimental caches
Expand Down Expand Up @@ -190,6 +214,8 @@ class ManagerConfig : public ConfigBase {
private:
const ManagerSettings& ms;
std::unordered_map<std::string, std::optional<TelemetryConfig>> telemetry_configs;
Validators validators;
std::unique_ptr<nlohmann::json_schema::json_validator> draft7_validator;

///
/// \brief loads and validates the manifest of the module \p module_id using the provided \p module config
Expand Down Expand Up @@ -328,14 +354,15 @@ class Config : public ConfigBase {
/// \brief loads the config.json and manifest.json in the schemes subfolder of
/// the provided \p schemas_dir
///
/// \returns the config and manifest schemas
static schemas load_schemas(const fs::path& schemas_dir);
/// \returns the loaded configs and related validators
static SchemaValidation load_schemas(const fs::path& schemas_dir);

///
/// \brief loads and validates a json schema at the provided \p path
///
/// \returns the loaded json schema as a json object
static nlohmann::json load_schema(const fs::path& path);
/// \returns the loaded json schema as a json object as well as a related schema validator
static std::tuple<nlohmann::json, std::unique_ptr<nlohmann::json_schema::json_validator>>
load_schema(const fs::path& path);

///
/// \brief loads all module manifests relative to the \p main_dir
Expand All @@ -348,25 +375,14 @@ class Config : public ConfigBase {
///
/// \returns a set of object keys
static std::set<std::string> keys(const nlohmann::json& object);

///
/// \brief A simple json schema loader that uses the builtin draft7 schema of
/// the json schema validator when it encounters it, throws an exception
/// otherwise
static void loader(const nlohmann::json_uri& uri, nlohmann::json& schema);

///
/// \brief An extension to the default format checker of the json schema
/// validator supporting uris
static void format_checker(const std::string& format, const std::string& value);
};
} // namespace Everest

NLOHMANN_JSON_NAMESPACE_BEGIN
template <> struct adl_serializer<Everest::schemas> {
static void to_json(nlohmann::json& j, const Everest::schemas& s);
template <> struct adl_serializer<Everest::Schemas> {
static void to_json(nlohmann::json& j, const Everest::Schemas& s);

static void from_json(const nlohmann::json& j, Everest::schemas& s);
static void from_json(const nlohmann::json& j, Everest::Schemas& s);
};
NLOHMANN_JSON_NAMESPACE_END

Expand Down
Loading