-
Notifications
You must be signed in to change notification settings - Fork 55
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
feat: Entity registration HTTP API #3230
base: main
Are you sure you want to change the base?
feat: Entity registration HTTP API #3230
Conversation
Robot Results
Failed Tests
|
@@ -1,4 +1,6 @@ | |||
//TODO Rename this module to tedge_server |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I would prefer http_server
, stressing this is about HTTP while tedge is obvious in the context of tedge-agent
.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Will be renamed as suggested, right before merging this PR.
pub mod actor; | ||
pub mod error; | ||
mod file_transfer_controller; |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Why not simply file_transfer
?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I've renamed it as you suggested.
It actually started with entity_store_controller
, as I didn't want to name it entity_store
to differentiate this from the original entity store and hence ended up adding the controller
suffix. Did the the same for file_transfer
as well.
But, now I'm wondering if naming them file_transfer_service
and entity_store_service
would have been clearer/better.
e0b458e
to
2151a99
Compare
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I suggest to move the responsibilities of managing the entity store from the file_transfer_service
to the entity_manager
, forwarding HTTP entity registration to the latter.
match EntityRegistrationMessage::try_from(&input) { | ||
Ok(register_message) => { | ||
info!("Hippo: Received reg message: {:?}", register_message); | ||
if let Err(e) = self.entity_store.lock().unwrap().update(register_message) { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Seeing the implementation of this actor - that is really minimal, I wonder if it wouldn't be better to swap the roles of the EntityManagerActor
and the file_transfer_server::entity_store
?
I.e.:
- keep the
file_transfer_server
module mostly focused on file transfer with only one extra responsibility that is to forward entity registration to the entity manager. - give to the entity manager actor the plain responsibility of entity registration and persistence.
So:
- the responsibilities better match the actor names
- the use of mutex makes more sense as on the HTTP side a response is expected, while a channel would be enough with the current design as no response is expected by the MQTT clients.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Addressed in 20f71f4
/// Creates a entity metadata for the main device. | ||
pub fn main_device(device_id: String) -> Self { | ||
Self { | ||
topic_id: EntityTopicId::default_main_device(), |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This is out-of-scope of this PR, but we must really find a way to support non default topic-id schema.
} | ||
|
||
#[derive(Debug, Clone, PartialEq, Eq)] | ||
pub struct EntityMetadata { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
There are now two slightly different definitions of struct EntityMetadata
; the former definition has been kept in entity_store.rs
.
What's the plan?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
As mentioned in #3230 (comment), the entity store and entity cache needs to maintain their own versions of entity metadata as the latter would be much lighter with minimal metadata maintained at the mapper level. Ideally the EntityMetadata
maintained by the mapper should have been as minimal as the XID. But, I was forced to store a bit more info like the entity type, display name and display type just because of the way we map service health status messages by publishing the service creation message itself(which requires all that metadata) for status updates as well. One way to eliminate caching this additional metadata at the mapper level is by making it query the entity store contents from the agent, whenever it requires all that metadata. But that extra HTTP call during the mapping complicates the conversion logic and I just chose caching extra metadata instead of that added complexity. But, open to revisit that decision in a follow-up PR dedicated for that change as the unit test impact would be quite big.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
But that extra HTTP call during the mapping complicates the conversion logic and I just chose caching extra metadata instead of that added complexity.
Agree, caching registration messages is simpler than late-requests to the entity store.
But could we share the type of cached data? Say raw registration messages? Even if some fields are not currently used in the mapper.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
But could we share the type of cached data? Say raw registration messages? Even if some fields are not currently used in the mapper.
Yeah, that is an option. I introduced a different struct to clearly represent the minimal set of things that a mapper would need to do the mapping. And also, there were minor things like the external id (@id
) being an Option
in the actual EntityRegistrationMessage
, but the same being a mandatory field in the entity cache. But yeah, if we want to avoid having too many "metadata representations" across the crates, I can reuse some of the existing ones with implicit assumptions about fields like @id
.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
We really have to reduce the number of similar but different structs.
@@ -3,7 +3,7 @@ use std::collections::VecDeque; | |||
|
|||
/// A bounded buffer that replaces older values with newer ones when full | |||
#[derive(Debug)] | |||
pub(crate) struct RingBuffer<T> { | |||
pub struct RingBuffer<T> { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
One has to avoid making public internal types. This pub
is required only because PendingEntityStore ::telemetry_cache
is pub
. Both can be pub(crate)
.
pub struct RingBuffer<T> { | |
pub(crate) struct RingBuffer<T> { |
@@ -149,6 +152,9 @@ pub enum CumulocityMapperError { | |||
#[error(transparent)] | |||
FromEntityStore(#[from] tedge_api::entity_store::Error), | |||
|
|||
#[error(transparent)] | |||
FromEntityCacheError(#[from] crate::entity_cache::Error), | |||
|
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Why do we have to maintain two error types with the same error cases?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The errors emitted by the entity cache would be similar, but a much smaller set compared to the entity store, as it maintains far less metadata. That's why I didn't want them to share the same error type. Eventually I see the entity store structure moving from tedge_api
to tedge_agent
and the entity cache being mainained by the tedge_mapper
. So, avoiding any direct dependency between them would be better for that move.
NonDefaultTopicScheme(EntityTopicId), | ||
} | ||
|
||
pub(crate) struct EntityCache { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
A comment telling the purpose of this cache would be really helpful.
- What the difference of this cache compared to the entity store?
- How both cooperate?
- What's the content of the pending entities?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Yes, I'll be adding a detailed comment for the EntityCache
and the newly introduced EntityMetadata
structs. Here is the quick summary though:
What the difference of this cache compared to the entity store?
While EntityStore
, owned by the agent, maintains all the metadata associated with an entity like its registration message, device twin data, supported operations etc, the EntityCache
was envisaged as a lightweight map maintained by the mappers to maintain the external id mapping and a minimal set of additional metadata required to do the appropriate mapping to the cloud twins.
How both cooperate?
The entity cache is updated every time a new entity is successfully registered with the entity store(when an entity registration message is emitted either directly by the client or by the agent in response to a registrartion attempt over HTTP). When the mapper receives an entity registration message either way, it assumes that it was successfully registered and adds it to its entity cache.
What's the content of the pending entities?
When the mapper receives an entity registration message with a parent, and if that parent is not registered yet at that point, it is added to the pending entity store and such pending entities are added to the entity cache only when their parent is registered. So, whenever a new entity is registered, the mapper checks its entity cache to find any of its children that arrived earlier, waiting to be registered and registers them along with the parent.
let file_transfer_server_builder = FileTransferServerBuilder::try_bind( | ||
self.config.http_config, | ||
&mut entity_store_actor_builder, | ||
&mut mqtt_actor_builder, | ||
) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I'm surprised to see the file_transfer_server
listening on both HTTP and MQTT.
I would expect the entity_store_actor
to listen to MQTT plus a channel of requests forwarded by the file_transfer_server
.
- What has been your drivers for this design?
- Is this because on the HTTP side a response is expected?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
What has been your drivers for this design?
It might be just a naming confusing during this transition period. Even though the entire HTTP code is currently kept under the module named file_transfer_server
, the plan is to rename this module into http_server
and associated structs into HttpServerBuilder
,HttpServerActor
, etc. Then this HttpActor
would delegate HTTP requests to file_transfer_service
and entity_store_service
respectively. When that happens, the file_transfer_service
will no have any MQTT aspects in it.
While publishing to a nested child device, publish directly to the SmartREST topic of that device (`c8y/s/us/<xid>`), without specifying all of all its ancestors.
db17d27
to
a3fa213
Compare
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Really nice to see that you reach a point with a working implementation. Moving the entity store from the mapper to the mapper was really not an easy task.
I'm still a bit confused by:
- actor naming and branching in the agent
- the two definitions of
struct EntityMetadata
- the two similar error types
c8y_mapper_ext::CumulocityMapperError
andc8y_mapper_ext::ConversionError
mqtt.skip(1).await; // Skip child device registration messages | ||
let msg = mqtt.recv().await.unwrap(); | ||
dbg!(&msg); | ||
// mqtt.skip(1).await; // Skip child device registration messages |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
To be removed before merge.
EntityStoreResponse::Delete(deleted_entities) | ||
} | ||
EntityStoreRequest::MqttMessage(mqtt_message) => { | ||
self.process_mqtt_message(mqtt_message).await; |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I'm surprised that there is a lot more to do when a message is received from MQTT vs HTTP.
Why that?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The difference is in the "pending entities" caching aspect. When the registration is done over MQTT, due to the lack of feedback to the client, we are forced to cache any early child reg messages, until the parents are also registered. But, with the HTTP, when a client tries to register before its parent, we can always immediately send the response back that the parent is not available so that the client can retry later. We don't need to do caching in this case. But, we can if we want to.
Yes, we can discuss the details. Renaming some of the modules are already planned as described here
The
Yeah, this has been a legacy issue that can be fixed. But, I'd prefer to do that in a separate PR as this PR is already huge and resolving merge conflicts is a pain every time someone else touches the mapper. |
Proposed changes
entity_store
usage from the mappers, extracting only the external ID aspects from itfile_transfer_server
module intotedge_server
Types of changes
Paste Link to the issue
Checklist
cargo fmt
as mentioned in CODING_GUIDELINEScargo clippy
as mentioned in CODING_GUIDELINESFurther comments