feat: Entity registration HTTP API

[albinsuresh]

Proposed changes

• Refactor file-transfer logic into its own module
• Add entity store endpoints to existing tedge HTTP server
• Integrate the HTTP endpoints with the entity store
• Refactor file transfer API logic to its own module
• Publish entities registered via HTTP APIs to the MQTT broker
• Accept registration messages via MQTT as well
• Remove entity_store usage from the mappers, extracting only the external ID aspects from it
• Handle auto-registration
• Rename existing file_transfer_server module into tedge_server

Types of changes

• Bugfix (non-breaking change which fixes an issue)
• New feature (non-breaking change which adds functionality)
• Improvement (general improvements like code refactoring that doesn't explicitly fix a bug or add any new functionality)
• Documentation Update (if none of the other choices apply)
• Breaking change (fix or feature that would cause existing functionality to not work as expected)

Paste Link to the issue

Checklist

• I have read the CONTRIBUTING doc
• I have signed the CLA (in all commits with git commit -s)
• I ran cargo fmt as mentioned in CODING_GUIDELINES
• I used cargo clippy as mentioned in CODING_GUIDELINES
• I have added tests that prove my fix is effective or that my feature works
• I have added necessary documentation (if appropriate)

Further comments

[codecov]

Codecov Report

Attention: Patch coverage is 82.82123% with 246 lines in your changes missing coverage. Please review.

Files with missing lines	Patch %	Lines
...ates/core/tedge_agent/src/entity_manager/server.rs	0.00%	117 Missing ⚠️
...s/core/tedge_agent/src/http_server/entity_store.rs	91.08%	6 Missing and 26 partials ⚠️
crates/extensions/c8y_mapper_ext/src/converter.rs	78.43%	8 Missing and 14 partials ⚠️
...ates/extensions/c8y_mapper_ext/src/entity_cache.rs	93.30%	10 Missing and 6 partials ⚠️
crates/core/tedge_agent/src/agent.rs	0.00%	14 Missing ⚠️
crates/core/tedge_api/src/entity.rs	77.41%	10 Missing and 4 partials ⚠️
.../tedge_config/src/tedge_config_cli/tedge_config.rs	0.00%	0 Missing and 12 partials ⚠️
crates/core/tedge_agent/src/http_server/actor.rs	71.42%	2 Missing and 4 partials ⚠️
...xtensions/c8y_mapper_ext/src/operations/convert.rs	85.71%	0 Missing and 4 partials ⚠️
crates/core/tedge_api/src/entity_store.rs	93.18%	2 Missing and 1 partial ⚠️
... and 6 more

Additional details and impacted files

📢 Thoughts on this report? Let us know!

[github-actions]

Robot Results

✅ Passed	❌ Failed	⏭️ Skipped	Total	Pass %	⏱️ Duration
556	0	2	556	100	1h32m59.355476s

[didier-wenzek]

I would prefer http_server, stressing this is about HTTP while tedge is obvious in the context of tedge-agent.

[albinsuresh]

Will be renamed as suggested, right before merging this PR.

[didier-wenzek]

Why not simply file_transfer?

[albinsuresh]

I've renamed it as you suggested.

It actually started with entity_store_controller, as I didn't want to name it entity_store to differentiate this from the original entity store and hence ended up adding the controller suffix. Did the the same for file_transfer as well.

But, now I'm wondering if naming them file_transfer_service and entity_store_service would have been clearer/better.

[didier-wenzek]

I suggest to move the responsibilities of managing the entity store from the file_transfer_service to the entity_manager, forwarding HTTP entity registration to the latter.

[didier-wenzek]

This is out-of-scope of this PR, but we must really find a way to support non default topic-id schema.

[didier-wenzek]

There are now two slightly different definitions of struct EntityMetadata; the former definition has been kept in entity_store.rs.

What's the plan?

[albinsuresh]

As mentioned in #3230 (comment), the entity store and entity cache needs to maintain their own versions of entity metadata as the latter would be much lighter with minimal metadata maintained at the mapper level. Ideally the EntityMetadata maintained by the mapper should have been as minimal as the XID. But, I was forced to store a bit more info like the entity type, display name and display type just because of the way we map service health status messages by publishing the service creation message itself(which requires all that metadata) for status updates as well. One way to eliminate caching this additional metadata at the mapper level is by making it query the entity store contents from the agent, whenever it requires all that metadata. But that extra HTTP call during the mapping complicates the conversion logic and I just chose caching extra metadata instead of that added complexity. But, open to revisit that decision in a follow-up PR dedicated for that change as the unit test impact would be quite big.

[didier-wenzek]

But that extra HTTP call during the mapping complicates the conversion logic and I just chose caching extra metadata instead of that added complexity.

Agree, caching registration messages is simpler than late-requests to the entity store.

But could we share the type of cached data? Say raw registration messages? Even if some fields are not currently used in the mapper.

[albinsuresh]

But could we share the type of cached data? Say raw registration messages? Even if some fields are not currently used in the mapper.

Yeah, that is an option. I introduced a different struct to clearly represent the minimal set of things that a mapper would need to do the mapping. And also, there were minor things like the external id (@id) being an Option in the actual EntityRegistrationMessage, but the same being a mandatory field in the entity cache. But yeah, if we want to avoid having too many "metadata representations" across the crates, I can reuse some of the existing ones with implicit assumptions about fields like @id.

[didier-wenzek]

We really have to reduce the number of similar but different structs.

[albinsuresh]

Resolved by 8751a76 . Even though a single struct is easier to maintain, I'm still tempted to introduce a dedicated EntityMetadataDto struct to capture the message payloads, without overloading this internal EntityMetadata representation with serde aspects. By using the same struct for everything, I'm a bit worried that it'll make future changes to this internal metadata representation more difficult as it'd affect the external representation as well.

[didier-wenzek]

I don't fully get your concerns.

• The extra lines for serde are minimal and only related to naming.
• Is this really using the same struct for everything? These metadata are exchanged over MQTT and stored in-memory. What's wrong?

The issues have been fixed.

[didier-wenzek]

This PR can be merged once renamed the file_transfer_server module.

Thank you for the sustained effort!

Required follow-up tasks:

• PUT and PATCH implementation
• Query over registered entities
• Publishing entity properties as twin metadata
• Propagating entity deletions to the cloud, controlled by a config setting