Vision statement for thin-edge core

[didier-wenzek]

Signed-off-by: Didier Wenzek didier.wenzek@free.fr

Proposed changes

Here a proposal for a vision statement for the core of thin-edge.

The purpose of this statement is to set a frame around the discussions on what has been our vision so far, how to improve it and the direction proposed by @TheNeikos and @matthiasbeyer.

The focus is on what and why, letting the how to other discussions.

• Up to now, the focus has been set on flexibility and openness with a large set of features. Do we have to move to a smaller and statically defined core? Where is the balance point?
• What are the criteria to decide for an approach or another.
• What are the key ideas that make the backbone of the project ?
• Is MQTT part of the core or not ?
• Do we need to specify and maintain a thin-edge Json schema?

On purpose:

• The proposal is written as if an agreement has already been reached. Feel free to comment.
• Put aside the docs folder even if I think this vision should be a key element of the documentation. So we can merge it even, if not implemented yet.

Types of changes

• Design Documentation

Checklist

• I have read the CONTRIBUTING doc
• I have signed the CLA (in all commits with git commit -s)

[matthiasbeyer]

rendered version 😉

[cstoidner]

More precise: It does not necesarily provide the plugins (maybe for some), but provides especially plugin interafces, whose make it possible to hook in specific plugins.

[cstoidner]

What do you see in that context exactly as the scope of that "IoT application"? What is part of that application?

1. thin-edge platform (i.E. the core)?
2. plugin and external added components?
3. customer specific configurations?
4. customer specific application with business logic?
5. more?

[didier-wenzek]

First, it's important to differentiate the IoT application from the executables. The IoT application encompasses not only the devices but also the cloud and other systems. Hence, I take your question as related to thin-edge executables running on a device.

So what is included in a thin-edge executable:

1. The core engine (the plugin API and the mechanism to build, connect and run plugins).
2. Any Rust/internal plugin builder that has been explicitly added as a dependency of the executable.
3. The associated configuration has to provide a configuration entry for each of the plugin instances to be created.
4. Customer specific application logic can be implemented as a plugin and included into the executable (at compile time).
5. Customer specific application logic don't have to be included in such an executable - but for that to work the executable must include a bridge/connector component for the protocol used by the customer specific process.

For instance, to implement somethings akin to the current c8y mapper, one needs to assemble these plugins:

• An MQTT connector plugins that defines the misc thin-edge JSON payload and topics
• A c8y plugin that is roughly what we name currently the c8y convertor.
• The configuration will be mostly related to the c8y end-point.

To implement a tedge_agent, one needs:

• The MQTT connector plugin
• An sm management plugin that handles software-management
• A rust plugin connector used to connect an external sm plugin via the command line
• The configuration gives the location of the external sm-plugins.

A really nice benefit of this architecture is that the same plugins can be arranged into an executable tuned for a specific use case. For instance, the agent and the mapper can be combine into a single executable:

• The MQTT connector plugin
• The c8y plugin
• The sm management plugin
• The sm cli plugin connector
• The configuration gives the location of the external sm-plugins and of c8y.

And for a very specific and tuned use-case, say no MQTT external clients and only APT packages. The same can be achieved with a few adjustements:

• A plugin for direct MQTT connection to the cloud (no thin-edge json, no mosquitto).
• The c8y plugin
• The sm management plugin
• A rust implementation of the APT sm plugin.
• The configuration gives the location of c8y.

[albinsuresh]

Is cloud agnostic device twin creation/provisioning and connectivity, a goal of thin-edge? I mean, providing a mechanism to define the device properties like device id, name, type and other metadata which can be interpreted differently by different cloud mappers, which in turn uses that info to create the respective device twins in the cloud?

[didier-wenzek]

Is cloud agnostic device twin creation/provisioning and connectivity, a goal of thin-edge? I mean, providing a mechanism to define the device properties like device id, name, type and other metadata which can be interpreted differently by different cloud mappers, which in turn uses that info to create the respective device twins in the cloud?

Defining ways to extend thin-edge to various cloud platforms is a goal. But the solution might be different from a generic data-model with cloud specific translations. The risk would be to have a common denominator that is too narrow. I believe thin-edge should provide developer tools rather than ready to use features. To be more concrete on this twin example, the role of thin-edge could be just to provide conventions to build an active JSON document from a set of JSON fragments provided by independent components and to use this outcome as the twin representation.

[albinsuresh]

Representing thin-edge JSON as a box here might be a bit confusing as it might appear like an independent component/plugin where as it's just one data format used for the communication. So, in my head, it's just one of labels on these connecting arrows, not a box.

Or were you trying to represent a real app component that produces data in thin-edge JSON format?

[didier-wenzek]

thin-edge JSON is represented as a box because the intent is to provide support for thin-edge json as an independent component/plugin.

• The thin-edge JSON box is a component that establishes the connection between the internal rust components and the external MQTT components.
• On the north bound arrows, flow internal messages (i.e rust in-memory struct), for Measurement, Alarm, Operation.
• On the south bound arrows, flow external messages (a topic + a JSON payload).
• The thin-edge JSON box is an active component that consumes messages on one side and publishes translated messages the other side.
• The internal components (say a c8y mapper or a sm plugin) have then no dependencies to thin-edge JSON nor MQTT. They process Measurement or Operation.
• The thin-edge JSON box can be replaced by another box (say XML over XMPP) without any code impact on the internal components.

[itsroshan]

I really think this needs a visual representation also.

[didier-wenzek]

For now, I just have ascii art. I will add SVG diagrams when an agreement will be reached.

[TheNeikos]

I've added some comments to the upper part, as I consider the 'implementation' part of the discussion fleshed out enough. Either of the two approaches described are fine individually, but I do not believe that supporting two (imo competing) approaches on how to compose "thin-edge.io powered" applications is in scope of the framework.

In fact, I consider the idea of having decoupled executables that are "meant to coordinate" an anti-pattern for its intended application: low-resource and data constrained devices at the edge. Even as Rust itself has a fairly small runtime overhead, there is still a non-negligible binary size. Having multiple of these up can already strain some use-cases. Then, on top comes the communication overhead of such a system as well as the latency.

While IPC is fairly low-latency, copying data is still wasteful and does IMO not offset any form of claims that such a system is more approachable or extensible. Then, comes all the book-keeping, and suddenly the low-latency IPC is not so low anymore.

If anything, my experience shows me that if a user is given too much freedom in how they implement things, it will be used in the 'easiest' way possible without regards for the bigger picture. As a framework, that is undesirable, since any private API is now also a public API, but you now have users that might mess up your private data flow, use it in unintended ways etc...

Being fast and loose has certainly worked for some projects, think PHP, Javascript, Ruby on Rails, but people quickly realized that no, users should not be trusted for their own sake and instead a solid groundwork should be put up. While sounding paradoxical, it is much more human-friendly to be stern on known issues than trusting everyone to do the right thing.

Of course, this does not prevent the user to build such an amalgamation themselves. But I do not think it should be pushed as the way of building a thin-edge.io application.

[TheNeikos]

I do not think that the thin-edge.io framework needs to talk about executables or transport protocols. If users want to go down a microservices route, they are free to do so, but I don't think the framework should be opinionated about this.

[didier-wenzek]

The project of thin-edge has been built around the idea of loose-coupling over MQTT. After the discussion you opened, I'm now convinced that this is not enough and that one also needs strong coupling leveraging Rust features. However, loose-coupling cannot be removed from thin-edge : this would be another project. And not only because some wants to use microservices but also and mainly to interact with tools provided by others and to connect with peripherical devices.

[TheNeikos]

However, loose-coupling cannot be removed from thin-edge : this would be another project. And not only because some wants to use microservices but also and mainly to interact with tools provided by others and to connect with peripherical devices.

Please point out how loose-coupling of external tools or peripheral devices is negatively impacted by what I said.

[TheNeikos]

Same as above, going this hard on MQTT as the basic building block is IMO not the best solution for something that is meant to stay around even if the next best thing comes out. It's fine to mention MQTT as a current supported focus, and mention its compatibility (after all, it is a widely used standard) but there is no technical reason for it to be the sole focus of any kind of vision.

After all the only thing that MQTT does is allow setting a QoS and be an ordered byte transport. Why wouldn't thin-edge.io be compatible with TCP too?

[didier-wenzek]

MQTT is not chosen because of some kind of technical excellence (I see many pain points and misleading promises as QoS) but because this is a lingua franca in the IoT world.

[TheNeikos]

To what point are you exactly responding? So far, this does not seem to counter or expand on any points I made? Do you agree with what I said?

[andrej-schreiner]

@didier-wenzek I propose to rename this whole document to thin-edge.io Technology Vsion & Strategy or Design Principles and to link it the the vision.md:

1. In the vision md explains our motivation, target group/users and their needs, high level summary of the framework capabilities
2. this document should explain how we get there in terms of technology choices , design principles etc . The audience of this doc should be anyone who is willing to user, extend or contribute to thin-edge.io

[didier-wenzek]

@AndrejS321 I'm okay with your proposal.

• I will rename this document Design Principles.
• I will also move here some requirements you removed from the vision doc.
• I still have to address some of the comments.

[6367f766]

What is a low-resource device? Is this still a device running linux?

Maybe this is not the place for this next question but: what are the minimum specs the device should have? How much RAM for example

[albinsuresh]

Should we add one more point, which is the "seamless conversion of a Rust-based actor into an independent MQTT component"?

[didier-wenzek]

Not sure that this can be seamless and even meaningful for all actors. I will try to add a point about that, though.