Make it possible to implement transactions and feeds (aka data driven intents, RPC etc.) via existing FDC3 concepts

[kriswest]

Enhancement Request

There are many use-cases for (and a number of past discussions held and issues raised about) making it possible to return data from a raised intent, e.g. Issues #201 #203 and PR #287. There are two competing proposals currently (use the AppInstances concept introduced in #287 or merge large parts of the Plexus specification into FDC3). Both proposals introduce new concepts into FDC3 that add to its complexity significantly and, in the case of AppInstances, are counter-intuitive and/or flawed (see review comments on #287).

However, there exists a desire to keep FDC3 as simple as possible to both understand and implement - it is believed that this property has been whats lead to the relative success of the FDC3 standard thus far. Solutions to (asynchronously) returning data from raised intents are possible using the 'channels' concept and API that already exists in FDC3 and only a minor modification to raiseIntent's return type IntentResolution.

This would be achieved by having apps that wish to return data respond with a channel name, on which they will return either a single or stream of responses. By allowing the app handling the response to control the name of the response channel the developer can determine whether to generate a new channel for a response (e.g. to tie it to a particular intent response or transaction, e.g. for the creation of an order and return of a specific ID) or reuse an existing one (e.g. for returning a pricing feed, that could be consumed by multiple other apps).

Use Case:

• Transactions - Apps may need to request an action but then also receive a response. A transaction can be thought of as an Intent with a response (typically a piece of data). Transactions are essentially the desktop equivalent of the traditional query/response protocol used in client/server systems (such as the web). Apps can combine intents, channels, and context to create transactions.
• Feeds - Apps may wish to subscribe to a stream of data provided by another app. Feeds provide both a mechanism for requesting a stream and a mechanism for receiving that stream. Apps can combine intents, channels, and context to create feeds.

Proposed modification:

Update the definitions IntentResolution and ContextHandler to support the return of a channel name on which a response may be returned.

Current definitions

interface IntentResolution {
  source: TargetApp;
  /**
  * @deprecated not assignable from intent listeners
  */
  data?: object;
  version: string;
}

type ContextHandler = (context: Context) => void;

Proposed modifications

interface IntentResolution {
  source: TargetApp;
  responseChannel?: string;
  version: string;
}

type ContextHandler = (context: Context) => string | null;

When calling a context handler for an intent, a Desktop Agent implementation will be expected to check the return type of the handler function and if a string is found, use it to populate the channel name. Most context handlers will be implemented as some form of async function and should be able to return immediately with a channel name (or null if no data will be returned on a channel), before continuing to process the intent and potentially send back results on the indicated channel.

Other ways of returning a channel name from an intent listener may be possible and/or preferable and should be discussed with the Standards Working Group.

[rikoe]

I really like this @kriswest. It's simple, achieves the goal, and ties in to existing concepts and operations. It is also independent of app instances, as it should be.

[rikoe]

@kriswest thinking about it, how do we ensure the intent raising app not missing any data in this case? I.e. what if the app using the response channel sends the result before the app sets up a listener?

In other words, there is a race condition here as the receiving app does not know the name of the channel before hand, so it can only set up a channel listener after the intent is resolved.

[kriswest]

I'm still thinking on that TBH. Firstly, it is possible to resolve the intent (and return the channel name) before responding on the channel in many cases, also you can get the last context (any or a particular type) with channel.getCurrentContext() however, its not ideal to have to call that every time.

In Finsemble, when you subscribe to our pub/sub topics, you received the current state of the topic immediately (also they are always created with an initial state). It would be convenient if the same were true of channels in FDC3 as that would sidestep this issue neatly (for a single response).

The other obvious alternative solution is for the app raising the intent to specify the response channel... Instead of modifying the contextHandler's signature we'd have to modify the two raiseIntent methods to have an additional optional responseChannel argument. I'm less keen on that as it removes the ability for the responding app to reuse channels for feeds, instead putting the ability to reuse a channel into the hands of the requesting app.

[lspiro-Tick42]

This is so wrong, that I wrote a blog about it two years ago.
https://glue42.com/blog/10-common-pitfalls-during-the-life-of-an-in-house-request-response-implementation/

Kris, you are basically saying that any form of interop can be created once you have the equivalent of Pub/Sub. This is a true statement, which those of us who remember the TBCO Market Data system lived with for years.

in your case you are replacing the concept of a message subject in pub/sub with a named Context, but the problem still remains.

If you want to offer a reliable service like feeds, then define an API for Streaming, and then leave it to the implementation to deliver it. Once you try and define it 'exchange these kind of messages over Pub/Sub' it always eventually fails.

And of course, as you say the Plexus API contains a set of methods to support streaming, which deal with concepts like late joiners to a stream and recovery from overload.

If Transactions or Fees belong in FDC3 then let's define a functional interface for them, and leave the implementation of it in the FDC3 implementation aka Desktop Agent

[kriswest]

Hey @lspiro-Tick42 I'm struggling to see how most of the issues you raise in that article apply to this situation or are not already handled by the proposal - particularly as we're not proposing to use pre-shared topics, rather the responder is determining the topic and would (for example for a transaction) generate a new channel name allowing it to differentiate responses to different requests. Further, channels are not purely pub/sub topics, they are intended to allow for a stream of messages.

That said - I do personally prefer Finsemble's approach to a transactional model (the query responder) where a response function is passed in with the query and handles routing of the response back to the requester when called. There are no synchronisation issues - although a stream of responses is not supported. That said, I think there are big advantages to keeping the surface area of and the number of concepts in FDC3 down to a minimum.

I am firmly of the belief that both feeds and transactions are relevant concepts to FDC3 and if not handled by the standard will be built using it anyway. This issue is not yet a concrete proposal/PR - it was however intended to stimulate further discussion on the topic so lets keep this going and perhaps work through some specific problem scenarios or alternative proposals.

[lspiro-Tick42]

@kriswest on the plus side, we both agree that Transactions and Feeds are relevant concepts to FDC3.

If we agree that in terms of architecture, using named (and typically private) channel, whose name is returned by some method, is the same design pattern as using Pub/Sub messaging with a last value cache, then the analysis I wrote about the problems we typically see when people try and implement request/response over Pub/Sub are a guide to issues we are likely to see when attempting to implement Transactions using Intents with Private channels.

[kriswest]

@lspiro-Tick42 I think where our thinking differs is on who names the channel.

Regarding the relevance of your blog post: If the name of the channel is predetermined or reused then that would give rise to issues 1, 3, 4 & 10. 2 (method detection) is (in part) dealt with by the desktop agent's ability to use a resolver, although you have to know the intents available already which is now dealt with by raiseIntentForContext. 5 (Throttling) could affect any implementation of almost any functionality. 6 (Versioning) is dealt with by the use of a standard. 7 (Language support) is dealt with via implementation of FDC3 desktop agent clients in different languages. 8 (Supporting developers) is again dealt with via the use of a documented standard. 9 (Multi-machine interop) is out of scope for FDC3. 10 (Security) is somewhat out of scope too (as we are not dealing with a network use case), however, it is also better dealt with if the responder determines the channel name as other applications in the system won't know what channel name to listen in on to overhear the response.

If you foresee other potential issues could you walk us through how they arise so that we can refine the proposal or documentation to explicitly deal with them?

My only actual concern with this proposal is how to deal with the synchronization issue. I think the most elegant approach to that would be to specify that if a listener is added to channel that already has a context (of the relevant type if a type filter was specified) then the handler should be called with it immediately, this would negate the need for the developer to call channel.getCurrentContext(type?) in addition to adding a listener (which likely also applies to other situations). @rikoe would that deal with your concern?

[kriswest]

We have a refinement to the proposal for this that will deal with the race condition (without changing the behaviour of channel context handlers - although I think it's a valid change to call them immediately if there is an existing context). I'll add it to the thread next week when we've sense checked it this end.

[kriswest]

Superceded by #432 (for transactions/intents that return data)

[kriswest]

Superceded by #433 (for feeds)