Add the concept of a feed or stream to FDC3 to handle use-cases not covered by channels or intents

[kriswest]

Output from the Channels, Feeds & Transactions discussion group: #420
Supercedes #372

Enhancement Request

Channels (both 'App' and 'System' channels) provide a mechanism for applications to exchange messages with each other, enabling interop. However, they have a number of limitations that make them unsuitable for certain use-cases, in particular those where you need to communicate with a specific application or on a specific topic.

1. You don't know who sent a particular message over a channel.
   • The FDC3 specification currently relies on the fact that it's not possible to list all App channels to provide some measure of security for communication with a particular app.
   • However, this idea exists in tension with the idea that the names of app channels may be 'well known' and is not an effective means of securing the channel/determining that messages are coming from the expected source.
   • Messages are not marked up with the identity of the sender nor is the channel 'owned' or 'controlled' by the application that created it. Hence, any application in the system could be posting messages on the channel.
2. It is desirable to be able to request a channel to receive or broadcast a stream of data on a particular topic.
   • For example, you can currently create a 'pricing' channel, but you are more likely to want the channel to relate to pricing for a particular security (and it is impractical to create and post data to channels for every security.
3. There exist no events tracking subscription or subscription from a channel so its impossible to tell if anyone is listening
   • You probably don't want to waste resources sharing pricing information on particular securities if no one is listening.
   • You may not want to send information to the channel until a subscriber is added (for synchronisation purposes).
4. You may wish to retrieve a list of the available feeds (to enable feature discovery).

Summary of features needed

• Register a feed (and feed owner)
• Get list of feeds available
• Request a feed
• Subscribe to a feed
• Receive notice of subscription
• Unsubscribe from a feed
• Receive notice of unsubscription
• Publish to the feed (as owner)
• Publish to feed owner (as a subscriber)

Use-cases

• Request a feed from a particular application that provides Pricing (type of feed) of specific security (context).
• RFQs (type) for a particular market sector, client or security (context)
• Working with the container/Dekstop Agent itself, where the feeds are created and used by the desktop agent/container to implement functionality that interacts with the desktop, rather than a specific application.
  • Feeds might be used to implement functions such as:
    • Workspace state saving
    • Notification
    • Federated search
    • Authentication
    • Lifecycle management (startup dependencies, graceful shutdowns)
    • User preferences
  • Although FDC3 is not intended to provide an API for an application container, shared facilities like these are common parts of a multi-vendor Desktop and are not easy to implement with FDC3's channels and intents alone. The ability to integrate apps into a common platform is a key aspect of them working together.

[kriswest]

Other possible names for this concept are 'private channel' or 'extension'

[kriswest]

Proposal:

Desktop agent API additions for the feed owner:

/** Register a handler for a particular named feed that will receive 
    requests for the feed, create a new or return an existing feed channel 
    name and create handlers on it, as needed, for subscription events. 

    Although feeds are interacted with through the `Channel` interface, 
    their behaviour is different in that they must be requested (with 
    optional context) before they can be communicated with and they only 
    deliver communication between the feed owner and its subscribers (rather 
    than between all subscribers).

    If you need to know which specific subscriber sent a message back to the
    feed owner then ensure that you create a new feed for each subscriber.
*/
registerFeed(feedId: string, requestHandler: FeedRequestHandler): Promise<Void>;

/** Type representing a handler function for feed requests. 
    On receiving a feed request, a handler function should either
    Retrieve an existing channel object or create a channel feed object 
    and return it so that the Desktop Agent can respond to the requestor
    with the details. 
    @param An optional context Object that can be used to specify some 
    aspect of the feed requested. For example you might pass an
    `fdc3.instrument` to a `pricing` feed to specify that you want pricing
    for the security it represents. */
type FeedRequestHandler = (context?: Context) => Channel;


/** Function that should be called by a feed owner to create a new channel 
    to be returned to the requestor of a feed. The `Feed` Object returned
    Extends the capabilities of a channel to include the ability to listen 
    to subscription and unsubscription events, allowing a feed to be cleaned 
    up when no longer needed. */
createFeedChannel(
    feedId: string, 
    channelId: string
): Promise<Feed>;

Types:
/** Additional functionality a feed need to add for the owner, over what is
    already provided by a channel. */
interface Feed extends Channel {
	addSubscriptionHandler(handler: SubscriptionHandler): Promise<Listener>; 
};

/** Object providing metadata about a feed, including the application
    That registered it. May be used to describe both a registered feed
    and the channel Object returned when it is requested.
 */
interface FeedMetadata extends DisplayMetadata {
    owner: AppMetadata;
    feedId: string;
}

enum SubscriptionEventType {
    Subscribe = "subscribe",
    Unsubscribe = "unsubscribe"
};

type SubscriptionHandler = (
    channelId: string, 
    eventType: SubscriptionEventType
) => void;

Example usage:

let existingFeeds: Record<string, Channel> = {};
let subscriberCounts: Record<string, number> = {};
let subHandler: SubscriptionHandler = (channelId, eventType) => {
    let subscriberCount = subscriberCounts[channelId] ? 
        subscriberCounts[channelId] : 0;
    if(eventType == SubscriptionEventType.Subscribe){
        if (subscriberCount == 0) {
            //first subscriber, start pushing data
            // with `existingFeeds[channelId].broadcast(...);`
        }
        subscriberCounts[feedIdentifier] = ++subscriberCount;
    } else if (eventType == SubscriptionEventType.Unsubscribe){ 
        subscriberCounts[channelId] = --subscriberCount;
        if (subscriberCount <= 0) {
            //feed empty, stop pushing data
            ...
        }
    }
};
let feedHandler: FeedRequestHandler = (context: Context) => {
    //create an identifier for the feed that should be unique
    let securityIdentifier = getSecurityIdentifier(context);
    let feedIdentifier = "pricing-" + securityIdentifier;  
    
    if(existingFeeds[feedIdentifier]) {
        //feed already setup, including events counting subscribers
        return existingFeeds[feedIdentifier];
    } else {
        let theFeed = null;
        //ensure a unique channel name for the feed
        while(theFeed == null) {
            try {
                theFeed = await fdc.createFeedChannel(
                    "pricing", 
                    feedIdentifier
                );
            } catch (err) {
                //feedIdentifier is already taken as a channel name,            
                //  generate a different one
                feedIdentifier = ...
            }
        }

        //listen to subscription events
        let subListener = theFeed.addSubscriptionHandler(subHandler);

        //if listening for input from subscribers, do so here
        let contextListener = theFeed.addContextListener(null, (context) => { ... });`

        existingFeeds[feedIdentifier] = theFeed;
        return theFeed;
    }
};

fdc3.registerFeed("pricing", feedHandler);

Desktop agent API additions for the feed subscribers:

/** 
 * Request a feed, with an optional context. Returns a Channel object
   (rather than a Feed object) as only functions of a Channel are available
   to feed subscribers.
   @param feedId A string representing the feed type to request. To be 
   Resolved correctly, this type must already have been registered or 
   appear in the appD used by the resolver.
   @param context An optional context object that specifies some detail of 
   the feed requested, For example, when retrieving a pricing feed you might
   Pass an `fdc.instrument` object to specify the security it relates to.
 */
requestFeed(feedId: string, context?: Context): Promise<Channel>;

/** Return a list of feeds that have been registered, which includes details 
    of the Applications that registered them (or can do so from an AppD).
*/
getFeeds(): Promise<Array<FeedMetadata>>;

Example usage:

let instrument: Context = { 
    type: "fdc3.instrument", 
    name: "Apple Inc., 
    id: { ticker: "AAPL"} 
};
let feed: Channel = fdc3.requestFeed("pricing", instrument);
let handler: ContextHandler = (context: Context) => { … };
feed.addContextListener(null, handler);
//if you need to send a message to the feed owner:
feed.broadcast(context);

Client-side is simpler - you are just retrieving something that looks like a channel that you addContextListener() to. You can use the getCurrentContext() function to see if any messages have already been sent, however if the feed is new it shouldn't start sending data until after you add your context listener, hence, in most situations you will not need to do this.

[kriswest]

Alternative proposal that does not include a discovery or registration element, but instead relies of intents to retrieve a PrivateChannel:

export
let fdc3 : DesktopAgent;
let reuters : any;
type TargetApp = any;
type Context =  any;
type DisplayMetadata = any;
type ResolveError = any;
type Listener = any;
type AppIntent = any;
type ImplementationMetadata = any;
type ContextHandler = (context: Context) => void;

interface DesktopAgent {
	// NEW!
	// Used to create and establish a PrivateChannel within an intent handler
	createPrivateChannel() : Promise<PrivateChannel>

	open(app: TargetApp, context?: Context): Promise<void>;
	findIntent(intent: string, context?: Context): Promise<AppIntent>;
	findIntentsByContext(context: Context): Promise<Array<AppIntent>>;
	broadcast(context: Context): void;
	raiseIntent(intent: string, context: Context, app?: TargetApp): Promise<IntentResolution>;
	raiseIntentForContext(context: Context, app?: TargetApp): Promise<IntentResolution>;
	//Updated to use IntentHandler
	addIntentListener(intent: string, handler: IntentHandler): Listener;
	addContextListener(contextType: string | null, handler: ContextHandler): Listener;
	getSystemChannels(): Promise<Array<Channel>>;
	joinChannel(channelId: string): Promise<void>;
	getOrCreateChannel(channelId: string): Promise<Channel>;
	getCurrentChannel(): Promise<Channel | null>;
	leaveCurrentChannel(): Promise<void>;
	getInfo(): ImplementationMetadata;
}

interface Channel {
	readonly id: string;
  	readonly type: string;
  	readonly displayMetadata?: DisplayMetadata;
  	broadcast(context: Context): void;
  	getCurrentContext(contextType?: string): Promise<Context | null>;
  	addContextListener(handler: ContextHandler): Listener;
  	addContextListener(contextType: string | null, handler: ContextHandler): Listener;
}

// NEW! PrivateChannel is established via intents - it extends the functionality of channel for all parties
// using the private channel. Three new event handlers allow participants to manage their connection state.
// Desktop Agents SHOULD restrict external apps from listening or publishing on this channel
// Desktop Agents MUST prevent private channels from being retrieved via fdc3.getOrCreateChannel
// Desktop Agents MUST provide the `id` value for the channel as required by the Channel interface
// Desktop Agents DO NOT NEED to queue messages that are broadcast before a context listener is added
//     (as the intent app can wait on the onAddContextListener being called)
interface PrivateChannel extends Channel {
	// Called each time that the remote app invokes addContextListener on this channel.
	// Desktop Agents MUST call this for each invokation of addContextListener
	// on this channel before this handler was registered (to prevent race conditions).
	onAddContextListener(handler : (contextType ?: string) => void): Listener;

	// Desktop Agents MUST call this any time that the remote app invokes Listener.unsubscribe().
	onUnsubscribe(handler : (contextType ?: string) => void): Listener;

	// Called when the remote app is terminates (for example its window was closed or because disconnect was called)
	onDisconnect(handler : () => void): Listener;

	// May be called to indicate that a participant will no longer interact with this channel
	// Desktop Agents SHOULD prevent apps from broadcasting on this channel and SHOULD automatically 
	// call Listener.unsubscribe() for each listener they've added (causing any onUnsubscribe handler added 
	// by the other party to be called) before triggering any onDisconnect handler added by the other party.
	disconnect(): void;
}

// The promise returned from a call to raiseIntent() resolves when the desktop agent
// has delivered the intent to an intended app (e.g. the handler from addIntentListener()
// has been called on the intended app).
interface IntentResolution {
	readonly source: TargetApp;
	readonly version: string;

	// NEW #432
	// A client may subsequently await on getData() to receive data from the intent app.
	// If the intent app does not return a promise from its intent handler, then no
	// data will be returned and teh getData promise will resolve with void. If the Intent
	// handler throws an error the promise will be rejected (with details).
	getData : () => Promise<Context | void>;

	// NEW!
	// A client may also await on getChannel() to gain access to a PrivateChannel established
	// by the app resolving the intent. The client can then addContextListener() on that channel
	// to, for example, receive a stream of data. If the intent app does not return a PrivateChannel
	// object from its intent handler then this promise will resolve as void. If the Intent
	// handler throws an error the promise will be rejected (with details).
	getChannel : () => Promise<PrivateChannel | void> ;
}

// Intent listeners previously reused the ContextHandler type. However, that signature is not compatible with 
// returning data or a PrivateChannel. An new type should be introduced to accommodate optional data and channel 
// resolution - which should and can be backwards compatible with ContextHandler.
// By returning a Promise, the intent handler signals to the desktop agent that it will be returning data.
// By returning a PrivateChannel, the intent handler signals to the desktop agent that it has established a channel.
// By returning void, the intent handler signals that it will be doing neither of the above.
// Any errors thrown by the intent handler should be caught by the Desktop Agent and used to reject the
// promises returned by IntentResolution's getData and getChannel functions.
type IntentHandler = (context: Context) => void|Promise<any>|PrivateChannel;


// ---- Case 1: "Fire and Forget" (current use-case) ---- //

// Client Side
try {
    const resolution1 = await fdc3.raiseIntent("trade", { type: "tradeData", id : { symbol: "AAPL"}});
} catch (resolverError) {
	console.error(`Error: Intent was not resolved: ${resolverError}`);
}

// Intent App (Server Side)
// The intent app returns void. 
// The Desktop Agent knows this is "fire and forget" because the handler returns void.
// getData()'s and getChannel's returned promises will resolve with void if called.

 fdc3.addIntentListener("trade", (context) => {
	// go do something
	return;
});


// ---- Case 2: "Query / Response" (Make it possible to return data from a raised intent #432)  ---- //

// Client Side
try {
    const resolution2 = await fdc3.raiseIntent("trade", { type: "tradeData", id : { symbol: "AAPL"}});
    try{
	    const tradeResult = await resolution2.getData();
	    if (tradeResult) {
            console.log(`${resolution2.source} returned ${tradeResult}`);
         } else {
            console.warn(`${resolution2.source} did not return data`);
         }
    } catch(dataError){
	    console.log(`Error: ${resolution2.source} returned an error: ${dataError}`);
    }
} catch (resolverError) {
    console.error(`Error: Intent was not resolved: ${resolverError}`);
}

//Client side with less precise error handling
try {
    const resolution2 = await fdc3.raiseIntent("trade", { type: "tradeData", id : { symbol: "AAPL"}});
	const tradeResult = await resolution2.getData();
	console.log(`${resolution2.source} returned ${tradeResult}`);
} catch (error) {
    console.error(`Error: error occurred while raising intent for data: ${error}`);
}

// Intent App (Server Side)
// The intent app returns data to the client via a Promise (who is awaiting on getData() )
// The Desktop Agent knows that data is going to be returned because the handler returns type Promise

fdc3.addIntentListener("trade", (context) => {
	//return a Promise of Context data (or the data itself if handled synchronously)
	return new Promise<Context>((resolve, reject) => {
		// go place trade
		resolve({type: "fdc3.trade", id: {  }});
	});
});



// ---- Case 3: "Stream" (new use-case)---- //

// Client Side
try {
    const resolution3 = await fdc3.raiseIntent("QuoteStream", { type: "fdc3.instrument", id : { symbol: "AAPL"}});
    try {
	    const channel = await resolution3.getChannel();
        if (channel) {
            const listener = channel.addContextListener("price", (quote) => console.log(quote));
	         channel.onDisconnect(() => {
		          console.warn("Quote feed went down");
	         });

	         // Sometime later...
	         listener.unsubscribe();
        } else {
            console.warn(`${resolution3.source} did not return a channel`);
        }
     } catch(channelError){
	    console.log(`Error: ${resolution3.source} returned an error: ${channelError}`);
     }
} catch (resolverError) {
    console.error(`Error: Intent was not resolved: ${resolverError}`);
}

// Intent App (Server Side)
// The intent app establishes and returns a PrivateChannel to the client (who is awaiting on getChannel() )
// When the client calls addContextlistener() on that channel, the intent app receives the onAddContextListener()
// event and knows that the client is ready to start receiving quotes
// The Desktop Agent knows that a channel is being returned by inspecting the object returned from the handler (e.g. check constructor or look for private member)

fdc3.addIntentListener("QuoteStream", async (context) => {
	const channel : PrivateChannel = await fdc3.createPrivateChannel();
	const symbol = context.id.symbol;

	// This gets called when the remote side adds a context listener
	const addContextListener = channel.onAddContextListener((contextType) => {
		// broadcast price quotes as they come in from our fictional reuters quote feed
		reuters.onQuote(symbol, (price) => {
			channel.broadcast({ type: "price", price});
		})
	})

	// This gets called when the remote side calls Listener.unsubscribe()
	const unsubscriberListener = channel.onUnsubscribe((contextType) => {
		reuters.stop(symbol);
	})

	// This gets called if the remote side closes
	const disconnectListener = channel.onDisconnect(() => {
		reuters.stop(symbol);
	})

	return channel;
});

[pbaize]

Here's a naive implementation without introducing anything else to the spec. Hopefully it can help sort out what is needed in a separate api.

function registerIntentWithResultHandler(tansactionType, handler) {
    fdc3.addIntentListener(`fdc3.Transaction.${transactionType}`, async (context) => {
        const transactionChannel = await fdc3.getOrCreateChannel(context.transactionChannelId);
        handler({ ...context, transactionChannel });
    });
}

async function raiseIntentForResult(transactionType, context) {
    const id = utils.getGuid(); // Some sort of guid util
    const transactionChannel = await fdc3.getOrCreateChannel(id);
    const resolution = fdc3.raiseIntent(`fdc3.Transaction.${transactionType}`, { ...context, transactionChannelId: id});
    return responsePromise;
}

[kriswest]

@pbaize thanks for the above, which demonstrate that you could use raiseIntent to pass a context and channel id and then receive responses via the channel with that id. It does allow discovery via the intent resolver (in the same way as the second proposal above) - but also suffers from the same issue regarding resolution (you can't resolve to an app that provides a particular type of response).

A couple of things I note in the example:

• you need to add a context listener to the channel before you raise the intent (I think that was your intention, but it isn't in the example)
  • perhaps accept the context handler for the result as an argument to raiseIntentForResolution
  • also don't forget to await raiseIntent or return its promise
• there is no way to know that either party has left/disconnected so you can clean up
• you can support multiplexing and bi-directional comms
• the only security provided is the obscurity of the channel name (GUIDs are unlikely to collide, although it is technically possible) - an implementation could provided a greater level of security if we adopt an approach that differentiate the type of channel (e.g. PrivateChannel)

[kriswest]

Outcomes from #489

• the majority of participants preferred the second proposal, which introduces the concept of private channels
  • Consensus was achieved that further work on the second proposal should be done,
    • In particular, to allow a resolver to consider the desired response type