Beyond the pipeline - DAG / Flow Graph / Data Flow

[sirinath]

Is it possible to add support to create flows beyond a pipeline by being able to create a flow graph like in:

http://akka.io/news/2014/09/12/akka-streams-0.7-released.html

[palladin]

Thank you for your suggestion! Actually, I'm thinking of possible ways to combine the power of Streams with other libraries like Rx and TPL Dataflow. The API of Akka Streams looks very interesting and a good source for inspiration.

[sirinath]

Also have a look at: https://github.com/Hopac/Hopac

Main thing is ease of use. If you have a DSL like Akka this would be easy to use.

Also see:
http://www.nextflow.io/ (DataFlow)
http://noflojs.org/ (Flow Based Programming)

And to a lesser extent:
http://www.infoq.com/presentations/mantis (Rx, DAG, Functional Reactive Programming)

FBP:
http://nifi.incubator.apache.org/

DAG:
http://flink.incubator.apache.org/

[palladin]

Thanks for all the wonderful resources!

[sirinath]

Integration with Rx and TPL is very welcome and good. But I am not sure if they are the best performing options. Have a look at Hopac for better performance.

[palladin]

I love Vesa's work and Hopac is definitely on my radar!

[sirinath]

Great

[sirinath]

The docs have come out. http://doc.akka.io/docs/akka-stream-and-http-experimental/1.0-M2/scala.html?_ga=1.257897355.1234628154.1418042628

It wold be great if you can have a similar DSL.

[palladin]

Thanks for the info. I really like the (~>) as the F# idiomatic operator for the underlying TPL.Dataflow LinkTo method.

[sirinath]

Yes. This would be good for starters.

But perhaps may be explore if there are better performing option that TPL also.

[sirinath]

Design it so you mix and match the underling framework.

[sirinath]

Wondering how things are shaping up in the data flow front?

[palladin]

Currently my focus is on MBrace.Streams (CloudStreams) but the data flow integration is a big concern for us. Unfortunately, the local-streams data flow support must wait for the Cloud data flow plans to take shape.

[sirinath]

I am more interested in Cloud data flows. If this is taking time at least the local case.