Twister2 Release 0.8.0
This is a major release of Twister2.
You can download source code from Github
Features of this release
- Fault Tolerance enhancements
- Apache Beam integration is now official Beam Docs
- Improvements to configurations
- Major TSet API update
Minor features
Apart from these, we have done many code improvements and bug fixes.
Next Release
In the next release we are working to,
- Improve and release Table API
- TSQL; Adding SQL support
Components in Twister2
We support the following components in Twister2
- Resource provisioning component to bring up and manage parallel workers in cluster environments
- Standalone
- Kubernetes
- Mesos
- Slurm
- Nomad
- Parallel and Distributed Operators in HPC and Cloud Environments
- Twister2:Net - a data level dataflow operator library for streaming and large scale batch analysis
- Harp - a BSP (Bulk Synchronous Processing) innovative collective framework for parallel applications and machine learning at message level
- OpenMPI (HPC Environments only) at message level
- Task System
- Task Graph
- Create dataflow graphs for streaming and batch analysis including iterative computations
- Task Scheduler - Schedule the task graph into cluster resources supporting different scheduling algorithms
- Datalocality Scheduling
- Roundrobin scheduling
- First fit scheduling
- Executor - Execution of task graph
- Batch executor
- Streaming executor
- Task Graph
- TSet for distributed data representation (Similar to Spark RDD, Flink DataSet and Heron Streamlet)
- Iterative computations
- Data caching
- APIs for streaming and batch applications
- Operator API
- Task Graph based API
- TSet API
- Support for storage systems
- HDFS
- Local file systems
- NFS for persistent storage
- Web UI for monitoring Twister2 Jobs
- Apache Storm Compatibility API
- Apache BEAM API
- Connected DataFlow (Experimental)
- Supports creation of multiple dataflow graphs executing in a single job