Skip to content

Streaming application development and management system, based on Linkis and DSS, planning to provide the workflow-like graphical drag-and-drop development capability.

License

Notifications You must be signed in to change notification settings

WeDataSphere/Streamis

 
 

Repository files navigation

Streamis

License

English | 中文

Introduction

       Streamis is an jointed development project for Streaming application development and management established by WeBank, CtYun, Samoyed Financial Cloud and XianWeng Technology.

       Based on the framework capabilities of DataSphere Studio and the underlying docking Linkis 's Flink engine allows users to complete the development, debugging, release and production management of streaming applications at low cost.

       In the future, it is also planned to use a workflow-style graphical drag-and-drop development experience, and the streaming application will be based on the Source node, The Dimension node, Transform node, Sink node and Visualis nodes are connected in series to form a streaming workflow, allowing users to complete the development of streaming applications at a lower learning cost. Debug and release.


Core features

1. Based on DSS and DSS-Scriptis, to create a leading streaming application development management system.

       With Flink as the underlying computation engine, based on the architectural design pattern of the isolation between the development center and the production center, it completely isolates development permissions and publishing permissions, and isolates the development environment and production environment to ensure high stability and high security of streaming applications.

       The application development layer is organically integrated with the data application development process of DSS, providing simplier user experience.

       The application execution layer integrates Linkis to provide financial-level streaming application management capabilities with high concurrency, high availability, multi-tenant isolation, and resource management.

2. Powerful streaming application development and debugging capabilities.

       Based on DSS-Scriptis, provides streaming application development and debugging functions, and supports real-time debugging and result set display of FlinkSQL.

development center

3. Powerful streaming application production center capabilities.

       Supports multi-version management, full life cycle management, monitoring alarm, checkpoint and savepoint management capabilities of streaming jobs.

prod center

       Running information page:

Running information

       Configurations page:

Configurations Configurations

       For more features, please refer to: User Manual.


Depended ecosystems

Depended Component Description Streamis compatibility
DataSphereStudio Data application development management framework. With a unified UI, the workflow-like graphical drag-and-drop development experience meets the entire lifecycle of data application development from data import, desensitization cleaning, data analysis, data mining, quality inspection, visualization, scheduling to data output applications, etc. >= DSS1.1.2 (Released)
Linkis Apache Linkis, builds a layer of computation middleware, by using standard interfaces such as REST/WS/JDBC provided by Linkis, the upper applications can easily access the underlying engines such as MySQL/Spark/Hive/Presto/Flink, etc. >= Linkis1.4.0 (Released)

Demo Trial environment

       In progress, stay tuned!


Download

       Please go to the Streamis Releases Page to download a compiled version or a source code package of Streamis.


Compile and install deployment

please refer to Streamis Installation and Deployment Document for installing and deploying Streamis.


Examples and usage guidelines

       Please visit to User documentation, learn how to use Streamis quickly.


Features

Function Module Description Streamis
UI Integrated and convenient management interface and monitoring window Integrated
Installation and deployment Deployment difficulty and third-party dependencies One-click deployment, relying on Linkis Flink engine
Development Center FlinkSQL streaming application real-time development and debugging Support, need to integrate DSS
Production Center Streaming Application Management Operation and Maintenance Capability Support
Reuse Linkis computing governance capabilities Support
Support FlinkSQL and FlinkJar package release Support
Multi-version management capabilities Support
Configuration and alert management capabilities Support
Service high availability Multiple services,State snapshot for fault tolerance, failure does not affect the use Application high availability
System Management Node and Resource Management Support
Permission management Task operation permission control Support

Architecture

Architecture


Contributing

       Contributions are always welcomed, we need more contributors to build Streamis together. either code, or doc, or other supports that could help the community.


Communication contribution

       For any questions or suggestions, please kindly submit an issue.

       You can scan the QR code below to join our WeChat group to get more immediate response.

comminicate


License

       DSS is under the Apache 2.0 license. See the License file for details.

About

Streaming application development and management system, based on Linkis and DSS, planning to provide the workflow-like graphical drag-and-drop development capability.

Resources

License

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published

Languages

  • Java 84.5%
  • Scala 10.8%
  • SCSS 4.3%
  • Shell 0.4%