Webka

Rebuilding Kafka, for the web, and making it easier to use.
Explore the docs »

Report Bug · Request Feature

Table of Contents

1. About The Project
   • Built With
2. Getting Started
   • Prerequisites
   • Installation
3. Usage
4. Roadmap
5. License
6. Contact
7. Acknowledgments

About The Project

Description

This project is a custom implementation of a Kafka-like messaging system using HTTP, ASP.NET, C#, and Microsoft SQL Server. The goal is to replicate the core functionalities of Apache Kafka, providing a reliable and scalable platform for building real-time data pipelines and streaming applications. Features

• Cluster: Manage multiple clusters for organizational and scalability purposes.
• Topics: Create and manage topics where messages are published.
• Partitions: Divide topics into partitions for parallel processing and load balancing.
• Producers: Send messages to topics, with support for key-value pairs.
• Consumers: Subscribe to topics and consume messages, with configurable settings like auto-commit.
• Consumer Groups: Group multiple consumers together to manage load balancing and fault tolerance.
• Subscriptions: Define which consumers are subscribed to which topics.
• Messages: Store and manage messages within partitions, supporting retrieval and replay.
• HTTP Interface: Interact with the system using a RESTful API built with ASP.NET

(back to top)

Getting Started

To get a local copy up and running follow these simple steps.

Prerequisites

This is an example of how to list things you need to use the software and how to install them.

Installation

1. Clone the repo

   git clone https://github.com/bobrandy13/webka.git

2. Start the Database Server. Currently postgresql, and MS SQL are supported

3. Migrate the models

   dotnet ef migrations add Initial

4. Run the migrations to create the database.

   dotnet ef database update

5. Run the project in Visual Studio.

   dotnet run

6. Read the documentation to get started!

(back to top)

Usage

For more examples, please refer to the swagger documentation. Linked here here

(back to top)

Roadmap

Roadmap

• Set up project with ASP.NET and C#
• Implement Kafka-like messaging system
• Create models for Cluster, Topics, Partitions, Producers, Consumers, Consumer Groups, Subscriptions, and Messages
• Implement HTTP Interface with RESTful API
• Implement Producers
• Implement Consumers
• Implement Consumer Groups
• Implement Subscriptions
• Implement Messages
• Save data to MS SQL Server
• Allow producers to produce messages
• Producer must specify a topic, and specify partition
• Allow all types of messages to be produced, Objects, json, strings, int, etc
• Partition producer's data into partitions using a partition key and hashing algorithms
• Have a replication factor to ensure reliability.
• Allow consumers to consume messages
• Allow consumers to specify the offset to start consuming messages
• At least once, at most once, and exactly once delivery semantics, allow producer to specify.
• Save data across multiple brokers
• Add support for real-time updates using WebSockets or Server-Sent Events
• Allow consumers to connect to Kafka through Long Polling
• Implement a system for monitoring and logging activity
• Add testing.
• Implement authentication and authorization for API access

See the open issues for a full list of proposed features (and known issues).

(back to top)

Contributing

Contributions are what make the open source community such an amazing place to learn, inspire, and create. Any contributions you make are greatly appreciated.

If you have a suggestion that would make this better, please fork the repo and create a pull request. You can also simply open an issue with the tag "enhancement". Don't forget to give the project a star! Thanks again!

(back to top)

License

Distributed under the MIT License.

(back to top)

Contact

Project Link: https://github.com/bobrandy13/webka

(back to top)