NoriKV

A sharded, Raft-replicated, log-structured key–value store with portable SDKs and first-class observability.

Overview

NoriKV is a distributed key-value database designed for high availability, strong consistency, and operational transparency. Built with modern storage and consensus algorithms, it provides predictable performance with comprehensive observability.

Key Features

Sharded Architecture: Horizontal scaling with Jump Consistent Hashing
Raft Consensus: Strong consistency with leader-based replication
LSM Storage: Log-structured merge-tree with leveled compaction
Multi-Language SDKs: TypeScript, Python, Go, and Java clients
First-Class Observability: Built-in metrics, tracing, and live visualization
SWIM Membership: Fast failure detection and cluster health monitoring
High Performance: Zero-copy operations, optimized hot paths

Architecture

┌─────────────────────────────────────────────────────────────┐
│  Client SDKs (TypeScript, Python, Go, Java)                 │
│  - Smart routing to shard leaders                           │
│  - Automatic retry with exponential backoff                 │
│  - Connection pooling & health checking                     │
└─────────────────────────────────────────────────────────────┘
                            │
                            ▼
┌─────────────────────────────────────────────────────────────┐
│  NoriKV Cluster                                              │
│  ┌──────────────┐  ┌──────────────┐  ┌──────────────┐      │
│  │   Node 1     │  │   Node 2     │  │   Node 3     │      │
│  │  (Shard 0)   │  │  (Shard 1)   │  │  (Shard 2)   │      │
│  │              │  │              │  │              │      │
│  │  ┌────────┐  │  │  ┌────────┐  │  │  ┌────────┐  │      │
│  │  │  Raft  │  │  │  │  Raft  │  │  │  │  Raft  │  │      │
│  │  └────────┘  │  │  └────────┘  │  │  └────────┘  │      │
│  │  ┌────────┐  │  │  ┌────────┐  │  │  ┌────────┐  │      │
│  │  │  LSM   │  │  │  │  LSM   │  │  │  │  LSM   │  │      │
│  │  └────────┘  │  │  └────────┘  │  │  └────────┘  │      │
│  └──────────────┘  └──────────────┘  └──────────────┘      │
│                                                               │
│  SWIM Membership: Gossip-based failure detection            │
└─────────────────────────────────────────────────────────────┘

Quick Start

Installation

Go SDK (Production Ready)

go get github.com/norikv/norikv-go

import norikv "github.com/norikv/norikv-go"

client, _ := norikv.NewClient(ctx, norikv.DefaultClientConfig(
    []string{"localhost:9001", "localhost:9002"},
))
defer client.Close()

// Put a value
version, _ := client.Put(ctx, []byte("key"), []byte("value"), nil)

// Get a value
result, _ := client.Get(ctx, []byte("key"), nil)

TypeScript SDK

npm install @norikv/client

import { createClient } from '@norikv/client';

const client = createClient({
  nodes: ['localhost:9001', 'localhost:9002'],
});

await client.put('key', 'value');
const result = await client.get('key');

Python SDK

pip install norikv

from norikv import Client

async with Client(['localhost:9001', 'localhost:9002']) as client:
    await client.put('key', 'value')
    result = await client.get('key')

Server (Development)

# Build the server
cargo build --release -p norikv-server

# Run with default configuration
./target/release/norikv-server

Project Structure

Core Storage Crates (Rust)

Crate	Status	Description
nori-observe	Complete	Vendor-neutral observability framework
nori-wal	Complete	Write-ahead log with recovery
nori-sstable	Complete	Sorted string tables with bloom filters
nori-lsm	Complete	LSM tree engine with compaction
nori-swim	Complete	SWIM failure detection protocol
nori-raft	Complete	Raft consensus implementation

Client SDKs

Language	Status	Features	Tests
TypeScript	Production	Smart routing, retries, pooling, ephemeral server	100+ passing
Python	Production	Async/await API, type hints, ephemeral server	80+ passing
Go	Production	Connection pooling, topology watching, integration tests	102+ passing
Java	In Progress	Maven/Gradle, gRPC client	Pending

Server Components

Component	Status	Description
norikv-server	In Progress	Main server binary
norikv-placement	Complete	Shard assignment and routing
norikv-transport-grpc	In Progress	gRPC/HTTP transport layer
norikv-vizd	Planned	Visualization daemon
norikv-dashboard	Planned	Real-time web dashboard

SDK Features Comparison

All SDKs provide consistent functionality:

Smart Client Routing: Client-side shard assignment with Jump Consistent Hashing
Leader-Aware Operations: Direct requests to shard leaders with automatic failover
Retry Logic: Exponential backoff with jitter for transient failures
Connection Pooling: Efficient connection management per node
Conditional Operations: Compare-and-swap (CAS) with version matching
Consistency Levels: Lease-based, linearizable, or stale reads
Idempotency Keys: Safe retries for write operations
Cluster Topology: Dynamic cluster membership tracking
Ephemeral Server: In-memory server for testing (no external dependencies)

Hash Function Compatibility

Critical: All SDKs use identical hash functions to ensure consistent shard routing:

Key Hashing: xxhash64 (seed=0)
Shard Assignment: Jump Consistent Hash
Cross-Validated: Test vectors ensure identical results across all languages

Development

Prerequisites

Rust 1.75+ (for server and core crates)
Node.js 18+ (for TypeScript SDK)
Python 3.9+ (for Python SDK)
Go 1.21+ (for Go SDK)
Java 11+ (for Java SDK)

Building from Source

# Build all Rust crates
cargo build --all

# Run tests
cargo test --all

# Build specific SDK
cd sdks/go && go build ./...
cd sdks/typescript && npm install && npm run build
cd sdks/python && pip install -e .

Running Tests

# Rust core tests
cargo test --all

# Go SDK tests (unit + integration)
cd sdks/go && go test ./...

# TypeScript SDK tests
cd sdks/typescript && npm test

# Python SDK tests
cd sdks/python && pytest

Performance

Storage Engine (nori-lsm)

Point Reads: ~10µs (p99)
Point Writes: ~20µs (p99)
Bloom Filter Hit: ~80ns (zero allocation)
Compaction: Leveled strategy with size-tiered L0

Hash Functions (Cross-SDK)

xxhash64: ~2.5ns per operation (Go), ~8ns (Python/TypeScript)
Jump Consistent Hash: ~14ns per operation (Go)
Combined Routing: ~23ns (Go), <100ns (TypeScript/Python)

Observability

NoriKV is built with observability as a first-class concern:

Metrics

Vendor-Neutral: nori-observe trait for pluggable backends
Prometheus: Built-in Prometheus exporter
OTLP: OpenTelemetry support with trace exemplars
Low Overhead: <100ns per metric operation

Visualization

Live Dashboard: Real-time cluster visualization (planned)
VizEvent Stream: Typed events for custom tooling
Health Endpoints: HTTP health checks and readiness probes

Documentation

Architecture Guide: System design and components
Storage Layer: WAL, SSTable, and LSM details
Consensus: Raft implementation specifics
SDKs: Individual SDK documentation in each directory
Operations: Deployment and monitoring guides

Roadmap

Completed

Core storage engine (WAL, SSTable, LSM)
Raft consensus with read-index and leases
SWIM membership protocol
TypeScript, Python, and Go SDKs
Ephemeral servers for testing
Cross-SDK hash validation

In Progress

Java SDK
Server application and transport layer
gRPC/HTTP API implementation
Integration testing with real server

Planned

Live visualization dashboard
Multi-shard transactions
Streaming operations (watch API)
Backup and restore
Chaos testing framework

Contributing

Contributions are welcome! Please see CONTRIBUTING.md for guidelines.

Areas for Contribution

Java SDK: Complete implementation following Go/TypeScript patterns
Server Development: gRPC handlers, sharding coordinator
Dashboard: Real-time visualization UI
Documentation: Tutorials, examples, API reference
Testing: Property tests, chaos engineering
Performance: Benchmarking, optimization

License

This project is dual-licensed under MIT OR Apache-2.0.

See LICENSE-MIT and LICENSE-APACHE for details.

Related Projects

Python SDK: sdks/python/
TypeScript SDK: sdks/typescript/
Go SDK: sdks/go/
Java SDK: sdks/java/

Acknowledgments

Built with modern distributed systems research:

LSM Trees: Original LevelDB/RocksDB design
Raft Consensus: Diego Ongaro's dissertation
SWIM: Scalable Weakly-consistent Infection-style Process Group Membership Protocol
Jump Consistent Hash: Google's consistent hashing algorithm

Status: Active development | Stability: Alpha | Production Ready: SDKs only (server in progress)

Name		Name	Last commit message	Last commit date
Latest commit History 238 Commits
.github/workflows		.github/workflows
apps		apps
context		context
crates		crates
docker		docker
docs		docs
img		img
proto		proto
sdks		sdks
test-vectors		test-vectors
.editorconfig		.editorconfig
.gitattributes		.gitattributes
.gitignore		.gitignore
CHANGELOG.md		CHANGELOG.md
CLAUDE.md		CLAUDE.md
CODE_OF_CONDUCT.md		CODE_OF_CONDUCT.md
CONTRIBUTING.md		CONTRIBUTING.md
Cargo.toml		Cargo.toml
Dockerfile		Dockerfile
LICENSE		LICENSE
README.md		README.md
docker-compose.yml		docker-compose.yml
mkdocs.yml		mkdocs.yml
norikv.example.yaml		norikv.example.yaml
requirements-docs.txt		requirements-docs.txt

License

jeffhajewski/nori

Folders and files

Latest commit

History

Repository files navigation

NoriKV

Overview

Key Features

Architecture

Quick Start

Installation

Go SDK (Production Ready)

TypeScript SDK

Python SDK

Server (Development)

Project Structure

Core Storage Crates (Rust)

Client SDKs

Server Components

SDK Features Comparison

Hash Function Compatibility

Development

Prerequisites

Building from Source

Running Tests

Performance

Storage Engine (nori-lsm)

Hash Functions (Cross-SDK)

Observability

Metrics

Visualization

Documentation

Roadmap

Completed

In Progress

Planned

Contributing

Areas for Contribution

License

Related Projects

Acknowledgments

About

Resources

License

Code of conduct

Contributing

Uh oh!

Stars

Watchers

Forks

Releases

Packages 0

Contributors 2

Uh oh!

Languages

Packages