Skip to content

Latest commit

 

History

History
161 lines (130 loc) · 9.96 KB

Roadmap.md

File metadata and controls

161 lines (130 loc) · 9.96 KB
Raw
layout title nav_order description permalink
default
Project Roadmap
2000
Project Roadmap
/roadmap

Roadmap

This is a living document containing the FASTER team's priorities as well as release notes for previous releases. Items refer to FASTER C# unless indicated otherwise. For C++ info, scroll to C++ Porting Notes.

(Scroll to Release Notes)

Past and Future Work

The following is a summary of the FASTER team's past work and backlog for the next 6 months. Completed items are included to provide the context and progress of the work.

Past Work

General

  • Full Read, Upsert, RMW functionality
  • Bulk delete via key expiry: user-initiated truncation from head of log (ShiftBeginAddress)
  • Persistence support for larger-than-memory data
  • Scan operation support for scanning a specified range of log
  • Ability to resize (grow) the hash table in powers of two (user-initiated)
  • Separately configurable optional read cache for read-heavy workloads
  • Support for separate user-defined object hash/equality comparers and object serializers
  • Remove C# dynamic code-gen for quicker instantiation, stability, debuggability
  • Full support for .NET core and Linux/Mac
  • Individual key delete support
  • Support iteration over all and only live key-value pairs (different from log scan)
  • Full support for async interface to FasterKV
  • Full support for async/await threading model in C#: PR
  • IFunctions specified via sessions
  • Remove generic type constraints, add default serializers and comparers for common types

Log, Cache, and Storage

  • Variable sized keys and values using separate object log, see [here]
  • Two allocators (specializing for Blittable and Generic key-value types), with common extensible base class
  • Generic IDevice abstraction with out-of-the-box implementations for local storage
  • Segmented log on storage (log file broken into segments of configurable size)
  • Allocator support for copying reads to tail, useful for update-intensive workloads
  • Highly configurable allocator parameters (log and read cache) with respect to memory size, page size, mutable fraction of memory, etc.
  • Support for runtime shifting of address markers (e.g., begin, head, read-only) for dynamic tuning of memory contents of allocators (log and read cache).
  • Log compaction by rolling forward older active keys
  • Support for subscribing to the hybrid log (push-based, as record batches become read-only): PR
  • Support for callback when records in hybrid log become read-only: PR
  • Support for cloud storage, starting with Azure Page Blobs: PR
  • Support for tiered storage: PR
  • Support for sharded storage: PR
  • Support for FasterLog as an independent first-class abstraction: PR
  • Reduced log memory footprint significantly (min pages in memory reduced to 1)
  • Improved performance of fine-grained epoch protection
  • Support per-entry checksums and persistent iterators in FasterLog
  • Full support for async interface to FasterLog
  • Improved IDevice v2 interface without Overlapped, improved local storage scalability

Checkpoint and Recovery

  • CPR-based checkpointing and recovery (snapshot and fold-over modes), see [here] for details
  • Optional separate checkpointing of index and log (so that index may be checkpointed less frequently)
  • Auto-recover to most recent checkpoint
  • Make checkpointing use a pluggable user-specified interface for providing devices and performing metadata commit: PR
  • Generic state machine for checkpointing (internals)
  • Choose checkpoint type (Snapshot and FoldOver) on a per-checkpoint basis
  • Process all checkpoints through a IDevice-based checkpoint and log commit manager

Ongoing and Future Work

  • Client-server interface to FASTER
  • Read variant starting from given logical addresses
  • Limit number of preloaded pages during recovery
  • RDMA IDevice implementation
  • Support for Incremental Snapshot checkpoint type
  • Expose incremental data structures over FasterKV hash chains
  • Scale-out and elasticity support
  • Handle log logical addresses greater than 48 bit (up to 64 bit)
  • Checksums for storage pages

Release Notes

FASTER v2019.10.31.1

  • Support for FasterLog as an independent first-class abstraction: PR
  • Reduced log memory footprint significantly (min pages in memory reduced to 1)
  • Improved performance of fine-grained epoch protection
  • Support per-entry checksums and persistent iterators in FasterLog
  • Full support for async interface to FasterLog

FASTER v2019.8.27.1

  • Improved support for varlen blittable allocator (iteration, compaction): PR
  • BREAKING CHANGE: change return type for InPlaceUpdater and ConcurrentWriter functions to bool. User has to return true for usual in-place-update (IPU) behavior, and return false to force retry as read-copy-update (RCU).
  • Support for cloud storage, starting with Azure Page Blobs: PR
  • Support for tiered storage: PR
  • Make checkpointing use a pluggable user-specified interface for providing devices and performing metadata commit: PR
  • Support for sharded storage: PR

FASTER v2019.7.23.1

  • Object log recovery bug fix: PR
  • Option to enable file buffering for local storage device
  • Optimizing what is loaded to hybrid log memory during recovery (prior head address onwards only)
  • Removing direct call of callback when IO completes synchronously: PR
  • Fixed checkpoint recovery bug: PR
  • Adding FILE_SHARE_DELETE when deleteOnClose is used: PR
  • Support for callback when records in hybrid log become read-only: PR

FASTER v2019.4.24.4

  • Added support for variable sized (inline) structs without object log: PR
  • Removed statics from codebase to better support multiple instances: PR
  • Fixes related to scheduling pending operations: PR

FASTER v2019.4.1.1

  • Log compaction by rolling forward older active keys: PR
  • Individual key delete support: PR

FASTER v2019.3.16.1 (cumulative feature list)

  • Full Read, Upsert, RMW functionality
  • Bulk delete via key expiry: user-initiated truncation from head of log (ShiftBeginAddress)
  • Persistence support for larger-than-memory data
  • Scan operation support for scanning a specified range of log
  • Ability to resize (grow) the hash table in powers of two (user-initiated)
  • Separately configurable optional read cache for read-heavy workloads
  • Support for separate user-defined object hash/equality comparers and object serializers
  • Remove C# dynamic code-gen for quicker instantiation, stability, debuggability
  • Full support for .NET core and Linux/Mac
  • Experimental feature: DeleteFromMemory to delete recently added keys that are still in memory
  • Variable sized keys and values using separate object log, see [here]
  • Two allocators (specializing for Blittable and Generic key-value types), with common extensible base class
  • Generic IDevice abstraction with out-of-the-box implementations for local storage
  • Segmented log on storage (log file broken into segments of configurable size)
  • Allocator support for copying reads to tail, useful for update-intensive workloads
  • Highly configurable allocator parameters (log and read cache) with respect to memory size, page size, mutable fraction of memory, etc.
  • Support for runtime shifting of address markers (e.g., begin, head, read-only) for dynamic tuning of memory contents of allocators (log and read cache).
  • CPR-based checkpointing and recovery (snapshot and fold-over modes), see [here] for details
  • Optional separate checkpointing of index and log (so that index may be checkpointed less frequently)
  • Auto-recover to most recent checkpoint

C++ Porting Notes

FASTER C++ is a fairly direct port of FASTER C# using C++ based coding and style guidelines. It supports the following features as of now:

  • Full Read, Upsert, RMW functionality
  • Persistence support for larger-than-memory data
  • Variable sized payloads; no separate object log, see [here]
  • Log segments on storage, with truncation from head of log
  • CPR-based checkpointing and recovery (both snapshot and fold-over modes), see [here]
  • Ability to resize the hash table
  • C++: Added a new value_size() method to RmwContext for RCU operations: PR