A keen observer of changes that can also relay change events reliably to interested parties. Provides useful infrastructure for building Eventually Consistent data sources and systems. Growing list of change data source and destinations:
| Sources | Destinations | Notes |
|---|---|---|
| MySQL | MySQL, HBase, Elastic Search, Kafka | MySQL source |
| HBase | MySQL, HBase, Elastic Search, Kafka | [HBase source] (https://github.com/Flipkart/aesop/wiki/HBase-change-event-producer) |
| Any Data source | Any destination | [Pull Producer] (https://github.com/Flipkart/aesop/wiki/Pull-change-event-producer) |
| Release | Date | Description |
|---|---|---|
| Version 1.2.3-SNAPSHOT | May 2016 | Move to Trooper 2.0.0, log4j2 instead of logback |
| Version 1.2.1 | Apr 2016 | Bug fix release |
| Version 1.2.1-SNAPSHOT | Aug 2015 | Feature enhancements and bug fix release |
| Version 1.2.0 | Jul 2015 | Bug fix release |
Changelog can be viewed in CHANGELOG.md file (https://github.com/Flipkart/aesop/blob/master/CHANGELOG.md)
Data change propagation from source to consumers is a fairly common requirement in systems that automate business processes. An example is inventory updates on a warehousing system reflecting on product pages of an eCommerce portal. This example is an instance that requires the data updates to propagate with low latency and reliably. Few broad options exist:
- Application publishes changes asynchronously to a queue before/after writing data to persistent store - this is usually fire-and-forget in most implementations and therefore unreliable.
- Variants include those with retries, back-off and queue sidelining. Reliability can be enhanced using local transactions (not distributed) and message relaying.
- Batch extraction(snapshots) and load every few hours - affects data freshness in the consumer systems.
- Database supported replication - very common approach when source and consumer systems use the same data store technology(e.g. MySQL master-slave replication) and share the same data model / schema.
Aesop provides reliable, low-latency data change propagation for source and consumer systems that optionally use different data stores. It also supports snapshot based change detection. For consumers, it provides a unified interface for change data consumption that is independent of how the data change is sourced. More on the Change Propagation Approach
Console showing the relay and all connected consumers with SCN information per partition
Live metrics on producer,consumers progress highlighting best and worst performing partition per consumer against producer SCN
The Getting Started page has "5 minute" examples to help you start using Aesop.
Aesop project modules that start with "sample" - for e.g. sample-memory-relay, sample-client are example implementations. Documentation is continuously being added to the Wiki page of Aesop (https://github.com/Flipkart/aesop/wiki)
For discussion, help regarding usage, or receiving important announcements, subscribe to the Aesop users mailing list: http://groups.google.com/group/aesop-users
Aesop is licensed under : The Apache Software License, Version 2.0. Here is a copy of the license (http://www.apache.org/licenses/LICENSE-2.0.txt)
- Chandan Bansal (@chandanbansal)
- Jagadeesh Huliyar (@jagadeesh-huliyar)
- Kartik B Ukhalkar (@kartikssj)
- Prakhar Jain (@jprakhar77)
- Regunath B (@regunathb)
- Shoury (@Shoury)
- Yogesh Dahiya (@yogeshdfk)
- Arya Ketan (@aryaKetan)
- Flipkart Payments : The payment processing system uses a MySQL data store for holding transactional data. This data is needed in de-normalized form in the reconciliation data store(MySQL again) and an archive data store (HBase). Data changes need to reflect in the reconciliation and archive stores almost instantaneously. Aesop is used to do this. This system is in production and has processed thousands of events per second with insert rates of about 30K per second on HBase.
- Flipkart User data : The Aesop relay is used to propagate changes on Flipkart user accounts to the central analysis data system.
- Flipkart User Wishlist : The Aesop relay is used to propagate changes on Flipkart user wishlists to an Elastic Search secondary index used in searches.