All notable changes to this project will be documented in this file.
The format is based on Keep a Changelog, and this project adheres to Semantic Versioning.
- Fix error cardinality while push metric to Prometheus
- Add metrics when barito-flow failed connect to ES or Kafka
- Adjust log message when barito-flow failed connect to ES or Kafka
- Make barito-flow-consumer compatible with both ES 6.x and ES 7.x
- Add basic auth for xpack-security feature
- Remove index settings when creating new index
- Set document type to
_doc
- Add more metrics on producer
- Fixed: return early on error, before calling any method on nil value
- Reorder config to fetch kafka and elasticsearch value
- Support GRPC with gateway
- Add instrumentation for prometheus
- Enable consul to get multi elasticsearch urls, if urls not available then using environment variable
- Update instru to the latest version, to avoid race condition
- Add the ability to use bulk indexing when committing logs to Elasticsearch
- Only create Elasticsearch client once, instead of recreating it before every commit
- Retry connecting to kafka during startup if kafka cluster is not available (configurable)
- Enable toggle-able verbose mode
- Set round robin strategy as default for consumer rebalancing, also allow it to be configured
- Recreate cluster admin whenever barito-flow needs to create a new topic
- Update rate limiter behaviour for produce batch, now it counts rate per line in batch instead of per batch request
- Lower metadata refresh frequency
- Refresh metadata before creating new topic (see here)
- Rate limiter on produce batch should count per request, not per log
- Ability to process batch logs
- Use updated instru that avoid race condition problem on counter
- Implement Elasticsearch backoff functionality
- Upgrade sarama to 1.19.0
- Use sarama ClusterAdmin for creating topic
- Produce logs if flow has to return 5xx errors
- Use updated instru that avoid race condition problem
- Also send application secret from context when sending metrics
- Send metrics to market
- Fix bug when checking for new events
- Support for multiple topics and indexes
- Graceful Shutdown
- Using consul to get kafka brokers, if consul not available then using environment variable
- Using consul to get elasticsearch url, if url not available then using environment variable
- Rate limit trx per second (by default 100)
- Rename receiver to producer
- Rename forwarder to consumer