- Does Karafka require Ruby on Rails?
- Why there used to be an ApplicationController mentioned in the Wiki and some articles?
- Does Karafka require Redis and/or Sidekiq to work?
- Could an HTTP controller also consume a fetched message through the Karafka router?
- Does Karafka require a separate process running?
- Can I start Karafka process with only particular consumer groups running for given topics?
- Can I use
#seekto start processing topics partition from a certain point? - Why Karafka does not pre-initialize consumers so all the callbacks can be executed in their context?
- Racecar breaks Rails loading including Karafka when trying to migrate from one to another
- Why Karafka does not restart dead PG connections?
- Does Karafka require gems to be thread-safe?
No. Karafka is a fully independent framework that can operate in a standalone mode. It can be easily integrated with any Ruby-based application, including those written with Ruby on Rails. For more details, please follow the Integrating with Ruby on Rails and other frameworks Wiki section.
You can name the main application consumer with any name. You can even call it ApplicationController or anything else you want. Karafka will sort that out, as long as your root application consumer inherits from the Karafka::BaseConsumer. It's not related to Ruby on Rails controllers. Karafka framework used to use the *Controller naming convention up until Karafka 1.2 where it was changed because many people had problems with name colisions.
No. Karafka is a standalone framework, with an additional process that will be used to consume Kafka messages. You can however use Karafka Sidekiq Backend if you would like to outsource your Kafka messages processing to Sidekiq workers.
No. Kafka messages can be consumed only using Karafka consumers. You cannot use your Ruby on Rails HTTP consumers to consume Kafka messages, as Karafka is not an HTTP Kafka proxy. Karafka uses Kafka API for messages consumption.
Yes. Karafka requires a separate process to be running (Karafka server) to consume and process messages. You can read about it in the Consuming messages section of the Wiki.
Yes. Karafka allows you to listen with a single consumer group on multiple topics, which means, that you can tune up the number of threads that Karafka server runs, accordingly to your needs. You can also run multiple Karafka instances, specifying consumer groups that should be running per each process using the --consumer_groups server flag as followed:
bundle exec karafka server --consumer_groups group_name1 group_name3Karafka provices a #seek client method that can be used to do that. Due to the fact, that Karafka is a long-running process, this option needs to be set up as a callback executed before fetching starts per each consumer group. Please refer to the #before_fetch_loop Wiki section for more details.
Why Karafka does not pre-initialize consumers so all the callbacks can be executed in their context?
Because Karafka does not have knowledge about the whole topology of a given Kafka cluster. We work on what we receive dynamically building consumers when it is required.
This issue has been described here.
Racecar under the hood detects that you want Rails by... trying to load Rails. If it is able to, will run the whole app before other things are loaded. In most cases, this won't cause any trouble but from time to time... well it does.
To fix that (as long as you want to use both at the same time), please add the return unless Rails.application in your karafka.rb accordingly:
# Ruby on Rails setup
ENV['RAILS_ENV'] ||= 'development'
ENV['KARAFKA_ENV'] = ENV['RAILS_ENV']
require ::File.expand_path('../config/environment', __FILE__)
return unless Rails.application
Rails.application.eager_load!This is an expected behavior because Karafka is meant to be transaction supporting system with long living consumers.
Please see this troubleshooting page for a more extensive explanation of this behavior.
For most of the time no. The only case for which any library that you use should be thread-safe, is a case where you run multiple consumer groups from withing a single Karafka process. As long as you don't do that, neither your libraries nor code needs to be thread-safe.