Using the sidekiq executor backend

[anirbanmu]

Disclosure: this is less of an issue, more of a question. If there is a better avenue to ask these sorts of questions, please let me know and I can redirect.

I've been playing with dynflow to figure out if it fits some needs in our fairly standard Rails + Sidekiq application. I did have some confusion regarding the sidekiq executor backend though. Here is what I've done to get it mostly working in our application from seeing the example in the examples directory:

1. I've added the queue dynflow_orchestrator to the list of queues the sidekiq workers process.
2. I've added a block like this to my sidekiq initializer:

if Sidekiq.server?
  Dynflow::Config.new.tap do |config|
    config.persistence_adapter = persistence_adapter
    config.logger_adapter      = Dynflow::LoggerAdapters::Simple.new($stderr, Logger::DEBUG)
    config.auto_rescue         = false
    config.telemetry_adapter   = Dynflow::TelemetryAdapters::Dummy.new
    config.queues.add(:slow, :pool_size => 3)
    config.executor            = ::Dynflow::Executors::Sidekiq::Core
    config.process_role        = :worker
    config.connector           = Proc.new { |world| Dynflow::Connectors::Database.new(world) }
  end.then { |config| Dynflow::World.new(config) }
end

Is this correct so far?

I also have a question regarding what sort of Dynflow::World I should be creating for the Rails web application side of the app. This is the side that will need to trigger the Dynflow::Actions that will then get executed by sidekiq. I'm using this right now (one line commented out from above):

  world = Dynflow::Config.new.tap do |config|
    config.persistence_adapter = persistence_adapter
    config.logger_adapter      = Dynflow::LoggerAdapters::Simple.new($stderr, Logger::DEBUG)
    config.auto_rescue         = false
    config.telemetry_adapter   = Dynflow::TelemetryAdapters::Dummy.new
    config.queues.add(:slow, :pool_size => 3)
    config.executor            = ::Dynflow::Executors::Sidekiq::Core
    # config.process_role        = :worker
    config.connector           = Proc.new { |world| Dynflow::Connectors::Database.new(world) }
  end.then { |config| Dynflow::World.new(config) }

It does seem to be working but I'm unsure if I'm doing things correctly. Can anyone guide me in the right direction?

[adamruzicka]

Hi,
I'd suggest you take a look at the rails configuration in this repository, how it is used in Foreman[2], the primary Dynflow user, and how you can override the configuration[3]. Foreman currently uses the non-sidekiq way of running Dynflow, but it can give you some insights into how to configure the world without having to configure every single option by hand.

Then for running the orchestrator and workers, I'd suggest using something like this loader[4]. It is a small script which loads the rails environment, spawns the Dynflow world and then handles the control to Sidekiq. In general I'd avoid running an executor in the rails process, just setting it to false should be fine.

Does this somewhat fit into the way you were trying to run it?

Also I'm aruzicka on #theforeman-dev on freenode IRC, feel free to drop in and we can talk more about this in a more real time fashion.

[1] - https://github.com/Dynflow/dynflow/blob/master/lib/dynflow/rails/configuration.rb
[2] - https://github.com/theforeman/foreman/blob/develop/config/application.rb#L313
[3] - https://github.com/theforeman/foreman/blob/develop/lib/foreman/dynflow/configuration.rb
[4] - adamruzicka/foreman-tasks@deb4c6a

[anirbanmu]

Hi @adamruzicka,

Thanks for all the resources you provided; I'm looking through those now. I might jump on freenode after I look through those.

One immediate thing that came to mind upon looking at the loader at [4] was that, is it necessary to run two separate processes for orchestrator and worker? I'm not 100% clear on how everything works, but I was hoping to integrate this in a way such that my existing sidekiq worker processes can do the work for the dynflow orchestrator & workers. Is this not possible?

Anyhow, I will look into the resources you posted in more detail & come back! Thank you again.

[adamruzicka]

Running an orchestrator as a separate process is not strictly necessary, but it might be useful in the long run. Having the orchestrator in a separate process from the worker has several benefits. It allows you to safely restart the workers if needed. It is also better if you expect high amount of jobs to keep the latency low.