Salt Events and Reactor

Reactor System

Not a Module, configured separately, native part of Salt, used to react to different events.
Reactor is the go-to place to create 'self-healing' or 'fully-automated' solutions. Reactions are matched on the Salt Master.
The reaction sls files follow the same rules of compiling (they have default Jinja+YAML renderer), however they are limited in terms of available Dunder Dictionaries.
Additional: data dictionary is available for reactions which contains event data.

Reaction types

The sls reaction files must contain reaction type in front of execution_module_name.function_name

Local reaction

Runs Execution Module on targeted minions (not necessarily a "local"). Example:

highstate:
  local.state.highstate:
    - tgt: {{ data['id'] }}

This example runs highstate on data['id'] minion

Runner reaction

Runs Runner Modules on the Salt Master. Centralized Salt Masters view allows to create complex flows easily. Most widely used Runner Module for this purpose is the Orchestrate Runner.

redis_cluster_orchestrate:
  runner.state.orchestrate:
    - args:
      - mods:
        - redis.server._orchestrate
      - saltenv: server

Orchestrate Runner

In Salt terminology, the highstate is a collection of states applied to one minion.
Collection of states applied to multiple minions with inter-minion dependencies could be called Orchestration.
Orchestrate is more generic term than highstate.
As this is Runner Module it can be directly called from CLI: salt-run state.orchestrate kubernetes._orchestrate.cluster saltenv=server
Combined with Salt event system allows to create multi minion-aware reactions.

state.orchestrate Module accepts mods argument, this is the sls file list with actual orchestration logic. Example of redis.server._orchestrate:

refresh_pillar:
    salt.function:
    - name: saltutil.pillar_refresh_synchronous    # saltutil.pillar_refresh_synchronous doesn't exist, see below for explanation 
    - tgt: {{ salt['pillar.get']("redis:coordinator") }}

cluster_met:
    salt.state:
    - tgt: {{ salt['pillar.get']("redis:coordinator") }}
    - sls:
      - "redis.server._orchestrate.met"
    - queue: True
    - require:
      - salt: refresh_pillar

# some more logic
# ...

Orchestrate Runner accepts other sls'es evaluates them on Salt Master and invokes them on desired targets. These sls'es contain regular salt states/functions or even Runner Modules.
To simplify:

1. Some situation triggers event
2. Event is propagated to Salt Master
3. Salt Master checks if it can find reaction
4. Reaction is rendered if found.
5. Reaction executes Runner Orchestrate Module if runner.state.orchestrate
6. Runner Orchestrate Module renders mods on the Salt Master
7. Runner Orchestrate Module executes functions on desired targets.

The most typical orchestrate sls files will comprise mostly of salt.[function|state] calls as they accept the tgt parameter and thus can delegate the call to minions.

Aforementioned example contains unfortunate 'gotcha' in Salt.
Typically Salt Master may want Minions to refresh the Pillar data prior to invoking desired states. Thus calling saltutil.pillar_refresh prior to the states execution seems like viable solution.
However it may (and usually will) not work, because pillar_refresh function actually doesn't refresh the Pillar on Salt Minion. This particular function is asynchronous by default - which is inconsistent with most of the states that are synchronous.

Wheel reaction

Runs Wheel Modules on the Salt Master

Caller reaction

Used for Masterless Minions. The minion must be properly configured. Runs Execution Modules on the minion

Reactor state files limitations

Matching and redering reaction sls files is done sequentially in single process. Because of this, the reaction sls files should contain very few reactions. Also heavy jinja logic within reaction sls files can choke whole Reactor System.
Reactor doesn't support require or other requisite statements.
Pillar and grain data are not available.
Thus any time, some complex logic is required to handle event the state.orchestrate should be used as a reaction.

Example of typical flow (logic moved to orchestrator)