Specify dependencies between jobs

[fr33m0nk]

Hi team,

I recently started looking into clojure job executors.
I was curious to know if it's possible to specify dependencies between jobs.
e.g.

(def scheduled-job-dependency-graph
  {:start []
   :result1 [:start]
   :result2 [:start]
   :result3 [:result2]
   :finish [:result1 :result3]})

Libraries like overseer (not maintained and brittle) and juagerro (does not support scheduled jobs) support this by DAG implementation.

However, my use case demands DAG like dependency definition between different steps of a scheduled job, most of which are cron like periodic jobs.
I am happy to contribute if such a feature does not exist and is aligned with the goals of Goose.

[olttwa]

Hello @fr33m0nk,

Job-linking can be done by use of middlewares in Goose.

(ns goose.dag-jobs
  (:require [goose.brokers.rmq.broker :as rmq]
            [goose.client :as c]
            [goose.worker :as w]))

(defn my-dag-job
  [arg1 arg2]
  ;; Returns pre-known results.
  )

;;; Client-side code
(let [rmq-producer (rmq/new-producer rmq/default-opts)
      client-opts (assoc c/default-opts :broker rmq-producer)]
  (c/perform-async client-opts `my-dag-job :foo :bar))


;;; Worker-side code
(defn my-middlware
  [broker]
  (fn [next]
    (fn [opts job]
      ;; Linkage can be done based on name of the job.
      (condp = (:execute-fn-sym job)
        `my-dag-job
        (let [job-result (next opts job)
              client-opts (assoc c/default-opts :broker broker)]
          (condp = job-result
            :result1 (c/perform-async client-opts `job-one :foo :baz)
            :result2 (c/perform-in-sec client-opts 300 `job-two :foo :baz)))

        ;; Default case.
        (next opts job))
      )))

(let [rmq-producer (rmq/new-producer rmq/default-opts)
      ;; Inject a producer in the middleware that can
      ;; enqueue/schedule messages for background processing.
      dag-middleware (my-middlware rmq-producer)

      rmq-consumer (rmq/new-consumer rmq/default-opts)
      worker-opts (assoc w/default-opts :broker rmq-consumer
                                        :middlewares dag-middleware)
      worker (w/start worker-opts)]
  ;; Listen for sigint/sigterm...
  (w/stop worker))

[olttwa]

Does above code fulfil your requirement?

We'd like to keep Goose a plug-and-play library where it's power is derived from extending basic built-in features. For those reasons, I don't see a need for a specific DAG-style job linking feature in Goose.

Another reason is that results of a job are presumed to be dynamic, and above mentioned :result1, :result2 is a static design. If we were to bake a job-linking feature, it must be possible to modify it at runtime.

If you have any suggestions, do let me know.

[fr33m0nk]

Thanks a ton!!

I understand the rationale and your reasoning.
Honestly, your reply looks pretty much like what I desire.
I will take this for a spin and post back how it pans out.

Thanks again!