Yaad

Yaad is a high-throughput scheduleable-job queue. Jobs with different trigger times go in - jobs ordered by trigger by time come out.

• What is Yaad?
• Quickstart
  • Docker
  • Pre-built Binary
  • Running built-in load test
• Configuration
• Related work and inspiration

What is Yaad

Yaad is a job queue (like rabbitMQ) but for jobs that trigger at some time. You add jobs to it with a specific date/time trigger and they are made available to come off the queue at that time.

In it's simplest form, Yaad is an in-memory queue. Producers can enqueue jobs onto Yaad with a trigger time and Yaad will internally order them by their trigger time. Jobs with trigger time closest to now are dequeued before jobs with later trigger times. In other words, Yaad orders a stream of jobs by their trigger time.

Use cases

• Scheduled Emails/SMSs - send messages some time after an event occurs
• Abandoned shopping cart reminders
• Deadman switches - perform pre-set action if job isn't canceled before expiry
• Streaming Time-sorted messages - enqueue jobs in a window and consume a sorted stream
• Asynchronous work queue - run time-consuming tasks asynchronously

Quickstart

Docker

1. Pull latest image docker: docker pull urjitbhatia/goyaad:latest

2. Run server: docker run --rm --name goyaad -p 11301:11301 urjitbhatia/goyaad:latest -L server

3. Interact using the cli:

   docker run --rm urjitbhatia/goyaad:latest put --id "j1" --body "I should be ready second" --delay 45s
   docker run --rm urjitbhatia/goyaad:latest put --id "j2" --body "I should be ready first" --delay 2s
   docker run --rm urjitbhatia/goyaad:latest put --id "j3" --body "Cancel me" --delay 20m
   docker run --rm urjitbhatia/goyaad:latest next --timeout 20s
   docker run --rm urjitbhatia/goyaad:latest next --timeout 50s
   docker run --rm urjitbhatia/goyaad:latest cancel --id "j3"

Pre-built binary

1. Fetch binary

   1. Github releases https://github.com/urjitbhatia/goyaad/releases
   2. Or via go get: go get github.com/urjitbhatia/yaad

2. Run server: goyaad -L server

3. Interact using the cli:

   goyaad put --id "j1" --body "I should be ready second" --delay 45s
   goyaad put --id "j2" --body "I should be ready first" --delay 2s
   goyaad put --id "j3" --body "Cancel me" --delay 20m
   goyaad next --timeout 20s
   goyaad next --timeout 50s
   goyaad cancel --id "j3"

Running built-in load test

1. Docker:

      docker run --rm --name loadtest urjitbhatia/goyaad:latest \
         -L loadtest -e -d \
         --raddr \$(docker inspect goyaad -f '{{range .NetworkSettings.Networks}}{{.IPAddress}}{{end}}'):11301

2. Run a load test: goyaad -L loadtest -e -d

Configuration

Common configuration options

1. StatsD server address --statsAddr string Remote StatsD listener (host:port) (default ":8125")
2. Enable human-friendly coloured logs -l, --log-level string Set log level: INFO, DEBUG (default "INFO")
3. Set log level -L, --friendly-log Use a human-friendly logging style
4. RPC listen address --raddr string Bind RPC listener to (host:port) (default ":11301")
   1. For Server mode, it the address the server should advertize on
   2. For other modes, it is the address of the target server

Operation Mode: Server

Runs Yaad queue server. If restore is used, the server first attempts to restore jobs from the give snapshot location and then starts serving the queue. If the server receives a SIGUSR1, it exits gracefully by creating a new snapshot with any jobs that were held in memory at shutdown time. The snapshots can be copied to different machines and supplied to new server instances.

1. Restore jobs from a snapshot --restore
   1. Loads snapshots from --store-url
      1. Filesystem dir (default: PWD) or S3-style url. Examples: filesystemdir/subdir or {file|s3|gs|azblob}://bucket (default "/usr/local/bin")
      2. An optional --store-prefix can also be provided for S3 compatible addressing scheme
2. SpokeSpan is an advanced tuning parameter. It sets the bucket size for job ordering. -S, --spokeSpan duration Spoke span (golang duration string format) (default 10s) It configures the spread of job trigger times.

Operation Mode: Loadtest

Runs a loadtest that generated jobs with random payloads with their trigger times between the delayMin and the delayMax values (relative to when they're generated).

1. Set max concurrent producer connections -c, --con int Number of connections to use (default 5)
2. Max trigger time for test jobs. A random trigger time between delayMin and delayMax is assigned to test jobs -M, --delayMax int Max delay in seconds (Delay is random over delayMin, delayMax) (default 60)
3. Min trigger time for test jobs -N, --delayMin int Min delay in seconds (Delay is random over delayMin, delayMax)
4. Set dequeue mode -d, --dequeueMode Dequeue jobs
5. Set enqueue mode -e, --enqueue Enqueue jobs
6. Number of jobs to generated -n, --num int Number of total jobs (default 1000)
7. Fixed job payload size -z, --size int Job size in bytes (default 1000)

Operation Mode: Inspect

1. Number of Jobs to fetch -n, --num int Max Number of jobs to inspect (default 1)
   • If the target server has fewer than num jobs, it will return less than num jobs.
2. Inspect output file location -o, --out string Write output to outfile (default: stdout)

Related work and inspiration

• Beanstalkd
• RabbitMQ