Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

README typo #209

Open
wants to merge 15 commits into
base: master
Choose a base branch
from
116 changes: 49 additions & 67 deletions README.md
Original file line number Diff line number Diff line change
@@ -1,6 +1,8 @@
# gocraft/work [![GoDoc](https://godoc.org/github.com/gocraft/work?status.png)](https://godoc.org/github.com/gocraft/work)
# DispatchMe/go-work [![GoDoc](https://godoc.org/github.com/DispatchMe/go-work?status.png)](https://godoc.org/github.com/DispatchMe/go-work)

gocraft/work lets you enqueue and processes background jobs in Go. Jobs are durable and backed by Redis. Very similar to Sidekiq for Go.
**This is a fork of [gocraft/work](https://www.github.com/gocraft/work) that removes their strange usage of custom contexts. The use of reflection to support that was detracting from efficiency and didn't feel very "go-like", so we made this change**

DispatchMe/go-work lets you enqueue and processes background jobs in Go. Jobs are durable and backed by Redis. Very similar to Sidekiq for Go.

* Fast and efficient. Faster than [this](https://www.github.com/jrallison/go-workers), [this](https://www.github.com/benmanns/goworker), and [this](https://www.github.com/albrow/jobs). See below for benchmarks.
* Reliable - don't lose jobs even if your process crashes.
Expand All @@ -11,6 +13,10 @@ gocraft/work lets you enqueue and processes background jobs in Go. Jobs are dura
* Web UI to manage failed jobs and observe the system.
* Periodically enqueue jobs on a cron-like schedule.

## Run tests

Redis must be installed to avoid a panic when running tests.

## Enqueue new jobs

To enqueue jobs, you need to make an Enqueuer with a redis namespace and a redigo pool. Each enqueued job has a name and can take optional arguments. Arguments are k/v pairs (serialized as JSON internally).
Expand All @@ -19,8 +25,8 @@ To enqueue jobs, you need to make an Enqueuer with a redis namespace and a redig
package main

import (
"github.com/garyburd/redigo/redis"
"github.com/gocraft/work"
"github.com/gomodule/redigo/redis"
"github.com/DispatchMe/go-work"
)

// Make a redis pool
Expand All @@ -33,12 +39,18 @@ var redisPool = &redis.Pool{
},
}

type SendEmailJobParameters struct {
Address string
Subject string
CustomerID int
}

// Make an enqueuer with a particular namespace
var enqueuer = work.NewEnqueuer("my_app_namespace", redisPool)

func main() {
// Enqueue a job named "send_email" with the specified parameters.
_, err := enqueuer.Enqueue("send_email", work.Q{"address": "test@example.com", "subject": "hello world", "customer_id": 4})
_, err := enqueuer.Enqueue("send_email", &SendEmailJobParameters{Address: "test@example.com", Subject: "hello world", CustomerID: 4})
if err != nil {
log.Fatal(err)
}
Expand All @@ -55,8 +67,8 @@ In order to process jobs, you'll need to make a WorkerPool. Add middleware and j
package main

import (
"github.com/garyburd/redigo/redis"
"github.com/gocraft/work"
"github.com/gomodule/redigo/redis"
"github.com/DispatchMe/go-work"
"os"
"os/signal"
)
Expand All @@ -71,27 +83,21 @@ var redisPool = &redis.Pool{
},
}

type Context struct{
customerID int64
}

func main() {
// Make a new pool. Arguments:
// Context{} is a struct that will be the context for the request.
// 10 is the max concurrency
// "my_app_namespace" is the Redis namespace
// redisPool is a Redis pool
pool := work.NewWorkerPool(Context{}, 10, "my_app_namespace", redisPool)
pool := work.NewWorkerPool(10, "my_app_namespace", redisPool)

// Add middleware that will be executed for each job
pool.Middleware((*Context).Log)
pool.Middleware((*Context).FindCustomer)
pool.Middleware(LogMiddleware)

// Map the name of jobs to handler functions
pool.Job("send_email", (*Context).SendEmail)
pool.Job("send_email", SendEmailHandler)

// Customize options:
pool.JobWithOptions("export", JobOptions{Priority: 10, MaxFails: 1}, (*Context).Export)
pool.JobWithOptions("export", JobOptions{Priority: 10, MaxFails: 1}, ExportHandler)

// Start processing jobs
pool.Start()
Expand All @@ -105,63 +111,45 @@ func main() {
pool.Stop()
}

func (c *Context) Log(job *work.Job, next work.NextMiddlewareFunc) error {
fmt.Println("Starting job: ", job.Name)
func LogMiddleware(ctx *work.Context, next work.NextMiddlewareFunc) error {
fmt.Println("Starting job: ", ctx.Job.Name)
return next()
}

func (c *Context) FindCustomer(job *work.Job, next work.NextMiddlewareFunc) error {
// If there's a customer_id param, set it in the context for future middleware and handlers to use.
if _, ok := job.Args["customer_id"]; ok {
c.customerID = job.ArgInt64("customer_id")
if err := job.ArgError(); err != nil {
return err
}
}

return next()
}

func (c *Context) SendEmail(job *work.Job) error {
func SendEmailHandler(ctx *work.Context) error {
// Extract arguments:
addr := job.ArgString("address")
subject := job.ArgString("subject")
if err := job.ArgError(); err != nil {
args := new(SendEmailJobParameters)
err := ctx.Job.UnmarshalPayload(args)
if err != nil {
return err
}

// Go ahead and send the email...
// sendEmailTo(addr, subject)
// sendEmailTo(args.Address, args.Subject)

return nil
}

func (c *Context) Export(job *work.Job) error {
func ExportHandler(ctx *work.Context) error {
return nil
}
```

## Special Features

### Contexts

Just like in [gocraft/web](https://www.github.com/gocraft/web), gocraft/work lets you use your own contexts. Your context can be empty or it can have various fields in it. The fields can be whatever you want - it's your type! When a new job is processed by a worker, we'll allocate an instance of this struct and pass it to your middleware and handlers. This allows you to pass information from one middleware function to the next, and onto your handlers.

Custom contexts aren't really needed for trivial example applications, but are very important for production apps. For instance, one field in your context can be your tagged logger. Your tagged logger augments your log statements with a job-id. This lets you filter your logs by that job-id.

### Check-ins

Since this is a background job processing library, it's fairly common to have jobs that that take a long time to execute. Imagine you have a job that takes an hour to run. It can often be frustrating to know if it's hung, or about to finish, or if it has 30 more minutes to go.

To solve this, you can instrument your jobs to "checkin" every so often with a string message. This checkin status will show up in the web UI. For instance, your job could look like this:

```go
func (c *Context) Export(job *work.Job) error {
func ExportHandler(ctx *work.Context) error {
rowsToExport := getRows()
for i, row := range rowsToExport {
exportRow(row)
if i % 1000 == 0 {
job.Checkin("i=" + fmt.Sprint(i)) // Here's the magic! This tells gocraft/work our status
ctx.Job.Checkin("i=" + fmt.Sprint(i)) // Here's the magic! This tells DispatchMe/go-work our status
}
}
}
Expand All @@ -170,9 +158,11 @@ func (c *Context) Export(job *work.Job) error {

Then in the web UI, you'll see the status of the worker:

```
| Name | Arguments | Started At | Check-in At | Check-in |
| --- | --- | --- | --- | --- |
| export | {"account_id": 123} | 2016/07/09 04:16:51 | 2016/07/09 05:03:13 | i=335000 |
```

### Scheduled Jobs

Expand All @@ -197,22 +187,22 @@ job, err = enqueuer.EnqueueUniqueIn("clear_cache", 300, work.Q{"object_id_": "78

### Periodic Enqueueing (Cron)

You can periodically enqueue jobs on your gocraft/work cluster using your worker pool. The [scheduling specification](https://godoc.org/github.com/robfig/cron#hdr-CRON_Expression_Format) uses a Cron syntax where the fields represent seconds, minutes, hours, day of the month, month, and week of the day, respectively. Even if you have multiple worker pools on different machines, they'll all coordinate and only enqueue your job once.
You can periodically enqueue jobs on your DispatchMe/go-work cluster using your worker pool. The [scheduling specification](https://godoc.org/github.com/robfig/cron#hdr-CRON_Expression_Format) uses a Cron syntax where the fields represent seconds, minutes, hours, day of the month, month, and week of the day, respectively. Even if you have multiple worker pools on different machines, they'll all coordinate and only enqueue your job once.

```go
pool := work.NewWorkerPool(Context{}, 10, "my_app_namespace", redisPool)
pool := work.NewWorkerPool(10, "my_app_namespace", redisPool)
pool.PeriodicallyEnqueue("0 0 * * * *", "calculate_caches") // This will enqueue a "calculate_caches" job every hour
pool.Job("calculate_caches", (*Context).CalculateCaches) // Still need to register a handler for this job separately
pool.Job("calculate_caches", CalculateCaches) // Still need to register a handler for this job separately
```

## Run the Web UI

The web UI provides a view to view the state of your gocraft/work cluster, inspect queued jobs, and retry or delete dead jobs.
The web UI provides a view to view the state of your DispatchMe/go-work cluster, inspect queued jobs, and retry or delete dead jobs.

Building an installing the binary:
```bash
go get github.com/gocraft/work/cmd/workwebui
go install github.com/gocraft/work/cmd/workwebui
go get github.com/DispatchMe/go-work/cmd/workwebui
go install github.com/DispatchMe/go-work/cmd/workwebui
```

Then, you can run it:
Expand Down Expand Up @@ -248,11 +238,11 @@ You'll see a view that looks like this:
* The worker will then run the job. The job will either finish successfully or result in an error or panic.
* If the process completely crashes, the reaper will eventually find it in its in-progress queue and requeue it.
* If the job is successful, we'll simply remove the job from the in-progress queue.
* If the job returns an error or panic, we'll see how many retries a job has left. If it doesn't have any, we'll move it to the dead queue. If it has retries left, we'll consume a retry and add the job to the retry queue.
* If the job returns an error or panic, we'll see how many retries a job has left. If it doesn't have any, we'll move it to the dead queue. If it has retries left, we'll consume a retry and add the job to the retry queue.

### Workers and WorkerPools

* WorkerPools provide the public API of gocraft/work.
* WorkerPools provide the public API of DispatchMe/go-work.
* You can attach jobs and middleware to them.
* You can start and stop them.
* Based on their concurrency setting, they'll spin up N worker goroutines.
Expand Down Expand Up @@ -298,7 +288,7 @@ You'll see a view that looks like this:
* "worker observation" - A snapshot made by an observer of what a worker is working on.
* "periodic enqueuer" - A process that runs with a worker pool that periodically enqueues new jobs based on cron schedules.
* "job" - the actual bundle of data that constitutes one job
* "job name" - each job has a name, like "create_watch"
* "job name" - each job has a name, like `create_watch`
* "job type" - backend/private nomenclature for the handler+options for processing a job
* "queue" - each job creates a queue with the same name as the job. only jobs named X go into the X queue.
* "retry jobs" - If a job fails and needs to be retried, it will be put on this queue.
Expand All @@ -308,26 +298,18 @@ You'll see a view that looks like this:

## Benchmarks

The benches folder contains various benchmark code. In each case, we enqueue 100k jobs across 5 queues. The jobs are almost no-op jobs: they simply increment an atomic counter. We then measure the rate of change of the counter to obtain our measurement.
The benches folder used to contain various benchmark code. In each case, we enqueued 100k jobs across 5 queues. The jobs were almost no-op jobs: they simply incremented an atomic counter. We then measured the rate of change of the counter to obtain our measurement. These were some test results:

| Library | Speed |
| --- | --- |
| [gocraft/work](https://www.github.com/gocraft/work) | **20944 jobs/s** |
| [DispatchMe/go-work](https://www.github.com/DispatchMe/go-work) | **20944 jobs/s** |
| [jrallison/go-workers](https://www.github.com/jrallison/go-workers) | 19945 jobs/s |
| [benmanns/goworker](https://www.github.com/benmanns/goworker) | 10328.5 jobs/s |
| [albrow/jobs](https://www.github.com/albrow/jobs) | 40 jobs/s |


## gocraft

gocraft offers a toolkit for building web apps. Currently these packages are available:

* [gocraft/web](https://github.com/gocraft/web) - Go Router + Middleware. Your Contexts.
* [gocraft/dbr](https://github.com/gocraft/dbr) - Additions to Go's database/sql for super fast performance and convenience.
* [gocraft/health](https://github.com/gocraft/health) - Instrument your web apps with logging and metrics.
* [gocraft/work](https://github.com/gocraft/work) - Process background jobs in Go.

These packages were developed by the [engineering team](https://eng.uservoice.com) at [UserVoice](https://www.uservoice.com) and currently power much of its infrastructure and tech stack.
The comparison benchmarks were run against repositories that were stale and unmaintained by fall of 2018. Invalid
import paths were causing tests to fail in go-lib, which has background and indexer packages that rely on this
repository. As the benchmarks were not currently needed, they were removed.

## Authors

Expand Down
123 changes: 0 additions & 123 deletions benches/bench_goworker/main.go

This file was deleted.

Loading