Background Jobs

Run work outside the request cycle — welcome emails, report generation, outbound webhooks, image processing — with retries, backoff, dead-lettering, and monitoring.

regius make job send-welcome-email

What It Generates

Creates workers/<name>.go: a typed payload struct, a handler with TODO markers, and an Enqueue<Name> helper that handlers/middleware call to schedule the job through app.App.Jobs
On the first run it also bootstraps workers/register.go (the RegisterAll hub), wires workers.RegisterAll(app.App.Jobs) into init.regius.go, and scaffolds the regius_jobs/regius_locks table migration for the DATABASE_TYPE dialect (delete the migration if JOBS_BACKEND stays memory or redis)
Later runs append a MustRegister line to register.go; duplicate names are refused
Hyphenated or underscored names become valid Go identifiers: send-welcome-email and send_welcome_email both produce the SendWelcomeEmail handler and the send_welcome_email job name

Quickstart

regius new demo && cd demo           # 1. create an app
regius make job send-welcome-email   # 2. scaffold the job (+ wiring + migration)

Fill in the TODO in workers/send_welcome_email.go
Turn the feature on in .env:

JOBS_ENABLED=true
JOBS_BACKEND=sql   # or redis; memory (default) keeps jobs only for the current process

If you chose sql, run ./regius migrate to create the regius_jobs table
Call workers.EnqueueSendWelcomeEmail(...) from a handler and start the app — the work now happens outside the request cycle, with retries if it fails

Where Each Piece Goes

A generated app has three files involved in background jobs:

<app>/
├── init.regius.go                # boot — calls workers.RegisterAll(app.App.Jobs)
├── handlers/
│   └── signup.go                 # dispatch — call workers.Enqueue<Name>(...) here
└── workers/
    ├── register.go               # registration — MustRegister calls live here
    └── send_welcome_email.go     # the handler + payload + enqueue helper

Code	File	Who writes it
`MustRegister("send_welcome_email", SendWelcomeEmail, jobs.Options{…})`	`workers/register.go` (inside `RegisterAll`)	`regius make job` writes it (with `MaxAttempts: 3`); you edit it to tune `MaxAttempts` / `Timeout` / `Backoff`
`func SendWelcomeEmail(ctx, j *jobs.Job) error { … }` (the handler)	`workers/send_welcome_email.go`	`regius make job` scaffolds it; you fill in the TODO
`workers.EnqueueSendWelcomeEmail(ctx, app.App.Jobs, payload)`	called from `handlers/*.go`	you call it where the work needs to happen
`workers.RegisterAll(app.App.Jobs)`	`init.regius.go`	`regius make job` wires it on first run
`app.App.Jobs.Cron("0 9 * * *", …)` / `Every(…)`	`workers/register.go` (inside `RegisterAll`)	you add them for recurring schedules

The Handler File

// workers/send_welcome_email.go
package workers

import (
    "context"

    "github.com/hbarral/regius/jobs"
)

type SendWelcomeEmailPayload struct {
    UserID int    `json:"user_id"`
    Email  string `json:"email"`
}

func SendWelcomeEmail(ctx context.Context, j *jobs.Job) error {
    var p SendWelcomeEmailPayload
    if err := j.Decode(&p); err != nil {
        return err
    }
    // TODO: send the email using p.UserID / p.Email
    return nil
}

func EnqueueSendWelcomeEmail(ctx context.Context, m *jobs.Manager, p SendWelcomeEmailPayload) (*jobs.Job, error) {
    return m.Enqueue(ctx, "send_welcome_email", p)
}

Dispatching Work

// handlers/signup.go
func (h *Handlers) Signup(w http.ResponseWriter, r *http.Request) {
    // ...create the user, hash the password, insert...

    // hand the welcome email off to a background worker
    workers.EnqueueSendWelcomeEmail(r.Context(), h.App.Jobs, workers.SendWelcomeEmailPayload{
        UserID: user.ID,
        Email:  user.Email,
    })

    // ...redirect...
}

Registration Hub

// workers/register.go
package workers

import "github.com/hbarral/regius/jobs"

func RegisterAll(m *jobs.Manager) {
    m.MustRegister("send_welcome_email", SendWelcomeEmail, jobs.Options{
        MaxAttempts: 5,
    })

    // additional jobs are registered here
}

Tuning Retries, Timeouts, and Backoff

Edit the generated registration line in workers/register.go:

m.MustRegister("sync_payments", SyncPayments, jobs.Options{
    MaxAttempts: 8,                                                   // default 3 (JOBS_MAX_ATTEMPTS)
    Timeout:     2 * time.Minute,                                     // per-attempt deadline; 0 = none
    Backoff:     jobs.ExponentialBackoff(30*time.Second, time.Hour),  // default: 30s base, 15m cap
})

FixedBackoff(d) waits exactly d between attempts; LinearBackoff(step) waits step × attempt; ExponentialBackoff(base, max) doubles from base up to max. All exponential delays get full jitter (random within [0, delay]) so a burst of failures doesn't retry in lockstep
A handler that exceeds Timeout — or returns any error — counts as a failed attempt; a panic is recovered and also counts as a failed attempt (it never kills the worker)
After the final attempt the job is dead and its last_error is recorded — find and resurrect it via the monitoring endpoints

Choosing a Backend

JOBS_BACKEND picks where jobs live:

Backend	Jobs survive a restart?	Multi-process?	Use it for
`memory` (default)	No — queue is empty on boot	No	dev, tests, apps where lost work is acceptable
`sql`	Yes — rows in your own database (`regius_jobs` table)	Yes	the default choice for production: one database, atomic claims, nothing extra to run
`redis`	Yes — keys under `JOBS_PREFIX`	Yes	apps already running Redis that don't want job rows in their primary database

# sql — jobs live in the app's own database (any DATABASE_TYPE)
JOBS_ENABLED=true
JOBS_BACKEND=sql
# then: ./regius migrate   (regius make job already scaffolded the migration)

# redis — jobs live in Redis, reusing the app's REDIS_* connection settings
JOBS_ENABLED=true
JOBS_BACKEND=redis
JOBS_PREFIX=regius:jobs   # key namespace; change it if several apps share one Redis

app.App.Jobs is always constructed (memory backend by default), so Enqueue works even with workers off — e.g. a web process that only dispatches
Workers and the scheduler run in a process only when JOBS_ENABLED=true; ListenAndServe starts them and drains them on shutdown within JOBS_GRACEFUL_TIMEOUT (in-flight attempts are cancelled and their jobs requeued)

Delivery Semantics

At-least-once: a crash between claiming and completing a job means it runs again after its lease expires — handlers must be idempotent
Retries with backoff (Fixed/Linear/Exponential with full jitter); after MaxAttempts a job is dead and visible in the monitoring endpoints
A maintenance loop reclaims expired leases (crash recovery) and prunes completed jobs past JOBS_RETENTION
Keep payloads small (a few KB of IDs and parameters); for large blobs, store the file via filesystems (S3, MinIO, ...) and pass a reference in the payload

Writing Idempotent Handlers

Because delivery is at-least-once, a handler may run twice for the same logical work. Guard the side effect instead of hoping it doesn't happen:

func SendWelcomeEmail(ctx context.Context, j *jobs.Job) error {
    var p SendWelcomeEmailPayload
    if err := j.Decode(&p); err != nil {
        return err
    }

    // dedup: each job run has a unique ID — record it wherever the side
    // effect lands (a "sent_welcome_emails" row, a cache key, ...) and
    // treat "already done" as success.
    if alreadySent(p.UserID) {
        return nil // idempotent no-op: the first attempt already did the work
    }

    return sendEmail(p.Email)
}

A returned error schedules the next attempt; returning nil marks the job completed. Handlers that respect ctx cancellation can be interrupted cleanly on shutdown.

Scheduling Recurring Jobs

Also in workers/register.go, inside RegisterAll, so it runs once at boot before the workers start:

func RegisterAll(m *jobs.Manager) {
    m.MustRegister("send_welcome_email", SendWelcomeEmail, jobs.Options{MaxAttempts: 5})
    m.MustRegister("send_daily_digest", SendDailyDigest, jobs.Options{MaxAttempts: 3})
    m.MustRegister("refresh_cache", RefreshCache, jobs.Options{})

    // recurring schedules fire on their own from here
    m.Cron("0 9 * * *", "send_daily_digest", nil)     // robfig/cron 5-field spec
    m.Every(5*time.Minute, "refresh_cache", nil)

    // additional jobs are registered here
}

Recurring schedules enqueue an ordinary job at each tick, so scheduled work shares the same retry/backoff/monitoring pipeline as ad-hoc jobs
A one-off at a specific time (At) is called from a handler instead, since it is triggered by a request:

// handlers/reports.go — generate tomorrow at 06:00
runAt := time.Now().AddDate(0, 0, 1).Truncate(24 * time.Hour).Add(6 * time.Hour)
app.App.Jobs.At(r.Context(), runAt, "generate_report", GenerateReportPayload{UserID: user.ID})

Delayed Jobs and Per-Job Overrides

EnqueueWithOptions schedules the first attempt in the future and can give one specific job its own attempt budget:

// retry a flaky import sooner and harder than the handler default
_, err := app.App.Jobs.EnqueueWithOptions(r.Context(), "import_csv", ImportPayload{Path: path}, jobs.EnqueueOptions{
    RunAt:       time.Now().Add(15 * time.Minute), // zero/past = as soon as possible
    MaxAttempts: 10,                               // overrides the handler's Options for this job only
})

The scaffolded Enqueue<Name> helpers wrap Enqueue for the common case; call EnqueueWithOptions directly when a job needs a delay or its own retry budget.

Monitoring

Opt-in via JOBS_DASHBOARD_ENABLED=true:

Method	Path	Action
`GET`	`/api/jobs/stats`	counts by status (pending/running/completed/dead)
`GET`	`/api/jobs?status=dead&name=…&limit=…`	list jobs (`limit` ≤ 200)
`POST`	`/api/jobs/{id}/retry`	resurrect a dead job (attempts reset)
`DELETE`	`/api/jobs/{id}`	remove a dead job

These routes live on the outer mux, so they bypass CSRF/sanitizer (no form body). They are off by default; layer APIKeyAuth or IPFilter on them in production.

Example round-trip against a running app (JOBS_DASHBOARD_ENABLED=true):

# how is the queue doing?
curl http://localhost:4000/api/jobs/stats
# {"data":{"pending":0,"running":0,"completed":41,"dead":2}}

# something died — inspect it (newest first)
curl "http://localhost:4000/api/jobs?status=dead&limit=10"

# fix the underlying issue, then give the job a fresh attempt budget
curl -X POST http://localhost:4000/api/jobs/cn4lq8v2m0d0c3f1/retry
# {"data":{"status":"pending"}}

# or discard it for good
curl -X DELETE http://localhost:4000/api/jobs/cn4lq8v2m0d0c3f1
# {"data":{"status":"dropped"}}

The same operations are available programmatically on app.App.Jobs (Stats, List, Get, Retry, Drop).

The Job Model

Field	Meaning
`id`	unique, sortable ID of this job run
`name`	the registered job name (`send_welcome_email`)
`payload`	the JSON you passed to `Enqueue` — decode with `j.Decode(&v)`
`status`	`pending` → `running` → `completed`, or `dead` after the final failed attempt
`attempts` / `max_attempts`	how many tries have run / the budget for this job
`run_at`	when the next attempt becomes eligible (now, or after backoff/delay)
`lease_until`	crash detection: a `running` job past this is reclaimed and requeued
`last_error`	the error string from the most recent failed attempt
`created_at` / `updated_at` / `completed_at`	bookkeeping timestamps

Deploying with Workers

The simplest setup runs one process: it serves HTTP and runs workers (JOBS_ENABLED=true) — right for most apps
To scale, run the same binary with different env: web processes with JOBS_ENABLED=false (they only enqueue; Jobs is always constructed so dispatch works) and one or more worker processes with JOBS_ENABLED=true
With sql/redis backends, claims are atomic — several worker processes can safely pull from the same queue
Recurring schedules (Cron/Every) fire on one process at a time when JOBS_SCHEDULER_LOCK=true (default)
On shutdown (SIGTERM), in-flight attempts are cancelled and their jobs requeued within JOBS_GRACEFUL_TIMEOUT; idempotent handlers make that safe

Environment Variables

#JOBS_ENABLED=false
#JOBS_BACKEND=memory
#JOBS_PREFIX=regius:jobs
#JOBS_WORKERS=4
#JOBS_POLL_INTERVAL=1s
#JOBS_LEASE=5m
#JOBS_MAX_ATTEMPTS=3
#JOBS_GRACEFUL_TIMEOUT=30s
#JOBS_RETENTION=24h
#JOBS_SCHEDULER_LOCK=true
#JOBS_DASHBOARD_ENABLED=false

What It Generates​

Quickstart​

Where Each Piece Goes​

The Handler File​

Dispatching Work​

Registration Hub​

Tuning Retries, Timeouts, and Backoff​

Choosing a Backend​

Delivery Semantics​

Writing Idempotent Handlers​

Scheduling Recurring Jobs​

Delayed Jobs and Per-Job Overrides​

Monitoring​

The Job Model​

Deploying with Workers​

Environment Variables​