dj_queue is a database-backed task queue backend for Django's django.tasks framework, inspired by Rails' Solid Queue.
Queue state, runtime process metadata, concurrency semaphores, recurring schedules, and task results live in Django-managed SQL tables. No Redis, RabbitMQ, or separate result store is required. PostgreSQL is the first-class production backend; MySQL, MariaDB, and SQLite are supported at Django's database version floors.
Django applications already treat the relational database as the durable system of record. dj_queue executes immediate, scheduled, recurring, and concurrency-limited work on that same store:
- application code defines callables with Django's standard
@taskdecorator (defandasync def) DjQueueBackendpersists jobs, execution states, and return values through a single transactional operations layer- polling is the correctness path on every supported database;
SELECT ... FOR UPDATE SKIP LOCKEDand PostgreSQLLISTEN/NOTIFYare optional latency and throughput optimizations - runtime supervision (
forkorasync), operational controls, Django admin views, health checks, and Prometheus/JSON telemetry ship in one package
For trade-offs against Celery, RQ, Procrastinate, Huey, and other django.tasks backends, see docs/COMPARISONS.md. For throughput and contention measurements across supported databases, see docs/benchmarks/.
- Installation
- Quick start
- Concepts
- Enqueueing and results
- Running the runtime
- Recurring tasks
- Concurrency controls
- Queue and job operations
- Admin dashboard
- Monitoring and health
- Configuration reference
- Multi-database setup and backend coexistence
- Embedded server mode
- Database support
- Rolling upgrades
- PostgreSQL maintenance
- Examples
dj_queue requires Python 3.12+ and Django 6.0 or 6.1. The distribution ships inline type annotations and a PEP 561 py.typed marker.
pip install dj-queueOptional extras:
pip install "dj-queue[postgres]" "psycopg[binary]" # PostgreSQL + LISTEN/NOTIFY
pip install "dj-queue[prometheus]" # /dj_queue/metrics endpointOn PostgreSQL, dj-queue[postgres] depends on psycopg>=3.3.3 without forcing a C extension build. Install psycopg[binary] (bundled libpq) or psycopg[c] (system libpq); running on pure-Python psycopg requires a system libpq, reduces throughput, and emits a psycopg_pure_python.warning log event at supervisor startup. For MySQL or MariaDB, install a standard Django-compatible database driver.
Configure INSTALLED_APPS, DATABASE_ROUTERS, and TASKS in settings.py:
INSTALLED_APPS = [
# ...
"dj_queue",
]
DATABASE_ROUTERS = ["dj_queue.routers.DjQueueRouter"]
TASKS = {
"default": {
"BACKEND": "dj_queue.backend.DjQueueBackend",
"QUEUES": [],
"OPTIONS": {},
},
}DjQueueRouter is optional when all queue tables live on DATABASES["default"], and required for a dedicated queue database. For mixing dj_queue with other django.tasks backends, see Backend coexistence.
Apply migrations:
python manage.py migrateDefine a task in your Django app:
# myapp/tasks.py
from django.tasks import task
@task
def add(a: int, b: int) -> int:
return a + bStart the runtime supervisor:
python manage.py dj_queueWith the default configuration, this starts one supervisor with one worker (3 threads, all queues "*"), one dispatcher, and a scheduler (because finished-job cleanup is on by default).
Enqueue the task and read its persisted result:
from myapp.tasks import add
result = add.enqueue(3, 7)
fresh = add.get_backend().get_result(result.id)
print(fresh.status, fresh.return_value)Once a worker executes the job, fresh.status is TaskResultStatus.SUCCESSFUL and fresh.return_value is 10.
dj_queue separates callable definitions, persisted records, and routing scopes:
- task: a Python callable (
deforasync def) decorated with@django.tasks.task. - job: one persisted record in
dj_queue_jobscreated when a task is enqueued. Its UUID primary key isTaskResult.id. - recurring task: a schedule definition in
dj_queue_recurring_tasksthat enqueues jobs over time. - queue (
queue_name): a named lane within a backend alias (default"default"). Enqueueing validatesqueue_nameagainstTASKS[backend_alias]["QUEUES"]when that list is non-empty. - backend alias: a key in
settings.TASKSwith"BACKEND": "dj_queue.backend.DjQueueBackend". Each backend alias scopes its own runtime topology, jobs, pauses, process rows, recurring tasks, admin views, and metrics. - database alias: a key in
settings.DATABASESselected byOPTIONS["database_alias"](default"default"). It determines where that backend's queue and control tables reside.
Common deployment layouts combine one backend alias on default (simplest start), one backend alias on a dedicated queue database (recommended for production), multiple backend aliases sharing one database (logical isolation without extra migrations), or multiple backend aliases on separate databases (full storage isolation).
Every live job has at most one execution-state row (dj_queue_*_executions). Finished jobs have finished_at set and no execution-state row; successful jobs are deleted instead when preserve_finished_jobs=False. Operator discards (discard_* helpers, QueueInfo.clear(), admin) delete the job row; on_conflict="discard" keeps it as finished.
| Job state | Persisted representation | TaskResult.status |
Transition summary |
|---|---|---|---|
ready |
ReadyExecution |
READY |
Eligible to be claimed by a worker matching queue_name. |
scheduled |
ScheduledExecution |
READY |
Deferred until scheduled_at <= now(), then promoted by a dispatcher. |
blocked |
BlockedExecution |
READY |
Waiting for a concurrency semaphore slot; promoted to ready on slot handoff or maintenance. |
claimed |
ClaimedExecution |
RUNNING |
Locked by a registered worker process and executing in its thread pool. |
failed |
FailedExecution |
FAILED |
Task raised, could not be imported, returned non-JSON data, or lost its worker process; kept until retried, discarded, or pruned. |
finished |
Job.finished_at != None |
SUCCESSFUL |
Completed normally (with return_value), or discarded at enqueue/dispatch by on_conflict="discard" (return_value=None). |
Due scheduled jobs and retried failed jobs re-enter the dispatch path (ready, scheduled, blocked, or finished on discard-conflict); blocked jobs move only to ready. No promotion goes directly to claimed. If external data corruption leaves a job with multiple execution-state rows or combines finished_at with an execution-state row, Job.status reports "invalid" and get_result() returns FAILED with a DjQueueError diagnostic.
| Component | Responsibility |
|---|---|
supervisor |
Registers a Supervisor process row, starts and monitors runners (fork child processes or async threads), replaces exited runners, fails orphaned claims, and prunes stale process rows. |
worker |
Claims ready jobs matching its queue selectors, executes tasks in a thread pool, records finished or failed transitions, and releases or hands off concurrency slots. |
dispatcher |
Promotes due scheduled jobs and due failed retries (retry_at <= now()) in batches, deletes unused expired semaphores, and retries expired blocked jobs. |
scheduler |
Syncs static recurring tasks at startup, fires due recurring tasks with database deduplication, and runs configured retention cleanup. |
DjQueueBackend sets supports_async_task = True, supports_defer = True, supports_get_result = True, and supports_priority = True. Both synchronous (enqueue, enqueue_all, get_result) and asynchronous (aenqueue, aenqueue_all, aget_result) backend methods are implemented.
Use .using(...) or @task(...) arguments to set queue_name, priority (integer -100 to 100, default 0; higher runs first), run_after (datetime for scheduled execution), backend (backend alias), and @task(takes_context=True) to receive a django.tasks.TaskContext:
from datetime import timedelta
from django.tasks import TaskContext, task
from django.utils import timezone
@task(queue_name="email", priority=10, takes_context=True)
def send_digest(context: TaskContext, kind: str) -> str:
return f"{context.task_result.id}:{kind}"
due = timezone.now() + timedelta(hours=1)
send_digest.using(run_after=due, priority=-5).enqueue("daily")DjQueueBackend.enqueue_all() and aenqueue_all() validate and insert a batch of (task, args, kwargs) tuples in one transaction, grouping concurrency slot acquisition and deferring wakeup notifications until commit:
from myapp.tasks import add
backend = add.get_backend()
results = backend.enqueue_all([(add, [i, 1], {}) for i in range(100)])If any item in the batch fails validation, the entire batch raises EnqueueError before inserting rows.
enqueue() writes to the queue database immediately and returns a persisted TaskResult. It never defers writes implicitly or returns placeholder IDs. When a task reads rows written in an open database transaction, defer the enqueue until that transaction commits with enqueue_on_commit():
from django.db import transaction
from dj_queue.api import enqueue_on_commit
from myapp.tasks import add
with transaction.atomic():
enqueue_on_commit(add, 3, 7)Pass using="<db_alias>" to enqueue_on_commit() when wrapping a transaction on a non-default application database. Because using selects the transaction.on_commit() database alias, tasks that take a keyword argument literally named using should call transaction.on_commit(lambda: task.enqueue(..., using=...), using=...) directly.
backend.get_result(result_id) and await backend.aget_result(result_id) load the persisted Job for that backend alias and return a django.tasks.TaskResult, or raise django.tasks.exceptions.TaskResultDoesNotExist if the job does not exist, belongs to another backend alias, or was deleted (preserve_finished_jobs=False, discarded, or pruned).
Async backend calls (aenqueue, aenqueue_all, aget_result) wrap the sync implementation via asgiref.sync.sync_to_async, closing obsolete Django connections before and after each call; see async_thread_sensitive and async_close_connections to customize threading and connection teardown.
Task args, kwargs, and return_value are stored in JSON columns and must round-trip strictly through JSON (NaN, Infinity, -Infinity, non-string dict keys, and arbitrary Python objects are rejected):
- non-JSON
argsorkwargsraiseEnqueueErrorat enqueue time - a non-JSON
return_valuetransitions the claimed job tofailed(ValueError: return value must be JSON round-trippable)
Pass primary keys or explicit primitive representations rather than ORM model instances or file handles.
DjQueueBackend.enqueue() and enqueue_all() raise EnqueueError when:
queue_nameis outside a non-emptyTASKS[backend_alias]["QUEUES"]allow-listpriorityis not anint(booleans are rejected) or is outside-100..100argsorkwargsfail payload serializationconcurrency_keyis set withoutconcurrency_limit, cannot be formatted from the bound arguments, or resolves to"", a non-string, or more than 255 characters (arguments that do not bind to the task signature raiseTypeError)concurrency_limitorconcurrency_durationis not a positive integer, oron_conflictis not"block"or"discard"
Exceptions defined in dj_queue.exceptions:
| Exception | Base | Description |
|---|---|---|
DjQueueError |
Exception |
Base exception class for all dj_queue-specific errors. |
EnqueueError |
DjQueueError |
Validation failure on enqueue or dynamic recurring registration, or an execution-state precondition conflict. |
DispatchPolicyError |
EnqueueError |
Persisted or imported task concurrency policy cannot be resolved during promotion, retry, or recovery. |
ProcessExitError |
DjQueueError |
Recorded on FailedExecution when a supervised runner exits or stops while holding a claimed job. |
ProcessMissingError |
DjQueueError |
Recorded on FailedExecution when a ClaimedExecution row has no registered Process row. |
ProcessPrunedError |
DjQueueError |
Recorded on FailedExecution when a runner's heartbeat expires (process_alive_threshold) and its process row is pruned. |
python manage.py dj_queue [--backend BACKEND] [--config CONFIG] [--mode {fork,async}] \
[--only-work | --only-dispatch] [--skip-recurring] [--migration-wait-timeout SECONDS]Before starting runners, dj_queue checks the backend's configured database_alias for unapplied dj_queue migrations and polls every 0.5s up to --migration-wait-timeout seconds so rolling deployments can finish migrate concurrently. If pending migrations remain at the deadline, the command exits with CommandError. See Precedence and validation for how CLI flags interact with environment variables and settings.
| Flag | Type | Default | Description |
|---|---|---|---|
--backend |
str |
"default" |
TASKS backend alias to supervise; must use DjQueueBackend. |
--config |
str | None |
None |
Path to a TOML options overlay file. |
--mode |
str | None |
None |
Standalone supervision mode ("fork" or "async"). |
--only-work |
bool |
False |
Start workers only, without dispatchers or scheduler; cannot be combined with --only-dispatch. |
--only-dispatch |
bool |
False |
Start dispatchers only, without workers or scheduler; cannot be combined with --only-work. |
--skip-recurring |
bool | None |
None |
Suppress the scheduler even when recurring or cleanup work is configured. |
--migration-wait-timeout |
float |
60 |
Seconds (>= 0) to wait for pending dj_queue migrations on database_alias (0 fails immediately). |
forkmode (default):ForkSupervisorspawns each configured worker process, dispatcher, and scheduler as a separate OS child process and monitors them withwaitpid().asyncmode:AsyncSupervisorruns workers, dispatchers, and the scheduler on managed threads inside a single OS process, lowering memory footprint at the cost of process-level isolation and sharing one Python GIL. Workerprocesses > 1is normalized to1with aUserWarning.
Crashed runners in both modes are replaced immediately the first time, then after 0.1s doubling up to 30s on repeated crashes; the backoff resets once a runner stays up for 60s.
Multiple hosts may run against the same queue database concurrently; workers, dispatchers, and schedulers coordinate through database locks and unique constraints. In multi-node deployments, keep static recurring definitions identical across full nodes and prefer one full manage.py dj_queue instance alongside additional manage.py dj_queue --only-work nodes. Add --only-dispatch nodes only when you need more scheduled-job promotion or concurrency-maintenance throughput. Set supervisor_pidfile to prevent duplicate standalone supervisors on the same host.
Worker queues accepts a string or an ordered sequence of selector strings:
"*"matches all queues (any selectors listed after"*"are unreachable and ignored)"<prefix>*"matches queue names starting with<prefix>(for example,"email*")"<name>"matches an exact queue name
Claiming drains selectors strictly in list order before considering priority:
- selector position in
queues(for["email", "default"], all unpausedemailwork is claimed beforedefault, regardless of priority) prioritydescending (100down to-100)- FIFO (
ReadyExecution.idascending)
Avoid combining multi-queue selector ordering with cross-queue priority expectations on the same worker; dedicate separate worker entries when distinct queue lanes need independent priority ordering.
Standalone supervisors (manage.py dj_queue in fork or async mode) own OS signal handling:
SIGTERMandSIGINTrequest graceful shutdown, stop new claims, wait up toshutdown_timeoutseconds (shared across all supervised runners), deregister process rows, releasesupervisor_pidfile, and exit with status0.- At the
shutdown_timeoutdeadline,ForkSupervisorsendsSIGKILLto remaining child processes and fails their claimed jobs withProcessExitError. Inasyncand embedded modes, worker threads cannot be forcibly terminated; a worker that exceedsshutdown_timeoutsetsshutdown_state="draining"inProcess.metadata, continues heartbeating until in-flight threads finish, and then deletes its process row. SIGQUITsetsshutdown_timeoutto0, runs process-row and pidfile cleanup (SIGKILLto remainingforkchildren; waits for active job threads inasyncmode), and exits with status1.- Unrequested supervisor termination (such as an unhandled supervisor poll failure or a pruned supervisor row) exits with status
1.
Supervisor housekeeping runs on a bounded heartbeat cadence (max(min(process_alive_threshold, max(process_heartbeat_interval, 1)), 1) seconds, or max(min(process_alive_threshold, 60), 1) when process_heartbeat_interval == 0). Each pass recovers at most 500 claimed jobs in total: orphaned claims first (ProcessMissingError), then claims held by stale process rows older than process_alive_threshold, which it prunes (ProcessPrunedError). When dj_queue owns the transaction, queue state transitions retry transient database deadlocks and lock-wait timeouts up to 3 attempts; task functions run outside terminal-transition retries and are never repeated.
Register callbacks in dj_queue.hooks during Django startup (for example, AppConfig.ready()). Each callback receives the runner's dj_queue.models.Process instance (name, kind, pid, hostname, backend_alias, metadata). On *.exit events, the row has already been deleted from dj_queue_processes and the detached Process model instance is passed.
from dj_queue.hooks import on_start, on_worker_start, register_hook
@on_start
def supervisor_started(process):
print(process.name, process.backend_alias)
@on_worker_start
def worker_started(process):
print(process.metadata["queues"])
@register_hook("scheduler.exit")
def scheduler_exited(process):
print(process.name)| Event | Decorator | Description |
|---|---|---|
supervisor.start, supervisor.stop, supervisor.exit |
on_start, on_stop, on_exit |
Fired after supervisor registration, when stop begins, and after deregistration. |
worker.start, worker.stop, worker.exit |
on_worker_start, on_worker_stop, on_worker_exit |
Fired for each worker runner lifecycle transition. |
dispatcher.start, dispatcher.stop, dispatcher.exit |
on_dispatcher_start, on_dispatcher_stop, on_dispatcher_exit |
Fired for each dispatcher runner lifecycle transition. |
scheduler.start, scheduler.stop, scheduler.exit |
on_scheduler_start, on_scheduler_stop, on_scheduler_exit |
Fired for each scheduler runner lifecycle transition. |
register_hook(event, fn=None) supports both decorator and direct call forms; clear_hooks(event=None) removes hooks for one event or all events. Callbacks run in registration order; exceptions are isolated and routed to on_thread_error without blocking subsequent hooks.
Recurring tasks are executed by the built-in scheduler runner; no separate beat service or external cron daemon is required. Each RecurringTask row caches its next_run_at timestamp so polls skip rows that are not due.
Declare static schedules under TASKS[backend_alias]["OPTIONS"]["recurring"]:
TASKS = {
"default": {
"BACKEND": "dj_queue.backend.DjQueueBackend",
"QUEUES": [],
"OPTIONS": {
"recurring": {
"nightly_cleanup": {
"task_path": "myapp.tasks.cleanup",
"schedule": "0 3 * * *",
"args": [],
"kwargs": {"dry_run": False},
"queue_name": "maintenance",
"priority": -5,
"description": "nightly cleanup",
},
},
},
},
}Static definitions are validated at config load (ImproperlyConfigured on bad task_path, schedule, priority, or disallowed queue_name) and synchronized into dj_queue_recurring_tasks (static=True) when the scheduler starts, removing stale static rows no longer in settings and refusing to overwrite an existing dynamic key. See Static recurring task options for all fields.
Enable dynamic polling with OPTIONS["scheduler"]["dynamic_tasks_enabled"] = True, then manage schedules at runtime via dj_queue.api:
from dj_queue.api import schedule_recurring_task, unschedule_recurring_task
schedule_recurring_task(
key="tenant_42_report",
task_path="myapp.tasks.send_report",
schedule="every day at 08:00 in Europe/Rome",
kwargs={"tenant_id": 42},
queue_name="reports",
priority=5,
)
unschedule_recurring_task("tenant_42_report")Recurring task key values are unique per backend_alias. schedule_recurring_task() updates existing dynamic rows in place (resetting next_run_at when schedule changes) and raises EnqueueError if key belongs to static configuration. unschedule_recurring_task() deletes only dynamic rows (static=False), returning 1 when deleted and 0 when missing or static.
Schedules are parsed by dj_queue's built-in Fugit-compatible parser. Call dj_queue.cron.validate_cron(schedule: str) -> None to validate an expression directly (ValueError or TypeError on invalid input).
- Cron fields and presets: 5 fields (
minute hour day-of-month month day-of-week) or 6 fields with a leadingsecondsfield. Supported presets are@yearly,@annually,@monthly,@weekly,@daily,@midnight,@noon, and@hourly. - Cron extensions: month and weekday names (
JAN–DEC,SUN–SAT), steps, lists, wraparound ranges,L/lastand negative monthdays (-1is the last day of the month), weekday#(1#2= second Monday,5#-1= last Friday), and weekday modulo%(1%2= every other Monday). Day-of-month and day-of-week use OR semantics unless&is suffixed to either field to require both to match. An IANA timezone or UTC offset may trail the expression (for example,"30 1 * * * America/New_York"or"+02:00"). - Natural-language expressions:
every,at,on,from, andin <timezone>clauses, ordinal monthdays, and weekday/hour ranges—such as"every 5 minutes","every day at noon","every weekday at five","every month on day 2 at 10:00","from monday to friday at 9","at minute 5", and multi-time unions like"every day at 16:15 and 18:30". Natural interval counts must fit within their cron field; misleading intervals such as"every 90 seconds","every 90 minutes", and"every 24 hours"are rejected. - Calendar validation and DST: schedules that can never fire (such as
"0 0 31 2 *"or"0 0 31 * 1#1&") are rejected. Timezone-suffixed schedules follow local wall-clock time: during a fall-back overlap, daily and hourly schedules fire once at the first occurrence while sub-hourly schedules fire across both hours; times skipped by a spring-forward transition do not fire that day.
The scheduler starts automatically whenever at least one of the following holds (unless --only-work, --only-dispatch, or --skip-recurring is active):
OPTIONS["recurring"]is non-emptyOPTIONS["scheduler"]["dynamic_tasks_enabled"]isTruepreserve_finished_jobsisTrueandclear_finished_jobs_afteris notNone(true by default:86400s)clear_failed_jobs_afteris notNoneclear_recurring_executions_afteris notNone
Multiple schedulers deduplicate each (backend_alias, task_key, run_at) slot in dj_queue_recurring_executions with an intended_job_id allocated once per slot and reused when publication is retried, publishing the Job and its reservation attachment in one transaction (500 slots max per poll). Internal retention cleanup runs at most once every 60s (500 rows max per class per pass) without creating user-visible recurring task rows.
Decorate tasks with @concurrency from dj_queue.api to enforce database-backed semaphore limits:
from django.tasks import task
from dj_queue.api import concurrency
@task
@concurrency(key="account:{account_id}", limit=1, duration=60, on_conflict="block")
def sync_account(account_id: int, action: str = "full") -> str:
return f"{account_id}:{action}"Both decorator orders (@task above @concurrency or @concurrency above @task) are supported.
| Name | Type | Default | Description |
|---|---|---|---|
key |
str | Callable[..., str] |
required | Format string bound against the task's call signature (excluding TaskContext when takes_context=True), or a callable invoked as key(*args, **kwargs). Must resolve to a non-empty string of <= 255 characters. |
limit |
int |
required | Maximum concurrent ready plus claimed jobs holding the resolved key (>= 1). |
duration |
int | None |
OPTIONS["default_concurrency_duration"] |
Semaphore lease TTL in seconds (>= 1) used by dispatcher maintenance to recover leaked slots if a worker dies without cleanup. |
on_conflict |
Literal["block", "discard"] |
"block" |
"block" inserts a BlockedExecution row until a slot frees or expires; "discard" marks the Job finished immediately (return_value=None, TaskResultStatus.SUCCESSFUL) without running the task. |
Concurrency policy (concurrency_key, concurrency_limit, concurrency_duration, concurrency_on_conflict) is snapshotted onto each Job row at enqueue time. When a claimed job finishes or fails, its slot is atomically handed to the highest-priority blocked waiter for the same backend alias (promoting that waiter to ready) or released back to dj_queue_semaphores.
dj_queue_semaphores rows are shared across the queue database (database_alias). When multiple backend aliases share one queue database and require isolated limits, prefix key with the backend alias.
QueueInfo inspects and mutates per-backend queue state:
from dj_queue.api import QueueInfo
orders = QueueInfo("orders", backend_alias="default")
print(orders.size, orders.latency, orders.paused)
orders.pause()
orders.resume()
deleted = orders.clear()
for queue in QueueInfo.all(backend_alias="default"):
print(queue.queue_name, queue.size, queue.paused)sizereturns thereadyjob count for(backend_alias, queue_name).latencyreturnsNonewhile the queue is paused,0.0when unpaused with noreadyjobs, or the age in seconds of the oldestreadyjob (excluding intervals spent paused beforeresume()).pause()prevents future claims on(backend_alias, queue_name)without blockingenqueue()or interruptingclaimedjobs;clear()discardsreadyjobs only (releasing held concurrency slots).QueueInfo.all()discovers queues across ready, scheduled, claimed, blocked, failed, paused, and recurring rows in one snapshot and cachessize,latency, andpausedon each returned instance untilpause(),resume(), orclear()is called on it.
dj_queue does not run automatic retry or backoff policies; failed jobs retain exception_class, message, and traceback in dj_queue_failed_executions until acted on or pruned. Retry, schedule a future retry (retry_at, promoted by a dispatcher when due), or discard jobs by state (claimed jobs are never discardable):
from datetime import timedelta
from django.utils import timezone
from dj_queue.api import (
claim_ready_jobs,
discard_blocked_jobs,
discard_failed_job,
discard_failed_jobs,
discard_ready_jobs,
discard_scheduled_jobs,
execute_claimed_job,
retry_failed_job,
retry_failed_jobs,
schedule_failed_job_retry,
)
from dj_queue.models import Job
failed_ids = list(Job.objects.failed().values_list("id", flat=True)[:2])
if failed_ids:
schedule_failed_job_retry(failed_ids[0], retry_at=timezone.now() + timedelta(minutes=15))
retry_failed_job(failed_ids[0])
if len(failed_ids) > 1:
discard_failed_job(failed_ids[1])
retry_failed_jobs(batch_size=500)
discard_failed_jobs(batch_size=500)
discard_ready_jobs(batch_size=500)
discard_scheduled_jobs(batch_size=500)
discard_blocked_jobs(batch_size=500)
for claimed in claim_ready_jobs(limit=1, queues=["default"]):
execute_claimed_job(claimed)For read-only ORM inspection, dj_queue.models.Job provides .status ("ready", "scheduled", "claimed", "blocked", "failed", "finished", "invalid", or None), boolean status properties, and Job.objects queryset filters (.ready(), .scheduled(), .claimed(), .blocked(), .failed(), .finished(), .invalid_execution_state()).
All symbols exported in dj_queue.api (for schedule validation, see dj_queue.cron.validate_cron):
| Symbol | Signature | Description |
|---|---|---|
ClaimedJob |
(job: Job, claimed_at: datetime, worker_ids: tuple[str, ...], process_id: int | None = None) |
Dataclass returned by claim_ready_jobs(). |
QueueInfo |
(queue_name: str, *, backend_alias: str = "default") |
Queue read/control handle (.size, .latency, .paused, .pause(), .resume(), .clear(*, batch_size=500), QueueInfo.all(*, backend_alias="default")). |
claim_ready_jobs |
(*, limit: int, queues: str | Sequence[str] | None = None, process: Process | None = None, backend_alias: str = "default", config=None) -> list[ClaimedJob] |
Atomically lock and claim up to limit unpaused ready jobs. |
execute_claimed_job |
(job: ClaimedJob | Job | UUID | str, *, backend_alias: str = "default", config=None) -> Job |
Run a claimed job's task callable and persist its finished or failed outcome. |
concurrency |
(*, key: str | Callable[..., str], limit: int, duration: int | None = None, on_conflict: Literal["block", "discard"] = "block") |
Decorator attaching concurrency metadata to a task. |
enqueue_on_commit |
(task: Task, *args: Any, using: str | None = None, **kwargs: Any) -> None |
Register task.enqueue(*args, **kwargs) via transaction.on_commit(using=using). |
schedule_recurring_task |
(*, key: str, task_path: str, schedule: str, args: Sequence[Any] = (), kwargs: Mapping[str, Any] | None = None, queue_name: str = "default", priority: int = 0, description: str = "", backend_alias: str = "default", config=None) -> RecurringTask |
Create or update a dynamic recurring task. |
unschedule_recurring_task |
(key: str, *, backend_alias: str = "default", config=None) -> int |
Delete a dynamic recurring task; returns deleted count (0 or 1). |
retry_failed_job |
(job_id: UUID | str, *, backend_alias: str = "default", config=None) -> Job |
Redispatch one failed job immediately. |
retry_failed_jobs |
(*, job_ids: Iterable[UUID | str] | None = None, batch_size: int = 500, backend_alias: str = "default", config=None) -> int |
Redispatch up to batch_size failed jobs; returns retried count. |
schedule_failed_job_retry |
(job_id: UUID | str, *, retry_at: datetime, backend_alias: str = "default", config=None) -> Job |
Set FailedExecution.retry_at without leaving the failed state until due. |
discard_failed_job |
(job_id: UUID | str, *, backend_alias: str = "default", config=None) -> int |
Delete one failed job and its FailedExecution row. |
discard_failed_jobs |
(*, job_ids: Iterable[UUID | str] | None = None, batch_size: int = 500, backend_alias: str = "default", config=None) -> int |
Delete up to batch_size failed jobs. |
discard_ready_jobs |
(*, job_ids: Iterable[UUID | str] | None = None, batch_size: int = 500, backend_alias: str = "default", config=None) -> int |
Delete up to batch_size ready jobs and release held concurrency slots. |
discard_scheduled_jobs |
(*, job_ids: Iterable[UUID | str] | None = None, batch_size: int = 500, backend_alias: str = "default", config=None) -> int |
Delete up to batch_size scheduled jobs. |
discard_blocked_jobs |
(*, job_ids: Iterable[UUID | str] | None = None, batch_size: int = 500, backend_alias: str = "default", config=None) -> int |
Delete up to batch_size blocked jobs. |
python manage.py dj_queue_prune [--backend BACKEND] [--older-than SECONDS] \
[--failed-older-than SECONDS] [--recurring-older-than SECONDS] [--batch-size N] \
[--task-path TASK_PATH] [--task-key TASK_KEY]Deletes at most --batch-size rows per row class in one invocation and prints a notice when any class hits --batch-size so callers can re-run until drained.
| Flag | Type | Default | Description |
|---|---|---|---|
--backend |
str |
"default" |
TASKS backend alias to prune. |
--older-than |
int | None |
clear_finished_jobs_after |
Minimum finished_at age in seconds (>= 0) for finished jobs; skipped when both flag and config option are None. |
--failed-older-than |
int | None |
clear_failed_jobs_after |
Minimum age in seconds (>= 0) since the job failed (FailedExecution.created_at); skipped when both flag and config option are None. |
--recurring-older-than |
int | None |
clear_recurring_executions_after |
Minimum run_at age in seconds (>= 0) for dj_queue_recurring_executions; skipped when both flag and config option are None. |
--batch-size |
int |
500 |
Maximum rows (>= 1) deleted per row class per invocation. |
--task-path |
str | None |
None |
Restrict finished- and failed-job deletion to one task_path. |
--task-key |
str | None |
None |
Restrict recurring-execution deletion to one task_key. |
When django.contrib.admin is installed, dj_queue mounts a backend-aware dashboard at /admin/dj_queue/dashboard/ alongside read-only model changelists and detail views for Job, FailedExecution, Process, RecurringTask, Pause, and Semaphore.
- Job, failed-execution, process, pause, and recurring-task views filter by the selected
backendquery parameter; semaphore views scope to that backend'sdatabase_alias.JobandFailedExecutionchangelists omit fullCOUNT(*)totals to keep large tables responsive. - Queue drill-down pages inspect
ready,scheduled,claimed,blocked,failed,finished, andinvalidtabs and execute bulk operations in batches of500job IDs per transaction. - Active staff users (
is_activeandis_staff) can view the dashboard overview; drill-down pages and mutating actions require explicit Django model permissions (403when unauthorized, with unauthorized controls hidden):
| Action | Permission | Surface |
|---|---|---|
view |
dj_queue.view_job |
Queue drill-down page (/admin/dj_queue/dashboard/queue/<name>/). |
enqueue |
dj_queue.add_job |
Enqueue a fresh copy from non-scheduled Job detail or finished drill-down bulk action. |
enqueue_copy_now |
dj_queue.add_job |
Enqueue an immediate (run_after=None) copy from a scheduled Job detail page. |
run_now |
dj_queue.change_job |
Dispatch an existing scheduled job immediately from its Job detail page. |
retry |
dj_queue.change_job |
Retry failed jobs from Job detail, FailedExecution detail/changelist, or failed drill-down. |
discard |
dj_queue.delete_job |
Discard jobs from Job/FailedExecution views or ready, scheduled, blocked, and failed drill-downs. |
clear |
dj_queue.delete_job |
Clear all ready jobs in a queue from dashboard or queue drill-down controls. |
pause |
dj_queue.add_pause |
Pause a queue from dashboard or queue drill-down controls. |
resume |
dj_queue.delete_pause |
Resume a queue from dashboard, queue drill-down, or Pause detail controls. |
unschedule |
dj_queue.delete_recurringtask |
Unschedule dynamic recurring tasks from RecurringTask changelist or detail views. |
Dashboard overview
Queue drill-down
python manage.py dj_queue_health [--backend BACKEND] [--max-age SECONDS] \
[--deep] [--require-version VERSION]Prints healthy to stdout and exits 0 when at least one Process row for --backend has last_heartbeat_at within --max-age (and all enabled deep checks pass); otherwise writes failure reasons to stderr and exits 1.
| Flag | Type | Default | Description |
|---|---|---|---|
--backend |
str |
"default" |
TASKS backend alias to check. |
--max-age |
int | None |
process_alive_threshold |
Maximum heartbeat age in seconds (>= 1) for a process row to count as live. |
--deep |
bool |
False |
Also verify live process rollout_protocol, invalid job states or concurrency policies, denormalized backend/queue mismatches, claimed rows with missing or stale processes, incomplete or mismatched recurring reservations, impossible semaphore counts, failed jobs exceeding clear_failed_jobs_after, and PostgreSQL dead-tuple bloat (>= 10,000 dead tuples and >= 20% dead-tuple ratio), reporting any sessions or slots pinning xmin when dead-tuple bloat is found. |
--require-version |
str | None |
None |
Implies --deep and additionally requires every live queue process to report dj_queue_version == VERSION. |
Mount dj_queue.urls to expose JSON statistics and Prometheus metrics for all dj_queue-managed backend aliases:
from django.urls import include, path
urlpatterns = [
# ...
path("dj_queue/", include("dj_queue.urls")),
]GET /dj_queue/stats.json: returns{"backends": [...]}with per-backend queue counts (ready_count,scheduled_count,claimed_count,blocked_count,failed_count), queue pause/latency/worker counts, runner liveness counts, recurring tasks, semaphores, failed-job retention metrics, and (on PostgreSQL) advisory read-onlypg_stat_user_tablestuple/vacuum stats pluspg_stat_activity,pg_replication_slots, andpg_prepared_xactsxmin-blocker diagnostics (omitting usernames and client addresses).GET /dj_queue/metrics: returns Prometheus text exposition (0.0.4) whendj-queue[prometheus]is installed, or HTTP503with an installation hint whenprometheus_clientis absent.- Authentication: set
DJ_QUEUE_OBSERVABILITY_TOKENinsettings.pyto requireAuthorization: Bearer <token>on both endpoints (returns401withWWW-Authenticate: Bearerotherwise). Leave it unset or""when access is restricted at the network or reverse-proxy layer.
| Metric | Type | Labels | Description |
|---|---|---|---|
dj_queue_queue_jobs |
gauge |
backend, queue, state |
Job count per queue and state (ready, scheduled, claimed, blocked, failed). |
dj_queue_queue_paused |
gauge |
backend, queue |
1 when the queue is paused, 0 otherwise. |
dj_queue_queue_latency_seconds |
gauge |
backend, queue |
Age in seconds of the oldest ready job; omitted when the queue is paused or has no ready jobs. |
dj_queue_queue_live_workers |
gauge |
backend, queue |
Live Worker processes whose selectors match the queue. |
dj_queue_runner_processes |
gauge |
backend, status |
Process row count (all kinds) by liveness (live, stale). |
dj_queue_runner_processes_by_kind |
gauge |
backend, kind, status |
Process row count by kind (Supervisor, Worker, Dispatcher, Scheduler) and liveness (live, stale); kinds with no rows emit no samples. |
dj_queue_recurring_tasks |
gauge |
backend |
Configured RecurringTask row count. |
dj_queue_semaphores |
gauge |
queue_database |
Semaphore row count deduplicated by queue database_alias. |
dj_queue_process_rows |
gauge |
backend |
Total Process row count (live plus stale). |
dj_queue_failed_jobs |
gauge |
backend |
Current FailedExecution row count. |
dj_queue_failed_job_oldest_age_seconds |
gauge |
backend |
Age in seconds of the oldest failed job; omitted when failed count is 0. |
dj_queue_failed_job_retention_seconds |
gauge |
backend |
Configured clear_failed_jobs_after seconds; omitted when None. |
dj_queue_failed_jobs_over_retention |
gauge |
backend |
Failed jobs older than clear_failed_jobs_after; omitted when None. |
dj_queue logs structured records on the "dj_queue" logger without modifying django.db.backends. Domain events attach record.event and record.dj_queue ({"backend_alias": ..., ...}); runtime infrastructure errors attach flat event, backend_alias, thread_error_context, and thread_error_type extras:
| Event | Level | Fields | Description |
|---|---|---|---|
job.enqueued |
INFO |
backend_alias, job_id, task_path, queue_name, priority |
Single job enqueued. |
jobs.enqueued |
INFO |
backend_alias, job_count, ready_count, scheduled_count, blocked_count, discarded_count |
Bulk batch enqueued via enqueue_all(). |
job.claimed |
INFO |
backend_alias, job_id, queue_name, priority |
Ready job claimed by a worker. |
job.executed |
INFO |
backend_alias, job_id, status |
Claimed job completed (status="success"). |
job.failed |
INFO |
backend_alias, job_id, exception_class, message, optional failure_kind |
Job marked failed from task import/execution/serialization, dispatch policy, or process recovery. |
job.retried |
INFO |
backend_alias, job_id, queue_name, priority |
Failed job redispatched immediately or via due retry_at. |
job.discarded |
INFO |
backend_alias, job_id, reason |
Job discarded from ready, scheduled, blocked, or failed state. |
job.dispatched_now |
INFO |
backend_alias, job_id, queue_name, priority, dispatched_as |
Scheduled job dispatched immediately from admin. |
job.unblocked |
INFO |
backend_alias, job_id, concurrency_key |
Blocked job promoted to ready. |
queue.paused |
INFO |
backend_alias, queue_name |
Queue paused. |
queue.resumed |
INFO |
backend_alias, queue_name |
Queue resumed. |
process.pruned |
INFO |
backend_alias, process_name, pid |
Stale process row pruned by supervisor housekeeping. |
process.replaced |
INFO |
backend_alias, process_name, kind, old_pid, new_pid, exit_status |
Exited runner replaced by supervisor (exit_status=None in async mode). |
notify.restored |
INFO |
backend_alias |
PostgreSQL LISTEN connection restored after a disconnect. |
connection_budget.warning |
WARNING |
backend_alias, database_alias, estimated_worker_connections, available_database_connections, max_connections, reserved_connections, threshold |
Persistent connection estimate reached 75% of PostgreSQL capacity at startup. |
psycopg_pure_python.warning |
WARNING |
backend_alias, database_alias, psycopg_implementation, install |
PostgreSQL runs on pure-Python psycopg (install="psycopg[binary] or psycopg[c]"). |
dj_queue.thread_error |
ERROR |
backend_alias, thread_error_context, thread_error_type |
Background runtime error when on_thread_error is unset or raised. |
dj_queue.thread_error_callback_failed |
ERROR |
backend_alias, thread_error_context, on_thread_error, thread_error_type |
Configured on_thread_error callback raised an exception. |
Set OPTIONS["on_thread_error"] to an importable dotted callable def report(exc: Exception) -> None to forward infrastructure exceptions (hook errors, heartbeat/liveness errors, LISTEN/NOTIFY errors, and runner crashes) to an error tracker. Task exceptions are recorded as failed jobs and are not sent to on_thread_error.
Configuration for a backend_alias resolves from highest to lowest precedence:
- CLI flags (
--mode,--only-work,--only-dispatch,--skip-recurring, and--configfor selecting the TOML path) - Environment variables (
DJ_QUEUE_MODE,DJ_QUEUE_SKIP_RECURRING, andDJ_QUEUE_CONFIGwhen--configis omitted) - TOML options file (
--configorDJ_QUEUE_CONFIG) TASKS[backend_alias]["OPTIONS"]in Django settings- Built-in defaults
All option mappings must be JSON-serializable. Unknown keys in OPTIONS or in the TOML overlay—including nested workers, dispatchers, scheduler, and recurring entries—raise ImproperlyConfigured naming the section and key (such as workers[0] contains unknown options: 'thread'); invalid values name the full path (such as workers[0].threads). Boolean options accept bool, 0/1, or case-insensitive "1", "true", "yes", "on", "0", "false", "no", "off"; numeric options reject bool. All sections are validated before --only-work, --only-dispatch, or --skip-recurring prunes the active topology, and config loading fails if no worker, dispatcher, or scheduler workload remains.
Configured under TASKS[backend_alias]["OPTIONS"]:
| Name | Type | Default | Description |
|---|---|---|---|
mode |
str |
"fork" |
Standalone supervisor mode: "fork" or "async". |
workers |
Sequence[Mapping] | Mapping |
[{"queues": "*", "threads": 3, "processes": 1, "polling_interval": 0.1, "prefetch_multiplier": 2}] |
Worker definitions (see Worker options). A single mapping is normalized to a one-item list. |
dispatchers |
Sequence[Mapping] | Mapping |
[{"batch_size": 500, "polling_interval": 1, "concurrency_maintenance": True, "concurrency_maintenance_interval": 600}] |
Dispatcher definitions (see Dispatcher options). A single mapping is normalized to a one-item list. |
scheduler |
Mapping | None |
{"dynamic_tasks_enabled": False, "polling_interval": 5} |
Scheduler settings (see Scheduler options). None uses the defaults. |
recurring |
Mapping[str, Mapping] | None |
{} |
Static recurring task definitions keyed by task key (see Static recurring task options). |
database_alias |
str |
"default" |
DATABASES key storing queue and runtime tables for this backend alias. |
preserve_finished_jobs |
bool |
True |
Retain finished Job rows for result lookup and retention cleanup; False deletes Job rows on completion. |
clear_finished_jobs_after |
int | None |
86400 |
Seconds (>= 0) before scheduler cleanup deletes finished jobs (when preserve_finished_jobs=True); None disables cleanup. |
clear_failed_jobs_after |
int | None |
None |
Seconds (>= 0) before scheduler cleanup deletes failed jobs; None disables cleanup. |
clear_recurring_executions_after |
int | None |
None |
Seconds (>= 0) before scheduler cleanup deletes dj_queue_recurring_executions rows; None disables cleanup. |
default_concurrency_duration |
int |
180 |
Default semaphore TTL in seconds (>= 1) when @concurrency omits duration. |
use_skip_locked |
bool |
True |
Enable FOR UPDATE SKIP LOCKED where the database supports it. |
listen_notify |
bool |
True |
Enable PostgreSQL LISTEN/NOTIFY worker wakeups on top of polling; the LISTEN connection uses the connection parameters of database_alias. |
async_thread_sensitive |
bool |
False |
Serialize aenqueue(), aenqueue_all(), and aget_result() in asgiref's thread-sensitive lane. |
async_close_connections |
bool |
False |
Close all Django connections after each async backend call instead of only obsolete connections. |
process_heartbeat_interval |
float |
60 |
Seconds (>= 0) between runner heartbeat writes (0 falls back to min(process_alive_threshold / 2, 60)). |
process_alive_threshold |
float |
300 |
Seconds (> 0) before a Process row is considered stale. |
shutdown_timeout |
float |
5 |
Seconds (>= 0) shared across runners to drain in-flight work on graceful stop. |
supervisor_pidfile |
str | None |
None |
Lock-backed pidfile path acquired by standalone supervisors (standalone=True). |
on_thread_error |
str | None |
None |
Importable dotted callable (exc) -> None for runtime infrastructure exceptions ("" treated as None). |
Each mapping in OPTIONS["workers"]:
| Name | Type | Default | Description |
|---|---|---|---|
queues |
str | Sequence[str] |
"*" |
Queue selector or ordered list of selectors ("*", exact name, or "prefix*"). |
threads |
int |
3 |
Thread-pool size per worker runner (>= 1). |
processes |
int |
1 |
OS processes launched for this entry in fork mode (>= 1); normalized to 1 in async mode. |
polling_interval |
float |
0.1 |
Seconds (> 0) between ready-queue polls. |
prefetch_multiplier |
int |
2 |
Upper bound (threads * prefetch_multiplier, >= 1) on in-flight plus queued claimed jobs; workers poll only while at least one thread is idle. |
Each mapping in OPTIONS["dispatchers"]:
| Name | Type | Default | Description |
|---|---|---|---|
batch_size |
int |
500 |
Maximum rows (>= 1) per scheduled promotion, due failed-retry promotion, expired semaphore cleanup, or expired blocked-job pass. |
polling_interval |
float |
1 |
Seconds (> 0) between dispatcher polls. |
concurrency_maintenance |
bool |
True |
Run expired semaphore cleanup and expired blocked-job promotion. |
concurrency_maintenance_interval |
float |
600 |
Seconds (>= 0) between concurrency maintenance passes; 0 runs on every poll. |
Configured under OPTIONS["scheduler"]:
| Name | Type | Default | Description |
|---|---|---|---|
dynamic_tasks_enabled |
bool |
False |
Poll dynamic recurring tasks (static=False) from dj_queue_recurring_tasks. |
polling_interval |
float |
5 |
Seconds (> 0) between scheduler polls. |
Each <key> mapping under OPTIONS["recurring"]:
| Name | Type | Default | Description |
|---|---|---|---|
task_path |
str |
required | Non-empty dotted import path referencing a Django Task. |
schedule |
str |
required | Valid Fugit-style cron or cronish schedule expression. |
args |
Sequence[Any] |
[] |
Positional arguments passed when enqueueing. |
kwargs |
Mapping[str, Any] |
{} |
Keyword arguments passed when enqueueing. |
queue_name |
str |
"default" |
Destination queue; must be in TASKS[backend_alias]["QUEUES"] when that allow-list is non-empty. |
priority |
int |
0 |
Job priority (-100 to 100). |
description |
str |
"" |
Operator-facing description. |
Pass --config /etc/dj_queue.toml or set DJ_QUEUE_CONFIG=/etc/dj_queue.toml to overlay OPTIONS without editing settings.py. The overlay replaces top-level options. A TOML workers, dispatchers, scheduler, or recurring value replaces the settings value entirely, and keys omitted inside it use built-in defaults. A file with backends may contain no other top-level keys; an alias missing from it gets no overlay. TOML has no null, so omit a top-level key to keep its settings value.
Flat overlay:
mode = "async"
database_alias = "queue"
[[workers]]
queues = ["default", "email*"]
threads = 8Per-backend overlay:
[backends.default]
mode = "fork"
database_alias = "queue"
clear_finished_jobs_after = 86400
[[backends.default.workers]]
queues = ["critical", "default", "email*"]
threads = 8
processes = 2
polling_interval = 0.1
prefetch_multiplier = 2
[[backends.default.dispatchers]]
batch_size = 500
polling_interval = 1
[backends.default.scheduler]
dynamic_tasks_enabled = true
polling_interval = 5
[backends.default.recurring.nightly_cleanup]
task_path = "myapp.tasks.cleanup"
schedule = "0 3 * * *"
queue_name = "maintenance"
priority = -5Environment variables read by dj_queue (see Precedence and validation):
| Name | Type | Default | Description |
|---|---|---|---|
DJ_QUEUE_CONFIG |
str |
unset | Path to a TOML options overlay file. |
DJ_QUEUE_MODE |
str |
unset | Standalone mode override: "fork" or "async". |
DJ_QUEUE_SKIP_RECURRING |
bool |
unset | Boolean string (1/true/yes/on or 0/false/no/off) suppressing the scheduler. |
Django settings.py keys used by dj_queue:
| Name | Type | Default | Description |
|---|---|---|---|
TASKS |
Mapping[str, Mapping] |
Django's ImmediateBackend "default" alias |
Django task backend definitions (BACKEND, QUEUES, OPTIONS); dj_queue needs at least one alias whose BACKEND is "dj_queue.backend.DjQueueBackend". |
DATABASE_ROUTERS |
Sequence[str] |
[] |
Include "dj_queue.routers.DjQueueRouter" for migration gating and single-queue-database ORM routing. |
DJ_QUEUE_OBSERVABILITY_TOKEN |
str | None |
None |
Optional bearer token protecting /dj_queue/stats.json and /dj_queue/metrics. |
Isolate queue tables and connection pools on a dedicated DATABASES alias via OPTIONS["database_alias"]:
DATABASES = {
"default": {"ENGINE": "django.db.backends.postgresql", "NAME": "app"},
"queue": {"ENGINE": "django.db.backends.postgresql", "NAME": "queue"},
}
DATABASE_ROUTERS = ["dj_queue.routers.DjQueueRouter"]
TASKS = {
"default": {
"BACKEND": "dj_queue.backend.DjQueueBackend",
"QUEUES": [],
"OPTIONS": {"database_alias": "queue"},
},
}Apply dj_queue migrations to queue and application migrations to default:
python manage.py migrate dj_queue --database queue
python manage.py migrateDjQueueRouter permits dj_queue migrations only on database aliases referenced by configured dj_queue backends (and on none when no dj_queue backend is configured). When all dj_queue backends share one database_alias, the router directs dj_queue ORM reads and writes to it; when multiple dj_queue backends point at different database_alias targets, migrate each target with --database <alias>—the router leaves dj_queue ORM calls unrouted while dj_queue operations pass using= explicitly.
dj_queue can share settings.TASKS with other django.tasks backends:
- when
TASKS = {},dj_queueexposes one implicit"default"backend alias with built-in defaults (leavingTASKSunset selects Django'sImmediateBackend, whichdj_queueignores) - when
TASKSis non-empty,dj_queuemanages only aliases whose"BACKEND"equals"dj_queue.backend.DjQueueBackend"; aliases pointing at another backend class are ignored bymanage.py dj_queue*, admin views,/dj_queue/stats.json, and/dj_queue/metrics, and targeting a non-dj_queueor missing-BACKENDalias raisesImproperlyConfigured
For most deployments, start with a standalone dj_queue process; reach for a dedicated queue database before embedded mode.
dj_queue can run an embedded AsyncSupervisor(standalone=False) inside an ASGI or Gunicorn server process, leaving OS signal handling to the host server. Startup runs synchronously so configuration errors surface immediately, and supervisor.run() executes on a daemon thread that enforces shutdown_timeout when the server exits. If the supervisor loop terminates, its runners deregister and it is not restarted inside that worker process.
Wrap your ASGI callable with DjQueueLifespan(app, *, backend_alias="default", forward_wrapped_lifespan=True):
from django.core.asgi import get_asgi_application
from dj_queue.contrib.asgi import DjQueueLifespan
django_application = get_asgi_application()
application = DjQueueLifespan(django_application, forward_wrapped_lifespan=False)Pass forward_wrapped_lifespan=False when the wrapped application does not implement the ASGI lifespan protocol (such as Django's plain get_asgi_application()); keep the default True when wrapping middleware or frameworks with their own lifespan hooks.
Import the worker lifecycle hooks in gunicorn.conf.py:
from dj_queue.contrib.gunicorn import post_fork, worker_exitAcross multiple Gunicorn workers in the same working directory and backend alias, post_fork coordinates via a non-blocking file lock in the system temp directory so only one Gunicorn worker runs the embedded supervisor at a time; sibling workers retry acquiring the lock every 1.0s and take over automatically when the active worker recycles or exits. Set dj_queue_backend_alias = "<alias>" on the Gunicorn server or worker object to target a non-default backend alias.
| Backend | Support level | Capabilities |
|---|---|---|
| PostgreSQL | First-class production backend | Polling, FOR UPDATE SKIP LOCKED, optional LISTEN/NOTIFY wakeups, and advisory table/xmin diagnostics. |
| MySQL | Supported | Polling and FOR UPDATE SKIP LOCKED. |
| MariaDB | Supported | Polling and FOR UPDATE SKIP LOCKED. |
| SQLite | Supported with limits | Polling only with serialized writes (no SKIP LOCKED, no LISTEN/NOTIFY); suited for local development, CI, and low-write single-host deployments. |
Minimum supported database versions follow the installed Django release:
| Django | PostgreSQL | MySQL | MariaDB | SQLite |
|---|---|---|---|---|
6.0 |
14+ |
8.0.11+ |
10.6+ |
3.31+ |
6.1 |
15+ |
8.4+ |
10.11+ |
3.37+ |
CI runs the Django 6.0 floor versions alongside PostgreSQL 18, MySQL 9.7 and 26.10, and MariaDB 12.3 and 13.0; Django 6.1 only runs with database versions it supports. Other Django database vendors raise ImproperlyConfigured during capability detection.
dj_queue supports live queue reads and writes across a one-generation N−1 -> N rolling deployment through an expand, bridge, activate, and contract migration sequence:
- Expand and bridge: release N adds nullable columns and writes both representations while keeping the old representation authoritative for N−1 writers. Expansion migrations keep one DDL operation per hot-table migration so lock conflicts can retry safely; constraints that a supported database cannot add online stay enforced in operations and
--deephealth until a later release. - Activate and contract: release N+1 switches authority after N−1 has left the fleet, and a later contract release drops retired compatibility columns or adds
NOT NULLconstraints once its minimum compatible N−1 release no longer references them. - Version
0.19.0compatibility: the minimum rollout-compatible previous release for0.19.0is0.18.0.0.17.xalso publishesrollout_protocol = 1, so protocol checks alone cannot detect it. The minimum N−1 patch must already retry transient database conflicts, because release N cannot change the behavior of a process still running N−1. Upgrading directly from0.17.xor earlier with live writers, running N−2 processes, or skipping bridge releases is unsupported.
Recommended deployment sequence:
- Ensure every running application producer and queue process is on at least the minimum rollout-compatible N−1 release (
0.18.0before0.19.0). - Verify queue health and the exact N−1 version with
python manage.py dj_queue_health --require-version 0.18.0, and confirm in your deployment platform that no older application producer remains (application producers do not registerdj_queue_processesrows). - Run
python manage.py migrate(ormigrate dj_queue --database <alias>) once for release N. - Roll application producers and
dj_queuesupervisors from N−1 to N; new supervisors automatically wait up to--migration-wait-timeoutseconds if started while step 3 is finishing. - Verify completion of the queue fleet rollout with
python manage.py dj_queue_health --require-version 0.19.0and confirm in your deployment platform that no N−1 producer remains.
If you roll application and queue code back from N (0.19.0) to its compatible N−1 (0.18.0), keep release N's applied database migrations in place; do not reverse migrations during a live rollback. Because persisted jobs can outlive a deployment, keep application task import paths and argument signatures backward-compatible across adjacent releases, introducing a new task function when a breaking signature change is required.
Maintainers verify N−1 -> N upgrades against isolated wheels and live background writers with bin/prerelease.py (backed by the Mixed-version upgrade check workflow on PostgreSQL 14, MySQL 8.0, and MariaDB 10.6):
bin/prerelease.py \
--from-ref v0.18.0 \
--to-ref v0.19.0 \
--backend postgres \
--django '>=6.0,<6.1'bin/prerelease.py creates an isolated database, validates immediate, scheduled, concurrency-limited, bulk, recurring, and fail/retry workloads across pre-migration, live-DDL, mixed-version, and N-only phases, drops the database on success unless --keep-database is passed, and writes wheels, manifests, and logs under benchmark-results/. It requires one shared rollout protocol and the same Django version; it is a correctness check, not a capacity or latency benchmark, and backend-specific DDL changes may need extra database versions.
Production guidance for high-throughput PostgreSQL deployments:
- Dedicated queue database and short transactions: place queue tables on a separate
database_aliasand keep transactions short across every connection touching that database;idle in transactionsessions, stale replication slots, and prepared transactions pinxminand block dead-tuple cleanup. - Connection reuse and capacity sizing: setting a positive
CONN_MAX_AGEon the queue database connection avoids per-poll and per-job connection churn. WhenCONN_MAX_AGEis positive orNone, each supervisor estimates its persistent connection footprint as2 * runners + worker_threads + listeners + 2(runners = worker_processes + dispatchers + (1 if scheduler else 0),listeners = worker_processeswhenlisten_notify=True) and logsconnection_budget.warningat startup if that estimate reaches75%ofmax_connections - reserved_connections. Sizemax_connectionsor your connection pooler across all web and queue hosts accordingly. - Retention vs deletion cost: bound table size with finite
clear_finished_jobs_after,clear_failed_jobs_after, andclear_recurring_executions_afterwindows or scheduleddj_queue_pruneruns. Settingpreserve_finished_jobs = Falseavoids storing finished jobs, but on PostgreSQL an unkeyed completion without retention runs3SQL statements inside aBEGIN...COMMITtransaction (5round trips, andJobpre_delete/post_deletesignals do not fire) versus1self-contained statement with no transaction block whenpreserve_finished_jobs = True. - Driver and optimizations: keep
use_skip_locked = Trueandlisten_notify = Truewithpsycopg[binary]orpsycopg[c]installed. - Observability before schema complexity: monitor dead tuples, autovacuum frequency, and long-running queries before reaching for partitioning or bulk-ingest paths.
- Autovacuum tuning: default PostgreSQL autovacuum thresholds (
20%scale factor) are too high for queue churn. Print reviewableALTER TABLEstatements (autovacuum_vacuum_scale_factor = 0.01,autovacuum_vacuum_threshold = 50,autovacuum_analyze_scale_factor = 0.02,autovacuum_analyze_threshold = 50) fordj_queue_jobs, the fivedj_queue_*_executionstables, anddj_queue_recurring_executions(exits withCommandErroron non-PostgreSQL databases;dj_queuenever applies storage parameters in migrations):
python manage.py dj_queue_postgres_autovacuum [--backend BACKEND]| Flag | Type | Default | Description |
|---|---|---|---|
--backend |
str |
"default" |
TASKS backend alias whose PostgreSQL queue tables should be emitted. |
Self-contained runnable scripts live in examples/ (SQLite by default; PostgreSQL, MySQL, or MariaDB via DB_BACKEND):
- examples/ex01_basic_enqueue.py — enqueue a task and inspect the persisted
Job - examples/ex07_basic_enqueue_on_commit.py — defer enqueue until
transaction.atomic()commits - examples/ex08_basic_recurring.py — static recurring tasks via
OPTIONS["recurring"] - examples/ex20_advanced_concurrency.py —
@concurrencyslot limits, blocking, andon_conflict="discard" - examples/ex21_advanced_queue_control.py —
QueueInfopause, resume, clear, size, and latency - examples/ex24_advanced_multi_db.py —
database_aliasandDjQueueRouteracross two databases - examples/ex25_advanced_asgi.py — embedded
DjQueueLifespanASGI startup and shutdown
See examples/README.md for the full 16-example progression (ex01–ex27, including live Uvicorn and Gunicorn examples and the bin/admin.py demo dashboard).
MIT — see LICENSE.

