Add persistent pause/resume queue

[art321ur]

Why

When we are using procrastinate, we found that sometimes a queue needs to stop temporarily (to allow other non-procrastinate services do some maintenance or allow them to use limited resources that procrastinate queue would consume).
Today the only options are to stop the worker (which also stops every other queue) or to let the jobs run and fail. This PR adds a way to pause one queue and leave the rest of the queues working.

I have tried to work around it (in simplified setup) in our codebase but the only way to solve this was to have some hacky fetch_job interception.

class PauseAwareConnector(PsycopgConnector):  
    async def execute_query_one_async(self, query: LiteralString, **arguments: Any) -> dict[str, Any]:
        if query == sql.queries["fetch_job"] and arguments.get("queues") is not None:
            arguments = {**arguments, "queues": await self._unpaused(arguments["queues"])}
        return await super().execute_query_one_async(query, **arguments)

    async def _unpaused(self, queues: Sequence[str]) -> list[str]:
        rows = await super().execute_query_all_async(_SELECT_PAUSED_QUEUES)
        paused = {row["queue_name"] for row in rows}
        return [queue for queue in queues if queue not in paused]

How

• the pause lives in the database, so it is shared by all workers and survives
  a worker restart
• procrastinate_fetch_job_v3 is a copy of _v2 with one extra predicate that skips jobs whose queue has a
  row
• jobs already running are not interrupted
• each pause is held under a pause key ("default" if not given). A queue stays paused until every key is released, so two independent processes (for example a deploy script and a maintenance task) can pause the same queue without one resuming the other by mistake
• cleanup of dead pauses is assumed to be DIY - but the paused_at column could be used for that

Migrations

• 03.10.00_01_pre_add_queue_pause.sql
• 03.10.00_50_post_add_queue_pause.sql

Successful PR Checklist:

• Tests
  • Unit, integration and acceptance tests
• Documentation
  • New how-to: docs/howto/advanced/pause_queue.md.

PR label(s):

• PR type: breaking 💥 Contains breaking changes
• PR type: feature ⭐️ Contains new features
• PR type: bugfix 🕵️ Contains bug fix
• PR type: miscellaneous 👾 Contains misc changes
• PR type: dependencies 🤖 Contains only dependencies updates
• PR type: documentation 📚 Contains documentation updates

Summary by CodeRabbit

• New Features

  • Queues can now be paused and resumed through synchronous and asynchronous APIs.
  • Paused queues stop receiving new jobs while running jobs finish normally.
  • Supports multiple pause keys, resuming individual keys or clearing all keys.
  • Added APIs to list paused queues with optional filters.
  • Workers automatically resume fetching jobs when a queue is resumed.

• Documentation

  • Added comprehensive queue pause and resume guidance.
  • Updated Django model documentation for paused queue records.

[coderabbitai]

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro

Run ID: 8137c5fb-273d-4166-846c-99acdff354c5

📥 Commits

Reviewing files that changed from the base of the PR and between 7feebed and d1c94e2.

📒 Files selected for processing (20)

• docs/howto/advanced.md
• docs/howto/advanced/pause_queue.md
• docs/howto/django/models.md
• procrastinate/contrib/django/migrations/0042_pre_add_queue_pause.py
• procrastinate/contrib/django/migrations/0043_post_add_queue_pause.py
• procrastinate/contrib/django/migrations/0044_add_paused_queue_model.py
• procrastinate/contrib/django/models.py
• procrastinate/jobs.py
• procrastinate/manager.py
• procrastinate/sql/migrations/03.10.00_01_pre_add_queue_pause.sql
• procrastinate/sql/migrations/03.10.00_50_post_add_queue_pause.sql
• procrastinate/sql/queries.sql
• procrastinate/sql/schema.sql
• procrastinate/testing.py
• procrastinate/worker.py
• tests/acceptance/test_async.py
• tests/integration/contrib/django/test_models.py
• tests/integration/test_manager.py
• tests/unit/test_manager.py
• tests/unit/test_worker.py

---

📝 Walkthrough

Walkthrough

Changes

Queue pause and resume

Layer / File(s)	Summary
Pause persistence and fetch filtering procrastinate/sql/..., procrastinate/contrib/django/..., procrastinate/contrib/django/migrations/..., docs/howto/django/models.md, tests/integration/contrib/django/test_models.py	Adds keyed pause storage, pause-aware job selection, resume triggers, Django migrations, and a read-only ProcrastinatePausedQueue model.
Manager and connector pause APIs procrastinate/manager.py, procrastinate/testing.py, procrastinate/sql/queries.sql, docs/howto/advanced.md, docs/howto/advanced/pause_queue.md, tests/unit/test_manager.py, tests/integration/test_manager.py	Adds synchronous and asynchronous pause, resume, and listing APIs with pause-key handling, filtering, and in-memory support.
Resume notifications and worker wake-up procrastinate/jobs.py, procrastinate/worker.py, tests/acceptance/test_async.py, tests/unit/test_worker.py	Adds queue_resumed notifications and causes workers to fetch jobs after a paused queue resumes.

Estimated code review effort: 4 (Complex) | ~45 minutes

Sequence Diagram(s)

sequenceDiagram
  participant JobManager
  participant QueuePauseStore
  participant QueueResumedTrigger
  participant Worker
  JobManager->>QueuePauseStore: Resume queue and delete pause key
  QueuePauseStore->>QueueResumedTrigger: Emit queue_resumed event
  QueueResumedTrigger->>Worker: Notify queue resumed
  Worker->>JobManager: Fetch newly available job

Loading

Suggested reviewers: medihack

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name	Status	Explanation	Resolution
Docstring Coverage	⚠️ Warning	Docstring coverage is 13.24% which is insufficient. The required threshold is 80.00%.	Write docstrings for the functions missing them to satisfy the coverage threshold.

✅ Passed checks (4 passed)

Check name	Status	Explanation
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title clearly summarizes the main change: adding persistent queue pause/resume support.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[art321ur]

Let me know if I should change or improve anything - PR description, docs etc- first time trying to contribute to OSS ;)