Skip to content
Merged
Show file tree
Hide file tree
Changes from 1 commit
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions .gitignore
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
.idea
247 changes: 106 additions & 141 deletions README.md
Original file line number Diff line number Diff line change
@@ -1,141 +1,106 @@
<!--
Use this template to structure your project's README.md file.
Not all sections are required, though we recommend including as many as possible.

* Project name (required)
* Overview (required)
* What's new (optional)
* Prerequisites (optional)
* Installation (required)
* Getting Started (optional)
* Running (optional)
* Examples (optional)
* Troubleshooting (optional)
* Contributing (required)
* License (required)
* Third-Party Dependencies (required)
* Additional Resources (required)

Consider using a table of contents to make lengthy and complex README files easier to navigate.
-->

# PROJECT_NAME
<!--
Replace `PROJECT_NAME` with your project's name.
Your project's name is the first thing people will see when browsing your project.
-->

## Overview
<!--
Include a brief project description, written from the perspective of the value your project provides users.
Be sure to define terms; links to resources where necessary and appropriate.
A good overview is clear, short, and to the point.
-->

### What's New
<!--
If applicable to your project, list new features you want users to be aware of.
This section might supplement the Changelog file from the repository and only highlight important changes.
-->

### Prerequisites
<!--
Provide guidelines on any prerequisites that may be useful in configuring the user's environment.
Prerequisites typically take the form of a list of required software that must be available.
Each piece of software might require its own setup steps.
Use lists and subtopics as appropriate.
-->

## Installation
<!--
Provide step-by-step instructions for installing your software project.
Use subtopics and screenshots as appropriate.
-->

### Getting Started
<!--
Provide users with initial steps for getting started using your project after they have installed it.
This is a good place to include screenshots, animated GIFs, or short example videos.
-->

### Running
<!--
Provide users with steps for running your project after they have installed it.
This is a good place to include screenshots, recordings, or short usage videos.
-->

### Examples
<!--
Provide additional examples of using the software, or point to further documentation.
Make learning and using your project as easy as possible!
-->

### Troubleshooting
<!--
Provide workarounds and solutions to known problems.
Organize troubleshooting information using subtopics, as appropriate.
-->

## Contributing
<!--
Specify whether your project accepts contributions.
Language you use in this section should mirror the language in your CONTRIBUTING.md file.
Use the default text below if you accept contributions.
If you do not accept contributions, note that here.
-->

<!-- Use this text if your project is not accepting contributions.-->
Maintainers are not currently accepting patches and contributions to this project.

<!--Use this text if your project is accepting contributions.-->
Maintainers are accepting patches and contributions to this project.
Please read [CONTRIBUTING.md](CONTRIBUTING.md) for details about submitting contributions to this project.

## License
<!--
Choose from the default text already in place below.
Do not alter the text without prior approval from SAS Legal and the Open Source Program Office.
-->

This project is licensed under the [Apache 2.0 License](LICENSE).

<!--
Substitute the text below if your project contains trademarked elements not intended for license under the open source license.
In this case, add all trademarked images to the same folder and replace `XXX` in the statement with the name of that folder.
-->
Except for the the contents of the `XXX` folder, this project is licensed under the [Apache 2.0 License](LICENSE). Elements in the `XXX` folder are owned by SAS and are not released under an open source license. SAS and all other SAS Institute Inc. product or service names are registered trademarks or trademarks of SAS Institute Inc. in the USA and other countries. ® indicates USA registration.

<!--
If your projects ships with/as a container image for a package registry, keep the following statement in place.
Otherwise, remove it.
-->
As with any container image, direct and indirect dependencies are governed by their own licenses.
Users of the published container image are responsible for ensuring that their use complies with all applicable licenses.

## Third-Party Dependencies
<!--
List all third-party dependencies required for using your project, then list and link to each dependency's license.
Use the table to organize your dependency list.
Do not include version numbers.
Consult (and then remove) the provided example.
-->
This project requires the following dependencies.

| Dependency | License |
| ---------- | ------- |
| `github.com/sassoftware/dpmm` | [Apache 2.0](https://github.com/sassoftware/dpmm/blob/main/LICENSE) |

## Additional Resources
<!--
Include any additional materials users may need or find useful when using your software. Additional resources might include:

* Documentation links
* Research papers
* Blog posts
* Articles from SAS Communities
* Other relevant documentation (i.e., from Tech Support, Education, etc.)
-->

* `Resource link`
* `Resource link`
* `Resource link`
# Argus

![Argus watchdog for RabbitMQ](logo.png)

Argus is a self-healing watchdog for RabbitMQ **quorum queues**. It monitors the broker's internal log exchange for Raft/quorum error signatures, evaluates whether a flagged queue is truly unrecoverable, and — when it is — automatically backs up the on-disk data, deletes the corrupted queue, carves recoverable messages from the backup files, and republishes them to the original queue.

## How it works

```text
amq.rabbitmq.log (warning/error)
│
▼
Log Monitor ──► pre-filter (ra/quorum/raft keywords)
│
▼
Queue name match against tracked quorum queues
│
▼
Health Evaluator
├─ GREEN → no action
├─ TRANSIENT_DOWN → no action (resource alarm, node restart, cluster-wide outage)
└─ UNRECOVERABLE → Recovery Pipeline
│
├─ 1. Locate Raft data directory on disk
├─ 2. Backup .segment / .wal files
├─ 3. Delete corrupted queue from broker
├─ 4. Carve AMQP payloads from backup (ETF byte-scan)
└─ 5. Republish to default exchange → original queue
```

### Health classification

| State | Condition | Action |
| --- | --- | --- |
| `GREEN` | Queue has a leader and is running | None |
| `TRANSIENT_DOWN` | Memory/disk alarm, host node down, or all neighbors also down | Wait — do not delete |
| `UNRECOVERABLE` | Management API returns 500 for this queue, **or** no Raft leader while neighbors are healthy | Run recovery pipeline |

## Configuration

All settings are provided via environment variables with sensible defaults for local development.

| Variable | Default | Description |
| --- | --- | --- |
| `ARGUS_AMQP_URL` | `amqp://guest:guest@localhost:5672/` | AMQP broker URL |
| `ARGUS_MGMT_URL` | `http://localhost:15672` | Management API base URL |
| `ARGUS_MGMT_USER` | `guest` | Management API username |
| `ARGUS_MGMT_PASS` | `guest` | Management API password |
| `ARGUS_QUORUM_PATH` | `/var/lib/rabbitmq/mnesia/rabbit@localhost/quorum/rabbit@localhost` | Base path of the quorum queue Raft storage on disk |
| `ARGUS_BACKUP_DIR` | `/var/lib/rabbitmq/argus-backups` | Directory where queue data is backed up before deletion |

## Building

```bash
go build -o argus .
```

## Running

```bash
export ARGUS_AMQP_URL="amqp://admin:secret@rabbitmq:5672/"
export ARGUS_MGMT_URL="http://rabbitmq:15672"
export ARGUS_MGMT_USER="admin"
export ARGUS_MGMT_PASS="secret"
export ARGUS_QUORUM_PATH="/var/lib/rabbitmq/mnesia/rabbit@rabbitmq/quorum/rabbit@rabbitmq"
export ARGUS_BACKUP_DIR="/data/argus-backups"

./argus
```

Argus runs until it receives `SIGINT` or `SIGTERM`.

## Requirements

- RabbitMQ 3.8+ with quorum queues enabled
- The process must run on the **same host** as the RabbitMQ node (for filesystem access to Raft data and `rabbitmqctl` fallback eviction)
- Go 1.21+ to build from source

## Project layout

```text
argus/
├── main.go # Config, entry point, recovery pipeline orchestration
├── monitor.go # Log monitor — subscribes to amq.rabbitmq.log
├── manager.go # DiagnosticsManager — RabbitMQ Management API client
├── health.go # EvaluateQueueHealth — three-gate health classification
├── locate.go # Find a queue's Raft data directory by decoding ETF meta files
├── carve.go # Extract AMQP payloads from .segment / .wal files
├── delete.go # HTTP queue deletion with rabbitmqctl fallback
└── republish.go # Republish recovered payloads via AMQP
```

## Limitations & caveats

- **Best-effort recovery.** The ETF carver scans raw bytes heuristically. Messages that span corrupted sectors may not be recoverable.
- **Single-node access required.** Argus must run on the RabbitMQ node that hosts the quorum queue data. In a multi-node cluster, run one Argus instance per node.
- **AMQP message metadata is not preserved.** Recovered messages are republished as `application/octet-stream` with `DeliveryMode=Persistent`. Original headers, content-type, and routing metadata are not reconstructed.
- **Duplicate delivery is possible.** If a message was already acknowledged before the crash, carving may recover and republish it again.

## Future
Comment thread
dhunt3800 marked this conversation as resolved.
Resolved

- Eliminate the limitation where original headers, content-type and routing metadata are not recovered.
- Ensure only one node in the RabbitMQ cluster attempts recovery of a queue.
- Manual mode to specify recovery of a specific queue.
- Remove orphaned queues.
- Possible classic queue support.
Loading