Skip to content
Merged
Show file tree
Hide file tree
Changes from 5 commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions .gitignore
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
.idea
27 changes: 10 additions & 17 deletions CONTRIBUTING.md
Original file line number Diff line number Diff line change
@@ -1,25 +1,18 @@
# Contributing
<!-- Use this text if your project is not accepting contributions, then delete the remaining file contents.-->
Maintainers are not currently accepting patches and contributions to this project.

<!-- Use this text if your project is accepting contributions. -->
Maintainers are accepting patches and contributions to this project.
Please note the following contribution guidelines, and see the [SAS Open Source Contributor Handbook](https://sassoftware.github.io/contributor-handbook.html) for additional guidance.
Please note the following contribution guidelines.

## Contributor Agreement
<!-- If accepting contributions, this section must remain in place. -->
Contributors to this project must comply with the [Contributor Agreement](ContributorAgreement.txt).
This agreement permits the project maintainers to use and redistribute your contributions as part of the project.
## Contributor License Agreement

Contributions to this project must be accompanied by a signed [Contributor Agreement](ContributorAgreement.txt).
You (or your employer) retain the copyright to your contribution.
This simply permits the project maintainers to use and redistribute your
contributions as part of the project.

## Code Reviews
<!-- Use this text if your project is accepting contributions. -->
All submissions, including submissions by project members, require review.
Maintainers use GitHub pull requests are used for this purpose.
Consult [GitHub Help](https://help.github.com/articles/about-pull-requests/) for more information about using pull requests.

## Update Documentation
<!-- Remove this section if not using the documentation website template. -->
Project documentation is stored in the `website` directory.
This project welcomes pull requests suggesting updates to this documentation.
See the documentation [README](./website/README.md) for details.
All submissions, including submissions by project members, require review.
GitHub pull requests are used for this purpose.
Consult [GitHub Help](https://help.github.com/articles/about-pull-requests/)
for more information about using pull requests.
258 changes: 126 additions & 132 deletions README.md
Original file line number Diff line number Diff line change
@@ -1,141 +1,135 @@
<!--
Use this template to structure your project's README.md file.
Not all sections are required, though we recommend including as many as possible.

* Project name (required)
* Overview (required)
* What's new (optional)
* Prerequisites (optional)
* Installation (required)
* Getting Started (optional)
* Running (optional)
* Examples (optional)
* Troubleshooting (optional)
* Contributing (required)
* License (required)
* Third-Party Dependencies (required)
* Additional Resources (required)

Consider using a table of contents to make lengthy and complex README files easier to navigate.
-->

# PROJECT_NAME
<!--
Replace `PROJECT_NAME` with your project's name.
Your project's name is the first thing people will see when browsing your project.
-->

## Overview
<!--
Include a brief project description, written from the perspective of the value your project provides users.
Be sure to define terms; links to resources where necessary and appropriate.
A good overview is clear, short, and to the point.
-->

### What's New
<!--
If applicable to your project, list new features you want users to be aware of.
This section might supplement the Changelog file from the repository and only highlight important changes.
-->

### Prerequisites
<!--
Provide guidelines on any prerequisites that may be useful in configuring the user's environment.
Prerequisites typically take the form of a list of required software that must be available.
Each piece of software might require its own setup steps.
Use lists and subtopics as appropriate.
-->

## Installation
<!--
Provide step-by-step instructions for installing your software project.
Use subtopics and screenshots as appropriate.
-->

### Getting Started
<!--
Provide users with initial steps for getting started using your project after they have installed it.
This is a good place to include screenshots, animated GIFs, or short example videos.
-->

### Running
<!--
Provide users with steps for running your project after they have installed it.
This is a good place to include screenshots, recordings, or short usage videos.
-->

### Examples
<!--
Provide additional examples of using the software, or point to further documentation.
Make learning and using your project as easy as possible!
-->

### Troubleshooting
<!--
Provide workarounds and solutions to known problems.
Organize troubleshooting information using subtopics, as appropriate.
-->
# Argus

![Argus watchdog for RabbitMQ](logo.png)

Argus is a self-healing watchdog for RabbitMQ **quorum queues**. It monitors the broker's internal log exchange for Raft/quorum error signatures, evaluates whether a flagged queue is truly unrecoverable, and — when it is — automatically backs up the on-disk data, deletes the corrupted queue, carves recoverable messages from the backup files, and republishes them to the original queue.

## How it works

```text
amq.rabbitmq.log (warning/error)
│
▼
Log Monitor ──► pre-filter (ra/quorum/raft keywords)
│
▼
Queue name match against tracked quorum queues
│
▼
Health Evaluator
├─ GREEN → no action
├─ TRANSIENT_DOWN → no action (resource alarm, node restart, cluster-wide outage)
└─ UNRECOVERABLE → Recovery Pipeline
│
├─ 1. Locate Raft data directory on disk
├─ 2. Backup .segment / .wal files
├─ 3. Delete corrupted queue from broker
├─ 4. Carve AMQP payloads from backup (ETF byte-scan)
└─ 5. Republish to default exchange → original queue
```

### Health classification

| State | Condition | Action |
| --- | --- | --- |
| `GREEN` | Queue has a leader and is running | None |
| `TRANSIENT_DOWN` | Memory/disk alarm, host node down, or all neighbors also down | Wait — do not delete |
| `UNRECOVERABLE` | Management API returns 500 for this queue, **or** no Raft leader while neighbors are healthy | Run recovery pipeline |

## Configuration

All settings are provided via environment variables with sensible defaults for local development.

| Variable | Default | Description |
| --- | --- | --- |
| `ARGUS_AMQP_URL` | `amqp://guest:guest@localhost:5672/` | AMQP broker URL |
| `ARGUS_MGMT_URL` | `http://localhost:15672` | Management API base URL |
| `ARGUS_MGMT_USER` | `guest` | Management API username |
| `ARGUS_MGMT_PASS` | `guest` | Management API password |
| `ARGUS_QUORUM_PATH` | `/var/lib/rabbitmq/mnesia/rabbit@localhost/quorum/rabbit@localhost` | Base path of the quorum queue Raft storage on disk |
| `ARGUS_BACKUP_DIR` | `/var/lib/rabbitmq/argus-backups` | Directory where queue data is backed up before deletion |

## Building

```bash
go build -o argus .
```

## Running

```bash
export ARGUS_AMQP_URL="amqp://admin:secret@rabbitmq:5672/"
export ARGUS_MGMT_URL="http://rabbitmq:15672"
export ARGUS_MGMT_USER="admin"
export ARGUS_MGMT_PASS="secret"
export ARGUS_QUORUM_PATH="/var/lib/rabbitmq/mnesia/rabbit@rabbitmq/quorum/rabbit@rabbitmq"
export ARGUS_BACKUP_DIR="/data/argus-backups"

./argus
```

Argus runs until it receives `SIGINT` or `SIGTERM`.

## Requirements

- RabbitMQ 3.8+ with quorum queues enabled
- The process must run on the **same host** as the RabbitMQ node (for filesystem access to Raft data and `rabbitmqctl` fallback eviction)
- Go 1.21+ to build from source

## Project layout

```text
argus/
├── main.go # Config, entry point, recovery pipeline orchestration
├── monitor.go # Log monitor — subscribes to amq.rabbitmq.log
├── manager.go # DiagnosticsManager — RabbitMQ Management API client
├── health.go # EvaluateQueueHealth — three-gate health classification
├── locate.go # Find a queue's Raft data directory by decoding ETF meta files
├── carve.go # Extract AMQP payloads from .segment / .wal files
├── delete.go # HTTP queue deletion with rabbitmqctl fallback
└── republish.go # Republish recovered payloads via AMQP
```

## Limitations & caveats

- **Best-effort recovery.** The ETF carver scans raw bytes heuristically. Messages that span corrupted sectors may not be recoverable.
- **Single-node access required.** Argus must run on the RabbitMQ node that hosts the quorum queue data. In a multi-node cluster, run one Argus instance per node.
- **AMQP message metadata is not preserved.** Recovered messages are republished as `application/octet-stream` with `DeliveryMode=Persistent`. Original headers, content-type, and routing metadata are not reconstructed.
- **Duplicate delivery is possible.** If a message was already acknowledged before the crash, carving may recover and republish it again.

## Future
Comment thread
dhunt3800 marked this conversation as resolved.
Resolved

- Eliminate the limitation where original headers, content-type and routing metadata are not recovered.
- Ensure only one node in the RabbitMQ cluster attempts recovery of a queue.
- Manual mode to specify recovery of a specific queue.
- Remove orphaned queues.
- Possible classic queue support.

---

## Contributing
<!--
Specify whether your project accepts contributions.
Language you use in this section should mirror the language in your CONTRIBUTING.md file.
Use the default text below if you accept contributions.
If you do not accept contributions, note that here.
-->

<!-- Use this text if your project is not accepting contributions.-->
Maintainers are not currently accepting patches and contributions to this project.

<!--Use this text if your project is accepting contributions.-->
Maintainers are accepting patches and contributions to this project.
Please read [CONTRIBUTING.md](CONTRIBUTING.md) for details about submitting contributions to this project.
Please read [CONTRIBUTING.md](CONTRIBUTING.md) for details about submitting
contributions to this project.

---

## Security Policy

Please see our [Security Policy](SECURITY.md) for details.

## License
<!--
Choose from the default text already in place below.
Do not alter the text without prior approval from SAS Legal and the Open Source Program Office.
-->

This project is licensed under the [Apache 2.0 License](LICENSE).

<!--
Substitute the text below if your project contains trademarked elements not intended for license under the open source license.
In this case, add all trademarked images to the same folder and replace `XXX` in the statement with the name of that folder.
-->
Except for the the contents of the `XXX` folder, this project is licensed under the [Apache 2.0 License](LICENSE). Elements in the `XXX` folder are owned by SAS and are not released under an open source license. SAS and all other SAS Institute Inc. product or service names are registered trademarks or trademarks of SAS Institute Inc. in the USA and other countries. ® indicates USA registration.

<!--
If your projects ships with/as a container image for a package registry, keep the following statement in place.
Otherwise, remove it.
-->
As with any container image, direct and indirect dependencies are governed by their own licenses.
Users of the published container image are responsible for ensuring that their use complies with all applicable licenses.

## Third-Party Dependencies
<!--
List all third-party dependencies required for using your project, then list and link to each dependency's license.
Use the table to organize your dependency list.
Do not include version numbers.
Consult (and then remove) the provided example.
-->
This project requires the following dependencies.

| Dependency | License |
| ---------- | ------- |
| `github.com/sassoftware/dpmm` | [Apache 2.0](https://github.com/sassoftware/dpmm/blob/main/LICENSE) |

## Additional Resources
<!--
Include any additional materials users may need or find useful when using your software. Additional resources might include:

* Documentation links
* Research papers
* Blog posts
* Articles from SAS Communities
* Other relevant documentation (i.e., from Tech Support, Education, etc.)
-->

* `Resource link`
* `Resource link`
* `Resource link`
---

## Third-party dependencies

<!-- markdownlint-disable MD013 -->
| Dependency | License |
|----------------------------------|---------------------------------------------------------------------|
| `github.com/DeedleFake/etf` | [LICENSE](https://github.com/DeedleFake/etf/blob/master/LICENSE) |
| `github.com/rabbitmq/amqp091-go` | [LICENSE](https://github.com/rabbitmq/amqp091-go/blob/main/LICENSE) |
<!-- markdownlint-enable MD013 -->
24 changes: 10 additions & 14 deletions SECURITY.md
Original file line number Diff line number Diff line change
@@ -1,18 +1,12 @@
<!--
A SECURITY.md outlines your project's security policy. It includes instructions on how to report a security vulnerability in your project.
If your project contains this file, link to it from the project's README.
-->
# Argus Security Policy

# PROJECT_NAME Security Policy
<!-- Replace PROJECT_NAME with the official name of your open source project. -->
Project maintainers and community contributors take security issues seriously.
Efforts to disclose potential issues responsibly are appreciated, and viable contributions will be acknowledged.
To aid investigation of any reported vulnerabilities, please follow the [reporting guidelines](#reporting-guidelines) when submitting your findings.
Efforts to disclose potential issues responsibly are appreciated, and viable
contributions will be acknowledged.
To aid investigation of any reported vulnerabilities, please follow the
[reporting guidelines](#reporting-guidelines) when submitting your findings.

## Reporting Guidelines
<!-- Project maintainers: When you project is public, activate GitHub's built-in private reporting:
https://docs.github.com/en/code-security/security-advisories/working-with-repository-security-advisories/configuring-private-vulnerability-reporting-for-a-repository
-->

To report a suspected security issue, use private vulnerability reporting.

Expand All @@ -22,9 +16,11 @@ To report a suspected security issue, use private vulnerability reporting.
Then provide the following information with suspected security issues:

* Your name and affiliation
* Version/build-date of project
* Version/build-date of project
* Issue description
* Steps to reproduce the issue
* Current public knowledge of this vulnerability (for example, related CVE, security advisory, and so on), if known
* Current public knowledge of this vulnerability (for example, related CVE,
security advisory, and so on), if known

The project release notes contain acknowledgments for contributors who provide security-related insights in their commits.
The project release notes contain acknowledgments for contributors who provide
security-related insights in their commits.
8 changes: 1 addition & 7 deletions SUPPORT.md
Original file line number Diff line number Diff line change
@@ -1,10 +1,4 @@
<!-- A SUPPORT.md outlines the ways project maintainers intended to interact with and assist users.
Update this file to provide these details. -->
# Support

## Support
<!-- Use this text if you plan to provide support for users. -->
Project maintainers use GitHub for tracking bugs and feature requests.
Please submit a GitHub issue or pull request to receive support.

<!-- Use this text if you do not intend to provide support for users. -->
Maintainers are not currently offering support for this project.
8 changes: 8 additions & 0 deletions go.mod
Original file line number Diff line number Diff line change
@@ -0,0 +1,8 @@
module github.com/sassoftware/argus

go 1.26.2

require (
github.com/DeedleFake/etf v0.0.0-20250818194526-023937b5970e
github.com/rabbitmq/amqp091-go v1.10.0
)
6 changes: 6 additions & 0 deletions go.sum
Original file line number Diff line number Diff line change
@@ -0,0 +1,6 @@
github.com/DeedleFake/etf v0.0.0-20250818194526-023937b5970e h1:4nmS17q152IglvLUo4rrtk2f8iCPypA9GSuzzFN1G0o=
github.com/DeedleFake/etf v0.0.0-20250818194526-023937b5970e/go.mod h1:BZQO3V9AXn7x3ocRxCo84FMZM6TKWidhH9i5u2alp7o=
github.com/rabbitmq/amqp091-go v1.10.0 h1:STpn5XsHlHGcecLmMFCtg7mqq0RnD+zFr4uzukfVhBw=
github.com/rabbitmq/amqp091-go v1.10.0/go.mod h1:Hy4jKW5kQART1u+JkDTF9YYOQUHXqMuhrgxOEeS7G4o=
go.uber.org/goleak v1.3.0 h1:2K3zAYmnTNqV73imy9J1T3WC+gmCePx2hEGkimedGto=
go.uber.org/goleak v1.3.0/go.mod h1:CoHD4mav9JJNrW/WLlf7HGZPjdw8EucARQHekz1X6bE=
Loading