OpsGuardian 🛡️

OpsGuardian – Cloud-Native Service Reliability & Observability Platform

OpsGuardian is a cloud-native, highly concurrent Site Reliability Engineering (SRE) monitoring service written in Go. It continuously polls infrastructure endpoints (HTTP, TCP, DNS, SSL), records metrics to Prometheus, tracks incident history in real-time, and dispatches automated alerts to external webhooks (Slack/Discord).

---

🌐 Live Demo

• Grafana Dashboard: http://52.62.127.151:3000 (Deployed live on AWS EC2)

---

🌟 Key Features & SRE Concepts

• Multi-Protocol Polling: Monitor HTTP APIs, TCP sockets, and DNS resolution.
• Resilience Engineering:
  • Circuit Breakers: Prevents cascading failures by failing fast when downstream services are degraded.
  • Exponential Backoff & Retry: Intelligently retries transient network errors.
• Observability (Prometheus): Exposes /metrics endpoint with Histograms (latency), Gauges (uptime), and Counters (errors).
• Incident History & Alert Manager: Maintains an in-memory state machine of incidents and dispatches webhooks when services transition from UP ↔ DOWN.
• Configuration as Code (Viper): Completely dynamic configuration via YAML or Environment Variables.
• Embedded Web Dashboard: Single binary includes an embedded, rich dark-mode UI for local demos.
• Kubernetes Native: Includes Liveness/Readiness probes and Prometheus scrape annotations.

---

💻 Tech Stack

• Language: Go (1.24/1.25) leveraging Goroutines for high concurrency
• Observability: Prometheus & Grafana
• Logging: Uber's zap (Structured JSON logging)
• Configuration: spf13/viper
• Containerization: Docker & Docker Compose
• Orchestration: Kubernetes
• CI/CD: GitHub Actions

---

📂 Folder Structure

opsguardian/
├── cmd/                # Entrypoints (CLI commands)
├── internal/           # Internal HTTP server & embedded UI
├── pkg/
│   ├── alerting/       # Webhooks & Notification dispatch
│   ├── config/         # Viper configuration parsing
│   ├── history/        # In-memory Incident state machine
│   ├── logger/         # Zap structured logging
│   ├── metrics/        # Prometheus metric definitions
│   ├── monitor/        # Concurrent scheduler & Checkers (HTTP, TCP, DNS)
│   └── resilience/     # Circuit Breaker & Exponential Backoff logic
├── k8s/                # Kubernetes Deployments & ConfigMaps
├── grafana/            # Grafana Dashboard JSON templates
├── .github/workflows/  # CI/CD pipelines
├── Dockerfile          # Multi-stage container build
└── Makefile            # Make targets for build, test, and deploy

---

🛠️ Architecture

              +--------------------+
              |   Monitored APIs   |
              +---------+----------+
                        |
                 Health Checks
                        |
                  OpsGuardian
      +---------+-------+---------+
      |         |                 |
 Prometheus   Logger        Incident Manager
      |                           |
      +-----------+---------------+
                  |
              Grafana

OpsGuardian uses a highly concurrent architecture. A central Scheduler parses the YAML configuration and spawns an independent Goroutine for every target. Each goroutine uses a time.Ticker to execute checks. The checks are wrapped in a Circuit Breaker and a Retry mechanism before making the actual network call. Results are piped into an Incident Store and a Metrics Registry, which are served by an isolated HTTP server.

---

🚀 Getting Started

Prerequisites

• Go 1.24+
• Docker & Docker Compose (Recommended)

Installation

Option 1: Full Observability Stack (Docker Compose)

The easiest way to run OpsGuardian alongside Prometheus and Grafana:

git clone https://github.com/ayushgupta704/OpsGuardian.git
cd OpsGuardian
docker-compose up -d --build

Grafana will be available at http://localhost:3000 (or your server's IP).

Option 2: Local Binary (Backend Only)

make build
make run

The embedded Web UI will be available at http://localhost:8080/

---

📈 Observability & Metrics

OpsGuardian exposes native Prometheus metrics at http://localhost:8080/metrics.

Example Output:

# HELP opsguardian_target_latency_seconds Latency of target requests in seconds.
# TYPE opsguardian_target_latency_seconds histogram
opsguardian_target_latency_seconds_bucket{protocol="http",target="Google API",le="0.1"} 0
opsguardian_target_latency_seconds_bucket{protocol="http",target="Google API",le="0.5"} 12
...
# HELP opsguardian_target_up Indicates if the target is currently up (1) or down (0).
# TYPE opsguardian_target_up gauge
opsguardian_target_up{protocol="tcp",target="Internal Database"} 1
opsguardian_target_up{protocol="dns",target="GitHub DNS"} 1

(Note: Import the provided grafana/dashboard.json into your local Grafana instance to instantly visualize Latency, Uptime, and Error Rates!)

---

🚀 Performance & Load Testing

OpsGuardian is built for high concurrency. We utilize k6 to validate the performance of our internal API and Dashboard rendering under load.

Performance Testing

This project includes load-testing support using k6. Benchmark results will be generated against the deployed application and documented after testing.

To run the load tests yourself:

k6 run k6/load_test.js

---

🐳 Docker & Kubernetes

OpsGuardian is built for the cloud. A multi-stage Dockerfile guarantees a minimal Alpine image.

Build Image:

make docker-build

Deploy to Kubernetes:

make k8s-apply

This applies a Deployment (with CPU/Memory limits and Health Probes), a Service, and a ConfigMap for dynamic configuration without rebuilding the image.

---

🔮 Future Work

• Distributed Polling: Implement a leader-election system via etcd/Redis to allow multiple OpsGuardian nodes to poll concurrently without duplicating alerts.
• Persistent Storage: Replace the in-memory history store with PostgreSQL or SQLite.
• Dynamic Configuration Reload: Listen to Kubernetes ConfigMap updates via fsnotify to hot-reload target configurations without restarting the pod.