LogMind

Lightweight log ingestion & real-time incident detection engine
Accept logs via HTTP, store in SQLite, get incidents when errors spike.

---

Features

• Zero-dependency Go backend — no web framework, no ORM, no message queue. Just stdlib, SQLite, and Prometheus.
• O(1) sliding-window detection — allocation-free ring buffer detects error bursts in constant time.
• Persistent & crash-recoverable — incident state rebuilds from SQLite on restart. No data loss.
• Batched async ingestion — configurable worker pool batches inserts for 3.5× throughput over individual writes.
• Per-IP rate limiting — token-bucket limiter protects the ingestion endpoint.
• Observability built-in — Prometheus metrics (/metrics), health checks (/healthz, /readyz), live dashboard, and pre-built Grafana dashboards.
• Docker Compose in 30 seconds — one command to run the full stack.

---

Architecture

  App / Service
       │
       ▼ POST /logs
 ┌─────────────┐
 │ Rate Limiter│  (per-IP token bucket)
 └──────┬──────┘
        ▼
 ┌──────────────┐     ┌──────────────────┐     ┌────────────────┐
 │  HTTP Handler│────▶│  Buffered Channel │────▶│  Worker Pool   │
 └──────────────┘     │   (cap 10,000)    │     │  (4 goroutines)│
                      └──────────────────┘     └───────┬────────┘
                                                        │ batch flush
                                                        ▼
                                               ┌────────────────┐
                                               │   SQLite (WAL) │
                                               └───────┬────────┘
                                                        │
                                               ┌────────▼────────┐
                                               │ Incident Engine │  (O(1) sliding window)
                                               └────────┬────────┘
                                                        │
                                                        ▼ GET /incidents

---

Quick Start

# Start the full stack (engine + Prometheus + Grafana)
docker compose up -d

# Send a test log
curl -X POST http://localhost:8080/logs \
  -H "Content-Type: application/json" \
  -d '{"service":"payment-api","level":"error","message":"Connection refused"}'

# Check for incidents
curl http://localhost:8080/incidents

That's it. Dashboard at http://localhost:8080, Grafana at http://localhost:3000.

---

Configuration

All configuration is through environment variables:

Variable	Default	Description
PORT	8080	HTTP listen port
LOGMIND_DB_PATH	logmind.db	SQLite database file path
MAX_WORKERS	4	Number of ingestion worker goroutines
BATCH_SIZE	1000	Logs per batch insert
LOG_CHANNEL_BUFFER	10000	In-memory channel capacity
SLIDING_WINDOW_SEC	60	Error detection window (seconds)
ERROR_THRESHOLD	3	Error count to trigger incident
ENABLE_SIMULATOR	false	Enable built-in traffic simulator
GOMEMLIMIT	96MiB	Go memory soft limit

---

API

Method	Path	Description
POST	/logs	Ingest a log entry
GET	/incidents	List active incidents
GET	/healthz	Health check (always 200)
GET	/readyz	Readiness check (pings SQLite)
GET	/metrics	Prometheus metrics

POST /logs payload:

{
  "service": "payment-api",
  "level": "error",
  "message": "Connection refused"
}

GET /incidents response:

{
  "incidents": [
    {
      "service": "payment-api",
      "error_count": 5,
      "window_seconds": 60,
      "first_seen": "2026-07-03T10:00:00Z",
      "last_seen": "2026-07-03T10:01:00Z"
    }
  ]
}

---

Example Clients

Runnable examples in multiple languages:

• curl
• Go
• Python
• Node.js
• Distributed order system demo (3 services)

---

Testing & Benchmarks

# Unit + integration + load tests
go test ./... -race

# Benchmarks
go test ./tests/bench/... -bench=. -benchmem

Benchmark	Result
Batch insert throughput	3.5× faster than individual
Sliding window ops	20M ops/sec, zero allocs
P99 ingestion latency	1.25 ms
Concurrent throughput	26,000 req/sec

Full results in benchmarks.md.

---

Project Structure

cmd/logmind/main.go        Entrypoint
internal/
├── api/                   HTTP router & handlers
├── config/                Env-based configuration
├── detection/             Incident detection engine
├── domain/                Log, Incident, ServiceMetrics types
├── ingestion/             HTTP handler + worker pool
├── observability/         Health checks & Prometheus metrics
├── simulator/             Built-in traffic simulator
└── storage/               SQLite repository
pkg/ratelimit/             Per-IP token bucket
deployments/               Prometheus & Grafana configs
tests/                     Unit, integration, load, UI, benchmarks
Doc/                       MkDocs documentation site

---

Contributing

See contributing.md. PRs welcome — please run go test -race ./... before submitting.

---

License

MIT — see LICENSE.