Log Monitoring Pipeline

A production-ready log monitoring pipeline built with Spring Boot, Kafka, Elasticsearch, Prometheus, and Grafana. This system ingests logs from multiple services, processes them in real-time, stores them for search and analytics, and provides comprehensive observability.

Architecture

[Spring Boot App]  →  [Kafka Producer]  →  [Kafka Topic: "app-logs"]
                                                     │
                                                     ▼
                                              [Kafka Consumer]
                                               ├── Parse & enrich log
                                               ├── Store in Elasticsearch
                                               └── Emit metrics to Prometheus
                                                        │
                                                        ▼
                                                  [Grafana Dashboard]
                                                  ├── Error rate over time
                                                  ├── Top error types
                                                  └── Alert if error rate > threshold

Tech Stack

• Spring Boot 3.2.0 - Application framework
• Spring Kafka - Kafka integration for log streaming
• Spring Data Elasticsearch - Elasticsearch integration for log storage
• Kafka 7.5.0 - Distributed event streaming platform
• Elasticsearch 8.11.0 - Search and analytics engine
• Prometheus - Metrics collection and monitoring
• Grafana - Visualization and alerting
• Docker Compose - Container orchestration
• Testcontainers - Integration testing with real containers
• Lombok - Reduce boilerplate code

Prerequisites

• Docker and Docker Compose
• Java 17+
• Maven 3.6+

Quick Start

1. Start Infrastructure

docker-compose up -d

This starts:

• Kafka (port 9092)
• Elasticsearch (port 9200)
• Prometheus (port 9090)
• Grafana (port 3000)

2. Build and Run Application

mvn clean package
docker-compose up --build log-monitoring-app

Or run locally:

mvn spring-boot:run

3. Verify Services

• Kafka: Check logs with docker-compose logs kafka
• Elasticsearch: curl http://localhost:9200
• Prometheus: Open http://localhost:9090
• Grafana: Open http://localhost:3000 (admin/admin)
• Application: curl http://localhost:8080/actuator/health

API Documentation

Ingest Logs

Single Log:

curl -X POST http://localhost:8080/api/logs \
  -H "Content-Type: application/json" \
  -d '{
    "level": "ERROR",
    "service": "payment-api",
    "message": "Connection timeout after 30000ms",
    "host": "pod-payment-5a3b"
  }'

Batch Logs:

curl -X POST http://localhost:8080/api/logs/batch \
  -H "Content-Type: application/json" \
  -d '[
    {
      "level": "ERROR",
      "service": "payment-api",
      "message": "Connection timeout"
    },
    {
      "level": "INFO",
      "service": "order-api",
      "message": "Order processed"
    }
  ]'

Search Logs

Basic Search:

curl "http://localhost:8080/api/search?level=ERROR&service=payment-api"

Full-text Search:

curl "http://localhost:8080/api/search?keyword=timeout"

Time Range Search:

curl "http://localhost:8080/api/search?from=2026-01-01T00:00:00Z&to=2026-01-31T23:59:59Z"

Paginated Results:

curl "http://localhost:8080/api/search?level=ERROR&page=0&size=10"

Analytics

Error Count by Service:

curl http://localhost:8080/api/search/stats/errors-by-service

Response:

{
  "payment-api": 42,
  "order-api": 18,
  "user-api": 7
}

Grafana Dashboard

Import the provided dashboard from grafana/dashboard.json or create manually:

1. Add Prometheus as data source: http://prometheus:9090
2. Create dashboard with panels:
   • Log Throughput: rate(logs_consumed_total[1m])
   • Error Rate: rate(logs_errors_total[1m])
   • Error Percentage: rate(logs_errors_total[5m]) / rate(logs_consumed_total[5m]) * 100
   • Processing Latency (P99): histogram_quantile(0.99, rate(logs_processing_time_seconds_bucket[5m]))

Design Decisions

Why Kafka over Direct Elasticsearch Writes?

• Decoupling: Producers don't need to know about Elasticsearch
• Backpressure: Kafka buffers logs during Elasticsearch downtime
• Replay: Can reprocess logs by replaying Kafka topics
• Scalability: Multiple consumers can process logs independently

Why service as Kafka Message Key?

Using the service name as the message key ensures all logs from the same service go to the same partition, maintaining order within each service. This is crucial for debugging and understanding the sequence of events.

Why Batch Indexing?

Bulk indexing to Elasticsearch is 10-100x faster than single-document indexing. This is essential for production log volumes and reduces load on the Elasticsearch cluster.

Why At-Least-Once Delivery?

The system uses at-least-once delivery semantics, which is acceptable for logs. Duplicate logs are preferable to lost logs, and Elasticsearch's idempotent nature prevents duplicate documents.

Why Custom Metrics over ES-Only Monitoring?

Prometheus+Grafana provides:

• Real-time metrics with low latency
• Powerful alerting capabilities
• Integration with other monitoring systems
• Better performance for high-frequency metrics

Production Considerations

For production deployment, consider adding:

• Schema Registry: For Kafka schema evolution
• Index Lifecycle Management (ILM): Time-based indices for better performance
• Authentication/Authorization: Security for Kafka, Elasticsearch, and APIs
• TLS/SSL: Encrypted communication between components
• High Availability: Multiple Kafka brokers and Elasticsearch nodes
• Monitoring: Enhanced alerting and notification channels
• Log Retention: Configurable retention policies based on compliance requirements

Testing

Run integration tests with Testcontainers:

mvn test

This spins up real Kafka and Elasticsearch containers for comprehensive testing.

Health Checks

Check application health:

curl http://localhost:8080/actuator/health

Response includes status of:

• Kafka connection
• Elasticsearch connection
• Application components

Metrics

Custom metrics exposed at /actuator/prometheus:

• logs_consumed_total: Total logs processed
• logs_errors_total: Total ERROR level logs
• logs_processing_time_seconds: Log processing latency

License

This project is created for demonstration purposes.