EventStats lambda with Stats endpoint

.github/copilot-instructions.md

@@ -4,27 +4,34 @@ Purpose
 AWS Lambda event gateway that receives messages via API Gateway and dispatches them to Kafka, EventBridge, and PostgreSQL.
 
 Structure
-- Entry point: `src/event_gate_lambda.py`
-- Handlers: `src/handlers/` (HandlerApi, HandlerToken, HandlerTopic, HandlerHealth)
+- Main lambda (event ingestion): `src/event_gate_lambda.py`
+- Stats lambda (read-only queries): `src/event_stats_lambda.py`
+- Shared config loading: `src/utils/config_loader.py`
+- Handlers: `src/handlers/` (HandlerApi, HandlerToken, HandlerTopic, HandlerHealth, HandlerStats)
 - Writers: `src/writers/` (inherit from `Writer` base class)
+- Readers: `src/readers/` (read-only database access for stats)
 - Config: `conf/config.json`, `conf/access.json`, `conf/topic_schemas/*.json`
 - Production Terraform scripts are not part of this repository; `terraform_examples/` for reference configurations only
 
 Python style
 - Python 3.13
 - Type hints for public functions and classes
+- Use built-in generics for type hints
 - Use `logging.getLogger(__name__)`, not print
 - Lazy % formatting in logging: `logger.info("msg %s", var)`
 - F-strings in exceptions: `raise ValueError(f"Error {var}")`
 - All imports at top of file (never inside functions)
 - Apache 2.0 license header in every .py file (including `__init__.py`)
 - Docstrings must start with a short summary line
+- No blank lines between docstring sections (summary, Args, Returns, Raises)
+- Use single backticks in docstrings (`value`), never double backticks (`` ``value`` ``)
+- Do not use `# -----------` separator comments to divide sections
 - End all log messages with a period: `logger.info("Message.")`
 
 Patterns
 - `__init__` methods must not raise exceptions; defer validation and connection to first use (lazy init)
 - Writers: inherit from `Writer(ABC)`, implement `write(topic, message) -> (bool, str|None)` and `check_health() -> (bool, str)`
-- Route dispatch via `ROUTE_MAP` dict mapping routes to handler functions in `event_gate_lambda.py`
+- Route dispatch via `ROUTE_MAP` dict mapping routes to handler functions in `event_gate_lambda.py` and `event_stats_lambda.py`
 - Separate business logic from environment access (env vars, file I/O, network calls)
 - No duplicate validation; centralize parsing in one layer where practical
 - Preserve existing formatting and conventions
@@ -33,13 +40,16 @@ Patterns
 
 Testing
 - Mirror src structure: `src/handlers/` -> `tests/unit/handlers/`
+- Test modules (`test_*.py`) must not have module-level docstrings
 - Unit tests: mock external services via `conftest.py` (Kafka, EventBridge, PostgreSQL, S3)
 - Integration tests: call `lambda_handler` directly with real containers (testcontainers-python for Kafka, PostgreSQL, LocalStack)
 - No real API/DB calls in unit tests
 - Use `mocker.patch("module.dependency")` or `mocker.patch.object(Class, "method")`
 - Assert pattern: `assert expected == actual`
 
 Quality gates (run after changes, fix only if below threshold)
+- Always run quality gates using the project virtual environment: `.venv/bin/python -m <tool>`
+- Once a quality gate passes, do not re-run it in different scenarios
 - black .
 - mypy .
 - pylint $(git ls-files '*.py') >= 9.5

DEVELOPER.md

@@ -18,6 +18,11 @@ git clone https://github.com/AbsaOSS/EventGate.git
 cd EventGate
 ```
 
+### Project Structure
+EventGate ships two Lambda functions:
+- **Event Gate Lambda** (`src/event_gate_lambda.py`) — the main API surface. Serves the OpenAPI spec, token provider redirect, health check, topic schema catalogue, and event ingestion (`POST /topics/{topicName}`).
+- **Event Stats Lambda** (`src/event_stats_lambda.py`) — serves read-only queries via `POST /stats/{topicName}` with filtering, sorting, and cursor-based pagination backed by PostgreSQL.
+
 ## Prerequisites
 - Python 3.13 (current required runtime)
 - PostgreSQL client dev package

README.md

@@ -45,7 +45,8 @@ High-level flow:
 5. Aggregated response returned: `202 Accepted` if all succeed; `500` with per-writer error list otherwise.
 
 Key files:
-- `src/event_gate_lambda.py` – main Lambda handler and routing
+- `src/event_gate_lambda.py` – main Lambda handler and routing (event ingestion)
+- `src/event_stats_lambda.py` – stats Lambda handler and routing (read-only queries)
 - `conf/*.json` – configuration files
 - `conf/topic_schemas/` – JSON topic schemas
 - `api.yaml` – OpenAPI 3 definition served at `/api`
@@ -62,6 +63,7 @@ All responses are JSON unless otherwise noted. The POST endpoint requires a vali
 | GET | `/topics` | none | Lists available topic names |
 | GET | `/topics/{topicName}` | none | Returns JSON Schema for the topic |
 | POST | `/topics/{topicName}` | JWT | Validates + forwards message to configured sinks |
+| POST | `/stats/{topicName}` | JWT | Queries ingested events with filtering, sorting, and cursor pagination |
 | POST | `/terminate` | (internal) | Forces Lambda process exit (used to trigger cold start & config reload) |
 
 Status codes:
@@ -107,6 +109,9 @@ Supporting configs:
 
 Environment variables:
 - `LOG_LEVEL` (optional) – defaults to `INFO`.
+- `CONF_DIR` (optional) – directory containing `config.json` and `access.json`. Defaults to `conf`.
+- `POSTGRES_SECRET_NAME` (optional) – AWS Secrets Manager secret name holding PostgreSQL connection credentials (host, port, dbname, username, password). Required for Postgres writer and stats reader.
+- `POSTGRES_SECRET_REGION` (optional) – AWS region of the Secrets Manager secret. Must be set together with `POSTGRES_SECRET_NAME`.
 
 ## Deployment
 Infrastructure-as-Code examples are provided in `terraform_examples/`. These are reference implementations that you can adapt to your environment. Variables are supplied via a `*.tfvars` file or CLI.
@@ -168,7 +173,7 @@ Configured via `kafka_bootstrap_server`. (Future: support auth properties / TLS
 Publishes events to the configured `event_bus_arn` using put events API.
 
 ### Postgres Writer
-Example writer (currently a placeholder if no DSN present) demonstrating extensibility pattern.
+Writes enriched event data (runs + jobs) into PostgreSQL tables. Connection credentials are loaded from AWS Secrets Manager using the `POSTGRES_SECRET_NAME` and `POSTGRES_SECRET_REGION` environment variables.
 
 ## Troubleshooting
 | Symptom | Possible Cause | Action |