Integration test

.github/copilot-instructions.md

@@ -19,6 +19,7 @@ Python style
 - All imports at top of file (never inside functions)
 - Apache 2.0 license header in every .py file (including `__init__.py`)
 - Docstrings must start with a short summary line
+- End all log messages with a period: `logger.info("Message.")`
 
 Patterns
 - `__init__` methods must not raise exceptions; defer validation and connection to first use (lazy init)
@@ -31,8 +32,9 @@ Patterns
 - Keep error response format stable: `{"success": false, "statusCode": int, "errors": [...]}`
 
 Testing
-- Mirror src structure: `src/handlers/` -> `tests/handlers/`
-- Mock external services via `conftest.py`: Kafka, EventBridge, PostgreSQL, S3
+- Mirror src structure: `src/handlers/` -> `tests/unit/handlers/`
+- Unit tests: mock external services via `conftest.py` (Kafka, EventBridge, PostgreSQL, S3)
+- Integration tests: call `lambda_handler` directly with real containers (testcontainers-python for Kafka, PostgreSQL, LocalStack)
 - No real API/DB calls in unit tests
 - Use `mocker.patch("module.dependency")` or `mocker.patch.object(Class, "method")`
 - Assert pattern: `assert expected == actual`

.github/workflows/check_python.yml

@@ -111,7 +111,32 @@ jobs:
         id: check-format
         run: black --check $(git ls-files '*.py')
 
-  pytest-test:
+  mypy-check:
+    name: Mypy Type Check
+    needs: detect
+    if: needs.detect.outputs.python_changed == 'true'
+    runs-on: ubuntu-latest
+    steps:
+      - name: Checkout repository
+        uses: actions/checkout@8e8c483db84b4bee98b60c0593521ed34d9990e8
+        with:
+          persist-credentials: false
+          fetch-depth: 0
+
+      - name: Set up Python
+        uses: actions/setup-python@83679a892e2d95755f2dac6acb0bfd1e9ac5d548
+        with:
+          python-version: '3.13'
+          cache: 'pip'
+
+      - name: Install dependencies
+        run: pip install -r requirements.txt
+
+      - name: Check types with Mypy
+        id: check-types
+        run: mypy .
+
+  unit-tests:
     name: Pytest Unit Tests with Coverage
     needs: detect
     if: needs.detect.outputs.python_changed == 'true'
@@ -123,7 +148,8 @@ jobs:
           persist-credentials: false
           fetch-depth: 0
 
-      - uses: actions/setup-python@83679a892e2d95755f2dac6acb0bfd1e9ac5d548
+      - name: Set up Python
+        uses: actions/setup-python@83679a892e2d95755f2dac6acb0bfd1e9ac5d548
         with:
           python-version: '3.13'
           cache: 'pip'
@@ -132,13 +158,14 @@ jobs:
         run: pip install -r requirements.txt
 
       - name: Check code coverage with Pytest
-        run: pytest --cov=. -v tests/ --cov-fail-under=80
+        run: pytest --cov=. -v tests/unit/ --cov-fail-under=80
 
-  mypy-check:
-    name: Mypy Type Check
+  integration-tests:
+    name: Pytest Integration Tests
     needs: detect
     if: needs.detect.outputs.python_changed == 'true'
     runs-on: ubuntu-latest
+    timeout-minutes: 15
     steps:
       - name: Checkout repository
         uses: actions/checkout@8e8c483db84b4bee98b60c0593521ed34d9990e8
@@ -155,9 +182,8 @@ jobs:
       - name: Install dependencies
         run: pip install -r requirements.txt
 
-      - name: Check types with Mypy
-        id: check-types
-        run: mypy .
+      - name: Run integration tests
+        run: pytest tests/integration/ -v --tb=short --log-cli-level=INFO
 
   noop:
     name: No Operation

DEVELOPER.md

@@ -5,8 +5,9 @@
 - [Run Pylint Tool Locally](#run-pylint-tool-locally)
 - [Run Black Tool Locally](#run-black-tool-locally)
 - [Run mypy Tool Locally](#run-mypy-tool-locally)
-- [Run Unit Test](#running-unit-test)
+- [Run Unit Test Locally](#run-unit-test-locally)
 - [Code Coverage](#code-coverage)
+- [Run Integration Test Locally](#run-integration-test-locally)
 
 ## Get Started
 
@@ -45,7 +46,7 @@ To run Pylint on a specific file, follow the pattern `pylint <path_to_file>/<nam
 
 Example:
 ```shell
-pylint src/writer_kafka.py
+pylint src/event_gate_lambda.py
 ``` 
 
 ## Run Black Tool Locally
@@ -68,7 +69,7 @@ To run Black on a specific file, follow the pattern `black <path_to_file>/<name_
 
 Example:
 ```shell
-black src/writer_kafka.py
+black src/writers/writer_kafka.py
 ``` 
 
 ### Expected Output
@@ -100,39 +101,39 @@ To run mypy on a specific file, follow the pattern `mypy <path_to_file>/<name_of
 
 Example:
 ```shell
-mypy src/writer_kafka.py
+mypy src/handlers/handler_token.py
 ```
 
-## Running Unit Test
+## Run Unit Test Locally
 
 Unit tests are written using pytest. To run the tests, use the following command:
 
 ```shell
-pytest tests/
+pytest tests/unit/
 ```
 
-This will execute all tests located in the tests directory.
+This will execute all unit tests located in the tests/unit/ directory.
 
 ### Focused / Selective Test Runs
 Run a single test file:
 ```shell
-pytest tests/test_writer_kafka.py -q
+pytest tests/unit/writers/test_writer_kafka.py
 ```
 Filter by keyword expression:
 ```shell
-pytest -k kafka -q
+pytest -k kafka
 ```
 Run a single test function (node id):
 ```shell
-pytest tests/test_event_gate_lambda.py::test_post_multiple_writer_failures -q
+pytest tests/unit/writers/test_writer_eventbridge.py::test_write_success
 ```
 
 ## Code Coverage
 
 Code coverage is collected using the pytest-cov coverage tool. To run the tests and collect coverage information, use the following command:
 
 ```shell
-pytest --cov=. -v tests/ --cov-fail-under=80 --cov-report=term-missing
+pytest --cov=. -v tests/unit/ --cov-fail-under=80 --cov-report=html
 ```
 
 This will execute all tests in the tests directory and generate a code coverage report with missing line details and enforce a minimum 80% threshold.
@@ -141,3 +142,65 @@ Open the HTML coverage report:
 ```shell
 open htmlcov/index.html
 ```
+
+## Run Integration Test Locally
+
+Integration tests validate EventGate against real service dependencies using testcontainers-python.
+
+### Integration Test Approach
+
+EventGate uses a **direct invocation approach** for integration testing:
+- **Lambda handler is called directly** in Python (not run in a container)
+- **External dependencies run in Docker containers**: Kafka, PostgreSQL, LocalStack (EventBridge)
+- **Mock JWT provider runs in-process** as a background thread (no container)
+- Test configuration is dynamically generated and injected via environment variables
+
+This approach is faster (~8s vs 30s+), more reliable, and easier to debug than container-based Lambda testing.
+
+### Prerequisites
+- Docker running (Docker Desktop on macOS/Windows, or Docker Engine on Linux)
+- Python 3.13 with dependencies installed
+
+### Run Integration Tests
+
+Containers start and stop automatically:
+```shell
+pytest tests/integration/ -v
+```
+
+With detailed logging:
+```shell
+pytest tests/integration/ -v --log-cli-level=INFO
+```
+
+### Run Specific Integration Tests
+
+Run a single test file:
+```shell
+pytest tests/integration/test_health_endpoint.py -v
+```
+
+Run a specific test function:
+```shell
+pytest tests/integration/test_topics_endpoint.py::TestPostEventEndpoint::test_post_event_with_valid_token_returns_202 -v
+```
+
+### Troubleshooting
+
+If containers fail to start, check Docker is running:
+```shell
+docker info
+```
+
+If image pulls fail with TLS or timeout errors, pre-pull the required images manually:
+```shell
+docker pull testcontainers/ryuk:0.8.1
+docker pull postgres:16
+docker pull confluentinc/cp-kafka:7.6.0
+docker pull localstack/localstack:latest
+```
+
+View container logs in pytest output by increasing log level:
+```shell
+pytest tests/integration/ -v --log-cli-level=DEBUG
+```

README.md

@@ -146,10 +146,9 @@ Use when Kafka access needs Kerberos / SASL_SSL or custom `librdkafka` build.
 | Static code analysis (Pylint) | [Run Pylint Tool Locally](./DEVELOPER.md#run-pylint-tool-locally)     |
 | Formatting (Black)            | [Run Black Tool Locally](./DEVELOPER.md#run-black-tool-locally)       |
 | Type checking (mypy)          | [Run mypy Tool Locally](./DEVELOPER.md#run-mypy-tool-locally)         |
-| Terraform Linter (TFLint)     | [Run TFLint Tool Locally](./DEVELOPER.md#run-tflint-tool-locally)     |
-| Security Scanner (Trivy)      | [Run Trivy Tool Locally](./DEVELOPER.md#run-trivy-tool-locally)       |
-| Unit tests                    | [Running Unit Test](./DEVELOPER.md#running-unit-test)                 |
+| Unit tests                    | [Run Unit Test Locally](./DEVELOPER.md#run-unit-test-locally)         |
 | Code coverage                 | [Code Coverage](./DEVELOPER.md#code-coverage)                         |
+| Integration tests             | [Run Integration Test Locally](./DEVELOPER.md#run-integration-test-locally) |
 
 ## Security & Authorization
 - JWT tokens must be RS256 signed; current and previous public keys are fetched at cold start from `token_public_keys_url` as DER base64 values (list `keys[*].key`, with single-key fallback `{ "key": "..." }`).

conf/access.json

@@ -1,12 +1,5 @@
 {
-    "public.cps.za.runs": [
-        "FooBarUser"
-    ],
-    "public.cps.za.dlchange": [
-        "FooUser",
-        "BarUser"
-    ],
-    "public.cps.za.test": [
-        "TestUser"
-    ]
-}
+    "public.cps.za.runs": ["FooBarUser", "IntegrationTestUser"],
+    "public.cps.za.dlchange": ["FooUser", "BarUser"],
+    "public.cps.za.test": ["TestUser"]
+}

requirements.txt

@@ -1,4 +1,4 @@
-pytest==8.4.2
+pytest==9.0.2
 pytest-cov==6.3.0
 pytest-mock==3.15.0
 pylint==3.3.8
@@ -12,5 +12,7 @@ PyJWT==2.10.1
 requests==2.32.5
 boto3==1.40.25
 confluent-kafka==2.12.1
+testcontainers==4.14.1
+docker==7.1.0
 # psycopg2-binary==2.9.10 # Ideal for local development, but not for long-term production use
 psycopg2==2.9.10

tests/integration/__init__.py

@@ -0,0 +1,15 @@
+#
+# Copyright 2026 ABSA Group Limited
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#