feat: authorization

.gitignore

@@ -167,3 +167,6 @@ cython_debug/
 
 # tds-db
 pg_data
+
+# TDS custom authz module
+lib/*

.vscode/launch.json

@@ -8,6 +8,7 @@
             "name": "Python Debugger (TDS): Dev Container Attach",
             "type": "debugpy",
             "request": "attach",
+            "justMyCode": false,
             "connect": {
                 "host": "0.0.0.0",
                 "port": 9511

README.md

@@ -36,6 +36,14 @@ docker compose -f ./docker-compose.dev.yaml down
 
 You can then attach VS Code to the `tds` container, and use the preconfigured `Python Debugger (TDS)` for interactive debugging.
 
+## Authorization plugin
+
+The Transcriptomics-Data-Service is meant to be a reusable microservice that can be integrated in existing 
+stacks. Since authorization schemes vary across projects, TDS allows adopters to code their own authorization plugin, 
+enabling adopters to leverage their existing access control code, tools and policies.
+
+See the [authorization docs](./docs/authz.md) for more information on how to create and use the authz plugin with TDS.
+
 ## Endpoints
 
 * /service-info

docker-compose.dev.yaml

@@ -9,7 +9,7 @@ services:
     depends_on:
       - tds-db
     environment:
-      - BENTO_UID=1001
+      - BENTO_UID=${UID}
       - DATABASE_URI=postgres://tds_user:tds_password@tds-db:5432/tds_db
       - CORS_ORIGINS="*"
       - BENTO_AUTHZ_SERVICE_URL=""

docker-compose.yaml

@@ -10,6 +10,8 @@ services:
       - DATABASE_URI=postgres://tds_user:tds_password@tds-db:5432/tds_db
       - CORS_ORIGINS="*"
       - BENTO_AUTHZ_SERVICE_URL=""
+    volumes:
+      - $PWD/lib:/tds/lib
     ports:
       - "5000:5000"
 

docs/authz.md

@@ -0,0 +1,158 @@
+# Authorization plugin
+
+Although TDS is part of the Bento platform, it is meant to be reusable in other software stacks.
+
+Since authorization requirements and technology vary wildy across different projects, 
+TDS allows adopters to write their own authorization logic in python.
+
+For Bento, we rely on API calls to a custom authorization service, 
+see [etc/bento.authz.module.py](./etc/bento.authz.module.py) for an example.
+
+For different authorization requirements, you could choose to write a custom module that performs authorization checks based on:
+* An API key in the request header or in a cookie
+* A JWT bearer token, for example you could:
+  * Allow/Deny simply based on the token's validity (decode + TTL)
+  * Allow/Deny based on the presence of a scope in the token
+  * Allow/Deny based on the presence of a group membership claim
+* The results of API calls to an authorization service
+* Policy engine evaluations, like OPA or Casbin
+
+![AuthZ plugin overview](./authz-plugin.png)
+
+## Implementing an authorization plugin
+
+When starting the TDS container, the FastAPI server will attempt to dynamicaly load the authorization plugin 
+middleware from `lib/authz.module.py`.
+
+If authorization is enabled and there is no file at `lib/authz.module.py`, an exception will be thrown and the server
+will not start.
+
+Furthermore, the content of the file must follow some implementation guidelines:
+- You MUST declare a concrete class that extends [BaseAuthzMiddleware](./transcriptomics_data_service/authz/middleware_base.py)
+- In that class, you MUST implement the required functions from `BaseAuthzMiddleware`:
+- Finally, the script should expose an instance of your concrete authz middleware, named `authz_middleware`.
+
+Looking at [bento.authz.module.py](./etc/bento.authz.module.py), we can see an implementation that is specific to 
+Bento's authorization service and libraries.
+
+Rather than directly implementing the `attach`, `dispatch` and other authorization logic, we rely on the `bento-lib` 
+`FastApiAuthMiddleware`, which already provides a reusable authorization middleware for FastAPI.
+
+The only thing left to do is to implement the authorization check functions.
+
+The next sections cover the specific methods that can be implemented to perform authorization.
+
+### Dependency injection
+
+The authorization middleware plugin leverages [FastAPI's dependency injection mechanisms](https://fastapi.tiangolo.com/tutorial/dependencies/).
+
+This simple yet powerful pattern allows us to integrate custom parametrized authorization checks at different levels of the application.
+
+In FastAPI, dependencies can be injected at different levels:
+- App
+  - Affect ALL requests
+- Routers
+  - Only affect the router's requests
+- Endpoints
+  - Only affect the endpoint's requests
+
+The wiring of the dependency injection system is already in place, 
+adopters only need to implement the authorization checks as injectable dependencies in their implementation of `BaseAuthMiddleware`.
+
+### Lifecycle methods
+
+These methods determine the authz middleware's lifecycle behaviour. That is to say, how it attaches to the FastAPI app and how it handles
+the incoming requests.
+
+| Lifecycle methods | Requires Implementation | Description                                                                                 |
+| ----------------- | ----------------------- | ------------------------------------------------------------------------------------------- |
+| `dispatch`        | YES                     | Middleware dispatch executed for all requests. Handle  authorization errors/exceptions here |
+| `attach`          | NO                      | Attaches the middleware to the FastAPI app                                                  |
+| `mark_authz_done` | NO                      | Marks that the authz check on a request was performed for later handling in `dispatch`.     |
+
+The `dispatch` function is the most important among them, since it is responsible for handling the authorization exceptions raised by the
+authorization functions. A poor implementation could lead to broken access-control.
+
+While `attach` and `mark_authz_done` already have a default implementation, they can be overriden if needed.
+
+### App and router dependency methods
+
+These methods define dependencies that will be injected in the FastAPI app itself, or its routers, above the endpoints layer.
+Authorization checks can be performed at this level rather than at the endpoints level.
+
+This is useful for all-or-nothing authorization logic, such as API key authorization, 
+where the API key should be present in all requests.
+
+If you are performing authz checks using these methods, make sure to raise an exception on unauthorized requests.
+
+| App/router dependency methods  | Description                                                                                      |
+| ------------------------------ | ------------------------------------------------------------------------------------------------ |
+| `dep_app`                      | Returns a list of injectables that will be added as app dependencies, covering ALL paths         |
+| `dep_ingest_router`            | Returns a list of injectables for the ingest router, covers `/ingest` and `/normalize` endpoints |
+| `dep_expression_router`        | Returns a list of injectables for the expression router, covers `/expressions` endpoints         |
+| `dep_experiment_result_router` | Returns a list of injectables for the expression router, covers `/experiment` endpoints          |
+
+### Endpoints authorization methods
+
+These methods define dependencies that will be injected on specific endpoints.
+If an endpoint must be protected by authz, implement the authz check for the appropriate endpoint.
+
+Again, make sure to raise exceptions on unauthorized requests and that the `dispatch` method handles said exceptions correctly.
+
+| Endpoint authorization methods       | Description                                                                          |
+| ------------------------------------ | ------------------------------------------------------------------------------------ |
+| `dep_public_endpoint`                | Returns injectable authz functions for public endpoints (applied to `/service-info`) |
+| `dep_authz_ingest`                   | Returns injectable authz functions for the `/ingest` endpoint                        |
+| `dep_authz_normalize`                | Returns injectable authz functions for the `/normalize` endpoint                     |
+| `dep_authz_expressions_list`         | Returns injectable authz functions for the `/expressions` endpoint                   |
+| `dep_authz_delete_experiment_result` | Returns injectable authz functions for the `/experiment (DELETE)` endpoint           |
+| `dep_authz_get_experiment_result`    | Returns injectable authz functions for the `/experiment (GET)` endpoint              |
+
+## Using an authorization plugin
+
+When using the production image, the authz plugin must be mounted correclty on the container.
+Assuming you implemented an authz plugin at `~/custom_authz_lib/authz.module.py`, mount the host directory
+to the container's `/tds/lib` directory.
+
+```yaml
+services:
+  tds:
+    image: transcriptomics_data_service:latest
+    container_name: tds
+    # ...
+    volumes:
+      # Mount the directory containing authz.module.py, NOT the file itself
+      - ~/custom_authz_lib:/tds/lib
+
+  tds-db:
+    # ... Omitted for simplicity
+```
+
+## Providing extra configurations for a custom authorization plugin
+
+You can add custom settings for your authorization plugin.
+Following the API key authorization plugin [example](../etc/example.authz.module.py), 
+you will notice that the API key is not hard coded in a variable, but imported from the pydantic config.
+
+The TDS pydantic settings are configured to load a `.env` file from the authz plugin mount.
+After the `.env` is loaded, you can access the extra settings with: `config.model_extra.get(<lowercase .env var name>)`.
+
+In other scenarios, you could store any configuration values required for your authorization logic.
+
+## Defining additional python dependencies
+
+When implementing an authorization plugin, you may realize that the default python modules used in TDS are not enough
+for your needs.
+
+Maybe you want to use OPA's Python client to evaluate policies, or an in-house Python library your team made for this
+purpose.
+
+While the dependencies declared in [pyproject.toml](../pyproject.toml) are fixed for a given TDS release,
+you can still speficy extra dependencies to be installed when the container starts!
+
+To do so, add a `requirements.txt` file to the authz plugin mount.
+During the initialization of the container, these additional dependencies will be installed,
+allowing your plugin to use them.
+
+**Note: It is the plugin implementer's responsibility to ensure that these additional dependencies
+don't conflict with those in [pyproject.toml](../pyproject.toml)**

etc/authz-api-key-extra-env/authz.module.py

@@ -0,0 +1,93 @@
+from logging import Logger
+from typing import Annotated, Any, Awaitable, Callable, Coroutine, Sequence
+from fastapi import Depends, FastAPI, HTTPException, Header, Request, Response, status
+from fastapi.responses import JSONResponse
+
+from transcriptomics_data_service.authz.middleware_base import BaseAuthzMiddleware
+from transcriptomics_data_service.config import Config, get_config
+from transcriptomics_data_service.logger import get_logger
+
+config = get_config()
+logger = get_logger(config)
+
+
+"""
+CUSTOM PLUGIN CONFIGURATION
+Extra configurations can be added to the config object by adding
+a '.env' file in the plugin mount directory.
+Variables placed there will be loaded as lowercase properties
+
+This variable's value can be accessed with: config.api_key
+API_KEY="fake-super-secret-api-key"
+"""
+
+
+class ApiKeyAuthzMiddleware(BaseAuthzMiddleware):
+    """
+    Concrete implementation of BaseAuthzMiddleware to authorize requests based on the provided API key.
+    """
+
+    def __init__(self, config: Config, logger: Logger) -> None:
+        super().__init__()
+        self.enabled = config.bento_authz_enabled
+        self.logger = logger
+
+        # Load the api_key from the config's extras
+        self.api_key = config.model_extra.get("api_key")
+        if self.api_key is None:
+            # prevents the server from starting if misconfigured
+            raise ValueError("Expected variable 'API_KEY' is not set in the plugin's .env")
+
+    # Middleware lifecycle
+
+    def attach(self, app: FastAPI):
+        app.middleware("http")(self.dispatch)
+
+    async def dispatch(
+        self, request: Request, call_next: Callable[[Request], Awaitable[Response]]
+    ) -> Coroutine[Any, Any, Response]:
+        if not self.enabled:
+            return await call_next(request)
+
+        # Request was not checked for authz yet
+        try:
+            res = await call_next(request)
+        except HTTPException as e:
+            # Catch exceptions raised by authz functions
+            self.logger.error(e)
+            return JSONResponse(status_code=e.status_code, content=e.detail)
+
+        return res
+
+    # API KEY authorization
+
+    def _dep_check_api_key(self):
+        """
+        Dependency injection for the API key authorization.
+        The inner function checks the x_api_key header to validate the API key.
+        Raises an exception that should be caught and handled in the dispatch func.
+        """
+
+        async def _inner(x_api_key: Annotated[str, Header()]):
+            if x_api_key != self.api_key:
+                raise HTTPException(status_code=403, detail="Unauthorized: invalid API key")
+
+        return Depends(_inner)
+
+    def dep_ingest_router(self) -> Sequence[Depends]:
+        # Require API key check on the ingest router
+        return [self._dep_check_api_key()]
+
+    def dep_expression_router(self) -> Sequence[Depends]:
+        # Require API key check on the expressions router
+        return [self._dep_check_api_key()]
+
+    def dep_experiment_result_router(self) -> Sequence[Depends]:
+        # Require API key check on the experiment_result router
+        return [self._dep_check_api_key()]
+
+    # NOTE: With an all-or-nothing authz mechanism like an API key,
+    # we can place the authz checks at the router level to have a more concise module.
+
+
+authz_middleware = ApiKeyAuthzMiddleware(config, logger)

etc/authz-extra-dep/authz.module.py

@@ -0,0 +1,97 @@
+from logging import Logger
+from typing import Annotated, Any, Awaitable, Callable, Coroutine
+from fastapi import Depends, FastAPI, HTTPException, Header, Request, Response
+from fastapi.responses import JSONResponse
+
+from transcriptomics_data_service.authz.middleware_base import BaseAuthzMiddleware
+from transcriptomics_data_service.config import Config, get_config
+from transcriptomics_data_service.logger import get_logger
+
+config = get_config()
+logger = get_logger(config)
+
+
+"""
+CUSTOM PLUGIN DEPENDENCY
+Extra dependencies can be added if the authz plugin requires them.
+In this example, the authz module imports the OPA agent.
+Since OPA does not ship with TDS, a requirements.txt file must be placed under 'lib'.
+"""
+
+from opa_client.opa import OpaClient
+
+
+class ApiKeyAuthzMiddleware(BaseAuthzMiddleware):
+    """
+    Concrete implementation of BaseAuthzMiddleware to authorize requests based on the provided API key.
+    """
+
+    def __init__(self, config: Config, logger: Logger) -> None:
+        super().__init__()
+        self.enabled = config.bento_authz_enabled
+        self.logger = logger
+
+        # Get custom OPA configs from lib/.env
+        opa_host = config.model_extra.get("opa_host")
+        opa_port = int(config.model_extra.get("opa_host_port"))
+
+        # Init the OPA client with the server
+        self.opa_client = OpaClient(host=opa_host, port=opa_port)
+        try:
+            # Commented out as this is not pointing to a real OPA server
+            # self.logger.info(self.opa_client.check_connection())
+            pass
+        except:
+            raise Exception("Could not establish connection to the OPA service.")
+
+    # Middleware lifecycle
+
+    def attach(self, app: FastAPI):
+        app.middleware("http")(self.dispatch)
+
+    async def dispatch(
+        self, request: Request, call_next: Callable[[Request], Awaitable[Response]]
+    ) -> Coroutine[Any, Any, Response]:
+        if not self.enabled:
+            return await call_next(request)
+
+        try:
+            res = await call_next(request)
+        except HTTPException as e:
+            # Catch exceptions raised by authz functions
+            self.logger.error(e)
+            return JSONResponse(status_code=e.status_code, content=e.detail)
+
+        return res
+
+    # OPA authorization
+    def _dep_check_opa(self):
+        async def inner(request: Request):
+            # Check the permission using the OPA client.
+            # We assume true for the sake of the demonstration
+            # authz_result = await self.opa_client.check_permission()
+            authz_result = True
+            if not authz_result:
+                raise HTTPException(status_code=403, detail="Unauthorized: policy evaluation failed")
+
+        return Depends(inner)
+
+    # Authz logic: only check for valid API key
+
+    def dep_authz_ingest(self):
+        return [self._dep_check_opa()]
+
+    def dep_authz_normalize(self):
+        return [self._dep_check_opa()]
+
+    def dep_authz_delete_experiment_result(self):
+        return [self._dep_check_opa()]
+
+    def dep_authz_expressions_list(self):
+        return [self._dep_check_opa()]
+
+    def dep_authz_get_experiment_result(self):
+        return [self._dep_check_opa()]
+
+
+authz_middleware = ApiKeyAuthzMiddleware(config, logger)