Skip to content

Commit 3ca1b08

Browse files
authored
Merge pull request #12 from jqnatividad/tests-hermetic-env
test: keep .env out of the suite; report the real interpreter in notebooks
2 parents 9aaa067 + dd50c72 commit 3ca1b08

9 files changed

Lines changed: 63 additions & 22 deletions

File tree

‎.github/workflows/ci.yml‎

Lines changed: 3 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -46,7 +46,9 @@ jobs:
4646
# No env vars and no secrets: the suite is hermetic. Every outbound call
4747
# is mocked, storage is redirected at a temp dir by tests/conftest.py,
4848
# and the notebook-verification tests get their Jupyter kernel from the
49-
# venv's own kernelspec (ipykernel is a dependency).
49+
# venv's own kernelspec (ipykernel is a dependency). conftest.py also
50+
# sets DATA_CONCIERGE_ENV_FILE="" so a local .env — which core/config.py
51+
# reads for every field — cannot change what the suite asserts.
5052
#
5153
# Do not add `working-directory` here — mcp/registry.py resolves its
5254
# config relative to the process cwd, and four tests read it.

‎.serena/memories/suggested_commands.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -20,7 +20,7 @@ Two traps:
2020
- **Always run from the repo root.** `mcp/registry.py` sets `MCP_CONFIG_DIR = Path.cwd() / "configs"` at module scope, and several tests read it. This is why `.github/workflows/ci.yml` carries a comment forbidding `working-directory`.
2121
- **`addopts` in `pyproject.toml` includes `--cov=src/data_concierge --cov-report=term-missing`**, so a bare `pytest` is slower than CI and prints a coverage table. Pass `--no-cov` to match CI.
2222

23-
The suite is hermetic: no API key, no network, no `.env`. `asyncio_mode = "auto"`, so `async def` tests need no marker. Notebook-verification tests really execute notebooks (the venv's own ipykernel), so a full run is slower than a typical unit suite.
23+
The suite is hermetic: no API key, no network, no `.env`. It actively ignores `.env` — `core/config.py` reads that file for every setting, so `tests/conftest.py` sets `DATA_CONCIERGE_ENV_FILE=""` before anything imports the `settings` singleton. Assert shipped defaults via `Settings.model_fields["x"].default`, never the live `settings`, which still reflects the process environment (a shell `FOO=bar pytest` bypasses the conftest guard). `asyncio_mode = "auto"`, so `async def` tests need no marker. Notebook-verification tests really execute notebooks (the venv's own ipykernel), so a full run is slower than a typical unit suite.
2424

2525
## Lint / format / types
2626
```bash

‎CONTRIBUTING.md‎

Lines changed: 6 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -13,9 +13,13 @@ pytest
1313

1414
The full local setup — configuration, sign-in, Docker — is in the [README](README.md).
1515

16-
**The test suite is hermetic.** ~640 tests, under a minute, and it needs no API key, no
16+
**The test suite is hermetic.** ~800 tests, under a minute, and it needs no API key, no
1717
network and no `.env`: every outbound call is mocked, and a fixture points storage at a
18-
temporary directory so a run never writes into your checkout. If a change you make can only be
18+
temporary directory so a run never writes into your checkout. It also *ignores* your `.env` —
19+
`core/config.py` reads that file for every setting, so `tests/conftest.py` sets
20+
`DATA_CONCIERGE_ENV_FILE=""` to keep your local configuration out of the run. Assert shipped
21+
defaults against `Settings.model_fields[...]`, not the live `settings` singleton, which still
22+
reflects the process environment. If a change you make can only be
1923
tested with a live key or a running service, the test is at the wrong boundary — mock the
2024
boundary, or make it a manual script.
2125

‎README.md‎

Lines changed: 5 additions & 9 deletions
Original file line numberDiff line numberDiff line change
@@ -260,9 +260,8 @@ All settings come from `.env` (see [`.env.example`](.env.example)). **Only
260260
Notebook verification executes generated code. It's contained: a subprocess with a hard
261261
timeout, a minimal environment allowlist that withholds every credential, shell-escape cells
262262
skipped, and an egress guard blocking loopback, private, link-local and cloud-metadata
263-
addresses from inside the kernel. To turn it off, pass `NOTEBOOK_VERIFICATION_ENABLED=false`
264-
in the environment for that run rather than adding it to `.env` — see *Notebook verification
265-
never completes* under Troubleshooting for why.
263+
addresses from inside the kernel. Set `NOTEBOOK_VERIFICATION_ENABLED=false` to turn it off —
264+
in `.env` for a persistent local default, or on the command line for a single run.
266265

267266
</details>
268267

@@ -444,12 +443,9 @@ depending on the notebook. To skip it, pass the flag for that run:
444443
NOTEBOOK_VERIFICATION_ENABLED=false ./run_web.sh
445444
```
446445

447-
Pass it in the environment, not in `.env`. `core/config.py` sets `env_file=".env"`, and the
448-
test suite reads the same settings singleton — so a `.env` entry also disables verification
449-
under `pytest` and fails
450-
`tests/unit/test_notebook_verifier.py::TestFeatureFlag::test_ships_enabled`, the test that
451-
pins the shipped default to on. A shell variable applies only to the command you prefix,
452-
which is what you want here.
446+
Or set it in `.env` to make it your local default. Either is safe: `tests/conftest.py` sets
447+
`DATA_CONCIERGE_ENV_FILE=""` before anything imports the settings singleton, so the test suite
448+
ignores your `.env` and a local override cannot turn it red.
453449

454450
</details>
455451

‎src/data_concierge/agents/notebook_generator.py‎

Lines changed: 6 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -11,6 +11,7 @@
1111
"""
1212

1313
import json
14+
import platform
1415
from datetime import datetime
1516
from typing import Any
1617

@@ -300,7 +301,10 @@ async def process(self, state: GraphState) -> GraphState:
300301
},
301302
"language_info": {
302303
"name": "python",
303-
"version": "3.11",
304+
# The interpreter that actually ran this, not a hardcoded one:
305+
# the notebook is the deliverable and its provenance has to be
306+
# true. A pinned "3.11" outlived support for 3.11 entirely.
307+
"version": platform.python_version(),
304308
"codemirror_mode": {"name": "ipython", "version": 3},
305309
"file_extension": ".py",
306310
"mimetype": "text/x-python",
@@ -1287,7 +1291,7 @@ def generate_notebook_from_trace(
12871291
},
12881292
"language_info": {
12891293
"name": "python",
1290-
"version": "3.11",
1294+
"version": platform.python_version(),
12911295
},
12921296
"data_concierge": {
12931297
"version": "0.1.0",

‎src/data_concierge/core/config.py‎

Lines changed: 7 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,6 @@
11
"""Application configuration using Pydantic Settings."""
22

3+
import os
34
from functools import lru_cache
45
from typing import Literal
56

@@ -11,7 +12,12 @@ class Settings(BaseSettings):
1112
"""Application settings loaded from environment variables."""
1213

1314
model_config = SettingsConfigDict(
14-
env_file=".env",
15+
# `.env` feeds every field here, so it reaches a test run too: the suite
16+
# is otherwise only as hermetic as whatever happens to be in the working
17+
# copy's .env, and a developer following the README's own advice to
18+
# disable notebook verification turned the suite red. tests/conftest.py
19+
# sets this to "" so a run never depends on it. Empty means no file.
20+
env_file=os.environ.get("DATA_CONCIERGE_ENV_FILE", ".env") or None,
1521
env_file_encoding="utf-8",
1622
case_sensitive=False,
1723
extra="ignore",

‎tests/conftest.py‎

Lines changed: 9 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -14,8 +14,17 @@
1414
original.
1515
"""
1616

17+
import os
18+
1719
import pytest
1820

21+
# Set before anything imports `data_concierge.core.config`, which builds its
22+
# `settings` singleton at import time. `.env` feeds every field, so without this
23+
# the suite inherits whatever a developer happens to have configured locally —
24+
# which is how a documented `NOTEBOOK_VERIFICATION_ENABLED=false` turned the
25+
# suite red. Empty means "no env file"; export it to a path to opt back in.
26+
os.environ.setdefault("DATA_CONCIERGE_ENV_FILE", "")
27+
1928

2029
@pytest.fixture(autouse=True, scope="session")
2130
def _isolate_storage_from_the_repo(tmp_path_factory):

‎tests/unit/test_mcp_guards.py‎

Lines changed: 17 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -68,9 +68,15 @@ def test_metadata_refused_even_with_private_allowed(self) -> None:
6868
validate_server_url("http://169.254.169.254/", allow_private=True)
6969

7070
def test_ships_locked_down(self) -> None:
71-
from data_concierge.core.config import settings
71+
"""Asserts the declared default, not the live singleton.
7272
73-
assert settings.mcp_allow_private_urls is False
73+
`MCP_ALLOW_PRIVATE_URLS` exists precisely so local development can point
74+
at localhost MCP servers, so the one developer most likely to run this
75+
suite is the one most likely to have it set.
76+
"""
77+
from data_concierge.core.config import Settings
78+
79+
assert Settings.model_fields["mcp_allow_private_urls"].default is False
7480

7581

7682
class TestSseEndpointPinning:
@@ -133,13 +139,21 @@ def test_model_validator_rejects_metadata_url(self) -> None:
133139
url="http://169.254.169.254/latest/meta-data/",
134140
)
135141

136-
def test_model_validator_rejects_on_update_construction(self) -> None:
142+
def test_model_validator_rejects_on_update_construction(
143+
self, monkeypatch: "pytest.MonkeyPatch"
144+
) -> None:
137145
"""The update path rebuilds the model, so it is covered too."""
138146
import pytest
139147
from pydantic import ValidationError
140148

149+
from data_concierge.core import config
141150
from data_concierge.mcp.models import MCPServerConfig, MCPTransportType
142151

152+
# The validator consults the live setting, and MCP_ALLOW_PRIVATE_URLS
153+
# exists for local development — pin the default posture so an exported
154+
# variable cannot quietly turn this guard test into a failure.
155+
monkeypatch.setattr(config.settings, "mcp_allow_private_urls", False)
156+
143157
good = MCPServerConfig(
144158
id="x", name="x", transport=MCPTransportType.STREAMABLE_HTTP,
145159
url="https://example.com/mcp",

‎tests/unit/test_notebook_verifier.py‎

Lines changed: 9 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -206,10 +206,16 @@ def _score():
206206
)
207207

208208
def test_ships_enabled(self) -> None:
209-
"""Verification is ON (#131) — the egress guard makes it safe."""
210-
from data_concierge.core.config import settings
209+
"""Verification is ON (#131) — the egress guard makes it safe.
211210
212-
assert settings.notebook_verification_enabled is True
211+
Reads the declared default rather than the live `settings` singleton.
212+
conftest already keeps `.env` out of a test run, but the singleton also
213+
reflects the process environment, so asserting the class default is what
214+
makes this independent of the machine it runs on.
215+
"""
216+
from data_concierge.core.config import Settings
217+
218+
assert Settings.model_fields["notebook_verification_enabled"].default is True
213219

214220
def test_disabled_omits_the_factor_entirely(self, monkeypatch: pytest.MonkeyPatch) -> None:
215221
"""When off, no perpetual 'still running' for a check that never runs.

0 commit comments

Comments
 (0)