Fuzz the executor against a real Postgres, and record what it found

CHANGELOG.md

@@ -227,6 +227,15 @@ what shipped rather than what was written down at the time.
 
 ### Internal
 
+- `tools/fuzz` generates random `SELECT`s and compares sqlglot's executor against a
+  real PostgreSQL, minimising every divergence to the smallest query that still
+  shows it. The tripwires above were written by hand from bugs that happened to be
+  noticed; this looks for the ones nobody tripped over, and found fourteen on its
+  first run — six of them wrong answers rather than refusals, including
+  `count(DISTINCT x)` ignoring the DISTINCT and an outer join's NULL-padded row
+  surviving a `WHERE` that tests the padded column. Not part of any CI job: it needs
+  a live server, and a suite that fails on a new random seed is one people delete.
+  (#122)
 - `SessionState` is extracted and shared by the middleware and the session, so
   there is one place where per-connection state lives. (#61)
 - Strict-xfail tripwires assert what real Postgres answers for each sqlglot bug

docs/limitations.md

@@ -42,6 +42,15 @@
   Postgres makes it, so `select 3000000000` describes as `int8` rather than telling a binary client `int4`
   and crashing its decoder.
 
+  **Text is compared by code point, not by collation.** The executor compares strings the way Python
+  does, so `'abc' < 'ABC'` is false and `ORDER BY name` sorts every capital ahead of every lower-case
+  letter. A real PostgreSQL compares by its database collation, and on the usual locale-derived one
+  (`en_US.UTF-8`) both of those come out the other way round. This is a modelling gap rather than a
+  bug with a fix: the executor has no collation to consult, and Python's `str` ordering is the only
+  total order available to it. It matches a database created with `--locale=C` exactly, and differs
+  from any other on mixed-case or non-ASCII text. If your test asserts on the *order* of text rows,
+  that is a reason to want a real Postgres.
+
   `TABLESAMPLE` is refused outright, because the executor ignores it and nothing here can repair that.
   An `OFFSET` or `DISTINCT ON` nested inside a subquery is refused for the same reason: the
   repair reaches the query's own rows, and one buried in a subquery would be silently ignored as before.

tests/test_sqlglot_workarounds.py

@@ -23,6 +23,13 @@
 https://github.com/jbylund/pg_mimic/issues/49, and the ones reaching users
 untreated are https://github.com/jbylund/pg_mimic/issues/58
 
+The tests from `test_distinct_keeps_columns_that_share_a_name` down were found by
+`tools/fuzz`, which generates random SELECTs and compares the executor against a
+real PostgreSQL. Reading them as a group is the honest summary of how far the
+executor is from Postgres: it is a good query engine for the shapes sqlglot's own
+tests cover, and every one of these is a wrong answer or a refusal for SQL that a
+client has every reason to send.
+
 A failure here is never a pg_mimic regression: it means good news upstream.
 """
 
@@ -84,6 +91,10 @@ def test_descending_order_places_nulls_instead_of_raising():
     Workaround: `_rewrite_null_ordering` in tables.py.
 
     Ascending coincidentally matches Postgres; descending raises TypeError.
+
+    Fixed upstream but in no release, so the mark comes off when the floor moves:
+    https://github.com/jbylund/pg_mimic/issues/109 is the checklist. The daily
+    `Upstream sqlglot` job is already red on its `main` leg for this.
     """
     tables, schema = {"n": [{"a": 1}, {"a": None}]}, {"n": {"a": "INT"}}
     assert _rows("SELECT a FROM n ORDER BY a DESC", tables, schema) == [(None,), (1,)]
@@ -334,3 +345,227 @@ def test_full_outer_join_preserves_unmatched_rows():
     schema = {"x": {"a": "INT"}, "y": {"b": "INT"}}
     rows = _rows("SELECT x.a, y.b FROM x FULL OUTER JOIN y ON x.a = y.b", tables, schema)
     assert sorted(rows, key=str) == [(1, None), (2, 2), (None, 3)]
+
+
+# Everything below was found by tools/fuzz. See the module docstring.
+
+# A NULL in here on purpose: without one, count(DISTINCT a) and count(*) are the
+# same number, and the test cannot tell which of the two the executor answered.
+_DUPLICATES = ({"t": [{"a": 1}, {"a": 1}, {"a": 2}, {"a": None}]}, {"t": {"a": "INT"}})
+_MIXED = ({"m": [{"n": Decimal("2.5"), "v": 2.0}]}, {"m": {"n": "DECIMAL", "v": "DOUBLE"}})
+
+
+@pytest.mark.xfail(strict=True, reason=_UPSTREAM_FIXED)
+def test_distinct_keeps_columns_that_share_a_name():
+    """https://github.com/jbylund/pg_mimic/issues/58
+
+    Two output columns with the same name are silently merged into one, so the
+    query answers a *narrower* result than it was asked for -- and with DISTINCT
+    over a join, a shorter one too: nine rows of two columns become three of one.
+    Dropping a column the client's Describe already promised is worse than an
+    error, and no workaround in tables.py would catch it, since nothing peeks at
+    result rows.
+
+    `SELECT a, a` without DISTINCT is fine, which is what makes this a DISTINCT bug
+    rather than a projection one.
+    """
+    assert _rows("SELECT DISTINCT a, a FROM t", *_NUMBERS) == [(1, 1), (2, 2), (3, 3)]
+    rows = _rows("SELECT DISTINCT x.a, y.a FROM t AS x CROSS JOIN t AS y", *_NUMBERS)
+    assert len(rows) == 9
+
+
+@pytest.mark.xfail(strict=True, reason=_UPSTREAM_FIXED)
+def test_star_expands_columns_that_share_a_name():
+    """https://github.com/jbylund/pg_mimic/issues/58
+
+    The same merge-by-name as above, reached through `*` instead of DISTINCT, and
+    the reason `WITH c AS (SELECT 1, 1 FROM t) SELECT * FROM c` answers one column:
+    both are named `_col_0`. A CTE or derived table whose columns are expressions
+    rather than plain column references is the common way to hit it.
+    """
+    assert _rows("WITH c AS (SELECT a, a FROM t) SELECT * FROM c", *_NUMBERS) == [(1, 1), (2, 2), (3, 3)]
+
+
+@pytest.mark.xfail(strict=True, reason=_UPSTREAM_FIXED)
+def test_distinct_inside_an_aggregate_is_honored():
+    """https://github.com/jbylund/pg_mimic/issues/58
+
+    Every `agg(DISTINCT x)` is broken, not just `count`, and the cause is one line
+    of codegen: the Python generator renders `exp.Distinct` as `set()`, discarding
+    the expression inside it, so the aggregate is handed an empty set literal
+    instead of the column. `COUNT(DISTINCT a)` compiles to `COUNT(set())`.
+
+    `count` then answers `count(*)` -- 4 here, so both the DISTINCT *and* the
+    NULL-skipping that plain `count(a)` gets right are lost -- while `sum` and
+    `avg` raise on the empty set. `avg(DISTINCT a)` is 1.5 in Postgres and is left
+    out of the assertions only to keep them integer-exact.
+
+    The parser is fine: the `Distinct` node survives in the AST, in every dialect.
+    And the executor is wrong in every dialect, not only `postgres`, while
+    single-argument `count(DISTINCT x)` is SQL-92 and agrees exactly across
+    PostgreSQL 18.4, SQLite 3.51 and MySQL 8.0.46 -- so a fix upstream is not
+    dialect-gated and cannot break a dialect that wanted the old behaviour.
+    """
+    assert _rows("SELECT COUNT(a) FROM t", *_DUPLICATES) == [(3,)]  # correct today; the contrast
+    assert _rows("SELECT COUNT(DISTINCT a) FROM t", *_DUPLICATES) == [(2,)]
+    assert _rows("SELECT COUNT(DISTINCT 1) FROM t", *_DUPLICATES) == [(1,)]
+    assert _rows("SELECT SUM(DISTINCT a) FROM t", *_DUPLICATES) == [(3,)]
+
+
+@pytest.mark.xfail(strict=True, reason=_UPSTREAM_FIXED)
+def test_casting_to_an_integer_rounds_rather_than_truncates():
+    """https://github.com/jbylund/pg_mimic/issues/58
+
+    `env.cast` sends the value through `int()`, which truncates toward zero, where
+    Postgres rounds -- and by the same two rules as `round()`: an exact type goes
+    half away from zero, a binary float goes through rint() and lands half to even.
+    Verified against PostgreSQL 18.
+
+    So `CAST(0.5 AS INT)` is 1 rather than 0, and `CAST(-1.5::double AS INT)` is -2
+    rather than -1. Off-by-one in a value, silently.
+    """
+    exact = {"d": [{"n": Decimal(text)} for text in ("0.5", "-0.5", "2.5")]}
+    assert _rows("SELECT CAST(n AS INT) FROM d", exact, {"d": {"n": "DECIMAL"}}) == [(1,), (-1,), (3,)]
+    binary = {"d": [{"v": 0.5}, {"v": 2.5}, {"v": -1.5}]}
+    assert _rows("SELECT CAST(v AS INT) FROM d", binary, {"d": {"v": "DOUBLE"}}) == [(0,), (2,), (-2,)]
+
+
+@pytest.mark.xfail(strict=True, reason=_UPSTREAM_FIXED)
+def test_round_to_a_scale_goes_half_away_from_zero():
+    """https://github.com/jbylund/pg_mimic/issues/58
+
+    The two-argument `round()`, split from
+    `test_round_on_numeric_goes_half_away_from_zero` above because it is a separate
+    code path with a separate symptom: ENV["ROUND"] passes the scale to Python's
+    builtin, so 0.25 to one place is 0.2 where Postgres says 0.3.
+    """
+    assert _rows("SELECT ROUND(n, 1) FROM d", {"d": [{"n": Decimal("0.25")}]}, {"d": {"n": "DECIMAL"}}) == [(Decimal("0.3"),)]
+
+
+@pytest.mark.xfail(strict=True, reason=_UPSTREAM_FIXED)
+def test_like_honors_a_backslash_escape():
+    """https://github.com/jbylund/pg_mimic/issues/58
+
+    Postgres' LIKE takes backslash as its default escape character, so
+    `LIKE 'a\\%b'` matches the literal text `a%b` and nothing else. The executor
+    treats the backslash as a literal, matches nothing, and answers no rows.
+
+    Distinct from the anchoring and metacharacter bugs fixed in v30.17.0 and
+    covered by test_like_is_anchored_and_takes_metacharacters_literally: escaping
+    is the part that still differs.
+    """
+    tables, schema = {"q": [{"s": "a%b"}, {"s": "axb"}]}, {"q": {"s": "TEXT"}}
+    assert _rows("SELECT s FROM q WHERE s LIKE 'a\\%b'", tables, schema) == [("a%b",)]
+
+
+@pytest.mark.xfail(strict=True, reason=_UPSTREAM_FIXED)
+def test_exists_subquery_runs():
+    """https://github.com/jbylund/pg_mimic/issues/58
+
+    `EXISTS (subquery)` emits invalid Python, in a WHERE clause or a select list,
+    correlated or not -- the client gets a SyntaxError out of generated code it
+    never wrote. The same failure mode as the scalar subquery above and probably
+    the same cause, but its own tripwire because EXISTS is the more common of the
+    two by far: it is how an ORM asks whether a related row exists, and how
+    `NOT IN (subquery)` is rewritten before it runs.
+    """
+    tables, schema = {"t": [{"a": 1}], "u": [{"b": 1}]}, {"t": {"a": "INT"}, "u": {"b": "INT"}}
+    assert _rows("SELECT a FROM t WHERE EXISTS (SELECT 1 FROM u)", tables, schema) == [(1,)]
+
+
+@pytest.mark.xfail(strict=True, reason=_UPSTREAM_FIXED)
+def test_arithmetic_mixes_numeric_and_double():
+    """https://github.com/jbylund/pg_mimic/issues/58
+
+    A NUMERIC column against a DOUBLE one -- or against a float literal -- raises
+    `unsupported operand type(s) for *: 'decimal.Decimal' and 'float'`, because the
+    executor does Python arithmetic on the storage types and Python refuses that
+    pair. Postgres resolves both to double precision and answers 5.
+
+    The one on this list most likely to be the first thing a user hits, since it
+    needs no subquery, no join and no unusual function: two numeric columns of
+    different types multiplied together is enough.
+    """
+    assert _rows("SELECT n * v FROM m", *_MIXED) == [(5.0,)]
+
+
+@pytest.mark.xfail(strict=True, reason=_UPSTREAM_FIXED)
+def test_in_a_subquery_is_null_for_a_null_left_operand():
+    """https://github.com/jbylund/pg_mimic/issues/58
+
+    `NULL IN (SELECT ...)` is NULL in Postgres, not FALSE -- three-valued logic,
+    and the reason `NOT IN (subquery)` excludes a NULL row rather than keeping it.
+    The executor answers FALSE, so the negation keeps it, and a `NOT IN` against a
+    nullable column returns rows Postgres would not.
+    """
+    tables, schema = {"t": [{"a": 1}], "u": [{"b": 1}]}, {"t": {"a": "INT"}, "u": {"b": "INT"}}
+    assert _rows("SELECT NULL IN (SELECT b FROM u) FROM t", tables, schema) == [(None,)]
+
+
+@pytest.mark.xfail(strict=True, reason=_UPSTREAM_FIXED)
+def test_an_outer_join_pads_rows_a_where_clause_then_removes():
+    """https://github.com/jbylund/pg_mimic/issues/58
+
+    The NULL-padded row an outer join adds survives a WHERE clause that tests the
+    padded column, so `WHERE x.a = 1` keeps a row whose `x.a` is NULL. Extra rows
+    out of a filter that should have removed them, which is the worst shape a
+    wrong answer can take: nothing about the result looks suspicious.
+    """
+    tables = {"x": [{"a": 1}, {"a": 2}], "y": [{"b": 9}]}
+    schema = {"x": {"a": "INT"}, "y": {"b": "INT"}}
+    assert _rows("SELECT x.a FROM x FULL JOIN y ON NULL WHERE x.a = 1", tables, schema) == [(1,)]
+
+
+@pytest.mark.xfail(strict=True, reason=_UPSTREAM_FIXED)
+def test_the_trim_and_reverse_functions_exist():
+    """https://github.com/jbylund/pg_mimic/issues/58
+
+    Missing from ENV, like `length()` was before v30.17.0, and fixable the same
+    way. Grouped into one test because they are one gap: whichever of them is
+    added first, the rest are a line each.
+
+    Worth a tripwire despite being a plain omission because these are not exotic --
+    `btrim` is what Postgres compiles a bare `trim(x)` to.
+    """
+    tables, schema = {"q": [{"s": "  ab  "}]}, {"q": {"s": "TEXT"}}
+    assert _rows("SELECT BTRIM(s), LTRIM(s), RTRIM(s), REVERSE(BTRIM(s)) FROM q", tables, schema) == [("ab", "ab  ", "  ab", "ba")]
+
+
+@pytest.mark.xfail(strict=True, reason=_UPSTREAM_FIXED)
+def test_extract_takes_a_day_of_week():
+    """https://github.com/jbylund/pg_mimic/issues/58
+
+    `EXTRACT(DOW FROM ts)` reaches `datetime.dow`, an attribute datetime does not
+    have, so it raises rather than answering. YEAR, MONTH, DAY and HOUR all work,
+    which is what makes this a per-field gap rather than a missing EXTRACT.
+
+    2024-03-15 is a Friday, and Postgres numbers Sunday 0, so 5.
+    """
+    tables, schema = {"d": [{"ts": datetime.datetime(2024, 3, 15)}]}, {"d": {"ts": "TIMESTAMP"}}
+    assert _rows("SELECT EXTRACT(DOW FROM ts) FROM d", tables, schema) == [(5,)]
+
+
+@pytest.mark.xfail(strict=True, reason=_UPSTREAM_FIXED)
+def test_dividing_null_is_null():
+    """https://github.com/jbylund/pg_mimic/issues/58
+
+    `NULL / 1` raises `int() argument must be a string, a bytes-like object or a
+    real number, not 'NoneType'`: integer division coerces with `int()` before
+    checking for NULL. Any nullable integer column divided by anything is enough,
+    and the other operators handle NULL correctly, so it is division specifically.
+    """
+    assert _rows("SELECT a / 1 FROM n", {"n": [{"a": None}]}, {"n": {"a": "INT"}}) == [(None,)]
+
+
+@pytest.mark.xfail(strict=True, reason=_UPSTREAM_FIXED)
+def test_is_not_null_answers_a_boolean():
+    """https://github.com/jbylund/pg_mimic/issues/58
+
+    `x IS NOT NULL` answers `x` itself when x is not null, rather than TRUE -- the
+    generated code is `x` where it should be `x is not None`. It reads as correct
+    for as long as the value happens to be truthy, and a column of strings in a
+    boolean-typed output column is a wire encoding error rather than a wrong row.
+
+    `IS NULL` is fine; only the negation is wrong.
+    """
+    assert _rows("SELECT (CASE WHEN a = 1 THEN 'x' END) IS NOT NULL FROM t", *_NUMBERS) == [(True,), (False,), (False,)]