State in the catalog the size of every string MEOS returns

Every public MEOS function returning a char * takes a size_t * through
which it writes the size of the string, its terminating null character
included, and the catalog lists that parameter in shape.outParams; the
new tests/test_string_size.py refuses a public char * return whose
shape.outParams lists no size_t *. The catalog's own readers of a string
writer set that size aside as an out-parameter: build_type_encodings
leaves it out of the inputs an encoder takes, as _trailing of
parser/codecs.py does, and a function returning a string is exposable,
its size no wire param and named by size_outparam on its result, which
the service reads through the new invoke_string, copying the string for
the size MEOS writes rather than scanning it for its terminator.

Witness: against MobilityDB master 8b0886986d, which writes the size,
the suite parsing its installed headers crashes in
test_each_hex_writer_round_trips_through_encode, which calls the text
writers without their size, and fails two tests reading typeEncodings,
where TimestampTz, Timestamp, TimeADT, DateADT and TimeTzADT write
through no function and TSequence is missing; master's parser refuses
each of the 140 public functions returning a string with
array-or-out-param:size_out.

Why: the owner's decision of 2026-11-02 (IO-S2 of the input and output
plan, MobilityDB #3353), that no binding scan a string MEOS returns for
its terminator. The catalog is where every binding reads which parameter
holds that size, so it states it for every string and a test keeps it
stated.

Measured: over the installed headers of MobilityDB master 3dd98cc90e
built with -DALL=ON, the 144 public char * returns all list a size_t *
in shape.outParams, and 3641 functions are stateless-exposable, where
master's parser states 3360, 140 of them returning a string. Against
the catalog of MobilityDB a76d98f21a, before the size was written, the
contract test of tests/test_string_size.py fails on 118 functions; the
new test of tests/test_enrich.py on build_type_encodings fails without
its change. The suite passes 665 tests, the two of
tests/test_object_model_parity.py needing PyMEOS skipped.

Author: estebanzimanyi

docs/enrichment.md

@@ -82,7 +82,11 @@ literals are in the catalog.
   **encoder** — a public `T (+ aux) → char *` (`*_out`, `*_as_text`, `*_as_ewkt`,
   `*_as_mfjson`, `*_as_hexwkb`, `*_as_hexewkb`), or `T (+ aux) → uint8_t *` in `wkb` and
   `ewkb`. A size the function writes back is an out-parameter, stated in its
-  `shape.outParams` as for any other function.
+  `shape.outParams` as for any other function. Every public function returning a
+  `char *` writes the size of its string, the terminating null character included,
+  through a `size_t *` its `shape.outParams` lists, so a binding copies the string
+  with that size rather than scanning it for its terminator
+  (`tests/test_string_size.py`).
 - **A PostgreSQL type passed by value** (`TimestampTz`, `Timestamp`, `TimeADT`,
   `DateADT`) is a class like a pointer type, its decoder returning it and its
   encoder taking it without a pointer (`timestamptz_in` / `timestamptz_out`).
@@ -229,6 +233,13 @@ Otherwise `exposable` is `false` and `reason` lists the blockers
 > maps to *no value* (HTTP 204). This recovers the public `*_value_n`,
 > box-bound, and `*_value_at`/geo-accessor families.
 
+> **String results.** A function `char *f(.., size_t *size_out)` whose
+> `shape.outParams` lists its trailing `size_t *` writes through it the size
+> of the string it returns, the terminating null character included. The
+> size is no wire param: the result is the JSON string, with
+> `size_outparam` naming the size the string is read for (`tfloat_out`,
+> `geo_as_text`, `geom_relate`).
+
 > **Input-array builders.** A builder taking `(Elem **arr, int count)`
 > becomes one wire param of `kind: "array"` (element = the serialized
 > `Elem`); the `count` is implicit (the JSON array length). This recovers

parser/enrich.py

@@ -263,13 +263,15 @@ def build_type_encodings(functions: list, structs: set,
     first arg a char-like string). An *encoder* turns an object into a wire
     string (returns ``char *``, first arg ``const struct T *``). Trailing
     *formatting* scalars are allowed and defaulted (see ``_aux_specs``); a
-    non-defaultable trailing arg disqualifies the helper. Only declared
-    structs qualify, so primitives never register by accident.
+    non-defaultable trailing arg disqualifies the helper, an out-parameter the
+    function's ``shape.outParams`` lists, as the ``size_t *size_out`` of a string
+    writer, being no input. Only declared structs qualify, so primitives never
+    register by accident.
 
     A name in ``values`` is a type passed by value, as PostgreSQL passes its
     ``TimestampTz`` and ``DateADT``: its decoder returns it and its encoder takes it
     without a pointer (``TimestampTz timestamptz_in(const char *, int32_t)``,
-    ``char *timestamptz_out(TimestampTz)``).
+    ``char *timestamptz_out(TimestampTz, size_t *)``).
     """
     enc: dict[str, dict] = {}
 
@@ -286,7 +288,11 @@ def slot(b: str) -> dict:
         p0 = params[0]["canonical"]
         rb, rd = _base(ret), _ptr_depth(ret)
         pb, pd = _base(p0), _ptr_depth(p0)
-        aux = _aux_specs(params[1:])     # None => non-defaultable trailing arg
+        # None => non-defaultable trailing arg; the out-parameters, a string
+        # writer's size_out among them, are set aside as #_trailing of
+        # parser/codecs.py sets them aside
+        out = set((fn.get("shape") or {}).get("outParams") or ())
+        aux = _aux_specs([p for p in params[1:] if p["name"] not in out])
 
         # Decoder: const char* (+ defaultable scalar aux) -> opaque struct
         if (aux is not None and (rd >= 1 or rb in values) and rb in structs
@@ -415,6 +421,24 @@ def classify_category(fn: dict) -> str:
 # human-authored tag and cannot drift.
 
 
+def _string_size(fn: dict):
+    """The name of the ``size_t *`` through which ``char *f(.., size_t *size_out)``
+    writes the size of the string it returns, its terminating null character
+    included, when its ``shape.outParams`` lists it as its last parameter; else None.
+    The size is set aside as the presence flag of #_outparam is: no input, it is read
+    back with the result."""
+    ps = fn.get("params", [])
+    ret = fn["returnType"]["canonical"]
+    if not ps or _base(ret) != "char" or _ptr_depth(ret) != 1 or "const" in ret:
+        return None
+    last = ps[-1]
+    out = set((fn.get("shape") or {}).get("outParams") or ())
+    if (last["name"] in out and _base(last["canonical"]) == "size_t"
+            and _ptr_depth(last["canonical"]) == 1):
+        return last["name"]
+    return None
+
+
 def _outparam(fn: dict, enums: set, type_encodings: dict):
     """A MEOS accessor of the form ``bool f(.., T *result)`` returns its
     value through a trailing out-parameter, with the ``bool``/``int`` return
@@ -534,6 +558,9 @@ def assess(fn: dict, type_encodings: dict, enums: set) -> tuple:
 
     out_lead, out_p, out_mode = _outparam(fn, enums, type_encodings)
     eff = out_lead if out_p is not None else fn.get("params", [])
+    size_out = _string_size(fn)
+    if size_out is not None:
+        eff = eff[:-1]
     arr_i, count_i = _array_param(eff, type_encodings)
     ret_count_name, ret_elem = _array_return(fn, type_encodings)
     ret_count_i = (next((i for i, p in enumerate(eff)
@@ -641,7 +668,8 @@ def assess(fn: dict, type_encodings: dict, enums: set) -> tuple:
     if scalar == "handled":
         pass
     elif scalar is not None:
-        wire_result = scalar
+        wire_result = (scalar if size_out is None
+                       else {**scalar, "size_outparam": size_out})
     elif _is_scalar_pointer(ret, enums):
         reasons.append(f"unsupported-return:{ret}")
         wire_result = {"kind": "unsupported"}

server/app.py

@@ -98,7 +98,8 @@ def build_routes(catalog: dict) -> dict:
                        "presence_return": r.get("presence_return", False),
                        "element": _with_outparams(r.get("element"), outs),
                        "count_outparam": r.get("count_outparam"),
-                       "count_param": r.get("count_param")},
+                       "count_param": r.get("count_param"),
+                       "size_outparam": r.get("size_outparam")},
         }
     return routes
 
@@ -209,6 +210,10 @@ def _dispatch(body: dict, route: dict, engine: Engine, enums: dict):
                                          el.get("encode_outparams", []))
                            for p in ptrs]}
 
+    if res.get("size_outparam"):
+        # char *f(.., size_t *size_out): the string read for the size MEOS writes.
+        return {"result": engine.invoke_string(route["name"], args)}
+
     rtag = "void" if res["kind"] == "void" else (
         "ptr" if res["kind"] == "serialized"
         else _RESULT_TAG.get(res["json"], "str"))

server/engine.py

@@ -60,6 +60,11 @@ def invoke_array(self, fn_name: str, args: list):
         """``Elem **f(.., int *count)`` — return a list of element handles."""
         raise NotImplementedError
 
+    def invoke_string(self, fn_name: str, args: list):
+        """``char *f(.., size_t *size_out)`` — return the string, read for the
+        size the function writes, as #invoke_array reads its trailing count."""
+        raise NotImplementedError
+
     def invoke_counted_array(self, fn_name: str, args: list, count: int):
         """``Elem **f(Elem **arr, int count, ..)`` — return the ``count`` element
         handles of an array as long as an array argument."""
@@ -88,6 +93,10 @@ def invoke(self, fn_name, args, result_tag):
     def encode(self, fn_name, handle, aux=(), outparams=()):
         return f"<stub:{fn_name}>"
 
+    def invoke_string(self, fn_name, args):
+        # the placeholder string #invoke answers for a "str" result
+        return ""
+
     def invoke_outparam(self, fn_name, args, out_ctype, presence):
         return True, 0
 
@@ -292,6 +301,26 @@ def invoke_array(self, fn_name, args):
         arr = ct.cast(ret, ct.POINTER(ct.c_void_p))
         return [arr[i] for i in range(count)]
 
+    def invoke_string(self, fn_name, args):
+        # char *f(leading..., size_t *size_out): MEOS writes the size of the
+        # string, its terminating NUL included, through the trailing byref
+        # size, as #invoke_array reads its count; the string is read for
+        # that many bytes rather than scanned for its terminator.
+        self._last_error = None
+        f = self._sym(fn_name)
+        ct = self._ct
+        n = ct.c_size_t()
+        f.argtypes = [self._argmap[t] for t, _ in args] + [
+            ct.POINTER(ct.c_size_t)]
+        f.restype = ct.c_void_p
+        cargs = [v.encode() if t == "str" and isinstance(v, str) else v
+                 for t, v in args]
+        ret = f(*cargs, ct.byref(n))
+        self._raise_if_error()
+        if not ret:
+            return None
+        return ct.string_at(ret, n.value - 1).decode()
+
     def invoke_counted_array(self, fn_name, args, count):
         # Elem **f(Elem **arr, int count, ...): MEOS allocates an array as long
         # as the argument array, so its length is the one the caller passed.

tests/test_engine_integration.py

@@ -49,22 +49,31 @@ def _aux(specs):
 class CtypesIntegrationTests(unittest.TestCase):
     @classmethod
     def setUpClass(cls):
+        from server.app import writer_outparams
         cls.eng = CtypesEngine(_LIB)
-        te = (json.loads(_CATALOG.read_text()).get("typeEncodings", {})
-              if _CATALOG.exists() else {})
+        catalog = json.loads(_CATALOG.read_text()) if _CATALOG.exists() else {}
+        cls.outs = writer_outparams(catalog)
+        cls.catalog = catalog
+        te = catalog.get("typeEncodings", {})
         t = te.get("Temporal", {})
         cls.readers = (t.get("readers") or {}).get("text", {})
         cls.reader_aux = (t.get("readerAux") or {}).get("text", {})
         cls.writers = (t.get("writers") or {}).get("text", {})
         cls.writer_aux = (t.get("writerAux") or {}).get("text", {})
 
+    def encode(self, writer, handle, aux=()):
+        """``writer`` called through #CtypesEngine.encode with the out-parameters the
+        catalog states for it, as the routes of server/app.py call it
+        (#writer_outparams): a string writer writes its size through one."""
+        return self.eng.encode(writer, handle, aux, self.outs.get(writer, []))
+
     def read(self, sqltype, literal):
         return self.eng.decode(self.readers[sqltype], literal,
                                _aux(self.reader_aux[sqltype]))
 
     def write(self, sqltype, handle):
-        return self.eng.encode(self.writers[sqltype], handle,
-                               _aux(self.writer_aux[sqltype]))
+        return self.encode(self.writers[sqltype], handle,
+                           _aux(self.writer_aux[sqltype]))
 
     def test_each_type_states_its_own_reader_and_writer(self):
         for sqltype in _LITERAL_BY_TYPE:
@@ -116,7 +125,7 @@ def test_opaque_outparam_round_trip(self):
         # that junk is negative — measured failing 4 runs in 6, with a
         # different value each time, and passing 6 in 6 once maxdd is supplied.
         self.assertIn("POINT",
-                      self.eng.encode("geo_as_ewkt", ptr, [("int", 15)]).upper())
+                      self.encode("geo_as_ewkt", ptr, [("int", 15)]).upper())
 
     def test_input_array_builder_round_trip(self):
         # Temporal *temporal_merge_array(Temporal **temparr, int count):
@@ -138,7 +147,7 @@ def test_array_return_round_trip(self):
             "tbool_in", "{[t@2000-01-01, f@2000-01-03], [t@2000-01-05]}")
         ptrs = self.eng.invoke_array("temporal_sequences", [("ptr", h)])
         self.assertEqual(len(ptrs), 2)          # two composing sequences
-        outs = [self.eng.encode("tsequence_out", p, [("int", 15)])
+        outs = [self.encode("tsequence_out", p, [("int", 15)])
                 for p in ptrs]
         self.assertTrue(all("@" in o for o in outs))
 
@@ -150,7 +159,7 @@ def test_counted_array_return_round_trip(self):
         ptrs = self.eng.invoke_counted_array(
             "temparr_round", [("ptrarray", [h1, h2]), ("int", 2), ("int", 2)], 2)
         self.assertEqual(len(ptrs), 2)
-        outs = [self.eng.encode("tfloat_out", p, [("int", 15)]) for p in ptrs]
+        outs = [self.encode("tfloat_out", p, [("int", 15)]) for p in ptrs]
         self.assertIn("1.23@", outs[0])
         self.assertIn("7.65@", outs[1])
 
@@ -184,8 +193,7 @@ def test_each_hex_writer_round_trips_through_encode(self):
         #CtypesEngine.encode, with the out-parameters the catalog states
         (#writer_outparams of server/app.py), read back to the value, as
         #test_each_type_round_trips_through_its_own_writer round-trips the text."""
-        from server.app import writer_outparams
-        outs = writer_outparams(json.loads(_CATALOG.read_text()))
+        outs = self.outs
         for reader, literal, writer, back, out in (
                 ("intset_in", "{1, 2, 3}", "set_as_hexwkb", "set_from_hexwkb",
                  "intset_out"),
@@ -195,11 +203,27 @@ def test_each_hex_writer_round_trips_through_encode(self):
                  "temporal_from_hexwkb", "tint_out")):
             self.assertEqual(outs[writer], [(2, "size_t *")], writer)
             h = self.eng.decode(reader, literal)
-            hexwkb = self.eng.encode(writer, h, [("int", 0)], outs[writer])
+            hexwkb = self.encode(writer, h, [("int", 0)])
             self.assertRegex(hexwkb, r"^(?:[0-9A-F]{2})+$", writer)
             aux = [("int", 15)] if out == "tint_out" else []
-            again = self.eng.encode(out, self.eng.decode(back, hexwkb), aux)
-            self.assertEqual(again, self.eng.encode(out, h, aux), writer)
+            again = self.encode(out, self.eng.decode(back, hexwkb), aux)
+            self.assertEqual(again, self.encode(out, h, aux), writer)
+
+    def test_a_string_route_reads_the_size_meos_writes(self):
+        """``tfloat_out(temp, maxdd, size_out)`` served as its route, the size an
+        out-parameter of its result (#_string_size of parser/enrich.py), answers the
+        string #test_each_type_round_trips_through_its_own_writer writes."""
+        from server.app import _dispatch, build_routes
+        route = build_routes(self.catalog)["/tfloat_out"]
+        self.assertEqual([p["name"] for p in route["params"]], ["temp", "maxdd"])
+        self.assertEqual(route["result"]["size_outparam"], "size_out")
+        # the value crosses the wire as a client sends it: written by the out of its
+        # class, which the route's decode reads back
+        t = self.catalog["typeEncodings"]["Temporal"]
+        h = self.read("tfloat", _TFLOAT)
+        wire = self.encode(t["out"], h, _aux(t["out_aux"]))
+        got = _dispatch({"temp": wire, "maxdd": 15}, route, self.eng, {})
+        self.assertEqual(got["result"], self.write("tfloat", h))
 
     def test_bad_input_raises_not_exits(self):
         with self.assertRaises(MeosError):

tests/test_enrich.py

@@ -179,6 +179,20 @@ def test_defaultable_aux_accepted_type_tag_rejected(self):
         # Weird gets no decoder at all.
         self.assertNotIn("Weird", self.te)
 
+    def test_a_size_out_parameter_is_no_input(self):
+        """A string writer stating its size through a ``size_t *`` its
+        ``shape.outParams`` lists is an encoder as #test_struct_prefix_stripped_and_round_trip
+        reads one; the same parameter left unlisted is a pointer input and disqualifies it."""
+        listed = dict(fn("temporal_out", "char *", (T, "temp"), ("int", "maxdd"),
+                         ("size_t *", "size_out")), shape={"outParams": ["size_out"]})
+        te = build_type_encodings([FUNCTIONS[0], listed], {"Temporal"})
+        self.assertEqual(te["Temporal"]["out"], "temporal_out")
+        self.assertEqual(te["Temporal"]["out_aux"],
+                         [{"name": "maxdd", "kind": "integer", "default": 15}])
+        unlisted = fn("temporal_out", "char *", (T, "temp"), ("size_t *", "size_out"))
+        te = build_type_encodings([FUNCTIONS[0], unlisted], {"Temporal"})
+        self.assertIsNone(te["Temporal"]["out"])
+
     def test_byte_codec_beside_the_wire_encodings(self):
         # the byte-codec twin of #test_struct_prefix_stripped_and_round_trip
         self.assertEqual(self.te["Temporal"]["bytes"],
@@ -241,6 +255,28 @@ def test_out_param_not_exposable(self):
         r2 = self.n("tjsonb_value_at_timestamptz")["reason"]
         self.assertIn("array-or-out-param:value", r2)
 
+    def test_a_string_result_reads_its_size(self):
+        """``char *f(.., size_t *size_out)`` whose ``shape.outParams`` lists the size is
+        exposable, the size no input and stated on the result, as
+        #test_array_return_as_long_as_an_array_argument states its count; the same
+        size left unlisted is refused as #test_out_param_not_exposable refuses one."""
+        listed = dict(fn("geom_relate", "char *", ("const struct Jsonb *", "js"),
+                         ("int", "maxdd"), ("size_t *", "size_out")),
+                      shape={"outParams": ["size_out"]})
+        unlisted = fn("geom_relate2", "char *", ("const struct Jsonb *", "js"),
+                      ("size_t *", "size_out"))
+        fns = by_name(enrich_idl({
+            "functions": [listed, unlisted,
+                          fn("jsonb_in", "struct Jsonb *", ("const char *", "str"))],
+            "structs": [{"name": "Jsonb", "fields": []}], "enums": []}))
+        f = fns["geom_relate"]
+        self.assertTrue(f["network"]["exposable"])
+        self.assertEqual([p["name"] for p in f["wire"]["params"]], ["js", "maxdd"])
+        self.assertEqual(f["wire"]["result"],
+                         {"kind": "json", "json": "string", "size_outparam": "size_out"})
+        self.assertIn("array-or-out-param:size_out",
+                      fns["geom_relate2"]["network"]["reason"])
+
     def test_array_param_and_missing_encoder(self):
         r = self.n("tsequence_make")["reason"]
         self.assertFalse(self.n("tsequence_make")["exposable"])