cfscript: support a new expression as a function-listener target (#98)

CHANGELOG.md

@@ -3,6 +3,16 @@
 ## [Unreleased]
 
 ### cfscript
+- **Support a `new` expression as a function-listener target** — `threadName = new Query():function( result, error ) { … };` ([#98](https://github.com/cfmleditor/tree-sitter-cfml/issues/98), Lucee `FunctionListener.cfc`). That completes the eleven forms in Lucee's [Function Listeners](https://docs.lucee.org/recipes/function-listeners.html) recipe; the other ten landed in #96 and #97. **+53 parse states**, one declared conflict, corpus **644 → 642 error nodes across 120 → 119 files**, zero changed trees in both grammars. Probe `cfscript/function_listener_new.cfc` flips to `pass`.
+
+  **The `+591` this issue was parked on had expired.** It was measured when `new_expression` could complete on the bare keyword `new`; requiring its `arguments` — shipped separately — removed that, and the same widening then measured **+14**. A cost taken before a related change is not evidence about after it, which is the transferable part.
+
+  **What actually blocked it was precedence, not size.** With the target widened, `new Foo()` before a contested `:` is either a listener target or a ternary consequence, and only an open `?` tells them apart. The rule shipped in #87 sits at `prec.right('call', …)`, and `'call'` binds tighter than `'ternary'`, so the listener reading won inside `c ? new Foo() : obj` and that ternary stopped parsing — the failure every earlier attempt hit, including the +260 variant that restricted the listener side to a function literal.
+
+  So the `new` target is a **second arm below `'ternary'`**, with the conflict `[$.expression, $.function_listener_expression]` that `tree-sitter generate` asks for. Where a `?` is open the ternary wins; where none is, nothing competes for the colon. The call-target arm keeps `'call'` and is untouched.
+
+  **The conflict was benchmarked, as a declared conflict must be.** cfscript moved **+0.5%** against untouched controls spanning 3.8 points (cfml +0.9%, cfquery −2.9%) — inside the noise floor, which is what the prefix predicts: the conflict is live on a `new_expression` followed by `:` and nowhere else. Both ternary controls are pinned by corpus tests, because they are what every earlier attempt broke.
+
 - **Support a return type written between two modifiers** — `component { public struct static function f() {} }` ([#117](https://github.com/cfmleditor/tree-sitter-cfml/issues/117), Lucee `test/general/modifiers/All.cfc`). The type was already accepted at either end of the modifier run; this is the third position. **Corpus 645 → 644 error nodes across 121 → 120 files** — `All.cfc`, the file whose whole purpose is enumerating modifier spellings, goes to zero — **+54 parse states** (5382 → 5436), no new conflicts, and the tree-shape diff reports **zero** changed files in both grammars. Probe `cfscript/interleaved_return_type.cfc` added.
 
   **The hazard the rule's own comments name fired on the first attempt.** Spelling it as `repeat($.access_type)` after the optional type costs +45 states and makes the construct parse — and breaks `function static( … )`, a function *named* `static` from Mura's `MuraScope.cfc`. Allowing a modifier to follow the word `function` is what re-lexes `static` as `_kw_static`; the same trap is recorded in that rule for the type-first spelling, where it bit once before.

LIMITATIONS.md

@@ -165,20 +165,15 @@ assessment, including how many files each affects and what fixing it would cost.
   **`'${'` was already a token of the grammar** (`template_substitution`, inside a backtick string), so this admits an existing lexical form in a new position rather than adding one. Every other `${…}` in the corpus sits inside a string literal — Java-style placeholders in Slatwall's shipping URLs, JS template literals in TestBox's coverage browser — and a string lexes as one token, so none of them reach the rule. The single exception is `cfwheels`' `tools/vscode-ext/assets/templates/controller.cfc`, a **VS Code snippet template rather than CFML**, whose `${modelNamePlural}` placeholders sit in code position; it failed to parse before this change (12 error nodes) and still does (18), with the shape of its recovery moved rather than its validity.
 - **A numeric struct key by dot notation in write position — fixed** ([#86](https://github.com/cfmleditor/tree-sitter-cfml/issues/86)). `myNumb.4 = "4";` now yields an ordinary `member_expression` assignment. Kept here because the *cause* is the reusable part, and neither the issue nor the earlier note here had it. It is not the property rule: `identifier` is a permissive negated character class that matches `4` and `4b` quite happily, which is why the read position always worked. It is the **number** token. `.4` is a legal leading-dot float — `x = .5` is real CFML, Mura writes `imageQuality=.95` and Lucee has a ticket (LDEV4480) about treating `.0` as a number — so as the longer match it out-lexed `.` followed by a property wherever a number was *also* valid. At a statement head the `tag_statement` reading keeps a number valid, which is the whole of the difference: `var myNumb.4 = 1` and `a.b.4 = 1` both parsed on master, because `var` and the first `.` each commit the parser and take the tag reading out of play. The fix splits the leading-dot form into its own token at `prec(-1)`, so `.` wins wherever `.` is valid and the float wins everywhere a number literal actually appears. **+8 parse states, no new conflicts.** Note this also fixed a silent misparse the issue never mentioned: `myNumb.4.5 = "x"` was error-free and wrong, reading `myNumb` as a *tag name* and `4.5` as a member expression of the number `4`.
 - **A statement followed by a tag island — fixed** ([#118](https://github.com/cfmleditor/tree-sitter-cfml/issues/118)). `thread name="x" { … }` then a ` ``` ` block (Lucee LDEV4157) parses, and so does every other statement in front of a fence. Kept here because the entry it replaces named the wrong construct. It was filed, and recorded here, as a `thread`-specific gap — "each parses on its own; only the combination fails" — and `thread` had nothing to do with it. The fence never fired the **automatic semicolon**: `scan_automatic_semicolon` in `cfscript/src/scanner.c` descends from the JavaScript scanner, where a backtick opens a template literal and so must *not* take a semicolon in front of it, and that case survived the port. CFScript has no template literal, and its only backtick is the fence of a tag island — a statement of its own, never a continuation. So anything relying on insertion in front of a fence came out `MISSING ";"`: `thread name="t" { … }`, `lock { … }`, and a bare `var q = queryNew( "col" )`, which is the separate probe `cfscript/cfml_template_after_bare_statement.cfc` — the same cause, recorded twice as two unrelated symptoms. Deleting the case is the whole fix: **no change to `STATE_COUNT`** (scanner-only, `parser.c` byte-identical), corpus **654 → 648 error nodes across 127 → 124 files**, and the tree-shape diff reports **zero** changed files in both grammars. `common/scanner.h` carries the same inherited case and is deliberately left alone: the `cfml`/`cfquery` grammars have no ` ``` ` rule at all, so there the backtick is unreachable rather than wrong.
-- **Function-listener callback on a `new` *target*** — `threadName = new Query():function( result, error ) { … };` ([#98](https://github.com/cfmleditor/tree-sitter-cfml/issues/98), split out of [#87](https://github.com/cfmleditor/tree-sitter-cfml/issues/87); probe `cfscript/function_listener_new.cfc`). The last of the eleven forms in Lucee's [Function Listeners](https://docs.lucee.org/recipes/function-listeners.html) recipe (`since` 6.1) that does not parse. **Correction to the entry this replaces:** that one said a `new` on *either* side was unsupported. The listener side — `mySuccess():new component { … }` and its no-listener variant — now parses. It was worth separating, because the two halves are not remotely the same price:
+- **Function-listener callback on a `new` *target* — fixed** ([#98](https://github.com/cfmleditor/tree-sitter-cfml/issues/98), split out of [#87](https://github.com/cfmleditor/tree-sitter-cfml/issues/87)). `threadName = new Query():function( result, error ) { … };` parses, completing the eleven forms in Lucee's [Function Listeners](https://docs.lucee.org/recipes/function-listeners.html) recipe. **+53 parse states**, one declared conflict, corpus 644 → 642, zero changed trees.
 
-  | side | form | cost |
-  |---|---|---|
-  | listener | `f():new component { … }` | **+27 parse states (+0.5%), no conflicts** — shipped |
-  | target | `new Query():f(){ … }` | **+591 parse states (+11.3%), 2 conflicts** — not shipped |
-
-  **Correction, from measuring it properly:** the cost is *not* about which side of the colon the `new` sits on. That was the obvious reading and it is wrong. Two controls, measured from the rule as shipped: widening the target with `subscript_expression` costs **+46** states, widening it with `new_expression` costs **+591**, and widening it with a `new_expression` whose `arguments` are **required** costs **−2**. Widening the target is cheap. What is expensive is admitting a rule that can complete on a **bare keyword**: `new_expression` has both its constructor and its arguments optional, so `new` alone is already a complete expression, and putting that before a contested `:` makes every state that can precede a colon carry the `New`-as-label and `New`-as-property-name readings as well. Remove the self-completion and the whole 591 disappears. Rejected on the same basis as [#75](https://github.com/cfmleditor/tree-sitter-cfml/issues/75): 2 corpus nodes in one file, against +2.4 MB of committed C that every binding compiles.
+  **This entry's cost figures were all measured before the fix that made them obsolete**, and the history is worth keeping because it is a lesson about *when* a measurement expires. The `+591` recorded here was taken when `new_expression` could complete on the bare keyword; requiring its `arguments` (shipped separately) removed that, and the same widening then measured **+14**. A number from before a related change is not evidence about after it.
 
-  **Correction, and one of the four narrowings has now shipped.** This entry previously said that requiring `new_expression`'s `arguments` "changes how `new` *lexes*" so that `isNull(o) ? new() : o` reads its consequence as a `call_expression`. **That was wrong.** Requiring the arguments *on its own* keeps that ternary correct, takes **16 states out** of the table (5274 → 5258), prunes three stale conflicts, and fixes the `new.foo` misparse — it is shipped. What actually breaks the ternary is the **combination**: admitting `new_expression` as a listener *target* makes `new():y` a competing reading of `? new() : y`, and the constructor-less form loses. The blocker is an interaction between the two halves, not a property of either.
+  **What actually blocked it was never table size.** With the target widened, `new Foo()` before a contested `:` is either a listener target or a ternary consequence, and only an open `?` tells them apart — which is precisely what GLR carries. The rule as shipped in #87 sat at `prec.right('call', …)`, and `'call'` binds tighter than `'ternary'`, so the listener reading won inside `c ? new Foo() : obj` and that ternary stopped parsing. Every attempt recorded here failed the same way, including the +260 variant that restricted the listener side to a function literal.
 
-  That reframing also names why it is hard. A constructor-less `new( … )` is **real CFML** — a call to a user-defined function named `new`, as in Slatwall's `BaseDAO`/`BaseService` and RustCFML's own `test_new_udf_dispatch_and_null_call.cfm` — so both readings genuinely have to survive. The natural fix is a listener target that requires a constructor, and that **cannot generate**: it collides with `new_expression` at every `new (`, five interpretations of `'New' • '('`, the ambiguous-prefix shape the cost table warns about. `prec.dynamic` on the arguments at 1, 2 and 4 does not move it either.
+  **The fix is a second arm at a lower precedence**, below `'ternary'`, plus the conflict `[$.expression, $.function_listener_expression]` that `tree-sitter generate` asks for. Where a `?` is open the ternary wins; where none is, nothing competes for the colon and the listener wins. The call-target arm keeps `'call'` and is untouched. Both ternary controls — `c ? new Foo() : obj` and `isNull(o) ? new() : o` — are pinned by corpus tests, because they are what every earlier attempt broke.
 
-  The other narrowings still stand as recorded: requiring the constructor *as well* saves 70 of the 591 but breaks `new()` outright; and confining the whole listener rule to statement and assignment position — the trick that made script-syntax tag calls affordable — came out **worse** at +669, because the rule then has to be spelled at each site.
+  **Reserving `new` stays ruled out** for the reasons recorded before: 395 `.new(` call sites across 160 corpus files, and Lucee's documented reserved-word list is `null`, `true`, `false`.
 - **`savecontent` as an expression** — `greeting = savecontent { writeOutput("hi") };` ([#82](https://github.com/cfmleditor/tree-sitter-cfml/issues/82), 3 Lucee files). Attempted. The statement form has no `savecontent` rule at all — it goes through the generic `tag_statement` as an ordinary identifier — so an expression form needs a `keyword('Savecontent')`, and that is the trap this file's own "keyword extraction is lexical" section describes. Adding it generates cleanly and costs only 31 states, then breaks the *working* statement form and every ordinary use of the word: `savecontent = 1`, `x = savecontent`, `x = savecontent.foo` and `savecontent()` all become ERRORs, and `npm test` fails. A `_reserved_identifier` entry does not rescue it, because the keyword out-lexes the identifier before the parser can reach the fallback. The non-keyword alternative — admitting `identifier statement_block` as an expression — is far too broad, since it is live at every identifier followed by `{`.
 - **`function` as a bare value** — `h = function.foo;`. `function` is accepted as
   an assignable name (`admin ... function="" ...`) and as a property

cfscript/grammar.js

@@ -135,6 +135,13 @@ module.exports = grammar({
   ],
 
   conflicts: ($) => [
+    // `x = new Foo() : cb` against `c ? new Foo() : x` — a `new` expression
+    // before a contested `:` is a listener target or a ternary consequence,
+    // and only the presence of an open `?` settles it, which is exactly what
+    // GLR carries. Live on `new_expression` followed by `:` and nowhere else;
+    // the benchmark puts the cost inside the noise floor, with the two
+    // untouched grammars spanning more than the subject moved (#98).
+    [$.expression, $.function_listener_expression],
     [$.object, $.object_pattern],
     [$.primary_expression, $.pattern],
     [$.assignment_expression, $.pattern],
@@ -1290,11 +1297,28 @@ module.exports = grammar({
     // and far commoner reading, so it wins; nothing else reaches a state where
     // both survive, because the ternary's own `:` is required and the listener
     // reading leaves it dangling.
-    function_listener_expression: ($) => prec.dynamic(-1, prec.right('call', seq(
-      field('target', $.call_expression),
-      ':',
-      field('listener', choice($.primary_expression, $.new_expression)),
-    ))),
+    function_listener_expression: ($) => prec.dynamic(-1, choice(
+      prec.right('call', seq(
+        field('target', $.call_expression),
+        ':',
+        field('listener', choice($.primary_expression, $.new_expression)),
+      )),
+      // `threadName = new Query():function( … ) { … };` — a listener on a
+      // component instantiation, the last of the eleven forms in Lucee's
+      // Function Listeners recipe (#98).
+      //
+      // It is a SEPARATE arm from the call target above, and its precedence is
+      // the whole reason. At 'call' — the precedence the call-target arm uses,
+      // which binds tighter than 'ternary' — the listener reading wins inside
+      // `c ? new Foo() : obj` and that ternary stops parsing. Below 'ternary'
+      // the ternary wins where a `?` is open, and the listener still wins
+      // where there is none, because then nothing competes for the colon.
+      prec.right('elvis', seq(
+        field('target', $.new_expression),
+        ':',
+        field('listener', choice($.primary_expression, $.new_expression)),
+      )),
+    )),
 
     member_expression: $ => prec('member', seq(
       field('object', choice($.expression, $.primary_expression)),

cfscript/src/grammar.json

@@ -4186,42 +4186,85 @@
       "type": "PREC_DYNAMIC",
       "value": -1,
       "content": {
-        "type": "PREC_RIGHT",
-        "value": "call",
-        "content": {
-          "type": "SEQ",
-          "members": [
-            {
-              "type": "FIELD",
-              "name": "target",
-              "content": {
-                "type": "SYMBOL",
-                "name": "call_expression"
-              }
-            },
-            {
-              "type": "STRING",
-              "value": ":"
-            },
-            {
-              "type": "FIELD",
-              "name": "listener",
-              "content": {
-                "type": "CHOICE",
-                "members": [
-                  {
+        "type": "CHOICE",
+        "members": [
+          {
+            "type": "PREC_RIGHT",
+            "value": "call",
+            "content": {
+              "type": "SEQ",
+              "members": [
+                {
+                  "type": "FIELD",
+                  "name": "target",
+                  "content": {
                     "type": "SYMBOL",
-                    "name": "primary_expression"
-                  },
-                  {
+                    "name": "call_expression"
+                  }
+                },
+                {
+                  "type": "STRING",
+                  "value": ":"
+                },
+                {
+                  "type": "FIELD",
+                  "name": "listener",
+                  "content": {
+                    "type": "CHOICE",
+                    "members": [
+                      {
+                        "type": "SYMBOL",
+                        "name": "primary_expression"
+                      },
+                      {
+                        "type": "SYMBOL",
+                        "name": "new_expression"
+                      }
+                    ]
+                  }
+                }
+              ]
+            }
+          },
+          {
+            "type": "PREC_RIGHT",
+            "value": "elvis",
+            "content": {
+              "type": "SEQ",
+              "members": [
+                {
+                  "type": "FIELD",
+                  "name": "target",
+                  "content": {
                     "type": "SYMBOL",
                     "name": "new_expression"
                   }
-                ]
-              }
+                },
+                {
+                  "type": "STRING",
+                  "value": ":"
+                },
+                {
+                  "type": "FIELD",
+                  "name": "listener",
+                  "content": {
+                    "type": "CHOICE",
+                    "members": [
+                      {
+                        "type": "SYMBOL",
+                        "name": "primary_expression"
+                      },
+                      {
+                        "type": "SYMBOL",
+                        "name": "new_expression"
+                      }
+                    ]
+                  }
+                }
+              ]
             }
-          ]
-        }
+          }
+        ]
       }
     },
     "member_expression": {
@@ -10168,6 +10211,10 @@
     }
   ],
   "conflicts": [
+    [
+      "expression",
+      "function_listener_expression"
+    ],
     [
       "object",
       "object_pattern"