Repository navigation
Conversation
check_fstring_conversion compared the '!' mark token and the conversion
character token using .lineno/.col_offset, attributes that exist on
CPython's internal pegen tokenizer struct but not on Python's own
tokenize.TokenInfo (which exposes .start/.end (row, col) tuples and
.line instead). Every use of a conversion specifier (!s, !r, !a) in an
f-string inside a .enaml file therefore crashed the parser with:
AttributeError: 'TokenInfo' object has no attribute 'lineno'
instead of parsing successfully, or -- for a genuinely invalid case --
raising the intended "f-string: conversion type must come right after
the exclamanation mark" SyntaxError.
The check's real intent is adjacency: the conversion character must
immediately follow the '!' with nothing in between. Rewrite it as
mark.end != name.start, which is the correct translation using
TokenInfo's actual fields, matching how every other method in this
file already branches on isinstance(node, tokenize.TokenInfo) to use
.start/.end for token nodes.
fstring_conversion (enaml.gram) is declared to return int, and the
FormattedValue action it feeds already treats the no-conversion cases
as ints (b'r'[0] and -1). But check_fstring_conversion returned the
raw name TokenInfo, and the FormattedValue action then called
conversion.decode()[0] on it, expecting bytes. TokenInfo has no
.decode(), so any syntactically valid conversion specifier (!s, !r,
!a) that got past the adjacency check added in the previous commit
still failed to compile, with the same class of AttributeError this
time coming from fstring_replacement_field instead of
check_fstring_conversion.
Fix check_fstring_conversion to return ord(s) -- the int the grammar
rule's own type annotation already promised -- and drop the stray
.decode() call in the FormattedValue action now that conversion is
already an int. Regenerated enaml_parser.py from enaml.gram via
generate_enaml_parser.py; the only functional change in the generated
file is that one line.
Extends test_f_strings.py with regression coverage: parsing f'{x!s}',
f'{x!r}' and f'{x!a}' against the equivalent ast.parse() output, plus
a case confirming a non-adjacent conversion mark (e.g. f'{x! r}')
still raises the intended SyntaxError rather than crashing.
The test itself works fine below Python 3.12: without FSTRING_START/MIDDLE/END tokens, f-strings are parsed as a single STRING token via the legacy generate_ast_for_string() path, which produces the same AST as ast.parse() for conversion specifiers. Only the adjacency-error test needs the skip, since it asserts on enaml's own error wording, which is specific to the check_fstring_conversion() code path introduced by PEP 701 tokenization. Verified passing on Python 3.11 (legacy STRING path) and 3.14 (PEG path).
On Python 3.12+, enaml's PEG parser dropped the literal "x=" text CPython
injects as a leading ast.Constant for a self-documenting f-string debug
expression. f'{x=}' compiled without error but produced a JoinedStr
missing that prefix, so the rendered string silently lost its "x=" label:
src: a = f'{x=}'
py: JoinedStr(values=[Constant(value='x='), FormattedValue(...)])
en: JoinedStr(values=[FormattedValue(...)]) # missing Constant('x=')
CPython's rule: the injected Constant is the raw source text between '{'
and whatever follows the '=' (the conversion '!', the format spec ':', or
the closing '}'), taken verbatim including interior whitespace, e.g.
f'{ x = }' -> ' x = '.
Fixed by capturing the '{' token and, for the '='? group, the token
immediately following it (peeked, not consumed, via a new
fstring_debug_expr sub-rule) -- giving enough position information to
slice that raw text out of the source without depending on which of
conversion/format/rbrace comes next. fstring_replacement_field now
returns [Constant(...), FormattedValue(...)] instead of a bare
FormattedValue when a debug expression is present; fstring_mid* and
fstring_format_spec* results are flattened accordingly before being used
as a JoinedStr's values (base_python_parser.py: make_fstring_replacement_field,
_fstring_debug_prefix, flatten_fstring_parts, build_fstring_format_values).
Also fixes a second, closely related bug in the same conversion-default
logic: the implicit !r conversion for a bare debug expression was applied
even when an explicit format spec was present (f'{x=:>3}' incorrectly got
conversion=114 instead of -1). CPython only defaults to !r when there is
no format spec at all -- with one, the value is formatted directly, not
repr'd first.
Regenerated enaml_parser.py via
`python -m enaml.core.parser.generate_enaml_parser`; the only functional
changes are in fstring_replacement_field, the new fstring_debug_expr rule,
and the two JoinedStr-building call sites.
Like the f-string conversion bug in the sibling branch, this only affects
Python 3.12+: below 3.12 there's no FSTRING_START token, so f-strings
never reach fstring_replacement_field at all and go through the legacy
generate_ast_for_string() path instead, which already handles debug
expressions correctly (verified on Python 3.11 with pegen 0.3.0).
Testing: added test_f_string_debug_expr to tests/core/parser/test_f_strings.py,
parametrized over bare/whitespace/binop/format-spec/multi-line debug-expr
forms, validated against ast.parse() via the existing validate_ast helper.
Full parser suite (tests/core/parser): 437 passed, 2 skipped. Full suite
(tests): 4336 passed, 5 skipped, 0 failed.
Note: f'{x=!r}' (an explicit conversion combined with a debug expression)
could not be exercised here -- it hits the unrelated, already-known
check_fstring_conversion TokenInfo.lineno crash (fixed separately on
fix/fstring-conversion-tokeninfo-attrs). The debug-prefix and
conversion-default logic here don't depend on that fix and should compose
correctly with it once both land.
f'{x:{y}}' compiled without error but its format_spec JoinedStr carried an
extra, spurious ast.Constant(value='') CPython's own parser never produces:
src: a = f'{x:{y}}'
py: format_spec=JoinedStr(values=[FormattedValue(value=Name('y'))])
en: format_spec=JoinedStr(values=[FormattedValue(value=Name('y')), Constant(value='')])
Root cause: the tokenizer emits an FSTRING_MIDDLE token with an empty
string for the zero-length run of literal text between a nested
replacement field and whatever follows it (another field, or the closing
'}'). fstring_format_spec turns every FSTRING_MIDDLE into an ast.Constant
regardless of content, so this degenerate token became a real, spurious
element. build_fstring_format_values only special-cased a *lone* empty
element (a bare ':' with nothing after it), not one trailing other spec
parts, so it survived unfiltered.
Fixed by dropping every empty-string Constant from the flattened parts
before building the format spec's JoinedStr, regardless of position --
matching CPython, which never emits degenerate empty-text nodes into a
JoinedStr. A genuinely empty format spec (f'{x:}') still correctly
produces an empty values list, now via the same general filter rather than
a separate special case.
Confined to fstring_full_format_spec's nested-field handling; the
top-level fstring_mid* loop (the outer JoinedStr) was checked and does not
exhibit the same issue in practice -- verified f'{x}{y}' (two adjacent
replacement fields with no literal in between) already matches ast.parse().
Testing: added test_f_string_nested_format_spec to
tests/core/parser/test_f_strings.py (4 cases: single nested field, two
adjacent nested fields, nested field followed by a literal, and three
adjacent nested fields), validated against ast.parse() via validate_ast.
Parser suite: 441 passed, 2 skipped. Full suite: 4340 passed, 5 skipped,
0 failed.
Pre-existing bug, unrelated to and independent of the debug-expr-prefix
fix earlier on this branch -- only touches build_fstring_format_values,
which the debug-expr fix introduced as a rename/refactor of the prior
inline ternary but did not change the underlying (buggy) filtering logic.
…info-attrs Consolidates all three PEP-701 f-string-parsing fixes onto one branch/PR, since they all touch the same small area (fstring_replacement_field / fstring_full_format_spec and their helpers in base_python_parser.py) and textually conflict when merged separately: - check_fstring_conversion's wrong TokenInfo attributes (dc8f266, d35dfb3) - f'{x=}' losing its 'x=' debug-expr prefix (8434306) - spurious trailing empty Constant in nested format specs (9d138f6) Conflict resolution in fstring_replacement_field's grammar action: kept check_fstring_conversion's int return (this branch's fix) combined with the debug_expr-prefix branch's 'and format is None' guard on the implicit !r conversion default (its second fix). enaml_parser.py was regenerated from the merged enaml.gram rather than hand-merged. tests/test_f_strings.py conflicts were purely additive (interleaved test functions, no overlapping changes) and combined directly. Verified after resolution: parser suite 445 passed/2 skipped, full suite 4344 passed/0 failed/5 skipped. Also spot-checked every case that was previously blocked by one bug while exercising another -- f'{x=!r}' (debug expr + explicit conversion), f'{x!r:>{w}}' (conversion + nested format spec), f'{x=:{w}}' (debug expr + nested format spec) -- all now match ast.parse() exactly.
Codecov Report❌ Patch coverage is Additional details and impacted files@@ Coverage Diff @@
## main #629 +/- ##
==========================================
+ Coverage 66.07% 66.11% +0.04%
==========================================
Files 270 270
Lines 26584 26613 +29
Branches 3901 3906 +5
==========================================
+ Hits 17565 17596 +31
Misses 7975 7975
+ Partials 1044 1042 -2 🚀 New features to boost your workflow:
|
Member
|
Could you point Claude to pegen from which I extracted the parser to check if the bug need to fixed there too ? |
Contributor
Author
|
Looks like it, but most bugs were fixed by we-like-parsers/pegen#118. Is the fix in Enaml now moot? The one pending bug is the |
Member
Contributor
Author
|
Closing. Will keep the issue open so that it can be marked off once Enaml is updated from Pegen. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #628. All was done by Claude and reviewed by me. Claude wrote up the description.
Summary
.enamlfiles hit several bugs in the PEG parser's f-string handling onPython 3.12+ (the PEP 701 tokenizer path): a crash on every conversion
specifier (
!s/!r/!a), and two silent AST-correctness bugs thatcompile without error but produce a wrong result (a debug expression's
"x="prefix text, and a spurious empty node in nested format specs).This PR fixes all four bugs on that code path, one per commit, and adds
regression tests for each.
They're combined into one PR/branch rather than split up because they all
touch the same handful of lines in
fstring_replacement_field's grammaraction and its helpers -- split across separate PRs, the second to merge
would conflict with the first.
Bug 1 --
check_fstring_conversionused the wrong token attributesenaml/core/parser/base_python_parser.py,check_fstring_conversion(aroundline 224), checked adjacency between the
!mark token and the conversioncharacter token like this:
Both
markandnameare genuinetokenize.TokenInfoinstances (thismethod is called from the generated PEG parser,
enaml/core/parser/enaml_parser.py:5191, generated fromenaml/core/parser/enaml.gram:1583:conv_token="!" conv=NAME { self.check_fstring_conversion(conv_token, conv) },whenever the parser encounters
!s/!r/!ainside an f-string in a.enamlfile).tokenize.TokenInfohas no.lineno/.col_offset-- itexposes
.start/.end(each a(row, col)tuple) and.line. So thisline always raised:
for every conversion specifier, valid or not. The rest of this file
(
raise_syntax_error_known_location,raise_syntax_error_known_range,etc.) already branches on
isinstance(node, tokenize.TokenInfo)and uses.start/.endfor that case -- this method was the one outlier that nevergot that treatment.
The check's real intent is adjacency: the conversion character must come
immediately after the
!, with nothing (not even whitespace) in between.The correct translation using
TokenInfo's actual fields is:Bug 2 -- the conversion value that survives bug 1's fix is the wrong type
Fixing bug 1 alone still doesn't let a valid conversion specifier compile.
fstring_conversionis declared to returnint(
enaml.gram:1582,fstring_conversion[int]:), and theFormattedValueaction that consumes it already treats the "no conversion" cases as plain
ints (
b'r'[0]for a bare=debug expression,-1for none). Butcheck_fstring_conversionreturned the rawnameTokenInfo, and theaction then did:
TokenInfohas no.decode(), so a syntactically valid, adjacentconversion specifier still crashed -- same class of
AttributeError, nowraised from
fstring_replacement_fieldinstead ofcheck_fstring_conversion.Fixed by having
check_fstring_conversionreturnord(s)(the int thegrammar rule's own annotation already promised) and dropping the stray
.decode()call in theenaml.gramaction, then regeneratingenaml_parser.pyviapython -m enaml.core.parser.generate_enaml_parser.The regenerated file's only functional change is that one line.
Bug 3 -- f-string debug expressions (
f'{x=}') lost theirx=prefixfstring_replacement_field's grammar action never emitted the leadingast.Constant('x=')node CPython injects for a self-documenting debugexpression.
f'{x=}'compiled without error but produced aJoinedStrmissing that prefix, so the rendered string silently lost its
x=label:CPython's rule: the injected
Constantis the raw source text between{and whatever follows the
=(the conversion!, the format spec:, orthe closing
}), taken verbatim including interior whitespace, e.g.f'{ x = }'->' x = '.Fixed by capturing the
{token and, for the=? group, the tokenimmediately following it (peeked, not consumed, via a new
fstring_debug_exprsub-rule) -- giving enough position information toslice that raw text out of the source without depending on which of
conversion/format/rbrace comes next.
fstring_replacement_fieldnowreturns
[Constant(...), FormattedValue(...)]instead of a bareFormattedValuewhen a debug expression is present; thefstring_mid*and
fstring_format_spec*loop results are flattened accordingly beforebeing used as a
JoinedStr's values (new helpers inbase_python_parser.py:make_fstring_replacement_field,_fstring_debug_prefix,flatten_fstring_parts,build_fstring_format_values).Also fixes a second, closely related bug in the same conversion-default
logic: the implicit
!rconversion for a bare debug expression wasapplied even when an explicit format spec was present (
f'{x=:>3}'incorrectly got
conversion=114instead of-1). CPython only defaultsto
!rwhen there is no format spec at all -- with one, the value isformatted directly, not repr'd first.
Bug 4 -- spurious trailing empty
Constantin nested format specsA format spec ending in a nested replacement field, e.g.
f'{x:{y}}',compiled without error but its
format_specJoinedStrcarried an extra,spurious
ast.Constant(value='')CPython's own parser never produces:Root cause: the tokenizer emits an
FSTRING_MIDDLEtoken with an emptystring for the zero-length run of literal text between a nested
replacement field and whatever follows it (another field, or the closing
}).fstring_format_specturns everyFSTRING_MIDDLEinto anast.Constantregardless of content, so this degenerate token became areal, spurious element.
build_fstring_format_values(introduced by thebug 3 fix, as a rename/refactor of the prior inline ternary -- the
underlying filtering logic was unchanged) only special-cased a lone
empty element (a bare
:with nothing after it), not one trailing otherspec parts, so it survived unfiltered.
Fixed by dropping every empty-string
Constantfrom the flattened partsbefore building the format spec's
JoinedStr, regardless of position --matching CPython, which never emits degenerate empty-text nodes into a
JoinedStr. A genuinely empty format spec (f'{x:}') still correctlyproduces an empty values list, now via the same general filter rather than
a separate special case.
Confined to
fstring_full_format_spec's nested-field handling; thetop-level
fstring_mid*loop (the outerJoinedStr) was checked and doesnot exhibit the same issue in practice --
f'{x}{y}'(two adjacentreplacement fields with no literal in between) already matches
ast.parse().Testing
Added to
tests/core/parser/test_f_strings.py:test_f_string_conversion-- parametrized overs/r/a, parsesf'{x!<conv>}'and validates the resulting AST againstast.parse().Runs on all supported Python versions (3.10+): below 3.12 it exercises
the parser's legacy single-
STRING-token path instead, which alreadyhandled conversions correctly, so no version skip is needed.
test_f_string_conversion_not_adjacent_to_mark-- confirmsf'{x! r}'(whitespace between
!and the conversion character) still raises aSyntaxErrorwith the intended message, rather than anAttributeError. Skipped below 3.12: enaml's own error wording isspecific to this PEG code path, not reachable pre-3.12.
test_f_string_debug_expr-- 6 cases (bare, whitespace, binop,format-spec, empty format-spec, multi-line), validated against
ast.parse(). Skipped below 3.12 (fstring_replacement_fieldunreached).test_f_string_nested_format_spec-- 4 cases (single nested field, twoadjacent nested fields, nested field followed by a literal, three
adjacent nested fields), validated against
ast.parse(). Skipped below3.12 (
fstring_full_format_specunreached).Also spot-checked every case that's blocked by one bug while exercising
another, to confirm all four fixes compose correctly together:
f'{x=!r}'(debug expr + explicit conversion),f'{x!r:>{w}}'(conversion + nested format spec),
f'{x=:{w}}'(debug expr + nestedformat spec) -- all match
ast.parse()exactly after all four fixes areapplied together.
Ran the full parser suite (
python -m pytest tests/core/parser) on Python3.14 after all four fixes: 445 passed, 2 skipped (the 2 pre-existing skips
are an unrelated backslash-in-f-string test that only applies below 3.12).
Also ran the full test suite (
python -m pytest tests): 4344 passed, 5skipped, 0 failed.
Additionally spot-checked Python 3.11 (with
pegen>=0.3.0, the versionthis project requires) to confirm the legacy pre-3.12 path is unaffected
by any of the four fixes and already produces correct output for all of
them.
Repro (before this fix)
🤖 Generated with Claude Code
https://claude.ai/code/session_01DModRJvYu2WckV4FgT7vDP