Skip to content

Fix f-string parsing bugs on Python 3.12+ (conversion specifiers, debug expressions, nested format specs) - #629

Closed
bburan wants to merge 6 commits into
nucleic:mainfrom
bburan:fix/fstring-conversion-tokeninfo-attrs
Closed

bburan wants to merge 6 commits into
nucleic:mainfrom
bburan:fix/fstring-conversion-tokeninfo-attrs

Conversation

@bburan

@bburan bburan commented Sep 9, 2026

Copy link
Copy Markdown
Contributor

Fixes #628. All was done by Claude and reviewed by me. Claude wrote up the description.

Summary

.enaml files hit several bugs in the PEG parser's f-string handling on
Python 3.12+ (the PEP 701 tokenizer path): a crash on every conversion
specifier (!s/!r/!a), and two silent AST-correctness bugs that
compile without error but produce a wrong result (a debug expression's
"x=" prefix text, and a spurious empty node in nested format specs).
This PR fixes all four bugs on that code path, one per commit, and adds
regression tests for each.

They're combined into one PR/branch rather than split up because they all
touch the same handful of lines in fstring_replacement_field's grammar
action and its helpers -- split across separate PRs, the second to merge
would conflict with the first.

Bug 1 -- check_fstring_conversion used the wrong token attributes

enaml/core/parser/base_python_parser.py, check_fstring_conversion (around
line 224), checked adjacency between the ! mark token and the conversion
character token like this:

if mark.lineno != name.lineno or mark.col_offset != name.col_offset:

Both mark and name are genuine tokenize.TokenInfo instances (this
method is called from the generated PEG parser,
enaml/core/parser/enaml_parser.py:5191, generated from
enaml/core/parser/enaml.gram:1583:
conv_token="!" conv=NAME { self.check_fstring_conversion(conv_token, conv) },
whenever the parser encounters !s/!r/!a inside an f-string in a
.enaml file). tokenize.TokenInfo has no .lineno/.col_offset -- it
exposes .start/.end (each a (row, col) tuple) and .line. So this
line always raised:

AttributeError: 'TokenInfo' object has no attribute 'lineno'. Did you mean: 'line'?

for every conversion specifier, valid or not. The rest of this file
(raise_syntax_error_known_location, raise_syntax_error_known_range,
etc.) already branches on isinstance(node, tokenize.TokenInfo) and uses
.start/.end for that case -- this method was the one outlier that never
got that treatment.

The check's real intent is adjacency: the conversion character must come
immediately after the !, with nothing (not even whitespace) in between.
The correct translation using TokenInfo's actual fields is:

if mark.end != name.start:

Bug 2 -- the conversion value that survives bug 1's fix is the wrong type

Fixing bug 1 alone still doesn't let a valid conversion specifier compile.
fstring_conversion is declared to return int
(enaml.gram:1582, fstring_conversion[int]:), and the FormattedValue
action that consumes it already treats the "no conversion" cases as plain
ints (b'r'[0] for a bare = debug expression, -1 for none). But
check_fstring_conversion returned the raw name TokenInfo, and the
action then did:

conversion.decode()[0] if conversion else (b'r'[0] if debug_expr else -1)

TokenInfo has no .decode(), so a syntactically valid, adjacent
conversion specifier still crashed -- same class of AttributeError, now
raised from fstring_replacement_field instead of
check_fstring_conversion.

Fixed by having check_fstring_conversion return ord(s) (the int the
grammar rule's own annotation already promised) and dropping the stray
.decode() call in the enaml.gram action, then regenerating
enaml_parser.py via python -m enaml.core.parser.generate_enaml_parser.
The regenerated file's only functional change is that one line.

Bug 3 -- f-string debug expressions (f'{x=}') lost their x= prefix

fstring_replacement_field's grammar action never emitted the leading
ast.Constant('x=') node CPython injects for a self-documenting debug
expression. f'{x=}' compiled without error but produced a JoinedStr
missing that prefix, so the rendered string silently lost its x= label:

src: a = f'{x=}'
py:  JoinedStr(values=[Constant(value='x='), FormattedValue(...)])
en:  JoinedStr(values=[FormattedValue(...)])   # missing Constant('x=')

CPython's rule: the injected Constant is the raw source text between {
and whatever follows the = (the conversion !, the format spec :, or
the closing }), taken verbatim including interior whitespace, e.g.
f'{ x = }' -> ' x = '.

Fixed by capturing the { token and, for the =? group, the token
immediately following it (peeked, not consumed, via a new
fstring_debug_expr sub-rule) -- giving enough position information to
slice that raw text out of the source without depending on which of
conversion/format/rbrace comes next. fstring_replacement_field now
returns [Constant(...), FormattedValue(...)] instead of a bare
FormattedValue when a debug expression is present; the fstring_mid*
and fstring_format_spec* loop results are flattened accordingly before
being used as a JoinedStr's values (new helpers in
base_python_parser.py: make_fstring_replacement_field,
_fstring_debug_prefix, flatten_fstring_parts,
build_fstring_format_values).

Also fixes a second, closely related bug in the same conversion-default
logic: the implicit !r conversion for a bare debug expression was
applied even when an explicit format spec was present (f'{x=:>3}'
incorrectly got conversion=114 instead of -1). CPython only defaults
to !r when there is no format spec at all -- with one, the value is
formatted directly, not repr'd first.

Bug 4 -- spurious trailing empty Constant in nested format specs

A format spec ending in a nested replacement field, e.g. f'{x:{y}}',
compiled without error but its format_spec JoinedStr carried an extra,
spurious ast.Constant(value='') CPython's own parser never produces:

src: a = f'{x:{y}}'
py:  format_spec=JoinedStr(values=[FormattedValue(value=Name('y'))])
en:  format_spec=JoinedStr(values=[FormattedValue(value=Name('y')), Constant(value='')])

Root cause: the tokenizer emits an FSTRING_MIDDLE token with an empty
string for the zero-length run of literal text between a nested
replacement field and whatever follows it (another field, or the closing
}). fstring_format_spec turns every FSTRING_MIDDLE into an
ast.Constant regardless of content, so this degenerate token became a
real, spurious element. build_fstring_format_values (introduced by the
bug 3 fix, as a rename/refactor of the prior inline ternary -- the
underlying filtering logic was unchanged) only special-cased a lone
empty element (a bare : with nothing after it), not one trailing other
spec parts, so it survived unfiltered.

Fixed by dropping every empty-string Constant from the flattened parts
before building the format spec's JoinedStr, regardless of position --
matching CPython, which never emits degenerate empty-text nodes into a
JoinedStr. A genuinely empty format spec (f'{x:}') still correctly
produces an empty values list, now via the same general filter rather than
a separate special case.

Confined to fstring_full_format_spec's nested-field handling; the
top-level fstring_mid* loop (the outer JoinedStr) was checked and does
not exhibit the same issue in practice -- f'{x}{y}' (two adjacent
replacement fields with no literal in between) already matches
ast.parse().

Testing

Added to tests/core/parser/test_f_strings.py:

  • test_f_string_conversion -- parametrized over s/r/a, parses
    f'{x!<conv>}' and validates the resulting AST against ast.parse().
    Runs on all supported Python versions (3.10+): below 3.12 it exercises
    the parser's legacy single-STRING-token path instead, which already
    handled conversions correctly, so no version skip is needed.
  • test_f_string_conversion_not_adjacent_to_mark -- confirms f'{x! r}'
    (whitespace between ! and the conversion character) still raises a
    SyntaxError with the intended message, rather than an
    AttributeError. Skipped below 3.12: enaml's own error wording is
    specific to this PEG code path, not reachable pre-3.12.
  • test_f_string_debug_expr -- 6 cases (bare, whitespace, binop,
    format-spec, empty format-spec, multi-line), validated against
    ast.parse(). Skipped below 3.12 (fstring_replacement_field unreached).
  • test_f_string_nested_format_spec -- 4 cases (single nested field, two
    adjacent nested fields, nested field followed by a literal, three
    adjacent nested fields), validated against ast.parse(). Skipped below
    3.12 (fstring_full_format_spec unreached).

Also spot-checked every case that's blocked by one bug while exercising
another, to confirm all four fixes compose correctly together:
f'{x=!r}' (debug expr + explicit conversion), f'{x!r:>{w}}'
(conversion + nested format spec), f'{x=:{w}}' (debug expr + nested
format spec) -- all match ast.parse() exactly after all four fixes are
applied together.

Ran the full parser suite (python -m pytest tests/core/parser) on Python
3.14 after all four fixes: 445 passed, 2 skipped (the 2 pre-existing skips
are an unrelated backslash-in-f-string test that only applies below 3.12).
Also ran the full test suite (python -m pytest tests): 4344 passed, 5
skipped, 0 failed.

Additionally spot-checked Python 3.11 (with pegen>=0.3.0, the version
this project requires) to confirm the legacy pre-3.12 path is unaffected
by any of the four fixes and already produces correct output for all of
them.

Repro (before this fix)

from enaml.core.parser import parse
parse("a = f'{x!r}'")
AttributeError: 'TokenInfo' object has no attribute 'lineno'. Did you mean: 'line'?

🤖 Generated with Claude Code

https://claude.ai/code/session_01DModRJvYu2WckV4FgT7vDP

check_fstring_conversion compared the '!' mark token and the conversion
character token using .lineno/.col_offset, attributes that exist on
CPython's internal pegen tokenizer struct but not on Python's own
tokenize.TokenInfo (which exposes .start/.end (row, col) tuples and
.line instead). Every use of a conversion specifier (!s, !r, !a) in an
f-string inside a .enaml file therefore crashed the parser with:

    AttributeError: 'TokenInfo' object has no attribute 'lineno'

instead of parsing successfully, or -- for a genuinely invalid case --
raising the intended "f-string: conversion type must come right after
the exclamanation mark" SyntaxError.

The check's real intent is adjacency: the conversion character must
immediately follow the '!' with nothing in between. Rewrite it as
mark.end != name.start, which is the correct translation using
TokenInfo's actual fields, matching how every other method in this
file already branches on isinstance(node, tokenize.TokenInfo) to use
.start/.end for token nodes.
fstring_conversion (enaml.gram) is declared to return int, and the
FormattedValue action it feeds already treats the no-conversion cases
as ints (b'r'[0] and -1). But check_fstring_conversion returned the
raw name TokenInfo, and the FormattedValue action then called
conversion.decode()[0] on it, expecting bytes. TokenInfo has no
.decode(), so any syntactically valid conversion specifier (!s, !r,
!a) that got past the adjacency check added in the previous commit
still failed to compile, with the same class of AttributeError this
time coming from fstring_replacement_field instead of
check_fstring_conversion.

Fix check_fstring_conversion to return ord(s) -- the int the grammar
rule's own type annotation already promised -- and drop the stray
.decode() call in the FormattedValue action now that conversion is
already an int. Regenerated enaml_parser.py from enaml.gram via
generate_enaml_parser.py; the only functional change in the generated
file is that one line.

Extends test_f_strings.py with regression coverage: parsing f'{x!s}',
f'{x!r}' and f'{x!a}' against the equivalent ast.parse() output, plus
a case confirming a non-adjacent conversion mark (e.g. f'{x! r}')
still raises the intended SyntaxError rather than crashing.
The test itself works fine below Python 3.12: without FSTRING_START/MIDDLE/END
tokens, f-strings are parsed as a single STRING token via the legacy
generate_ast_for_string() path, which produces the same AST as ast.parse()
for conversion specifiers. Only the adjacency-error test needs the skip,
since it asserts on enaml's own error wording, which is specific to the
check_fstring_conversion() code path introduced by PEP 701 tokenization.

Verified passing on Python 3.11 (legacy STRING path) and 3.14 (PEG path).
On Python 3.12+, enaml's PEG parser dropped the literal "x=" text CPython
injects as a leading ast.Constant for a self-documenting f-string debug
expression. f'{x=}' compiled without error but produced a JoinedStr
missing that prefix, so the rendered string silently lost its "x=" label:

    src: a = f'{x=}'
    py:  JoinedStr(values=[Constant(value='x='), FormattedValue(...)])
    en:  JoinedStr(values=[FormattedValue(...)])   # missing Constant('x=')

CPython's rule: the injected Constant is the raw source text between '{'
and whatever follows the '=' (the conversion '!', the format spec ':', or
the closing '}'), taken verbatim including interior whitespace, e.g.
f'{ x = }' -> ' x = '.

Fixed by capturing the '{' token and, for the '='? group, the token
immediately following it (peeked, not consumed, via a new
fstring_debug_expr sub-rule) -- giving enough position information to
slice that raw text out of the source without depending on which of
conversion/format/rbrace comes next. fstring_replacement_field now
returns [Constant(...), FormattedValue(...)] instead of a bare
FormattedValue when a debug expression is present; fstring_mid* and
fstring_format_spec* results are flattened accordingly before being used
as a JoinedStr's values (base_python_parser.py: make_fstring_replacement_field,
_fstring_debug_prefix, flatten_fstring_parts, build_fstring_format_values).

Also fixes a second, closely related bug in the same conversion-default
logic: the implicit !r conversion for a bare debug expression was applied
even when an explicit format spec was present (f'{x=:>3}' incorrectly got
conversion=114 instead of -1). CPython only defaults to !r when there is
no format spec at all -- with one, the value is formatted directly, not
repr'd first.

Regenerated enaml_parser.py via
`python -m enaml.core.parser.generate_enaml_parser`; the only functional
changes are in fstring_replacement_field, the new fstring_debug_expr rule,
and the two JoinedStr-building call sites.

Like the f-string conversion bug in the sibling branch, this only affects
Python 3.12+: below 3.12 there's no FSTRING_START token, so f-strings
never reach fstring_replacement_field at all and go through the legacy
generate_ast_for_string() path instead, which already handles debug
expressions correctly (verified on Python 3.11 with pegen 0.3.0).

Testing: added test_f_string_debug_expr to tests/core/parser/test_f_strings.py,
parametrized over bare/whitespace/binop/format-spec/multi-line debug-expr
forms, validated against ast.parse() via the existing validate_ast helper.
Full parser suite (tests/core/parser): 437 passed, 2 skipped. Full suite
(tests): 4336 passed, 5 skipped, 0 failed.

Note: f'{x=!r}' (an explicit conversion combined with a debug expression)
could not be exercised here -- it hits the unrelated, already-known
check_fstring_conversion TokenInfo.lineno crash (fixed separately on
fix/fstring-conversion-tokeninfo-attrs). The debug-prefix and
conversion-default logic here don't depend on that fix and should compose
correctly with it once both land.
f'{x:{y}}' compiled without error but its format_spec JoinedStr carried an
extra, spurious ast.Constant(value='') CPython's own parser never produces:

    src: a = f'{x:{y}}'
    py:  format_spec=JoinedStr(values=[FormattedValue(value=Name('y'))])
    en:  format_spec=JoinedStr(values=[FormattedValue(value=Name('y')), Constant(value='')])

Root cause: the tokenizer emits an FSTRING_MIDDLE token with an empty
string for the zero-length run of literal text between a nested
replacement field and whatever follows it (another field, or the closing
'}'). fstring_format_spec turns every FSTRING_MIDDLE into an ast.Constant
regardless of content, so this degenerate token became a real, spurious
element. build_fstring_format_values only special-cased a *lone* empty
element (a bare ':' with nothing after it), not one trailing other spec
parts, so it survived unfiltered.

Fixed by dropping every empty-string Constant from the flattened parts
before building the format spec's JoinedStr, regardless of position --
matching CPython, which never emits degenerate empty-text nodes into a
JoinedStr. A genuinely empty format spec (f'{x:}') still correctly
produces an empty values list, now via the same general filter rather than
a separate special case.

Confined to fstring_full_format_spec's nested-field handling; the
top-level fstring_mid* loop (the outer JoinedStr) was checked and does not
exhibit the same issue in practice -- verified f'{x}{y}' (two adjacent
replacement fields with no literal in between) already matches ast.parse().

Testing: added test_f_string_nested_format_spec to
tests/core/parser/test_f_strings.py (4 cases: single nested field, two
adjacent nested fields, nested field followed by a literal, and three
adjacent nested fields), validated against ast.parse() via validate_ast.
Parser suite: 441 passed, 2 skipped. Full suite: 4340 passed, 5 skipped,
0 failed.

Pre-existing bug, unrelated to and independent of the debug-expr-prefix
fix earlier on this branch -- only touches build_fstring_format_values,
which the debug-expr fix introduced as a rename/refactor of the prior
inline ternary but did not change the underlying (buggy) filtering logic.
…info-attrs

Consolidates all three PEP-701 f-string-parsing fixes onto one branch/PR,
since they all touch the same small area (fstring_replacement_field /
fstring_full_format_spec and their helpers in base_python_parser.py) and
textually conflict when merged separately:

- check_fstring_conversion's wrong TokenInfo attributes (dc8f266, d35dfb3)
- f'{x=}' losing its 'x=' debug-expr prefix (8434306)
- spurious trailing empty Constant in nested format specs (9d138f6)

Conflict resolution in fstring_replacement_field's grammar action: kept
check_fstring_conversion's int return (this branch's fix) combined with
the debug_expr-prefix branch's 'and format is None' guard on the implicit
!r conversion default (its second fix). enaml_parser.py was regenerated
from the merged enaml.gram rather than hand-merged. tests/test_f_strings.py
conflicts were purely additive (interleaved test functions, no overlapping
changes) and combined directly.

Verified after resolution: parser suite 445 passed/2 skipped, full suite
4344 passed/0 failed/5 skipped. Also spot-checked every case that was
previously blocked by one bug while exercising another -- f'{x=!r}'
(debug expr + explicit conversion), f'{x!r:>{w}}' (conversion + nested
format spec), f'{x=:{w}}' (debug expr + nested format spec) -- all now
match ast.parse() exactly.
@codecov

codecov Bot commented Sep 9, 2026 •

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 94.11765% with 2 lines in your changes missing coverage. Please review.
✅ Project coverage is 66.11%. Comparing base (51aaab0) to head (7f8d5b7).
⚠️ Report is 2 commits behind head on main.

Additional details and impacted files
@@            Coverage Diff             @@
##             main     #629      +/-   ##
==========================================
+ Coverage   66.07%   66.11%   +0.04%     
==========================================
  Files         270      270              
  Lines       26584    26613      +29     
  Branches     3901     3906       +5     
==========================================
+ Hits        17565    17596      +31     
  Misses       7975     7975              
+ Partials     1044     1042       -2     
🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@MatthieuDartiailh

Copy link
Copy Markdown
Member

Could you point Claude to pegen from which I extracted the parser to check if the bug need to fixed there too ?

@bburan

bburan commented Sep 16, 2026

Copy link
Copy Markdown
Contributor Author

Looks like it, but most bugs were fixed by we-like-parsers/pegen#118. Is the fix in Enaml now moot? The one pending bug is the f'{x=}'. See https://github.com/bburan/pegen/tree/fix/py312-fstring-bugs for the fix. If you think that should be included, I can issue a PR.

@MatthieuDartiailh

Copy link
Copy Markdown
Member

Thanks @bburan but I opened pegen in the meantime. Once merged I will update the parser in enaml from the fixed pegen version.

@bburan

bburan commented Sep 16, 2026

Copy link
Copy Markdown
Contributor Author

Closing. Will keep the issue open so that it can be marked off once Enaml is updated from Pegen.

@bburan bburan closed this Sep 16, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

f-string parsing bugs on Python 3.12+: conversion specifiers, debug expressions (f'{x=}'), and nested format specs

2 participants