Skip to content

fix(parser): preserve markdown fences and comments inside strings#605

Merged
Aditya-thesys merged 3 commits into
thesysdev:mainfrom
shogun444:fix/markdown-string-parsing
Jun 8, 2026
Merged

fix(parser): preserve markdown fences and comments inside strings#605
Aditya-thesys merged 3 commits into
thesysdev:mainfrom
shogun444:fix/markdown-string-parsing

Conversation

@shogun444

@shogun444 shogun444 commented Jun 5, 2026

Copy link
Copy Markdown
Contributor

What

This PR fixes two parser preprocessing issues where markdown content inside string literals could be incorrectly modified before parsing.

Changes

  • Made stripFences string-aware so fenced code blocks inside string literals are preserved.
  • Preserved string context across line boundaries in stripComments.
  • Added regression tests covering both issues.

Test Plan

  • Not applicable (explain why)
  • Verified locally
pnpm --filter @openuidev/lang-core test

Checklist

  • I linked a related issue, if applicable
  • I updated docs/README when needed
  • I considered backwards compatibility

Closes #581

Closes #596

@Aditya-thesys Aditya-thesys left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

All looks good, could you fix those two small changes. Also maybe update the tests for single quote.

Comment thread packages/lang-core/src/parser/parser.ts Outdated
// Not in string
if (c === '"') {
inStr = true;
if (c === '"' || c === "'") {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Adding single quote tracking will cause issues with Apostrophes.
LLMs many times generates at the start
Here's the code:
Which now will not strip the following fences.
Could you for now scope it to double quotes only.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice catch!|
I'll do this right away.

Comment thread packages/lang-core/src/parser/parser.ts Outdated
Comment on lines +257 to +269
let inStr: false | '"' | "'" = false;
let fenceStart = -1;
while (i < input.length) {
const c = input[i];
if (inStr) {
if (c === "\\" && i + 1 < input.length) {
i += 2; // skip escaped character
continue;
}
if (c === inStr) {
inStr = false;
}
i++;

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: the inStr and closeStr tracking code is almost identical, we can deduplicate them

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yeah i will use a helper here.

@shogun444

shogun444 commented Jun 5, 2026

Copy link
Copy Markdown
Contributor Author

Done!

  • Scoped fence tracking to double quotes only
  • Deduplicated the scanner logic with a helper
  • Added a regression test for apostrophes before fenced code blocks

@shogun444 shogun444 requested a review from Aditya-thesys June 6, 2026 05:11
@Aditya-thesys

Copy link
Copy Markdown
Contributor

Thanks! for the fix, merging it.

@Aditya-thesys Aditya-thesys merged commit 96aaed4 into thesysdev:main Jun 8, 2026
1 check passed
ankit-thesys added a commit that referenced this pull request Jun 10, 2026
….0.8

Version bumps for the first publish since 2026-05-20 (4b663b9):

- @openuidev/react-ui 0.11.8 -> 0.12.0 (minor): component CSS now ships
  in `@layer openui` (#589/#621) and react-syntax-highlighter moved to
  ^16.1.1 (#577, fixes prismjs CVE-2024-53382)
- @openuidev/lang-core 0.2.5 -> 0.2.6 (patch): parser preserves markdown
  fences and comments inside string props (#605)
- @openuidev/cli 0.0.7 -> 0.0.8 (patch): cross-platform template build
  (#601, #627); no functional changes to the published CLI

Remaining packages have no consumer-visible changes since the last
publish and are not republished; the lang wrappers pick up lang-core
0.2.6 transitively via their ^0.2.5 ranges.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Markdown text that includes // in multiline strings is not visible Markdown code blocks are not visible (also in https://www.openui.com/playground)

2 participants