feat(thinking): add tag selection API with include/exclude filtering (#1790) by Abtiotm · Pull Request #1795 · embabel/embabel-agent

Abtiotm · 2026-07-18T16:41:14Z

Summary

Add tag selection to thinking extraction: thinking(include, exclude) selects which XML-style tags are extracted as reasoning blocks, per the consensus reached in #1790. Previously every XML tag was dynamically detected, so ordinary markup like <div>...</div> could surface as a thinking block.

Changes

ThinkingTagSelection (new): include/exclude filtering for TAG blocks; non-TAG blocks (prefix/untagged) are always retained; tags validated as XML names; a tag cannot be in both sets
PromptRunner.thinking(include) / thinking(include, exclude) / thinking(tag): thinking(tag) is a shorthand for thinking(include = setOf(tag)). Calling thinking() without arguments preserves default dynamic detection
Warn on missing tags: when tag selection is used, a WARN log entry is emitted if a declared tag is absent from the system prompt, making silent hasThinking() == false diagnosable
Docs: "Selecting Thinking Tags" section in thinking reference with Java/Kotlin examples

Usage

// Select only the reasoning tag, suppress markup tags
runner.thinking(include = setOf("reasoning"), exclude = setOf("div"))
    .createObject(prompt, MonthItem::class.java)

// Single-tag shorthand
runner.thinking("decision_reasoning").createObject(prompt, TravelPlan::class.java)

// Default dynamic detection unchanged
runner.thinking().createObject(prompt, TravelPlan::class.java)

Tag selection does not modify the prompt — the LLM only produces reasoning inside <tag>...</tag> if the prompt asks for it.

Testing

Added/updated tests in DelegatingThinkingTest (tag selection delegation + filtering), LlmInteractionSerializationTest (ThinkingTagSelection serialization + missing-tag warning), ChatClientLlmOperationsThinkingTest, DelegatingStreamingPromptRunnerTest. All pass.

Closes #1790

igordayen · 2026-07-18T20:22:44Z

@Abtiotm - thank you for your contribution! Could you please comment on one inquiry in the issue?

But what IF a user-created prompt with reasoning includes tag A, but some-tag=tag B?
Trust that the user is well aware and will use only one of the mechanisms: regular prompting OR auto-generated system prompt?

Thank you.

looping @azanux and @jorander

igordayen

@Abtiotm - thanks for contribution, added comment, and also there are some open inquiries on issue itself.

igordayen · 2026-07-18T20:25:36Z

+
+    override fun contribution(): String {
+        return buildString {
+            append("You MUST generate reasoning inside <$tag>...</$tag> tags ")


Should it be a system prompt?

igordayen · 2026-07-18T20:43:27Z

And what is the proper interpretation of thinking () ==> default tag "think" or no tag at all, remains unchanged?

and also needs to be added to the respective asciidoc (.adoc) in the thinking section.
Thanks

azanux

@Abtiotm thanks for your contribution , all good , just some few comments.

azanux · 2026-07-20T05:02:06Z

+        }
+    }
+
+    override fun promptContribution(): PromptContribution {


is this override needed?

promptContribution() duplicates the default impl (PromptContributor.kt:85-91), except it forces role = "thinking_tag_instruction". But that role is never read for this contributor.

may be just you just to provide location and role data via overriding those properties: override val role = "thinking_tag_instruction". ?

or maybe the custom role is not need as It's never read for this contributor: the only callers of promptContribution() (OperationContext.kt:98, ActionContext.kt:169) run at PromptRunner build time, while the injector is appended later via withPromptContributors

azanux · 2026-07-20T05:18:48Z

+        get() = PromptContributionLocation.END
+
+    override fun contribution(): String {
+        return buildString {


agree with @igordayen what is user mispell tag ?

tag is interpolated raw ("<$tag>...</$tag>"). "", "my tag", "a><b" all produce a malformed instruction. Add a require(...) in init matching [a-zA-Z][a-zA-Z0-9_-]*

so the failure surfaces at the call site instead of as a silent empty result.

and the injection side accepts anything the extraction side can't read:

"" → <></>, meaningless instruction

"my tag" → model complies, closing tag fails the regex → thinkingBlocks comes back empty, no error, after a paid LLM call (diificult to debug for user)

"a><b" → arbitrary markup injected into the system prompt

azanux · 2026-07-20T05:33:44Z

+            val result = operations.createObjectIfPossible(messages, outputClass)
+
+            // Then: Should have called withPromptContributors with ThinkingTagInjector
+            verify { mockDelegate.withPromptContributors(any()) }


the test doesn't check what its name promises

verify { mockDelegate.withPromptContributors(any()) }

any() accepts anything. Change ThinkingTagInjector(thinkingTag) to ThinkingTagInjector("think") at DelegatingThinking.kt:59 - the caller's tag is silently dropped and this test still passes

The assertEquals(expectedResponse, result) lines don't help either: expectedResponse is what the test told the mock to return.

you need to capture the argument instead so we are sure that we have the good tag at the end

azanux · 2026-07-20T05:46:32Z

+        delegateForTag().evaluateConditionWithThinking(condition, context, confidenceThreshold)
+
+    private fun delegateForTag(): PromptExecutionDelegate {
+        if (thinkingTag == null) return delegate


this answer @igordayen question Bare thinking() injects nothing; only thinking(tag) adds the instruction.

Good call - backward compatible , so update the doc , otherwise people need to read the source code to know how it works

also : PromptRunner.kt:502

fun thinking(tag: String): Thinking = thinking()

While updating the docs, PromptRunner.kt:502 (thinking(tag) = thinking()) could use a line noting that implementations overriding thinking() should override this too.

Abtiotm · 2026-07-20T11:23:07Z

Regarding the tag A / tag B case: thinking(tag) only adds a system-level instruction requesting the specified tag. It does not inspect or rewrite user-provided prompts, and extraction remains unchanged: all recognized thinking blocks are returned rather than only blocks matching the supplied tag.

If the user prompt requests tag A while thinking(\"tagB\") is used, the instructions may conflict and model behavior is not guaranteed. Callers should therefore use either explicit prompt instructions with thinking(), or automatic injection with thinking(tag), and avoid specifying different tags through both mechanisms.

We are intentionally not adding prompt conflict detection in this PR, as reliably inferring tag intent from arbitrary natural-language prompts is outside its scope.

igordayen · 2026-07-20T14:12:10Z

@Abtiotm - could you please comment on the comments and mark "resolved" as needed? Thank you

jorander · 2026-07-20T15:14:24Z

@Abtiotm @igordayen I think this is mixing different types of thinking/reasoning. Please see my comment on the original issue (#1790) why I think this is a bad idea and how I think this type of application level reasoning/motivation should be implemented.

igordayen · 2026-07-21T00:22:27Z

@Abtiotm To evaluate the solution:

Could you please plug in the example I posted on the issue itself and validate the output?

Test 1: Instead of my system prompt, use yours, auto-injected.
Test 2: Use 2 system prompts: auto-injected and mine, but remove any references to tag <decision_reasoning>...</decision_reasoning>
Test 3: Use 2 system prompts as-is.

As we learned from philosophy: experiment is the criterion of truth:) - Pierre Duhem.

Thank you.

Abtiotm · 2026-07-21T04:55:44Z

Thanks @jorander, @igordayen, and @azanux for the feedback.

I agree that native/internal model thinking and application-level reasoning or motivation are different concepts. The intention of this PR is not to expose or redefine native model thinking. It only adds an opt-in prompt instruction for the tagged, visible reasoning blocks that the existing thinking() extraction mechanism already supports.

However, I understand the concern that naming this API thinking(tag) may blur that distinction.

Before making further implementation changes, could the maintainers please confirm which direction is preferred?

Keep thinking(tag) as an opt-in convenience for the existing tagged-block extraction mechanism; or
Move the prompt-injection behavior to a separate application-level reasoning API or PromptContributor, leaving thinking() exclusively for internal/native model thinking.

If option 1 is accepted, I will address the current review comments by:

validating tag names;
removing the redundant promptContribution() override;
strengthening the tests to verify the exact injected tag;
documenting the difference between thinking() and thinking(tag); and
running the three requested experiments with the TravelPlan example and reporting the model, prompt combination, extracted blocks, and structured result.

I will avoid expanding the API further until the intended semantic boundary is confirmed.

igordayen · 2026-07-21T16:13:10Z

@Abtiotm - please see the discussion on the issue.

If I'm reading the discussion correctly, it's largely pointing to Option 1.
But let's wait for consensus before proceeding further.
Thanks.

alexheifetz · 2026-07-24T17:27:29Z

@Abtiotm is PR intended for Embabel 1.5.0 (Boot 4.1 / Spring AI 2.0.0) or Embabel 1.0.1 (Boot 3.5, Spring AI 1.1.x)
If intended for Embabel 1.1.x please close and reraise against 1.0.x branch

igordayen · 2026-07-30T00:12:12Z

@Abtiotm - could you please check the issue? We reached consensus. Thank you

igordayen · 2026-07-31T00:12:05Z

@Abtiotm - there is not much activity on PR recently. What is your timeline? thanks

…mbabel#1790) Add ThinkingTagSelection with include/exclude filtering for XML-style thinking blocks. Non-TAG blocks (prefix/untagged) are always retained. Missing tags trigger a WARN log to make silent hasThinking()==false diagnosable. thinking(tag) convenience shorthand for single-tag extraction.

igordayen · 2026-08-07T20:49:28Z

@Abtiotm - thanks for moving this forward.

I just realized that there would be yet another option to express the same, namely through:

class Thinking private constructor(
    val enabled: Boolean = false,
    val tokenBudget: Int? = null,
    val extractThinking: Boolean = false,
    val tagsInclusion...
    val tagsExclusion
)

It would be good to have consistency in API usage.

@jorander @azanux @arnabnandy7

Another approach - to keep the current design "as-is" but internally convert it into LLMOptions.

You may not need to change "create"-API signatures if you convert to LLMOptions.Thinking.

Thank you

Abtiotm force-pushed the codex/issue-1790-thinking-prompt-tag branch from f1b6883 to 77c9c4c Compare July 18, 2026 20:10

igordayen reviewed Jul 18, 2026

View reviewed changes

igordayen added the release-2.0 label Jul 19, 2026

azanux reviewed Jul 20, 2026

View reviewed changes

Abtiotm force-pushed the codex/issue-1790-thinking-prompt-tag branch from 77c9c4c to 7f68fca Compare August 7, 2026 05:58

Abtiotm changed the title ~~feat: add thinking(tag: String) API to auto-inject reasoning instruction~~ feat(thinking): add tag selection API with include/exclude filtering (#1790) Aug 7, 2026

Uh oh!

Conversation

Abtiotm commented Jul 18, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Changes

Usage

Testing

Uh oh!

igordayen commented Jul 18, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

igordayen left a comment

Choose a reason for hiding this comment

Uh oh!

igordayen Jul 18, 2026

Choose a reason for hiding this comment

Uh oh!

igordayen commented Jul 18, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

azanux left a comment

Choose a reason for hiding this comment

Uh oh!

azanux Jul 20, 2026

Choose a reason for hiding this comment

Uh oh!

azanux Jul 20, 2026

Choose a reason for hiding this comment

Uh oh!

azanux Jul 20, 2026

Choose a reason for hiding this comment

Uh oh!

azanux Jul 20, 2026

Choose a reason for hiding this comment

Uh oh!

Abtiotm commented Jul 20, 2026

Uh oh!

igordayen commented Jul 20, 2026

Uh oh!

jorander commented Jul 20, 2026

Uh oh!

igordayen commented Jul 21, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Abtiotm commented Jul 21, 2026

Uh oh!

igordayen commented Jul 21, 2026

Uh oh!

alexheifetz commented Jul 24, 2026

Uh oh!

igordayen commented Jul 30, 2026

Uh oh!

igordayen commented Jul 31, 2026

Uh oh!

igordayen commented Aug 7, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

Abtiotm commented Jul 18, 2026 •

edited

Loading

igordayen commented Jul 18, 2026 •

edited

Loading

igordayen commented Jul 18, 2026 •

edited

Loading

igordayen commented Jul 21, 2026 •

edited

Loading