Skip to content

NO-SNOW: cast numeric prompt arg in AI complete test, drop expired date xfails - #4319

Merged
sfc-gh-bkogan merged 4 commits into
mainfrom
bkogan-xfail-ai-complete-named-placeholders
Aug 17, 2026
Merged

sfc-gh-bkogan merged 4 commits into
mainfrom
bkogan-xfail-ai-complete-named-placeholders

Conversation

@sfc-gh-bkogan

@sfc-gh-bkogan sfc-gh-bkogan commented Aug 17, 2026 •

Copy link
Copy Markdown
Collaborator

Summary

test_dataframe_ai_complete_with_named_placeholders has been failing in every daily precommit job (26/26, all clouds) since the blanket date-based xfails in src/conftest.py and tests/integ/test_dataframe_ai.py expired on 2026-06-04. Deterministic, not a flake.

Root cause: the test passes an integer rating column into PROMPT(). Official docs say NUMBER is allowed ("any type coercible to a string (for example, VARCHAR, NUMBER, etc.), or FILE"), but the runtime stores the number verbatim and rejects it with unsupported argument type. With return_error_details off, that surfaces as a NULL completion.

Minimal repro:

SELECT AI_COMPLETE('llama3.1-8b', PROMPT('{0}', 1), {'return_error_details': true});
-- {"value": null, "error": "unsupported argument type"}

Sibling llama3.1-8b tests pass because they only pass string columns.

Changes

  • Cast rating to StringType in input_columns so this test covers named-placeholder rewriting instead of tripping over the server-side type bug.
  • Remove the two expired date-based xfails (doctest hook in src/conftest.py, module-level mark in test_dataframe_ai.py).

Server-side follow-up for the AI SQL team: either cast string-coercible args in PROMPT() as documented, or have the prompt parser stringify scalars. Separately, a type error collapsing to silent NULL is a bad failure mode.

Test plan

  • Daily precommit passes with test_dataframe_ai_complete_with_named_placeholders green.
  • Other dataframe_ai tests and doctests now run unmasked; confirm no other latent failures.

Verified the casted SQL against a live deployment: all three rows return non-empty completions. Pytest integ run is covered by CI (no local snowpark integ credentials here).

… xfails

test_dataframe_ai_complete_with_named_placeholders fails deterministically
(26/26 daily precommit jobs, all clouds, through 5 pytest reruns) because
AI_COMPLETE returns NULL for its prompt. Scope the xfail to that one test so
the ~50 other AI tests in the file keep gating, and use strict=False so an
XPASS surfaces once the underlying issue is resolved.

The blanket date-based xfails in the module pytestmark and the doctest hook
expired on 2026-06-04, so they were dead code masking nothing.
@sfc-gh-bkogan sfc-gh-bkogan added the NO-CHANGELOG-UPDATES This pull request does not need to update CHANGELOG.md label Aug 17, 2026

@sfc-gh-bkogan sfc-gh-bkogan left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

SHIP WITH NITS

Clean, well-scoped test-only change: expired blanket xfails removed with no dangling references, and the narrow xfail matches the file's existing convention. One tradeoff worth acknowledging inline about strict=False masking non-target failures.

Comment thread tests/integ/test_dataframe_ai.py Resolved
Comment thread src/conftest.py Resolved
Reproduced on a live deployment: PROMPT() stores an integer argument as a
number, and the executor only accepts text/file arguments, so the row comes
back as a NULL completion. Point the xfail at that instead of "infrastructure".
@sfc-gh-bkogan sfc-gh-bkogan changed the title NO-SNOW: xfail AI complete named-placeholders test, drop expired date xfails NO-SNOW: xfail AI complete named-placeholders test (PROMPT() rejects non-string args) Aug 17, 2026
PROMPT() only accepts string arguments, so the integer rating column reached
the server as a number and every row came back as a NULL completion. Cast it
so the test exercises named placeholders instead of tripping over arg typing.
@sfc-gh-bkogan sfc-gh-bkogan changed the title NO-SNOW: xfail AI complete named-placeholders test (PROMPT() rejects non-string args) NO-SNOW: cast numeric prompt arg in AI complete test, drop expired date xfails Aug 17, 2026
Docs allow NUMBER prompt args; the runtime currently rejects them. Make the
comment match that rather than implying strings are the only supported type.
@sfc-gh-bkogan
sfc-gh-bkogan marked this pull request as ready for review August 17, 2026 23:05
@sfc-gh-bkogan
sfc-gh-bkogan requested a review from a team as a code owner August 17, 2026 23:05
@codecov-commenter

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 95.24%. Comparing base (1c389a0) to head (94660c5).

Additional details and impacted files
@@           Coverage Diff           @@
##             main    #4319   +/-   ##
=======================================
  Coverage   95.23%   95.24%           
=======================================
  Files         171      171           
  Lines       44720    44720           
  Branches     7676     7676           
=======================================
+ Hits        42591    42593    +2     
+ Misses       1341     1340    -1     
+ Partials      788      787    -1     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@snowflake-security-bot snowflake-security-bot Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Snowflake Security Review

Security grade: A — Passed ✅

This PR was classified as LOW risk by the automated pre-screen.

@sfc-gh-bkogan
sfc-gh-bkogan merged commit 6f88fc5 into main Aug 17, 2026
32 of 35 checks passed
@sfc-gh-bkogan
sfc-gh-bkogan deleted the bkogan-xfail-ai-complete-named-placeholders branch August 17, 2026 23:28
@github-actions github-actions Bot locked and limited conversation to collaborators Aug 17, 2026
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Labels

NO-CHANGELOG-UPDATES This pull request does not need to update CHANGELOG.md

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants