feat: LLM check type support in contract JSON schema#2613
Draft
LaurenDebruyn wants to merge 11 commits intomainfrom
Draft
feat: LLM check type support in contract JSON schema#2613LaurenDebruyn wants to merge 11 commits intomainfrom
LaurenDebruyn wants to merge 11 commits intomainfrom
Conversation
Add core infrastructure for the new LLM check type:
- JSON schema definition for llm check in data contracts
- Default check name registration ("LLM validation passes")
- Remediation block parsing (sql/llm strategies with references and tools)
- Contract verification model extensions for remediation metadata
- Unit tests for remediation YAML parsing
Co-Authored-By: Claude Opus 4.6 <[email protected]>
…lasses Remediation is not used by Soda Core at runtime, so there's no need to parse its internal structure. Store it as an opaque dict for downstream consumers to interpret. Co-Authored-By: Claude Opus 4.6 <[email protected]>
for more information, see https://pre-commit.ci
Copy the latest schema from soda-server which includes remediation, reconciliation, diagnostics, failed_rows, check_attributes, and updated descriptions with doc links. Co-Authored-By: Claude Opus 4.6 <[email protected]>
for more information, see https://pre-commit.ci
Remediation must be properly parsed so it can be ingested by Soda Cloud when contract verification results are pushed. Restore the structured dataclasses for parsing and add remediation serialization to the check result cloud dict. Co-Authored-By: Claude Opus 4.6 <[email protected]>
Remediation is handled by the remediation-cli (Gravity), which parses the YAML itself. Keeping parsing in soda-core added dead code that nothing consumed at runtime. Removed: - RemediationYaml / RemediationStrategyYaml dataclasses and _parse_remediation() - Remediation / RemediationStrategy / RemediationReference / RemediationTool from Check - _build_remediation() in CheckImpl and _build_remediation_cloud_dict() in SodaCloud - $defs/remediation and all 14 check-type references in the JSON schema - test_remediation_parsing.py Co-Authored-By: Claude Opus 4.6 <[email protected]>
LaurenDebruyn
changed the title
Rebuild JSON schema from main + only the LLM check type definition. Removes the large formatting/description rewrite that was obscuring the actual change in the PR diff. Co-Authored-By: Claude Opus 4.6 <[email protected]>
The schema is not used by soda-core at runtime (json_schema_verifier.py is fully commented out). Schema updates belong in the soda-webapp repo. Co-Authored-By: Claude Opus 4.6 <[email protected]>
Co-Authored-By: Claude Opus 4.6 <[email protected]>
…tabricks Co-Authored-By: Claude Sonnet 4.6 <[email protected]>
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
llmcheck type in data contracts (prompt, threshold, references)referencesproperty to thellmcheck schema for reference dataset lookups during LLM validationNote: Remediation contract parsing was added and then removed — remediation is handled entirely by the remediation-cli (Gravity), not soda-core.
Companion PR: sodadata/soda-extensions#298 —
feat/llm-checks-and-remediation(soda-llm package with tools, reference lookup, check implementation)Test plan
🤖 Generated with Claude Code