Repository navigation
FIX: reject dataset size limits a scenario would ignore - #3082
Open
Utkarsh Bahuguna (u7k4rs6) wants to merge 2 commits into
Open
Utkarsh Bahuguna (u7k4rs6) wants to merge 2 commits into
Utkarsh Bahuguna (u7k4rs6) wants to merge 2 commits into
Conversation
Richard Lundeen (richlundeen)
approved these changes
Oct 11, 2026
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #3081.
resolve_configurationnow checks the scenario before building a dataset override:dataset_config(exploitation), dataset names, filters or a limit get a clear "uses a fixed dataset" errorUSES_DATASET_SIZE_LIMIT = False, a non-defaultmax_dataset_sizeis rejected instead of ignored."default"and dataset names still work as before.Both launch and estimate go through the resolver, so the CLI and API get the same error. The GUI doesn't send a limit for these, so nothing changes there.
Same repro as the issue on this branch:
Also dropped
--max-dataset-size 1from the web_injection CLI example in the garak doc since it never did anything.Tests: the three ignoring scenarios with 3 and "all", dataset names still passing through, and exploitation with each override. They fail on main. Some backend tests use bare MagicMock scenarios, so I gave those mocks the base
supported_parameters(). tests/unit/backend, cli and scenario: 4422 passed.