Skip to content

FIX: reject dataset size limits a scenario would ignore - #3082

Open
Utkarsh Bahuguna (u7k4rs6) wants to merge 2 commits into
microsoft:mainfrom
u7k4rs6:fix/dataset-limit-not-applicable
Open

Utkarsh Bahuguna (u7k4rs6) wants to merge 2 commits into
microsoft:mainfrom
u7k4rs6:fix/dataset-limit-not-applicable

Conversation

@u7k4rs6

Copy link
Copy Markdown
Contributor

Fixes #3081.

resolve_configuration now checks the scenario before building a dataset override:

  • if the scenario doesn't take dataset_config (exploitation), dataset names, filters or a limit get a clear "uses a fixed dataset" error
  • if it has USES_DATASET_SIZE_LIMIT = False, a non-default max_dataset_size is rejected instead of ignored. "default" and dataset names still work as before.

Both launch and estimate go through the resolver, so the CLI and API get the same error. The GUI doesn't send a limit for these, so nothing changes there.

Same repro as the issue on this branch:

garak.system_prompt_extraction max_dataset_size=3: ValueError: Scenario 'garak.system_prompt_extraction' doesn't use a dataset size limit; ...
garak.exploitation max_dataset_size=3: ValueError: Scenario 'garak.exploitation' uses a fixed dataset, so it doesn't take dataset names, dataset filters or a dataset size limit.

Also dropped --max-dataset-size 1 from the web_injection CLI example in the garak doc since it never did anything.

Tests: the three ignoring scenarios with 3 and "all", dataset names still passing through, and exploitation with each override. They fail on main. Some backend tests use bare MagicMock scenarios, so I gave those mocks the base supported_parameters(). tests/unit/backend, cli and scenario: 4422 passed.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

BUG: max_dataset_size is silently ignored by some garak scenarios

2 participants