diff --git a/CHANGELOG.md b/CHANGELOG.md index 55bb558c0..18a05ce4f 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -7,6 +7,13 @@ Versioning: [Semantic Versioning](https://semver.org/spec/v2.0.0.html) --- +## [Unreleased] + +### Fixed +- **Multimodal (Audio)**: Raise explicit `ValueError` or `FileNotFoundError` from `Audio.from_url()` and `Audio.from_path()` instead of relying on bare `assert` statements that can disappear under `python -O`. ([#2361](https://github.com/567-labs/instructor/pull/2361)) + +--- + ## [1.15.5] - 2026-06-28 ### Fixed diff --git a/docs/blog/posts/semantic-validation-structured-outputs.md b/docs/blog/posts/semantic-validation-structured-outputs.md index 5dd2bf02a..386ff587e 100644 --- a/docs/blog/posts/semantic-validation-structured-outputs.md +++ b/docs/blog/posts/semantic-validation-structured-outputs.md @@ -334,7 +334,7 @@ Semantic validation represents a significant advancement in ensuring the quality As these techniques mature, we can expect to see semantic validation become a standard part of AI application development, especially in regulated industries where output quality is critical. -To get started with semantic validation in your projects, check out the [Semantic Validation documentation](https://python.useinstructor.com../../concepts/semantic_validation/.md) and explore the various examples and patterns. +To get started with semantic validation in your projects, check out the [Semantic Validation documentation](../../concepts/semantic_validation.md) and explore the various examples and patterns. This approach isn't just a technical improvement-it's a fundamental shift in how we think about validation, moving from rigid rules to intelligent understanding of content and context. @@ -345,4 +345,4 @@ This approach isn't just a technical improvement-it's a fundamental shift in how ## See Also - [Validation Deep Dive](validation-part1.md) - Foundation validation concepts - [Anthropic Prompt Caching](anthropic-prompt-caching.md) - Optimize validation costs -- [Monitoring with Logfire](logfire.md) - Track validation performance \ No newline at end of file +- [Monitoring with Logfire](logfire.md) - Track validation performance diff --git a/docs/concepts/patching.md b/docs/concepts/patching.md index 4fd3dd80c..aaae72123 100644 --- a/docs/concepts/patching.md +++ b/docs/concepts/patching.md @@ -37,7 +37,7 @@ Different providers support different modes for structured extraction. Instructo Uses the provider's function/tool calling API. This is the default for OpenAI. -Supported by: OpenAI, Anthropic (ANTHROPIC_TOOLS), Google (GENAI_TOOLS), Ollama (for supported models) +Supported by: OpenAI, Anthropic, Google, Ollama (for supported models) ### JSON Mode @@ -136,14 +136,14 @@ The patched client's `create()` method: ### Anthropic -- Default mode: `ANTHROPIC_TOOLS` (tool use) +- Default mode: `Mode.TOOLS` (tool use) - Uses Claude's native tool calling API ### Google Gemini -- Default mode: `GENAI_TOOLS` (function calling) +- Default mode: `Mode.TOOLS` (function calling) - Requires `jsonref` package for tool calling -- Some limitations with strict validation and enums +- Union types (except `Optional`) are not supported ### Ollama (Local Models) diff --git a/docs/integrations/genai.md b/docs/integrations/genai.md index 3cb312497..d60bdef6f 100644 --- a/docs/integrations/genai.md +++ b/docs/integrations/genai.md @@ -31,7 +31,7 @@ We currently have two modes for Gemini !!! note "Backwards Compatibility" - The provider-specific modes (`Mode.TOOLS`, `Mode.JSON`, `Mode.JSON`) are still supported but emit deprecation warnings and map to the generic modes (`Mode.TOOLS`, `Mode.JSON`). + The legacy provider-specific modes (`Mode.GENAI_TOOLS`, `Mode.GENAI_JSON`, `Mode.GENAI_STRUCTURED_OUTPUTS`) are still supported but emit deprecation warnings and automatically map to the generic modes (`Mode.TOOLS`, `Mode.JSON`). ## Installation @@ -41,9 +41,9 @@ pip install "instructor[google-genai]" ## Basic Usage -!!! warning "Unions and Optionals" +!!! warning "Union Types" - Gemini doesn't have support for Union and Optional types in the structured outputs and tool calling integrations. We currently throw an error when we detect these in your response model. + Gemini does not support Union types (except for `Optional`) in structured outputs and tool calling. Use separate response models or `Literal` types instead. `Optional` fields (i.e., `X | None`) work correctly. Getting started with Instructor and the genai SDK is straightforward. Just create a Pydantic model defining your output structure, patch the genai client, and make your request with a response_model parameter: @@ -547,11 +547,11 @@ print(response) !!! warning "Streaming Limitations" - **As of July 11, 2025, Google GenAI does not support streaming with tool/function calling or structured outputs for regular models.** + Google GenAI does not support streaming with tool/function calling (`Mode.TOOLS`) for regular models. To use streaming with Gemini: - - `Mode.TOOLS` and `Mode.JSON` do not support streaming with regular models - - To use streaming, you must use `Partial[YourModel]` explicitly or switch to other modes like `Mode.JSON` - - Alternatively, set `stream=False` to disable streaming + - Use `Mode.JSON` which supports streaming via `create_partial` and `create_iterable` + - Or use `Partial[YourModel]` explicitly with `Mode.JSON` + - Alternatively, set `stream=False` to disable streaming when using `Mode.TOOLS` Streaming allows you to process responses incrementally rather than waiting for the complete result. This is extremely useful for making UI changes feel instant and responsive. diff --git a/docs/integrations/google.md b/docs/integrations/google.md index e008ead78..27abe0fcb 100644 --- a/docs/integrations/google.md +++ b/docs/integrations/google.md @@ -321,13 +321,12 @@ for user in users: #> name='Mike' age=28 ``` -## Known Limitations (as of Nov 12, 2024) +## Known Limitations Google Gemini has the following known limitations when used with Instructor: -1. **Union Types**: Gemini does not support Union types (except for Optional). Use separate response models or Literal types instead. -2. **Enum Types**: Gemini returns string values instead of properly typed Enum instances. You may need to manually convert strings to enums after extraction. -3. **Union Streaming**: Streaming is not supported for Union types with Iterable. +1. **Union Types**: Gemini does not support Union types (except for `Optional`). Use separate response models or `Literal` types instead. +2. **Union Streaming**: Streaming is not supported for Union types with Iterable. These limitations are specific to Google Gemini and do not affect other providers like OpenAI or Anthropic. Tests automatically skip these features for Google to prevent failures. @@ -340,7 +339,7 @@ We provide several modes to make it easy to work with the different response mod !!! note "Backwards Compatibility" - Legacy provider-specific modes (for example `Mode.TOOLS`, `Mode.JSON`, `Mode.JSON`, `Mode.TOOLS`) are deprecated. They emit warnings and map to the generic modes. + Legacy provider-specific modes (such as `Mode.GENAI_TOOLS`, `Mode.GENAI_JSON`, `Mode.GENAI_STRUCTURED_OUTPUTS`) are deprecated. They emit warnings and automatically map to the generic modes (`Mode.TOOLS`, `Mode.JSON`). !!! info "Mode Selection" When using `from_provider`, the appropriate mode is automatically selected based on the provider and model capabilities. @@ -349,9 +348,9 @@ We provide several modes to make it easy to work with the different response mod Google offers several Gemini models: -- Gemini Flash (General purpose) -- Gemini Pro (Multimodal) -- Gemini Flash-8b (Coming soon) +- Gemini Flash (General purpose, fast inference) +- Gemini Pro (Advanced reasoning, multimodal) +- Gemini Flash-8b (Lightweight, cost-effective) ## Using Gemini's Multimodal Capabilities @@ -410,7 +409,9 @@ import vertexai from vertexai.generative_models import GenerativeModel vertexai.init(project="your-project", location="us-central1") -client = instructor.from_provider("google/gemini-2.5-flash", vertexai=True), +client = instructor.from_provider( + "google/gemini-2.5-flash", + vertexai=True, mode=instructor.Mode.TOOLS, ) ``` diff --git a/docs/integrations/vertex.md b/docs/integrations/vertex.md index 7934082cb..2813e780e 100644 --- a/docs/integrations/vertex.md +++ b/docs/integrations/vertex.md @@ -242,7 +242,9 @@ from vertexai.generative_models import GenerativeModel vertexai.init(project="your-project", location="us-central1") -client = instructor.from_provider("google/gemini-2.5-flash", vertexai=True), +client = instructor.from_provider( + "google/gemini-2.5-flash", + vertexai=True, mode=instructor.Mode.TOOLS, ) ``` diff --git a/examples/crm/run.py b/examples/crm/run.py index db931b923..98a0e8eb2 100644 --- a/examples/crm/run.py +++ b/examples/crm/run.py @@ -47,7 +47,7 @@ def query_crm(query: str) -> CRMSearchQuery: "role": "system", "content": """ You are a world class CRM search career generator. - You will take the user query and decompose it into a set of CRM queries queries. + You will take the user query and decompose it into a set of CRM queries. """, }, {"role": "user", "content": query}, diff --git a/instructor/v2/core/multimodal.py b/instructor/v2/core/multimodal.py index 6aa85d066..40aa046d4 100644 --- a/instructor/v2/core/multimodal.py +++ b/instructor/v2/core/multimodal.py @@ -329,9 +329,11 @@ def from_url(cls, url: str) -> Audio: return cls.from_gs_url(url) response = requests.get(url) content_type = response.headers.get("content-type") - assert content_type in VALID_AUDIO_MIME_TYPES, ( - f"Invalid audio format. Must be one of: {', '.join(VALID_AUDIO_MIME_TYPES)}" - ) + if content_type not in VALID_AUDIO_MIME_TYPES: + raise ValueError( + f"Unsupported audio format: {content_type}. " + f"Must be one of: {', '.join(VALID_AUDIO_MIME_TYPES)}" + ) data = base64.b64encode(response.content).decode("utf-8") return cls(source=url, data=data, media_type=content_type) @@ -340,7 +342,11 @@ def from_url(cls, url: str) -> Audio: def from_path(cls, path: Union[str, Path]) -> Audio: # noqa: UP007 """Create an Audio instance from a file path.""" path = Path(path) - assert path.is_file(), f"Audio file not found: {path}" + if not path.is_file(): + raise FileNotFoundError(f"Audio file not found: {path}") + + if path.stat().st_size == 0: + raise ValueError("Audio file is empty") mime_type = mimetypes.guess_type(str(path))[0] @@ -352,9 +358,11 @@ def from_path(cls, path: Union[str, Path]) -> Audio: # noqa: UP007 ): # <--- this is the case for aac audio files in Windows mime_type = "audio/aac" - assert mime_type in VALID_AUDIO_MIME_TYPES, ( - f"Invalid audio format. Must be one of: {', '.join(VALID_AUDIO_MIME_TYPES)}" - ) + if mime_type not in VALID_AUDIO_MIME_TYPES: + raise ValueError( + f"Unsupported audio format: {mime_type}. " + f"Must be one of: {', '.join(VALID_AUDIO_MIME_TYPES)}" + ) data = base64.b64encode(path.read_bytes()).decode("utf-8") return cls(source=str(path), data=data, media_type=mime_type) diff --git a/tests/v2/test_core_multimodal_runtime.py b/tests/v2/test_core_multimodal_runtime.py index b926ec037..0ac0e63a2 100644 --- a/tests/v2/test_core_multimodal_runtime.py +++ b/tests/v2/test_core_multimodal_runtime.py @@ -218,3 +218,41 @@ def test_audio_from_path_normalizes_windows_wav_and_aac_mime_types( assert wav_audio.media_type == "audio/wav" assert aac_audio.media_type == "audio/aac" + + +def test_audio_from_url_raises_value_error_for_unsupported_content_type( + monkeypatch: pytest.MonkeyPatch, +) -> None: + class _Response: + headers = {"content-type": "video/mp4"} + content = b"\x00\x01\x02" + + monkeypatch.setattr("requests.get", lambda *_args, **_kwargs: _Response()) + + with pytest.raises(ValueError, match="Unsupported audio format"): + Audio.from_url("https://example.com/clip.mp4") + + +def test_audio_from_path_raises_value_error_for_unsupported_format( + tmp_path: Path, +) -> None: + txt_path = tmp_path / "clip.txt" + txt_path.write_bytes(b"not audio") + + with pytest.raises(ValueError, match="Unsupported audio format"): + Audio.from_path(txt_path) + + +def test_audio_from_path_raises_value_error_for_empty_file(tmp_path: Path) -> None: + empty_path = tmp_path / "empty.wav" + empty_path.write_bytes(b"") + + with pytest.raises(ValueError, match="Audio file is empty"): + Audio.from_path(empty_path) + + +def test_audio_from_path_raises_file_not_found_for_missing_file(tmp_path: Path) -> None: + missing_path = tmp_path / "missing.wav" + + with pytest.raises(FileNotFoundError): + Audio.from_path(missing_path)