diff --git a/README.md b/README.md index 31118663..974f784d 100644 --- a/README.md +++ b/README.md @@ -261,7 +261,7 @@ _Not all options working now!_ For video URLs, the video language is used automatically. WARNING: setting this for non-matching songs will degrade alignment quality (e.g. --language en for German songs). - --whisper_batch_size Segments processed in parallel. Lower if low on GPU mem: slower, but transcription is UNCHANGED (the safe lever) >> ((default) is 16) + --whisper_batch_size Segments processed in parallel. 'auto' (default) scales to GPU VRAM: 16 (~8GB and larger) / 8 (5-7GB) / 4 (smaller). Lower manually if still low on GPU mem: slower, but transcription is UNCHANGED (the safe lever) --whisper_compute_type Change to "int8" to save more GPU mem at a small accuracy cost; use only if lowering the batch size is not enough >> ((default) is "float16" for cuda devices, "int8" for cpu) --keep_numbers Numbers will be transcribed as numerics instead of as words --vad_onset VAD (Voice Activity Detection) speech activation threshold (0.0-1.0). Lower @@ -854,12 +854,15 @@ uv pip install --index-url https://download.pytorch.org/whl/cu121 torch torchvis #### Crashes due to low VRAM -The default Whisper model (`large-v2`) needs more than 8 GB VRAM. On a smaller -GPU there are two independent ways to fit it, in order of preference: +At full batch size (16) the default Whisper model (`large-v2`) needs roughly +8 GB of VRAM. UltraSinger scales the batch size to the detected GPU memory +automatically (16 on ~8 GB and larger, 8 on 5-7 GB, 4 on smaller cards), so +most GPUs fit as-is. If it still runs out of memory, there are two independent +levers, in order of preference: -1. **Lower the batch size** (`--whisper_batch_size 4`, or 2/1). This processes - fewer audio segments in parallel: it is slower, but the transcription is - **unchanged** — so this is the safe lever, try it first. +1. **Lower the batch size further** (`--whisper_batch_size 2`, or 1). This + processes fewer audio segments in parallel: it is slower, but the + transcription is **unchanged** — so this is the safe lever, try it first. 2. **Switch to int8** (`--whisper_compute_type int8`). This halves the model's memory at a small accuracy cost — add it only if lowering the batch size alone is not enough. diff --git a/install/auto_install.bat b/install/auto_install.bat index 4e297713..ccd85640 100644 --- a/install/auto_install.bat +++ b/install/auto_install.bat @@ -220,13 +220,13 @@ if "!BUILD!"=="cpu" ( ) else ( if not defined GPU_VRAM ( echo GPU VRAM could not be verified ^(forced CUDA build^). - echo If your GPU has less than 8 GB VRAM, the default Whisper model - echo ^(large-v2^) may run out of memory. Two independent ways to fit it - echo ^(not set automatically^): - echo --whisper_batch_size 4 Fewer segments in parallel: slower, - echo but the transcription is UNCHANGED - echo ^(the safe lever; lower to 2 or 1 if it - echo still runs out^). + echo UltraSinger scales the Whisper batch size automatically to the + echo GPU memory it detects at runtime ^(16 on ~8 GB and larger, 8 on + echo 5-7 GB, 4 on smaller cards^), which usually makes the default + echo Whisper model ^(large-v2^) fit. If it still runs out of memory: + echo --whisper_batch_size 2 ^(or 1^) Fewer segments in parallel: + echo slower, but the transcription is + echo UNCHANGED ^(the safe lever^). echo --whisper_compute_type int8 Halves the model's memory at a echo small accuracy cost - add this only if echo lowering the batch size is not enough. @@ -238,13 +238,15 @@ if "!BUILD!"=="cpu" ( echo GUI users: enable this under Settings -^> 'Remote Speech-to-Text'. ) else ( if !GPU_VRAM! LSS 8192 ( - echo Your GPU has less than 8 GB VRAM. The default Whisper model - echo ^(large-v2^) may run out of memory. Two independent ways to fit - echo it ^(not set automatically^): - echo --whisper_batch_size 4 Fewer segments in parallel: slower, - echo but the transcription is UNCHANGED - echo ^(the safe lever; lower to 2 or 1 if - echo it still runs out^). + echo Your GPU has less than 8 GB VRAM. UltraSinger scales the + echo Whisper batch size automatically to the GPU memory ^(16 on + echo ~8 GB and larger, 8 on 5-7 GB, 4 on smaller cards^), which + echo usually makes the default Whisper model ^(large-v2^) fit. + echo If it still runs out of memory: + echo --whisper_batch_size 2 ^(or 1^) Fewer segments in + echo parallel: slower, but the + echo transcription is UNCHANGED + echo ^(the safe lever^). echo --whisper_compute_type int8 Halves the model's memory at a echo small accuracy cost - add this only echo if lowering the batch size is not diff --git a/install/auto_install.sh b/install/auto_install.sh index 3648fd56..f28dedff 100644 --- a/install/auto_install.sh +++ b/install/auto_install.sh @@ -246,13 +246,13 @@ case "$BUILD" in cuda) if [ -z "$GPU_VRAM" ]; then echo "GPU VRAM could not be verified (forced CUDA build)." - echo "If your GPU has less than 8 GB VRAM, the default Whisper model" - echo "(large-v2) may run out of memory. Two independent ways to fit it" - echo "(not set automatically):" - echo " --whisper_batch_size 4 Fewer segments in parallel: slower," - echo " but the transcription is UNCHANGED" - echo " (the safe lever; lower to 2 or 1 if" - echo " it still runs out)." + echo "UltraSinger scales the Whisper batch size automatically to the" + echo "GPU memory it detects at runtime (16 on ~8 GB and larger, 8 on" + echo "5-7 GB, 4 on smaller cards), which usually makes the default" + echo "Whisper model (large-v2) fit. If it still runs out of memory:" + echo " --whisper_batch_size 2 (or 1) Fewer segments in parallel:" + echo " slower, but the transcription is" + echo " UNCHANGED (the safe lever)." echo " --whisper_compute_type int8 Halves the model's memory at a" echo " small accuracy cost - add this only" echo " if lowering the batch size is not enough." @@ -263,13 +263,14 @@ case "$BUILD" in echo "runs transcription in the cloud instead of on your GPU." echo "GUI users: enable this under Settings -> 'Remote Speech-to-Text'." elif [ "$GPU_VRAM" -lt 8192 ]; then - echo "Your GPU has less than 8 GB VRAM. The default Whisper model" - echo "(large-v2) may run out of memory. Two independent ways to fit it" - echo "(not set automatically):" - echo " --whisper_batch_size 4 Fewer segments in parallel: slower," - echo " but the transcription is UNCHANGED" - echo " (the safe lever; lower to 2 or 1 if" - echo " it still runs out)." + echo "Your GPU has less than 8 GB VRAM. UltraSinger scales the" + echo "Whisper batch size automatically to the GPU memory (16 on" + echo "~8 GB and larger, 8 on 5-7 GB, 4 on smaller cards), which" + echo "usually makes the default Whisper model (large-v2) fit." + echo "If it still runs out of memory:" + echo " --whisper_batch_size 2 (or 1) Fewer segments in parallel:" + echo " slower, but the transcription is" + echo " UNCHANGED (the safe lever)." echo " --whisper_compute_type int8 Halves the model's memory at a" echo " small accuracy cost - add this only" echo " if lowering the batch size is not enough." diff --git a/pytest/gui/test_freestyle_threshold_widgets.py b/pytest/gui/test_freestyle_threshold_widgets.py index 47aef35c..16652d0f 100644 --- a/pytest/gui/test_freestyle_threshold_widgets.py +++ b/pytest/gui/test_freestyle_threshold_widgets.py @@ -204,3 +204,24 @@ def test_default_thresholds_round_trip_to_settings_defaults(self): if __name__ == "__main__": unittest.main() + + +class TestBatchSizeAutoArgs(unittest.TestCase): + """Batch size 0 means Auto: the flag must be omitted so the CLI can + scale it to the detected GPU memory; explicit values are forwarded.""" + + def setUp(self): + self.runner = UltraSingerRunner() + + def test_auto_omits_batch_size_flag(self): + config = dict(_DEFAULTS) + config["whisper_batch_size"] = 0 + args = self.runner.build_args(config, "test.mp3") + self.assertNotIn("--whisper_batch_size", args) + + def test_explicit_batch_size_is_forwarded(self): + config = dict(_DEFAULTS) + config["whisper_batch_size"] = 8 + args = self.runner.build_args(config, "test.mp3") + self.assertIn("--whisper_batch_size", args) + self.assertEqual(args[args.index("--whisper_batch_size") + 1], "8") diff --git a/pytest/modules/DeviceDetection/test_device_detection.py b/pytest/modules/DeviceDetection/test_device_detection.py new file mode 100644 index 00000000..6b8c9ccb --- /dev/null +++ b/pytest/modules/DeviceDetection/test_device_detection.py @@ -0,0 +1,66 @@ +"""Tests for the device detection module (auto Whisper batch size).""" + +from unittest.mock import MagicMock, patch + +from modules.DeviceDetection.device_detection import auto_whisper_batch_size + + +def _gpu_props(vram_gb: float): + props = MagicMock() + props.total_memory = int(vram_gb * 1024 ** 3) + return props + + +class TestAutoWhisperBatchSize: + """VRAM-based scaling: 16 on 8+ GB, 8 on 6 GB, 4 on 4 GB cards.""" + + def test_cpu_keeps_historic_default(self): + assert auto_whisper_batch_size("cpu") == 16 + + def test_cuda_unavailable_keeps_historic_default(self): + with patch("torch.cuda.is_available", return_value=False): + assert auto_whisper_batch_size("cuda") == 16 + + def test_4gb_card_gets_4(self): + with patch("torch.cuda.is_available", return_value=True), \ + patch("torch.cuda.get_device_properties", + return_value=_gpu_props(4.0)): + assert auto_whisper_batch_size("cuda") == 4 + + def test_6gb_card_gets_8(self): + with patch("torch.cuda.is_available", return_value=True), \ + patch("torch.cuda.get_device_properties", + return_value=_gpu_props(6.0)): + assert auto_whisper_batch_size("cuda") == 8 + + def test_7gb_card_gets_8(self): + """7 GB is not enough for batch 16 (which needs ~8 GB).""" + with patch("torch.cuda.is_available", return_value=True), \ + patch("torch.cuda.get_device_properties", + return_value=_gpu_props(7.0)): + assert auto_whisper_batch_size("cuda") == 8 + + def test_8gb_card_reporting_slightly_less_gets_16(self): + """True 8 GB cards often report a bit less than 8.0 GB.""" + with patch("torch.cuda.is_available", return_value=True), \ + patch("torch.cuda.get_device_properties", + return_value=_gpu_props(7.9)): + assert auto_whisper_batch_size("cuda") == 16 + + def test_8gb_card_gets_16(self): + with patch("torch.cuda.is_available", return_value=True), \ + patch("torch.cuda.get_device_properties", + return_value=_gpu_props(8.0)): + assert auto_whisper_batch_size("cuda") == 16 + + def test_12gb_card_gets_16(self): + with patch("torch.cuda.is_available", return_value=True), \ + patch("torch.cuda.get_device_properties", + return_value=_gpu_props(12.0)): + assert auto_whisper_batch_size("cuda") == 16 + + def test_query_failure_keeps_historic_default(self): + with patch("torch.cuda.is_available", return_value=True), \ + patch("torch.cuda.get_device_properties", + side_effect=RuntimeError("driver in a bad state")): + assert auto_whisper_batch_size("cuda") == 16 diff --git a/src/Settings.py b/src/Settings.py index 1adeb357..be1c0935 100644 --- a/src/Settings.py +++ b/src/Settings.py @@ -90,7 +90,7 @@ class Settings: whisper_model = WhisperModel.LARGE_V2 # Multilingual model tiny|base|small|medium|large-v1|large-v2|large-v3 # English-only model tiny.en|base.en|small.en|medium.en whisper_align_model = None # Model for other languages from huggingface.co e.g -> "gigant/romanian-wav2vec2" - whisper_batch_size = 16 # reduce if low on GPU mem + whisper_batch_size = None # None = auto: scaled to GPU VRAM (16 / 8 / 4); set an int to override whisper_compute_type = None # change to "int8" if low on GPU mem (may reduce accuracy) keep_numbers = False diff --git a/src/UltraSinger.py b/src/UltraSinger.py index 84df7502..5d20380b 100644 --- a/src/UltraSinger.py +++ b/src/UltraSinger.py @@ -97,7 +97,10 @@ from modules.musicbrainz_client import search_musicbrainz from modules.sheet import create_sheet from modules.ProcessData import ProcessData, ProcessDataPaths, MediaInfo -from modules.DeviceDetection.device_detection import check_gpu_support +from modules.DeviceDetection.device_detection import ( + auto_whisper_batch_size, + check_gpu_support, +) from modules.Image.image_helper import save_image from modules.ffmpeg_helper import ( is_ffmpeg_available, @@ -1923,6 +1926,19 @@ def main(argv: list[str]) -> None: def check_requirements() -> None: if not settings.force_cpu: settings.pytorch_device = check_gpu_support() + + # Resolve the automatic Whisper batch size now that the device (and + # its VRAM) is known. An explicit --whisper_batch_size wins. + if settings.whisper_batch_size is None: + whisper_device = ( + "cpu" if settings.force_whisper_cpu else settings.pytorch_device + ) + settings.whisper_batch_size = auto_whisper_batch_size(whisper_device) + print( + f"{ULTRASINGER_HEAD} Whisper batch size: " + f"{blue_highlighted(str(settings.whisper_batch_size))} " + f"(auto, scaled to GPU memory)" + ) print(f"{ULTRASINGER_HEAD} ----------------------") if not is_ffmpeg_available(settings.user_ffmpeg_path): @@ -1988,7 +2004,16 @@ def init_settings(argv: list[str]) -> Settings: elif opt in ("--whisper_align_model"): settings.whisper_align_model = arg elif opt in ("--whisper_batch_size"): - settings.whisper_batch_size = int(arg) + if arg.strip().lower() == "auto": + settings.whisper_batch_size = None + else: + val = int(arg) + if val < 1: + print( + f"{ULTRASINGER_HEAD} Error: --whisper_batch_size must " + f"be a positive integer or 'auto', got {val}") + sys.exit(1) + settings.whisper_batch_size = val elif opt in ("--whisper_compute_type"): settings.whisper_compute_type = arg elif opt in ("--keep_numbers"): diff --git a/src/gui/config.py b/src/gui/config.py index 44bc4791..67de8897 100644 --- a/src/gui/config.py +++ b/src/gui/config.py @@ -26,7 +26,7 @@ "output_folder": "", # Whisper "whisper_model": "large-v2", - "whisper_batch_size": 16, + "whisper_batch_size": 0, # 0 = Auto: UltraSinger scales it to GPU VRAM "whisper_compute_type": "", "whisper_align_model": "", # VAD / ASR (tuned for singing) diff --git a/src/gui/settings_tab.py b/src/gui/settings_tab.py index 1a54d75c..430a6a55 100644 --- a/src/gui/settings_tab.py +++ b/src/gui/settings_tab.py @@ -133,15 +133,21 @@ def _build_transcription_section(self): _DEFAULTS["whisper_model"])) self._whisper_batch_size = _NoScrollSpinBox() - self._whisper_batch_size.setRange(1, 64) - self._whisper_batch_size.setValue(self._config.get("whisper_batch_size", 16)) + self._whisper_batch_size.setRange(0, 64) + self._whisper_batch_size.setSpecialValueText("Auto") + self._whisper_batch_size.setValue( + self._config.get("whisper_batch_size", 0)) card.add_row("Batch Size", self._whisper_batch_size, "How many audio segments Whisper processes at once. " + "'Auto' (recommended) scales to your GPU memory: " + "16 on ~8 GB and larger, 8 on 5-7 GB, 4 on smaller " + "cards. " "Higher values are faster but use more GPU memory. " - "Lower it (4, or 2/1) if you get out-of-memory errors: " - "this only makes transcription slower — the result is " - "unchanged, so it is the safe lever to try first " - "(before switching Compute Type to int8).", + "Set a low value (4, or 2/1) if you still get " + "out-of-memory errors: this only makes transcription " + "slower — the result is unchanged, so it is the safe " + "lever to try first (before switching Compute Type to " + "int8).", reset_callback=lambda: self._whisper_batch_size.setValue( _DEFAULTS["whisper_batch_size"])) diff --git a/src/modules/DeviceDetection/device_detection.py b/src/modules/DeviceDetection/device_detection.py index 5051c3c8..6ab9dfad 100644 --- a/src/modules/DeviceDetection/device_detection.py +++ b/src/modules/DeviceDetection/device_detection.py @@ -92,6 +92,31 @@ def _enable_deterministic_mode(): ) +def auto_whisper_batch_size(device: str) -> int: + """Pick a Whisper batch size that fits the GPU memory. + + Field-tested scaling: the historic default of 16 needs roughly 8 GB + of VRAM with the large-v2 model; 8 runs stable on 6 GB cards and 4 + on 4 GB cards. Thresholds: below 5 GB -> 4, below 7.5 GB -> 8 + (so 7 GB cards are not pushed into the 8 GB tier, while true 8 GB + cards that report slightly less than 8 still get 16), else 16. + A lower batch size only slows transcription down - the result is + unchanged. On CPU there is no VRAM cliff, so the historic default + is kept. + """ + if device != "cuda" or not torch.cuda.is_available(): + return 16 + try: + vram_gb = torch.cuda.get_device_properties(0).total_memory / 1024 ** 3 + except RuntimeError: + return 16 + if vram_gb < 5: + return 4 + if vram_gb < 7.5: + return 8 + return 16 + + def __check_pytorch_support(): pytorch_gpu_supported = torch.cuda.is_available() if not pytorch_gpu_supported: @@ -105,6 +130,8 @@ def __check_pytorch_support(): print(f"{ULTRASINGER_HEAD} Found GPU: {blue_highlighted(gpu_name)} VRAM: {blue_highlighted(gpu_vram)} GB.") if gpu_vram < 6: print( - f"{ULTRASINGER_HEAD} {red_highlighted('GPU VRAM is less than 6GB. Program may crash due to insufficient memory.')}") + f"{ULTRASINGER_HEAD} GPU VRAM is less than 6GB - the Whisper " + f"batch size is scaled down automatically to fit (a manual " + f"--whisper_batch_size overrides this).") print(f"{ULTRASINGER_HEAD} {blue_highlighted('pytorch')} - using {red_highlighted('cuda')} gpu.") return pytorch_gpu_supported diff --git a/src/modules/common_print.py b/src/modules/common_print.py index f22bdf05..1eb8b2fd 100644 --- a/src/modules/common_print.py +++ b/src/modules/common_print.py @@ -65,7 +65,7 @@ def print_help() -> None: via Whisper tiny (~2-3s) instead of full transcription. WARNING: setting this for non-matching songs will degrade alignment quality (e.g. --language en for German songs). - --whisper_batch_size Segments processed in parallel. Lower if low on GPU mem: slower, but transcription is UNCHANGED (the safe lever) >> ((default) is 16) + --whisper_batch_size Segments processed in parallel. 'auto' (default) scales to GPU VRAM: 16 (~8GB and larger) / 8 (5-7GB) / 4 (smaller). Lower manually if still low on GPU mem: slower, but transcription is UNCHANGED (the safe lever) --whisper_compute_type Change to "int8" to save more GPU mem at a small accuracy cost; use only if lowering the batch size is not enough >> ((default) is "float16" for cuda devices, "int8" for cpu) --keep_numbers Numbers will be transcribed as numerics instead of as words >> True|False >> ((default) is False) --vad_onset VAD (Voice Activity Detection) speech activation threshold (0.0-1.0). Lower diff --git a/src/modules/init_interactive_mode.py b/src/modules/init_interactive_mode.py index 0aaf2cee..44953dcc 100644 --- a/src/modules/init_interactive_mode.py +++ b/src/modules/init_interactive_mode.py @@ -59,11 +59,18 @@ def configure_additional_options(console, settings, header): if additional_options_input == 'y': console.print(f"\n{header} [bold underline]Additional options:[/bold underline]\n") - # Whisper Batch Size - whipser_batch_size_response = console.input( - f"{header} Enter the [green]Whisper batch size[/green] (default [cyan]16[/cyan]): " - ).strip() - settings.whisper_batch_size = int(whipser_batch_size_response) if whipser_batch_size_response.isdigit() else 16 + # Whisper Batch Size + whisper_batch_size_response = console.input( + f"{header} Enter the [green]Whisper batch size[/green] (default [cyan]auto[/cyan] - scaled to GPU memory): " + ).strip() + # Only a positive integer selects an explicit batch size; empty, + # non-numeric and zero inputs keep the automatic VRAM-based mode. + settings.whisper_batch_size = ( + int(whisper_batch_size_response) + if whisper_batch_size_response.isdigit() + and int(whisper_batch_size_response) > 0 + else None + ) # Whisper Compute Type whisper_compute_choice = console.input(