Repository navigation
fix: submit monitors the GPU that serves the model (v1.1.6) - #14
Merged
Merged
Conversation
On multi-GPU machines submit monitored GPU 0 even when the engine put the model on another GPU (Kaggle T4 x2: 3 of 5 runs showed an idle GPU). Load the model with a 1-token request, pick the GPU with the most memory in use, monitor it, and record hardware.serving_gpu when there are several GPUs. Fix the battery warning's flag name. Bump to 1.1.6.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
On multi-GPU machines
servebench submitmonitored GPU 0 even when the engine loaded the model on another GPU. In the first Kaggle T4 x2 run, Ollama put 3 of 5 models on the second T4, so those results reported an idle GPU (0% busy, 585 MHz, 29 W). The speeds were right; the conditions described the wrong card.Changes
submitsends one 1-token request to load the model (load_model), picks the GPU with the most memory in use (pick_serving_gpu, pure), and monitors that GPU for telemetry and limits.hardware.serving_gpuonly when there is more than one GPU, so single-GPU results are unchanged and still validate on older RunsOnMine.battery).Pairs with Prajwal7-2002/runsonmine#2, which adds
serving_gputo the format. Merge that PR first.Test verification
pick_serving_gpu(second GPU holds the model, single GPU, empty and garbage input);serving_gpuis recorded only with several GPUs.serving_gpufield; conditions recorded (power limit 38.9 W, clock 1073 of 2100 MHz); RunsOnMine flags itbattery(146 tok/s vs 202 on AC).