Skip to content

docs: record smoke-test failure modes (cookbook git, 429, eval tokens) #10

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Draft
sfc-gh-kganesan wants to merge 3 commits into
base: main
Choose a base branch
from
Draft

docs: record smoke-test failure modes (cookbook git, 429, eval tokens) #10

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
556a902
docs: cookbook SSH timeout, 429, and post-cancel capacity
sfc-gh-kganesan Aug 28, 2026
3448292
docs: inference JSON schematic is not pasteable; eval needs enough to…
sfc-gh-kganesan Aug 28, 2026
76e1cd8
docs: drop machine-specific cookbook git SSH workaround
sfc-gh-kganesan Aug 31, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
10 changes: 10 additions & 0 deletions docs/reference/troubleshooting.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -14,3 +14,13 @@ For recipe failures, first capture:

Recipe-specific failure modes are documented beside the recipe rather than
accumulated on this page.

## HTTP 429 and capacity after cancel

`POST /cortex-training` can return **429 Too Many Requests** while jobs are
draining. `cortex-training capacity` may still show `in_use_gpus` at the account
ceiling after `cancel`, even when `cortex-training list --status running` is
empty. Wait and re-check `available_gpus` before the next recipe.

Default Math GRPO needs **8** GPUs (4 train + 4 sample). Conversational SFT
Qwen3-8B needs **4**. Inference `qwen3_8b_*` sampling configs request **2**.
5 changes: 5 additions & 0 deletions recipes/inference/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -47,6 +47,8 @@ with the Python client.

## Job config JSON

The schematic below uses placeholders and `//` comments; it is **not** valid JSON. Copy a file from `configs/`.

The recipe loads one create-job body with a single sampling sub-job. Pass a
shipped example or a copy with `job_config=JOB_CONFIG`.

Expand Down Expand Up @@ -129,6 +131,9 @@ python -m recipes.inference.generate \

### Evaluate (MATH-500)

Default `max_tokens` is 4096. A tiny cap (for example 64) usually scores 0% even when the endpoint is healthy.


```bash
python -m recipes.inference.evaluate \
config=/path/to/config.json \
Expand Down