Skip to content

Commit b5ef174

Browse files
authored
Compile SDPA with bounded tensor-dependent batch sizes (#1257)
# Summary Allow SDPA compilation when a tensor-dependent batch size is provably within the existing 65,535-entry chunk limit. # Description of the change PyTorch 2.14 CPU, default Inductor backend: ```python import torch from sdm.nn import SDPA attention = SDPA(1).eval() def run(query, ids): selected = ids.unique() torch._check(selected.numel() > 0) query = query.index_select(0, selected) return attention(query, query, query) compiled = torch.compile(run, fullgraph=True) with torch.no_grad(): compiled(torch.randn(8, 1, 1, 1), torch.arange(8) % 3) ``` ```text Could not extract specialized integer from data-dependent expression u0 Caused by: for i in range(0, query.size(0), 65_535) ``` The original Python loop requires a concrete batch size. Use the single iteration `(0,)` when at most one chunk is needed; retain the original loop for larger batches. Attention computation and chunk sizes are unchanged, with no custom operation. This covers provably bounded batches. Strict compilation across an unknown 65,535-entry boundary remains unsupported. # Validation - PyTorch 2.14 CPU Inductor: `fullgraph=False` and `True`, tensor-dependent batches changing 2 → 8 → 1, matching eager outputs. - 21 parity cases covering FP32/FP64/BF16, batches 1/65,535/65,536, masks, causal attention, dropout/RNG and gradients. - Existing SDPA tests: 8 passed, 7 CUDA-only skipped. Pre-commit passed. CUDA execution and performance remain unmeasured.
1 parent ec0203b commit b5ef174

1 file changed

Lines changed: 6 additions & 2 deletions

File tree

‎sdm/nn/attention.py‎

Lines changed: 6 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -171,7 +171,7 @@ def forward(
171171
key = key.transpose(-3, -2) # [B, Hkv, KV, C]
172172
value = value.transpose(-3, -2) # [B, Hkv, KV, C]
173173

174-
# CUDA attention kernels accept at most 65,535 batch entries per call:
174+
# CUDA attention kernels accept at most 65,535 batch entries per call.
175175
outs = [
176176
F.scaled_dot_product_attention(
177177
query=query[i : i + 65_535],
@@ -185,7 +185,11 @@ def forward(
185185
is_causal=is_causal,
186186
scale=self.scale,
187187
).transpose(-3, -2) # [B, Q, Hq, C]
188-
for i in range(0, query.size(0), 65_535)
188+
for i in (
189+
(0,)
190+
if query.size(0) <= 65_535
191+
else range(0, query.size(0), 65_535)
192+
)
189193
]
190194
out = outs[0] if len(outs) == 1 else torch.cat(outs)
191195

0 commit comments

Comments
 (0)