Running PyTorch on the GPU in Eugo
Audience: people running PyTorch on an NVIDIA GPU inside the Eugo container: training, fine-tuning, or inference.
The one thing to take away: run attention under mixed precision. Wrapping your forward pass in
torch.autocastputsscaled_dot_product_attention, and every transformer module built on top of it, on PyTorch's fused attention kernels instead of the float32 reference path. It is one line, it changes nothing about how you write your model, and it is the standard way to run attention on any PyTorch stack.How this guide was written: every command on this page was run on 2026-09-21 in Eugo's GB10 extended image (PyTorch
2.14.0a0+gita474146, CUDA 13.4) on an NVIDIA GB10, and each output block below is that run's own output, pasted verbatim. Version strings on your image may differ, and behavior on a different GPU may differ; the shape of the advice is what matters.
1. Why mixed precision, and why for attention in particularโ
PyTorch's default tensor type is torch.float32. A model you build with torch.nn, or load without asking for
anything else, therefore holds float32 weights and runs float32 activations unless you say otherwise.
Eugo's PyTorch runs its fused attention kernels in bfloat16 and float16. In float32,
scaled_dot_product_attention runs the reference implementation, which builds the full attention score matrix:
one value for every query/key pair, in every head, for every sequence in the batch. That is fine at a few hundred
tokens and it is the thing that bites at a few thousand, because its size grows with the square of the sequence
length. A long-context float32 forward pass can run out of memory on intermediates a fused kernel never materializes
at all.
Mixed precision removes that whole class of problem, and you do not have to rewrite anything to get it.
On the T4G (the NVIDIA GPU of the Graviton2 images, sm_75) the rule is different. PyTorch's flash and cuDNN
attention start at sm_80, so the T4G has at most one fused kernel: memory-efficient attention, which below sm_80
takes float16 and float32 but not bfloat16. So on the T4G:
- use
torch.float16withautocast, neverbfloat16: abfloat16attention call there runs the float32-style reference path (andtorch.cuda.is_bf16_supported()can still answerTrueon it, so the selector in ยง2 picks the wrong type there: setamp_dtype = torch.float16yourself).
Nothing on this page was measured on the T4G: the commands below ran on a GB10.
2. The one lineโ
Leave your tensors and your weights in float32. Wrap the work in torch.autocast and let PyTorch choose the precision
per operation:
import torch
from torch.nn.functional import scaled_dot_product_attention
# bfloat16 where the GPU supports it, float16 otherwise.
amp_dtype = torch.bfloat16 if torch.cuda.is_bf16_supported() else torch.float16
# Your tensors stay in float32 - autocast converts at the operation.
q = torch.randn(32, 12, 256, 64, device="cuda")
k = torch.randn(32, 12, 256, 64, device="cuda")
v = torch.randn(32, 12, 256, 64, device="cuda")
with torch.autocast(device_type="cuda", dtype=amp_dtype):
out = scaled_dot_product_attention(q, k, v)
print("in :", q.dtype)
print("out:", out.dtype, tuple(out.shape))
in : torch.float32
out: torch.bfloat16 (32, 12, 256, 64)
The inputs went in as float32 and the result came back as bfloat16. That is autocast doing the conversion at the
operation: you never called .to(), and the tensors you hold are untouched.
2.1. A whole model, weights left aloneโ
The same wrapper works around an entire forward pass. Nothing about the module changes:
model = (
torch.nn.TransformerEncoderLayer(
d_model=768, nhead=12, dim_feedforward=3072, batch_first=True
)
.cuda()
.eval()
)
x = torch.randn(8, 256, 768, device="cuda")
with torch.autocast(device_type="cuda", dtype=amp_dtype):
y = model(x)
print("weights:", model.linear1.weight.dtype, " input:", x.dtype, " output:", y.dtype)
print("shape :", tuple(y.shape))
weights: torch.float32 input: torch.float32 output: torch.float32
shape : (8, 256, 768)
The float32 output is expected, and it is not a sign the wrapper did nothing. Autocast picks a precision per operation: attention and the large matrix multiplies run in bfloat16, while normalization and the values that leave the block stay in float32, which is what keeps the numerics well behaved. You get the fused kernels without having to reason about where a cast belongs.
2.2. Or convert the model outrightโ
If everything downstream is happy in reduced precision, converting the module is equally valid and needs no wrapper at all:
model = model.to(amp_dtype)
y = model(torch.randn(8, 256, 768, device="cuda", dtype=amp_dtype))
print("weights:", model.linear1.weight.dtype, " output:", y.dtype)
weights: torch.bfloat16 output: torch.bfloat16
Use autocast when you want float32 weights and optimizer state (the usual choice for training); convert outright
when you are serving a model that was trained or published in reduced precision.
2.3. Hugging Face Transformers: keep the default attentionโ
Leave attn_implementation unset (or pass "sdpa"); do not ask for FlashAttention. On the NVIDIA GB10 (DGX
Spark) Transformers' default attention, "sdpa", already runs NVIDIA cuDNN's fused attention kernel, and
FlashAttention measured slower there: with a Llama-shaped bfloat16 model across 7 prefill and decode cases, neither
FlashAttention route was faster on any case. vLLM's FlashAttention-2 kernels were 1.3โ8.5 % slower on 4 cases and
FlashAttention-4 2.2โ33 % slower on 5, the rest tied (paired runs, sign test, Holm-corrected p โค 0.011 on every
slower case). On the GPU images for Ampere and newer GPUs (the GB10 and Grace Blackwell images) Eugo ships
Dao-AILab's FlashAttention-2 (flash_attn), so code written for attn_implementation="flash_attention_2" runs as
written, with no warning; that is for compatibility, not speed (this build has not been timed). If your code asks for
a FlashAttention that cannot run here (FlashAttention-3, which Eugo does not build; FlashAttention-2 on the Graviton
images; FlashAttention-4 without vLLM; or any of them with no NVIDIA GPU visible), Transformers logs one warning and
runs the model on the default attention instead of stopping with an ImportError. Remove the argument or pass
"sdpa" to silence that warning.
3. Training with attention dropout: use a multiple of 1/16โ
One detail worth knowing before you set a hyper-parameter by habit. The fused attention kernel takes its dropout
probability in sixteenths. 0.1 is not a sixteenth, so a training-mode forward pass with dropout=0.1 leaves the
fused path.
Your step still runs; it just runs on the reference path. You can tell which one you got without changing any code:
at dropout=0.1 PyTorch prints a warning mentioning memory efficient attention the first time the layer runs, and at
dropout=0.125 it prints nothing at all.
Pick 0.0625 (1/16), 0.125 (2/16), 0.1875, 0.25 and so on instead; the difference from 0.1 is not one your
model will notice:
model = torch.nn.TransformerEncoderLayer(
d_model=768, nhead=12, dropout=0.125, batch_first=True # 2/16
).cuda().train()
with torch.autocast(device_type="cuda", dtype=amp_dtype):
y = model(torch.randn(8, 256, 768, device="cuda"))
print("trained forward:", tuple(y.shape), y.dtype)
trained forward: (8, 256, 768) torch.float32
This only applies while dropout is actually active: a module in eval() has attention dropout off, so inference is
never affected whatever you set. It applies equally to a dropout_p= you pass to scaled_dot_product_attention
yourself.
If you train in float16 rather than bfloat16, pair autocast with torch.amp.GradScaler the way PyTorch
documents. bfloat16 does not need it.
4. Quick referenceโ
| Situation | What to do |
|---|---|
| Inference on a model you built or loaded in float32 | wrap the forward pass in torch.autocast(device_type="cuda", dtype=torch.bfloat16) |
Model already in bfloat16 / float16 | nothing for inference; in training, check the dropout row too |
Training in float32 with autocast | works as-is; add torch.amp.GradScaler only if you chose float16 |
| Training with attention dropout | set it to a multiple of 1/16: 0.0625, 0.125, 0.25 |
| Out of memory at long sequence length in float32 | wrap it in autocast; the float32 reference path's intermediates grow with the square of the sequence length |
On the T4G (the NVIDIA GPU of the Graviton2 images, sm_75) | use dtype=torch.float16, not bfloat16 (its one fused kernel, memory-efficient attention, takes no bfloat16 there; ยง1) |
PyTorch mentions was not compiled with memory efficient attention | that call did not reach a fused kernel. Either its inputs were float32 (wrap them in autocast) or you are training with a dropout that is not a multiple of 1/16 |
Transformers code passes attn_implementation="flash_attention_2" (or _3, _4) | remove it or pass "sdpa": on the GB10 the default already runs cuDNN's fused kernel, and FlashAttention was never faster there: slower on 4-5 of 7 cases, tied on the rest (ยง2.3) |
| Not sure whether the wrapper took effect | check a tensor's .dtype inside the block, as in ยง2: a bfloat16 result from float32 inputs is the proof |
You set torch._inductor.config.cuda.enable_ptxas_info = True to see each CUDA kernel's register and memory use | if every CUTLASS GEMM compile then fails at the link with ptxas fatal : Unknown option '-keep', leave the option off. Otherwise start the container with -e EUGO_PTXAS_DEBUG_LEVEL=2: PyTorch then logs each CUTLASS compile's compiler output as a warning, and the link's output carries ptxas's ptxas info lines (registers, barriers, spills). Below level 2 the image's ptxas wrapper drops the -v this option adds, so the log has no report. Level 2 can make large GPU kernel builds extremely slow (the wrapper prints a warning) |
vLLM refuses to start with KV connector ... is incompatible with PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True (you asked for KV offload or a KV connector) | the image adds expandable_segments:True to every process unless you set that key yourself. That default does not stop vLLM: when a KV connector is configured, vLLM switches it off for itself and the processes it starts, and logs one warning saying so; those processes give up cuda-checkpoint restore (cuda-checkpoint is NVIDIA's tool that saves a running GPU process's state and restores it later; this matters only if you use it). You still see this error when you set expandable_segments:True yourself (including after the image set it: any value of PYTORCH_CUDA_ALLOC_CONF other than the exact one the image composed counts as yours). Whenever you see it, start that process with -e PYTORCH_CUDA_ALLOC_CONF=expandable_segments:False, or pass vLLM's own --enable-cumem-allocator. The first gives up cuda-checkpoint restore for that process |
Comparing seeded torch.randn output on the CPU with stock PyTorch, bit for bit | with the same seed, torch.randn on the CPU on 16 or more float32/float64 elements is not bit-identical to stock PyTorch, because Eugo's PyTorch uses ArmPL's vector log/sin/cos (within Arm's documented accuracy); to compare against stock bit for bit, create the inputs once and load them in both |
See alsoโ
- The guide to installing Python packages, in this Guides section: installing your own packages alongside Eugo's bundled PyTorch.
- The EugoHPC Manager guide, in this Guides section: running the same code across a cluster.