Skip to main content

Running PyTorch on the GPU in Eugo

Audience: people running PyTorch on an NVIDIA GPU inside the Eugo container: training, fine-tuning, or inference.

The one thing to take away: run attention under mixed precision. Wrapping your forward pass in torch.autocast puts scaled_dot_product_attention, and every transformer module built on top of it, on PyTorch's fused attention kernels instead of the float32 reference path. It is one line, it changes nothing about how you write your model, and it is the standard way to run attention on any PyTorch stack.

How this guide was written: every command on this page was run on 2026-09-21 in Eugo's GB10 extended image (PyTorch 2.14.0a0+gita474146, CUDA 13.4) on an NVIDIA GB10, and each output block below is that run's own output, pasted verbatim. Version strings on your image may differ, and behavior on a different GPU may differ; the shape of the advice is what matters.


1. Why mixed precision, and why for attention in particularโ€‹

PyTorch's default tensor type is torch.float32. A model you build with torch.nn, or load without asking for anything else, therefore holds float32 weights and runs float32 activations unless you say otherwise.

Eugo's PyTorch runs its fused attention kernels in bfloat16 and float16. In float32, scaled_dot_product_attention runs the reference implementation, which builds the full attention score matrix: one value for every query/key pair, in every head, for every sequence in the batch. That is fine at a few hundred tokens and it is the thing that bites at a few thousand, because its size grows with the square of the sequence length. A long-context float32 forward pass can run out of memory on intermediates a fused kernel never materializes at all.

Mixed precision removes that whole class of problem, and you do not have to rewrite anything to get it.

On the T4G (the NVIDIA GPU of the Graviton2 images, sm_75) the rule is different. PyTorch's flash and cuDNN attention start at sm_80, so the T4G has at most one fused kernel: memory-efficient attention, which below sm_80 takes float16 and float32 but not bfloat16. So on the T4G:

  • use torch.float16 with autocast, never bfloat16: a bfloat16 attention call there runs the float32-style reference path (and torch.cuda.is_bf16_supported() can still answer True on it, so the selector in ยง2 picks the wrong type there: set amp_dtype = torch.float16 yourself).

Nothing on this page was measured on the T4G: the commands below ran on a GB10.

2. The one lineโ€‹

Leave your tensors and your weights in float32. Wrap the work in torch.autocast and let PyTorch choose the precision per operation:

import torch
from torch.nn.functional import scaled_dot_product_attention

# bfloat16 where the GPU supports it, float16 otherwise.
amp_dtype = torch.bfloat16 if torch.cuda.is_bf16_supported() else torch.float16

# Your tensors stay in float32 - autocast converts at the operation.
q = torch.randn(32, 12, 256, 64, device="cuda")
k = torch.randn(32, 12, 256, 64, device="cuda")
v = torch.randn(32, 12, 256, 64, device="cuda")

with torch.autocast(device_type="cuda", dtype=amp_dtype):
out = scaled_dot_product_attention(q, k, v)

print("in :", q.dtype)
print("out:", out.dtype, tuple(out.shape))
in : torch.float32
out: torch.bfloat16 (32, 12, 256, 64)

The inputs went in as float32 and the result came back as bfloat16. That is autocast doing the conversion at the operation: you never called .to(), and the tensors you hold are untouched.

2.1. A whole model, weights left aloneโ€‹

The same wrapper works around an entire forward pass. Nothing about the module changes:

model = (
torch.nn.TransformerEncoderLayer(
d_model=768, nhead=12, dim_feedforward=3072, batch_first=True
)
.cuda()
.eval()
)
x = torch.randn(8, 256, 768, device="cuda")

with torch.autocast(device_type="cuda", dtype=amp_dtype):
y = model(x)

print("weights:", model.linear1.weight.dtype, " input:", x.dtype, " output:", y.dtype)
print("shape :", tuple(y.shape))
weights: torch.float32  input: torch.float32  output: torch.float32
shape : (8, 256, 768)

The float32 output is expected, and it is not a sign the wrapper did nothing. Autocast picks a precision per operation: attention and the large matrix multiplies run in bfloat16, while normalization and the values that leave the block stay in float32, which is what keeps the numerics well behaved. You get the fused kernels without having to reason about where a cast belongs.

2.2. Or convert the model outrightโ€‹

If everything downstream is happy in reduced precision, converting the module is equally valid and needs no wrapper at all:

model = model.to(amp_dtype)
y = model(torch.randn(8, 256, 768, device="cuda", dtype=amp_dtype))
print("weights:", model.linear1.weight.dtype, " output:", y.dtype)
weights: torch.bfloat16  output: torch.bfloat16

Use autocast when you want float32 weights and optimizer state (the usual choice for training); convert outright when you are serving a model that was trained or published in reduced precision.

2.3. Hugging Face Transformers: keep the default attentionโ€‹

Leave attn_implementation unset (or pass "sdpa"); do not ask for FlashAttention. On the NVIDIA GB10 (DGX Spark) Transformers' default attention, "sdpa", already runs NVIDIA cuDNN's fused attention kernel, and FlashAttention measured slower there: with a Llama-shaped bfloat16 model across 7 prefill and decode cases, neither FlashAttention route was faster on any case. vLLM's FlashAttention-2 kernels were 1.3โ€“8.5 % slower on 4 cases and FlashAttention-4 2.2โ€“33 % slower on 5, the rest tied (paired runs, sign test, Holm-corrected p โ‰ค 0.011 on every slower case). On the GPU images for Ampere and newer GPUs (the GB10 and Grace Blackwell images) Eugo ships Dao-AILab's FlashAttention-2 (flash_attn), so code written for attn_implementation="flash_attention_2" runs as written, with no warning; that is for compatibility, not speed (this build has not been timed). If your code asks for a FlashAttention that cannot run here (FlashAttention-3, which Eugo does not build; FlashAttention-2 on the Graviton images; FlashAttention-4 without vLLM; or any of them with no NVIDIA GPU visible), Transformers logs one warning and runs the model on the default attention instead of stopping with an ImportError. Remove the argument or pass "sdpa" to silence that warning.

3. Training with attention dropout: use a multiple of 1/16โ€‹

One detail worth knowing before you set a hyper-parameter by habit. The fused attention kernel takes its dropout probability in sixteenths. 0.1 is not a sixteenth, so a training-mode forward pass with dropout=0.1 leaves the fused path.

Your step still runs; it just runs on the reference path. You can tell which one you got without changing any code: at dropout=0.1 PyTorch prints a warning mentioning memory efficient attention the first time the layer runs, and at dropout=0.125 it prints nothing at all.

Pick 0.0625 (1/16), 0.125 (2/16), 0.1875, 0.25 and so on instead; the difference from 0.1 is not one your model will notice:

model = torch.nn.TransformerEncoderLayer(
d_model=768, nhead=12, dropout=0.125, batch_first=True # 2/16
).cuda().train()

with torch.autocast(device_type="cuda", dtype=amp_dtype):
y = model(torch.randn(8, 256, 768, device="cuda"))

print("trained forward:", tuple(y.shape), y.dtype)
trained forward: (8, 256, 768) torch.float32

This only applies while dropout is actually active: a module in eval() has attention dropout off, so inference is never affected whatever you set. It applies equally to a dropout_p= you pass to scaled_dot_product_attention yourself.

If you train in float16 rather than bfloat16, pair autocast with torch.amp.GradScaler the way PyTorch documents. bfloat16 does not need it.

4. Quick referenceโ€‹

SituationWhat to do
Inference on a model you built or loaded in float32wrap the forward pass in torch.autocast(device_type="cuda", dtype=torch.bfloat16)
Model already in bfloat16 / float16nothing for inference; in training, check the dropout row too
Training in float32 with autocastworks as-is; add torch.amp.GradScaler only if you chose float16
Training with attention dropoutset it to a multiple of 1/16: 0.0625, 0.125, 0.25
Out of memory at long sequence length in float32wrap it in autocast; the float32 reference path's intermediates grow with the square of the sequence length
On the T4G (the NVIDIA GPU of the Graviton2 images, sm_75)use dtype=torch.float16, not bfloat16 (its one fused kernel, memory-efficient attention, takes no bfloat16 there; ยง1)
PyTorch mentions was not compiled with memory efficient attentionthat call did not reach a fused kernel. Either its inputs were float32 (wrap them in autocast) or you are training with a dropout that is not a multiple of 1/16
Transformers code passes attn_implementation="flash_attention_2" (or _3, _4)remove it or pass "sdpa": on the GB10 the default already runs cuDNN's fused kernel, and FlashAttention was never faster there: slower on 4-5 of 7 cases, tied on the rest (ยง2.3)
Not sure whether the wrapper took effectcheck a tensor's .dtype inside the block, as in ยง2: a bfloat16 result from float32 inputs is the proof
You set torch._inductor.config.cuda.enable_ptxas_info = True to see each CUDA kernel's register and memory useif every CUTLASS GEMM compile then fails at the link with ptxas fatal : Unknown option '-keep', leave the option off. Otherwise start the container with -e EUGO_PTXAS_DEBUG_LEVEL=2: PyTorch then logs each CUTLASS compile's compiler output as a warning, and the link's output carries ptxas's ptxas info lines (registers, barriers, spills). Below level 2 the image's ptxas wrapper drops the -v this option adds, so the log has no report. Level 2 can make large GPU kernel builds extremely slow (the wrapper prints a warning)
vLLM refuses to start with KV connector ... is incompatible with PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True (you asked for KV offload or a KV connector)the image adds expandable_segments:True to every process unless you set that key yourself. That default does not stop vLLM: when a KV connector is configured, vLLM switches it off for itself and the processes it starts, and logs one warning saying so; those processes give up cuda-checkpoint restore (cuda-checkpoint is NVIDIA's tool that saves a running GPU process's state and restores it later; this matters only if you use it). You still see this error when you set expandable_segments:True yourself (including after the image set it: any value of PYTORCH_CUDA_ALLOC_CONF other than the exact one the image composed counts as yours). Whenever you see it, start that process with -e PYTORCH_CUDA_ALLOC_CONF=expandable_segments:False, or pass vLLM's own --enable-cumem-allocator. The first gives up cuda-checkpoint restore for that process
Comparing seeded torch.randn output on the CPU with stock PyTorch, bit for bitwith the same seed, torch.randn on the CPU on 16 or more float32/float64 elements is not bit-identical to stock PyTorch, because Eugo's PyTorch uses ArmPL's vector log/sin/cos (within Arm's documented accuracy); to compare against stock bit for bit, create the inputs once and load them in both

See alsoโ€‹

  • The guide to installing Python packages, in this Guides section: installing your own packages alongside Eugo's bundled PyTorch.
  • The EugoHPC Manager guide, in this Guides section: running the same code across a cluster.