# Running PyTorch on the GPU in Eugo

Run attention under mixed precision so Eugo's PyTorch uses its fused GPU attention kernels, plus attention dropout, FlashAttention in Hugging Face Transformers, the T4G and other GPU notes.

Source: https://docs.eugo.io/docs/guides/pytorch-on-the-gpu/

---

> **Audience:** people running PyTorch on an NVIDIA GPU inside the Eugo container: training, fine-tuning, or
> inference.
>
> **The one thing to take away:** run attention under **mixed precision**. Wrapping your forward pass in
> `torch.autocast` puts `scaled_dot_product_attention`, and every transformer module built on top of it, on
> PyTorch's fused attention kernels instead of the float32 reference path. It is one line, it changes nothing about
> how you write your model, and it is the standard way to run attention on any PyTorch stack.
>
> **How this guide was written:** every command on this page was run on **2026-09-21** in Eugo's GB10 extended image
> (PyTorch `2.14.0a0+gita474146`, CUDA 13.4) on an NVIDIA GB10, and each output block below is that run's own output,
> pasted verbatim. Version strings on your image may differ, and behavior on a different GPU may differ; the shape of
> the advice is what matters.

---

## 1. Why mixed precision, and why for attention in particular

PyTorch's default tensor type is `torch.float32`. A model you build with `torch.nn`, or load without asking for
anything else, therefore holds float32 weights and runs float32 activations unless you say otherwise.

Eugo's PyTorch runs its **fused** attention kernels in `bfloat16` and `float16`. In float32,
`scaled_dot_product_attention` runs the reference implementation, which builds the **full attention score matrix**:
one value for every query/key pair, in every head, for every sequence in the batch. That is fine at a few hundred
tokens and it is the thing that bites at a few thousand, because its size grows with the **square** of the sequence
length. A long-context float32 forward pass can run out of memory on intermediates a fused kernel never materializes
at all.

Mixed precision removes that whole class of problem, and you do not have to rewrite anything to get it.

**On the T4G (the NVIDIA GPU of the Graviton2 images, `sm_75`) the rule is different.** PyTorch's flash and cuDNN
attention start at `sm_80`, so the T4G has at most one fused kernel: memory-efficient attention, which below `sm_80`
takes `float16` and `float32` but **not `bfloat16`**. So on the T4G:

- use `torch.float16` with `autocast`, never `bfloat16`: a `bfloat16` attention call there runs the float32-style
  reference path (and `torch.cuda.is_bf16_supported()` can still answer `True` on it, so the selector in §2 picks the
  wrong type there: set `amp_dtype = torch.float16` yourself).

Nothing on this page was measured on the T4G: the commands below ran on a GB10.

## 2. The one line

Leave your tensors and your weights in float32. Wrap the work in `torch.autocast` and let PyTorch choose the precision
per operation:

```python
import torch
from torch.nn.functional import scaled_dot_product_attention

# bfloat16 where the GPU supports it, float16 otherwise.
amp_dtype = torch.bfloat16 if torch.cuda.is_bf16_supported() else torch.float16

# Your tensors stay in float32 - autocast converts at the operation.
q = torch.randn(32, 12, 256, 64, device="cuda")
k = torch.randn(32, 12, 256, 64, device="cuda")
v = torch.randn(32, 12, 256, 64, device="cuda")

with torch.autocast(device_type="cuda", dtype=amp_dtype):
    out = scaled_dot_product_attention(q, k, v)

print("in :", q.dtype)
print("out:", out.dtype, tuple(out.shape))
```

```text
in : torch.float32
out: torch.bfloat16 (32, 12, 256, 64)
```

The inputs went in as float32 and the result came back as bfloat16. That is autocast doing the conversion at the
operation: you never called `.to()`, and the tensors you hold are untouched.

### 2.1. A whole model, weights left alone

The same wrapper works around an entire forward pass. Nothing about the module changes:

```python
model = (
    torch.nn.TransformerEncoderLayer(
        d_model=768, nhead=12, dim_feedforward=3072, batch_first=True
    )
    .cuda()
    .eval()
)
x = torch.randn(8, 256, 768, device="cuda")

with torch.autocast(device_type="cuda", dtype=amp_dtype):
    y = model(x)

print("weights:", model.linear1.weight.dtype, " input:", x.dtype, " output:", y.dtype)
print("shape  :", tuple(y.shape))
```

```text
weights: torch.float32  input: torch.float32  output: torch.float32
shape  : (8, 256, 768)
```

**The float32 output is expected, and it is not a sign the wrapper did nothing.** Autocast picks a precision per
operation: attention and the large matrix multiplies run in bfloat16, while normalization and the values that leave
the block stay in float32, which is what keeps the numerics well behaved. You get the fused kernels without having to
reason about where a cast belongs.

### 2.2. Or convert the model outright

If everything downstream is happy in reduced precision, converting the module is equally valid and needs no wrapper at
all:

```python
model = model.to(amp_dtype)
y = model(torch.randn(8, 256, 768, device="cuda", dtype=amp_dtype))
print("weights:", model.linear1.weight.dtype, " output:", y.dtype)
```

```text
weights: torch.bfloat16  output: torch.bfloat16
```

Use `autocast` when you want float32 weights and optimizer state (the usual choice for training); convert outright
when you are serving a model that was trained or published in reduced precision.

### 2.3. Hugging Face Transformers: keep the default attention

**Leave `attn_implementation` unset (or pass `"sdpa"`); do not ask for FlashAttention.** On the NVIDIA GB10 (DGX
Spark) Transformers' default attention, `"sdpa"`, already runs NVIDIA cuDNN's fused attention kernel, and
FlashAttention measured slower there: with a Llama-shaped bfloat16 model across 7 prefill and decode cases, neither
FlashAttention route was faster on any case. vLLM's FlashAttention-2 kernels were 1.3–8.5 % slower on 4 cases and
FlashAttention-4 2.2–33 % slower on 5, the rest tied (paired runs, sign test, Holm-corrected p ≤ 0.011 on every
slower case). On the GPU images for Ampere and newer GPUs (the GB10 and Grace Blackwell images) Eugo ships
Dao-AILab's FlashAttention-2 (`flash_attn`), so code written for `attn_implementation="flash_attention_2"` runs as
written, with no warning; that is for compatibility, not speed (this build has not been timed). If your code asks for
a FlashAttention that cannot run here (FlashAttention-3, which Eugo does not build; FlashAttention-2 on the Graviton
images; FlashAttention-4 without vLLM; or any of them with no NVIDIA GPU visible), Transformers logs one warning and
runs the model on the default attention instead of stopping with an `ImportError`. Remove the argument or pass
`"sdpa"` to silence that warning.

## 3. Training with attention dropout: use a multiple of 1/16

One detail worth knowing before you set a hyper-parameter by habit. The fused attention kernel takes its dropout
probability **in sixteenths**. `0.1` is not a sixteenth, so a training-mode forward pass with `dropout=0.1` leaves the
fused path.

Your step still runs; it just runs on the reference path. You can tell which one you got without changing any code:
at `dropout=0.1` PyTorch prints a warning mentioning memory efficient attention the first time the layer runs, and at
`dropout=0.125` it prints nothing at all.

Pick `0.0625` (1/16), `0.125` (2/16), `0.1875`, `0.25` and so on instead; the difference from `0.1` is not one your
model will notice:

```python
model = torch.nn.TransformerEncoderLayer(
    d_model=768, nhead=12, dropout=0.125, batch_first=True   # 2/16
).cuda().train()

with torch.autocast(device_type="cuda", dtype=amp_dtype):
    y = model(torch.randn(8, 256, 768, device="cuda"))

print("trained forward:", tuple(y.shape), y.dtype)
```

```text
trained forward: (8, 256, 768) torch.float32
```

This only applies while dropout is actually active: a module in `eval()` has attention dropout off, so inference is
never affected whatever you set. It applies equally to a `dropout_p=` you pass to `scaled_dot_product_attention`
yourself.

If you train in `float16` rather than `bfloat16`, pair `autocast` with `torch.amp.GradScaler` the way PyTorch
documents. `bfloat16` does not need it.

## 4. Quick reference

| Situation | What to do |
|---|---|
| Inference on a model you built or loaded in float32 | wrap the forward pass in `torch.autocast(device_type="cuda", dtype=torch.bfloat16)` |
| Model already in `bfloat16` / `float16` | nothing for inference; in training, check the dropout row too |
| Training in float32 with `autocast` | works as-is; add `torch.amp.GradScaler` only if you chose `float16` |
| Training with attention dropout | set it to a multiple of 1/16: `0.0625`, `0.125`, `0.25` |
| Out of memory at long sequence length in float32 | wrap it in `autocast`; the float32 reference path's intermediates grow with the square of the sequence length |
| On the T4G (the NVIDIA GPU of the Graviton2 images, `sm_75`) | use `dtype=torch.float16`, not `bfloat16` (its one fused kernel, memory-efficient attention, takes no bfloat16 there; §1) |
| PyTorch mentions `was not compiled with memory efficient attention` | that call did not reach a fused kernel. Either its inputs were float32 (wrap them in `autocast`) or you are training with a dropout that is not a multiple of 1/16 |
| Transformers code passes `attn_implementation="flash_attention_2"` (or `_3`, `_4`) | remove it or pass `"sdpa"`: on the GB10 the default already runs cuDNN's fused kernel, and FlashAttention was never faster there: slower on 4-5 of 7 cases, tied on the rest (§2.3) |
| Not sure whether the wrapper took effect | check a tensor's `.dtype` inside the block, as in §2: a `bfloat16` result from `float32` inputs is the proof |
| You set `torch._inductor.config.cuda.enable_ptxas_info = True` to see each CUDA kernel's register and memory use | if every CUTLASS GEMM compile then fails at the link with `ptxas fatal : Unknown option '-keep'`, leave the option off. Otherwise start the container with `-e EUGO_PTXAS_DEBUG_LEVEL=2`: PyTorch then logs each CUTLASS compile's compiler output as a warning, and the link's output carries ptxas's `ptxas info` lines (registers, barriers, spills). Below level 2 the image's ptxas wrapper drops the `-v` this option adds, so the log has no report. Level 2 can make large GPU kernel builds extremely slow (the wrapper prints a warning) |
| vLLM refuses to start with `KV connector ... is incompatible with PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True` (you asked for KV offload or a KV connector) | the image adds `expandable_segments:True` to every process unless you set that key yourself. That default does not stop vLLM: when a KV connector is configured, vLLM switches it off for itself and the processes it starts, and logs one warning saying so; those processes give up cuda-checkpoint restore (cuda-checkpoint is NVIDIA's tool that saves a running GPU process's state and restores it later; this matters only if you use it). You still see this error when you set `expandable_segments:True` yourself (including after the image set it: any value of `PYTORCH_CUDA_ALLOC_CONF` other than the exact one the image composed counts as yours). Whenever you see it, start that process with `-e PYTORCH_CUDA_ALLOC_CONF=expandable_segments:False`, or pass vLLM's own `--enable-cumem-allocator`. The first gives up cuda-checkpoint restore for that process |
| Comparing seeded `torch.randn` output on the CPU with stock PyTorch, bit for bit | with the same seed, `torch.randn` on the CPU on 16 or more float32/float64 elements is not bit-identical to stock PyTorch, because Eugo's PyTorch uses ArmPL's vector log/sin/cos (within Arm's documented accuracy); to compare against stock bit for bit, create the inputs once and load them in both |

## See also

- The guide to installing Python packages, in this Guides section: installing your own packages alongside Eugo's
  bundled PyTorch.
- The EugoHPC Manager guide, in this Guides section: running the same code across a cluster.
