# Lowering peak memory with one glibc setting on Graviton2

An opt-in glibc setting that can lower the peak memory of a memory-bound program on Graviton2 (AWS Lambda, AWS Fargate): what it saves, what it costs, and how to check that it took effect.

Source: https://docs.eugo.io/docs/guides/lowering-peak-memory/

---

> **Audience:** people running the Eugo image on AWS Lambda or AWS Fargate on Graviton2 whose program is limited by
> memory: it runs out of memory, or you pay for more memory than its CPU time needs.
>
> **The one thing to take away:** Eugo ships glibc's own default for how many heaps the memory allocator keeps, and
> we keep it that way. If your program is memory-bound, you can opt in to fewer heaps with
>
> ```
> GLIBC_TUNABLES=glibc.malloc.hugetlb=0:glibc.malloc.arena_max=2
> ```
>
> In our test it about halved the peak memory of many threads allocating small blocks, and it made another case
> slower and larger. It is a trade, so it is **not on by default**. Measure your own program before you keep it.
>
> **How this guide was written:** the second and third commands in section 3 were run on **2026-11-13** in an Eugo
> GB10 extended image, through the image's own start-up script, and their output is that run's own output, pasted
> verbatim (the start-up script's own lines left out). The measurements in section 4 come from a test session on
> three rented AWS machines on **2026-10-14**, with the same glibc (2.43, from Fedora 44) but not our image (details
> in section 5). The figures in section 4 are copied from that session's record by a script, not typed.

---

## 1. What the setting changes

glibc's `malloc` gives threads separate heaps, called arenas, each with its own lock, so that threads do not wait for
each other. By default it allows up to eight arenas per CPU core. More arenas means less waiting, and more memory held
in partly used heaps.

`glibc.malloc.arena_max=2` caps the count at two. Threads then share two heaps: memory is reused more and held less,
and threads wait for the shared locks more often.

## 2. The exact line, and why `hugetlb=0` is repeated in it

```
GLIBC_TUNABLES=glibc.malloc.hugetlb=0:glibc.malloc.arena_max=2
```

The image already sets `GLIBC_TUNABLES=glibc.malloc.hugetlb=0`. A value you set **replaces** that one; glibc does not
merge the two, and it prints no warning. If you write only `glibc.malloc.arena_max=2`, the image's `hugetlb=0` is
silently gone (section 3 shows it). That matters: on the glibc the image ships (2.43, on Arm), leaving `hugetlb` unset
makes `malloc` ask the kernel for 2 MiB huge pages, and on a busy machine the asking thread can wait while the kernel
makes room, the very wait the image's `hugetlb=0` turns off. So restate it, and keep your container identical to the
image's default apart from the one setting you chose.

Setting `GLIBC_TUNABLES` yourself also switches off a start-up rule of the image: when a container starts on a machine
where asking for huge pages cannot make a thread wait (the kernel's huge-page `defrag` setting is `defer` or `never`,
and its huge page is 2 MiB) and the container may use 16 GiB or more, the image turns huge pages on for it. Your own
value always wins, so with the line above huge pages stay off. If you want them in that situation, write
`glibc.malloc.hugetlb=1` in place of `glibc.malloc.hugetlb=0`, a separate trade, not measured here.

To keep that rule off without writing a `GLIBC_TUNABLES` of your own, set **`EUGO_AUTO_HUGEPAGES=0`**. Any value other
than `1` switches it off too (off is the safe choice); unset or `1` lets the rule decide. Set it when your memory limit
is on a parent of the container rather than on the container itself, for example an Amazon ECS task on EC2 that sets
memory for the task only: the container cannot see that limit, so the rule judges by the machine's size. A memory
limit on the container itself also works.

Set it the way you set any environment variable of your container: `docker run -e`, the `environment` list of an
Amazon ECS task definition (Fargate), or the environment variables of a Lambda function. Then check that it reached
glibc (section 3); do not assume it did.

## 3. Check that it took effect

glibc's loader prints the values it resolved for the current environment. Run it in the same container, with the same
environment, as your program (from a shell, or with `subprocess.run` from your Python code).

Print the variable too: on this glibc the loader shows `hugetlb` as `0x0` both when it is set to `0` and when it is
not set at all, so the loader alone cannot tell you whether the image's value survived.

First, the image as shipped:

```
$ docker run --rm <image> sh -c 'echo "GLIBC_TUNABLES=${GLIBC_TUNABLES-<unset>}"; /usr/bin/ld.so --list-tunables | grep -E "^glibc\.malloc\.(arena_max|hugetlb):"'
```

Its `arena_max` line reads `0x0`, which means unset, so glibc's default. Its first line shows the image's own
`GLIBC_TUNABLES`, or `glibc.malloc.hugetlb=1` if the start-up rule in section 2 turned huge pages on for this
container.

Then the opt-in, and the mistake section 2 warns about:

```
$ docker run --rm -e GLIBC_TUNABLES=glibc.malloc.hugetlb=0:glibc.malloc.arena_max=2 <image> sh -c 'echo "GLIBC_TUNABLES=${GLIBC_TUNABLES-<unset>}"; /usr/bin/ld.so --list-tunables | grep -E "^glibc\.malloc\.(arena_max|hugetlb):"'
GLIBC_TUNABLES=glibc.malloc.hugetlb=0:glibc.malloc.arena_max=2
glibc.malloc.arena_max: 0x2 (min: 0x1, max: 0xffffffffffffffff)
glibc.malloc.hugetlb: 0x0 (min: 0x0, max: 0xffffffffffffffff)
$ docker run --rm -e GLIBC_TUNABLES=glibc.malloc.arena_max=2 <image> sh -c 'echo "GLIBC_TUNABLES=${GLIBC_TUNABLES-<unset>}"; /usr/bin/ld.so --list-tunables | grep -E "^glibc\.malloc\.(arena_max|hugetlb):"'
GLIBC_TUNABLES=glibc.malloc.arena_max=2
glibc.malloc.arena_max: 0x2 (min: 0x1, max: 0xffffffffffffffff)
glibc.malloc.hugetlb: 0x0 (min: 0x0, max: 0xffffffffffffffff)
```

The first is the opt-in: `arena_max` is `0x2`, and the variable still carries `hugetlb=0`. The second is the mistake:
`arena_max` is set and `hugetlb=0` has gone from the variable; the loader still prints `0x0`, which is why the
variable line is part of the check.

(The image's start-up script first prints three lines of its own, beginning `[EUGO][INFO]`; they are left out above.
The runs above also passed `--runtime=runc`, which matters only on a machine whose Docker uses the NVIDIA runtime by
default.)

## 4. What it saves and what it costs, as measured

The test session ran each case 20 times with the setting and 20 times without it, paired, half of the pairs in each
order. In the tables, **B** is the run with `arena_max=2` and **A** the same run without it; both carry `hugetlb=1`
(section 5 says what that leaves unmeasured). The three cases are small test programs, many Python threads each:

- `arena_contention_small`: threads allocating and freeing blocks of **8–64 KiB**;
- `arena_contention_raw`: the same, with blocks of **288 KiB–2.6 MiB**;
- `malloc_churn`: threads cycling medium-sized allocations.

**Time.** Each cell gives the median B/A ratio over the 20 pairs (above 1 means B was slower), then the pairs in which
B was faster out of the untied pairs, then the p-value, Holm-corrected for the 27 time comparisons made on each
machine. Bold marks a result the test calls (it clears the bar described below, judged the same way for slower and
faster). The last two columns give the worst peak-memory ratio B/A across the three cases, and the verdict.

| machine | `malloc_churn` | `arena_contention_small` | `arena_contention_raw` | worst peak memory B/A | verdict |
|---|---|---|---|---|---|
| Graviton2 | 1.048, 3/20, p 6.18e-02 | 1.011, 6/20, p 1.00e+00 | **1.049, 0/20, p 5.15e-05** | 1.081 | B slower on `arena_contention_raw` |
| Graviton5 | 0.997, 11/20, p 1.00e+00 | **1.037, 2/20, p 1.01e-02** | **1.304, 0/20, p 5.15e-05** | 1.003 | B slower on `arena_contention_raw` and `arena_contention_small` |
| Graviton4 (standing in for Grace) | 0.984, 14/20, p 1.00e+00 | **1.022, 1/20, p 1.00e-03** | **1.141, 0/20, p 5.15e-05** | 1.007 | B slower on `arena_contention_raw` and `arena_contention_small` |

**Peak memory.** Each cell gives the median B/A ratio of the peak resident memory over the 20 pairs, then the pairs
in which B was smaller, then the sign-test p-value. These p-values are not corrected, because the memory rows are not
part of the 27 time comparisons (the session's two other settings in this table are settings never to use, and are
left out here):

| setting | machine | `arena_contention_small` | `arena_contention_raw` | `malloc_churn` |
|---|---|---|---|---|
| `arena_max=2` | Graviton2 | **0.549**, 20/20 smaller, p 1.9e-06 | **1.081**, 1/20 smaller, p 4.0e-05 | 0.926, 16/20, p 0.012 |
| `arena_max=2` | Graviton5 | **0.521**, 20/20, p 1.9e-06 | 1.003, 9/20, p 0.82 | **0.930**, 18/20, p 4.0e-04 |
| `arena_max=2` | Graviton4 | **0.523**, 20/20, p 1.9e-06 | 1.007, 7/20, p 0.26 | 0.940, 16/20, p 0.012 |

**How much the machines moved on their own.** Each machine also ran the same case with the same setting on both sides
(an A/A run); the largest median move of that run across the three cases is the machine's noise floor. A time result
is called only when it clears twice the floor (the bar), clears a corrected p of 0.05, and has the same sign in both
orders, the same way for slower and faster:

| machine | A/A floor | bar | A/A run per case (median %, pairs) |
|---|---|---|---|
| Graviton2 | 0.103 % | 0.205 % | −0.077 (9/20), −0.085 (10/20), +0.103 (14/20) |
| Graviton5 | 0.773 % | 1.546 % | −0.179 (9/20), +0.773 (12/20), −0.117 (8/20) |
| Graviton4 | 0.510 % | 1.020 % | −0.196 (7/20), −0.417 (7/20), −0.510 (6/20) |

The session had no such floor for peak memory, so the memory rows rest on the sign test alone.

**On Graviton2, the machine this opt-in is for** (each figure below is the B/A ratio, then the pairs, then the
p-value, copied from the tables above):

- **It saves memory on small blocks.** Peak memory of the 8–64 KiB case: **0.549**, 20/20 smaller, p 1.9e-06
  (uncorrected p).
- **It costs time, and memory, on large blocks.** Time of the 288 KiB–2.6 MiB case: **1.049, 0/20, p 5.15e-05**
  (corrected p; the middle figure counts the pairs in which B was faster). Its peak memory grew too: **1.081**, 1/20
  smaller, p 4.0e-05 (uncorrected p). If your program mostly allocates blocks of that size, this setting makes it
  slower and larger.
- **Not called either way:** the time of the 8–64 KiB case (1.011, 6/20, p 1.00e+00) and of `malloc_churn` (1.048,
  3/20, p 6.18e-02). Neither clears a corrected p of 0.05, so the test cannot say whether these moved. And
  `malloc_churn`'s peak memory (0.926, 16/20, p 0.012) is not bold in the session's record, so we do not count it as a
  saving.

**On Graviton5 and Graviton4 we do not suggest it** without measuring your own program: there the time cost was
larger (their rows in the first table), and the opt-in was decided for Graviton2 only.

## 5. What the numbers do not tell you

- **They are not your program.** The three cases are small synthetic tests in Python threads, which take turns more
  than native threads do. A real program can gain more, gain less, or do worse.
- **They are not our image.** The session ran stock Fedora 44 with the same glibc build (2.43) that the image ships,
  not the image's own Python or libraries.
- **They are not your machine size.** Each test machine had 8 vCPUs (Graviton2 was a `c6g.2xlarge`). Lambda and
  Fargate sizes were not measured, and the default arena count grows with the number of cores.
- **Not measured on the NVIDIA GB10 or on Grace.** Graviton4 stood in for Grace's CPU core; its memory system
  differs.
- **Not measured with `hugetlb=0`.** Both sides of the session carried `hugetlb=1`.
  The line this page recommends restates `hugetlb=0`, which only changes whether `malloc` asks for huge pages for
  blocks of 2 MiB or more; whether the saving above is the same with it has not been measured.

So before you keep the setting, compare your own program with and without it: its peak memory (on Lambda, the
`Max Memory Used` figure in each invocation's `REPORT` log line) and its run time, over several runs each way.

## 6. Quick reference

| Situation | What to do |
|---|---|
| Your program runs out of memory, or memory is what you pay for, on Graviton2 | try `GLIBC_TUNABLES=glibc.malloc.hugetlb=0:glibc.malloc.arena_max=2` and measure |
| Your program mostly allocates blocks of about 288 KiB–2.6 MiB | do not use it: that case got slower and larger |
| Your program is limited by time, not memory | do not use it: no case got faster with it in the session |
| You set any `GLIBC_TUNABLES` value of your own | always restate `glibc.malloc.hugetlb=0` (the image's default) in the same string, or `=1` if you want huge pages |
| You want the image never to turn huge pages on by itself (for example, your memory limit is set for an ECS task, not for its container) | set `EUGO_AUTO_HUGEPAGES=0` |
| You are not sure the setting reached glibc | run the check in section 3 (the variable and the loader) inside the same container |
