Lowering peak memory with one glibc setting on Graviton2
Audience: people running the Eugo image on AWS Lambda or AWS Fargate on Graviton2 whose program is limited by memory: it runs out of memory, or you pay for more memory than its CPU time needs.
The one thing to take away: Eugo ships glibc's own default for how many heaps the memory allocator keeps, and we keep it that way. If your program is memory-bound, you can opt in to fewer heaps with
GLIBC_TUNABLES=glibc.malloc.hugetlb=0:glibc.malloc.arena_max=2In our test it about halved the peak memory of many threads allocating small blocks, and it made another case slower and larger. It is a trade, so it is not on by default. Measure your own program before you keep it.
How this guide was written: the second and third commands in section 3 were run on 2026-11-13 in an Eugo GB10 extended image, through the image's own start-up script, and their output is that run's own output, pasted verbatim (the start-up script's own lines left out). The measurements in section 4 come from a test session on three rented AWS machines on 2026-10-14, with the same glibc (2.43, from Fedora 44) but not our image (details in section 5). The figures in section 4 are copied from that session's record by a script, not typed.
1. What the setting changes​
glibc's malloc gives threads separate heaps, called arenas, each with its own lock, so that threads do not wait for
each other. By default it allows up to eight arenas per CPU core. More arenas means less waiting, and more memory held
in partly used heaps.
glibc.malloc.arena_max=2 caps the count at two. Threads then share two heaps: memory is reused more and held less,
and threads wait for the shared locks more often.
2. The exact line, and why hugetlb=0 is repeated in it​
GLIBC_TUNABLES=glibc.malloc.hugetlb=0:glibc.malloc.arena_max=2
The image already sets GLIBC_TUNABLES=glibc.malloc.hugetlb=0. A value you set replaces that one; glibc does not
merge the two, and it prints no warning. If you write only glibc.malloc.arena_max=2, the image's hugetlb=0 is
silently gone (section 3 shows it). That matters: on the glibc the image ships (2.43, on Arm), leaving hugetlb unset
makes malloc ask the kernel for 2 MiB huge pages, and on a busy machine the asking thread can wait while the kernel
makes room, the very wait the image's hugetlb=0 turns off. So restate it, and keep your container identical to the
image's default apart from the one setting you chose.
Setting GLIBC_TUNABLES yourself also switches off a start-up rule of the image: when a container starts on a machine
where asking for huge pages cannot make a thread wait (the kernel's huge-page defrag setting is defer or never,
and its huge page is 2 MiB) and the container may use 16 GiB or more, the image turns huge pages on for it. Your own
value always wins, so with the line above huge pages stay off. If you want them in that situation, write
glibc.malloc.hugetlb=1 in place of glibc.malloc.hugetlb=0, a separate trade, not measured here.
To keep that rule off without writing a GLIBC_TUNABLES of your own, set EUGO_AUTO_HUGEPAGES=0. Any value other
than 1 switches it off too (off is the safe choice); unset or 1 lets the rule decide. Set it when your memory limit
is on a parent of the container rather than on the container itself, for example an Amazon ECS task on EC2 that sets
memory for the task only: the container cannot see that limit, so the rule judges by the machine's size. A memory
limit on the container itself also works.
Set it the way you set any environment variable of your container: docker run -e, the environment list of an
Amazon ECS task definition (Fargate), or the environment variables of a Lambda function. Then check that it reached
glibc (section 3); do not assume it did.
3. Check that it took effect​
glibc's loader prints the values it resolved for the current environment. Run it in the same container, with the same
environment, as your program (from a shell, or with subprocess.run from your Python code).
Print the variable too: on this glibc the loader shows hugetlb as 0x0 both when it is set to 0 and when it is
not set at all, so the loader alone cannot tell you whether the image's value survived.
First, the image as shipped:
$ docker run --rm <image> sh -c 'echo "GLIBC_TUNABLES=${GLIBC_TUNABLES-<unset>}"; /usr/bin/ld.so --list-tunables | grep -E "^glibc\.malloc\.(arena_max|hugetlb):"'
Its arena_max line reads 0x0, which means unset, so glibc's default. Its first line shows the image's own
GLIBC_TUNABLES, or glibc.malloc.hugetlb=1 if the start-up rule in section 2 turned huge pages on for this
container.
Then the opt-in, and the mistake section 2 warns about:
$ docker run --rm -e GLIBC_TUNABLES=glibc.malloc.hugetlb=0:glibc.malloc.arena_max=2 <image> sh -c 'echo "GLIBC_TUNABLES=${GLIBC_TUNABLES-<unset>}"; /usr/bin/ld.so --list-tunables | grep -E "^glibc\.malloc\.(arena_max|hugetlb):"'
GLIBC_TUNABLES=glibc.malloc.hugetlb=0:glibc.malloc.arena_max=2
glibc.malloc.arena_max: 0x2 (min: 0x1, max: 0xffffffffffffffff)
glibc.malloc.hugetlb: 0x0 (min: 0x0, max: 0xffffffffffffffff)
$ docker run --rm -e GLIBC_TUNABLES=glibc.malloc.arena_max=2 <image> sh -c 'echo "GLIBC_TUNABLES=${GLIBC_TUNABLES-<unset>}"; /usr/bin/ld.so --list-tunables | grep -E "^glibc\.malloc\.(arena_max|hugetlb):"'
GLIBC_TUNABLES=glibc.malloc.arena_max=2
glibc.malloc.arena_max: 0x2 (min: 0x1, max: 0xffffffffffffffff)
glibc.malloc.hugetlb: 0x0 (min: 0x0, max: 0xffffffffffffffff)
The first is the opt-in: arena_max is 0x2, and the variable still carries hugetlb=0. The second is the mistake:
arena_max is set and hugetlb=0 has gone from the variable; the loader still prints 0x0, which is why the
variable line is part of the check.
(The image's start-up script first prints three lines of its own, beginning [EUGO][INFO]; they are left out above.
The runs above also passed --runtime=runc, which matters only on a machine whose Docker uses the NVIDIA runtime by
default.)
4. What it saves and what it costs, as measured​
The test session ran each case 20 times with the setting and 20 times without it, paired, half of the pairs in each
order. In the tables, B is the run with arena_max=2 and A the same run without it; both carry hugetlb=1
(section 5 says what that leaves unmeasured). The three cases are small test programs, many Python threads each:
arena_contention_small: threads allocating and freeing blocks of 8–64 KiB;arena_contention_raw: the same, with blocks of 288 KiB–2.6 MiB;malloc_churn: threads cycling medium-sized allocations.
Time. Each cell gives the median B/A ratio over the 20 pairs (above 1 means B was slower), then the pairs in which B was faster out of the untied pairs, then the p-value, Holm-corrected for the 27 time comparisons made on each machine. Bold marks a result the test calls (it clears the bar described below, judged the same way for slower and faster). The last two columns give the worst peak-memory ratio B/A across the three cases, and the verdict.
| machine | malloc_churn | arena_contention_small | arena_contention_raw | worst peak memory B/A | verdict |
|---|---|---|---|---|---|
| Graviton2 | 1.048, 3/20, p 6.18e-02 | 1.011, 6/20, p 1.00e+00 | 1.049, 0/20, p 5.15e-05 | 1.081 | B slower on arena_contention_raw |
| Graviton5 | 0.997, 11/20, p 1.00e+00 | 1.037, 2/20, p 1.01e-02 | 1.304, 0/20, p 5.15e-05 | 1.003 | B slower on arena_contention_raw and arena_contention_small |
| Graviton4 (standing in for Grace) | 0.984, 14/20, p 1.00e+00 | 1.022, 1/20, p 1.00e-03 | 1.141, 0/20, p 5.15e-05 | 1.007 | B slower on arena_contention_raw and arena_contention_small |
Peak memory. Each cell gives the median B/A ratio of the peak resident memory over the 20 pairs, then the pairs in which B was smaller, then the sign-test p-value. These p-values are not corrected, because the memory rows are not part of the 27 time comparisons (the session's two other settings in this table are settings never to use, and are left out here):
| setting | machine | arena_contention_small | arena_contention_raw | malloc_churn |
|---|---|---|---|---|
arena_max=2 | Graviton2 | 0.549, 20/20 smaller, p 1.9e-06 | 1.081, 1/20 smaller, p 4.0e-05 | 0.926, 16/20, p 0.012 |
arena_max=2 | Graviton5 | 0.521, 20/20, p 1.9e-06 | 1.003, 9/20, p 0.82 | 0.930, 18/20, p 4.0e-04 |
arena_max=2 | Graviton4 | 0.523, 20/20, p 1.9e-06 | 1.007, 7/20, p 0.26 | 0.940, 16/20, p 0.012 |
How much the machines moved on their own. Each machine also ran the same case with the same setting on both sides (an A/A run); the largest median move of that run across the three cases is the machine's noise floor. A time result is called only when it clears twice the floor (the bar), clears a corrected p of 0.05, and has the same sign in both orders, the same way for slower and faster:
| machine | A/A floor | bar | A/A run per case (median %, pairs) |
|---|---|---|---|
| Graviton2 | 0.103 % | 0.205 % | −0.077 (9/20), −0.085 (10/20), +0.103 (14/20) |
| Graviton5 | 0.773 % | 1.546 % | −0.179 (9/20), +0.773 (12/20), −0.117 (8/20) |
| Graviton4 | 0.510 % | 1.020 % | −0.196 (7/20), −0.417 (7/20), −0.510 (6/20) |
The session had no such floor for peak memory, so the memory rows rest on the sign test alone.
On Graviton2, the machine this opt-in is for (each figure below is the B/A ratio, then the pairs, then the p-value, copied from the tables above):
- It saves memory on small blocks. Peak memory of the 8–64 KiB case: 0.549, 20/20 smaller, p 1.9e-06 (uncorrected p).
- It costs time, and memory, on large blocks. Time of the 288 KiB–2.6 MiB case: 1.049, 0/20, p 5.15e-05 (corrected p; the middle figure counts the pairs in which B was faster). Its peak memory grew too: 1.081, 1/20 smaller, p 4.0e-05 (uncorrected p). If your program mostly allocates blocks of that size, this setting makes it slower and larger.
- Not called either way: the time of the 8–64 KiB case (1.011, 6/20, p 1.00e+00) and of
malloc_churn(1.048, 3/20, p 6.18e-02). Neither clears a corrected p of 0.05, so the test cannot say whether these moved. Andmalloc_churn's peak memory (0.926, 16/20, p 0.012) is not bold in the session's record, so we do not count it as a saving.
On Graviton5 and Graviton4 we do not suggest it without measuring your own program: there the time cost was larger (their rows in the first table), and the opt-in was decided for Graviton2 only.
5. What the numbers do not tell you​
- They are not your program. The three cases are small synthetic tests in Python threads, which take turns more than native threads do. A real program can gain more, gain less, or do worse.
- They are not our image. The session ran stock Fedora 44 with the same glibc build (2.43) that the image ships, not the image's own Python or libraries.
- They are not your machine size. Each test machine had 8 vCPUs (Graviton2 was a
c6g.2xlarge). Lambda and Fargate sizes were not measured, and the default arena count grows with the number of cores. - Not measured on the NVIDIA GB10 or on Grace. Graviton4 stood in for Grace's CPU core; its memory system differs.
- Not measured with
hugetlb=0. Both sides of the session carriedhugetlb=1. The line this page recommends restateshugetlb=0, which only changes whethermallocasks for huge pages for blocks of 2 MiB or more; whether the saving above is the same with it has not been measured.
So before you keep the setting, compare your own program with and without it: its peak memory (on Lambda, the
Max Memory Used figure in each invocation's REPORT log line) and its run time, over several runs each way.
6. Quick reference​
| Situation | What to do |
|---|---|
| Your program runs out of memory, or memory is what you pay for, on Graviton2 | try GLIBC_TUNABLES=glibc.malloc.hugetlb=0:glibc.malloc.arena_max=2 and measure |
| Your program mostly allocates blocks of about 288 KiB–2.6 MiB | do not use it: that case got slower and larger |
| Your program is limited by time, not memory | do not use it: no case got faster with it in the session |
You set any GLIBC_TUNABLES value of your own | always restate glibc.malloc.hugetlb=0 (the image's default) in the same string, or =1 if you want huge pages |
| You want the image never to turn huge pages on by itself (for example, your memory limit is set for an ECS task, not for its container) | set EUGO_AUTO_HUGEPAGES=0 |
| You are not sure the setting reached glibc | run the check in section 3 (the variable and the loader) inside the same container |