# Memory for distributed jobs

How memory works when your code runs on Ray through eugo.hpc: what the object store is, how big it is and how to change it, what spilling to disk means, and what to do when Ray kills a task for running out of memory.

Source: https://docs.eugo.io/docs/guides/memory-for-distributed-jobs/

---

When your code runs on a Ray cluster through `eugo.hpc`, memory is used in two separate places on every node:

1. **The object store**, a block of memory that holds the objects your tasks pass around: the values tasks return, and anything you store with `eugo.hpc.put`. Ray keeps it in the node's shared memory when there is room for it (see [Shared memory and the object store](#shared-memory-and-the-object-store)).
2. **Your tasks' own memory**, the ordinary memory each worker process uses while your function runs (the lists, arrays and DataFrames it builds as it goes).

This page explains both: how big the object store is and how to change it, what happens when it fills up, and what it looks like when a task runs out of its own memory and what to do about it. It builds on [Installing Python packages](./installing-python-packages.md) and [EugoHPC Manager](./hpc-cluster-manager.md), which cover what runs on your cluster and how to connect to it.

> **Note:** `eugo.hpc` is Ray, and the settings on this page are Ray's own. The outputs shown were captured in an Eugo image, in a session with 8 GiB of memory and 64 MiB of shared memory, with Ray's own log lines left out. Your numbers depend on the memory of the machine you run on.

> **Note:** Run the examples on this page in a notebook that has not used **Connect**, or restart the kernel first. Each example starts Ray in your session with `eugo.hpc.init()` and ends with `eugo.hpc.shutdown()`. In a notebook that is connected to a cluster, that `eugo.hpc.shutdown()` ends your connection to the cluster, and the code you run after it runs in your session, not on your cluster; to connect again, click **Connect** on the cluster (see [The Connect button](./hpc-cluster-manager.md#the-connect-button)). To read your cluster's numbers while you are connected, run only the `cluster_resources()` and `available_resources()` lines of the first example, without `eugo.hpc.init()` and `eugo.hpc.shutdown()`.

## What the object store is

Every node in a Ray cluster runs one object store. When a task returns a value, or when you call `eugo.hpc.put(value)`, Ray stores the value there once and hands you a reference to it. Any task on that node can then read it without making its own copy; a large NumPy array is read in place. When a task on another node needs the object, Ray copies it across to that node's store.

The object store is not where your function's working memory lives. A task that builds a 3 GB list inside its own function body uses its worker process's memory, not the object store, and that is the memory that can run out (see [When a task runs out of memory](#when-a-task-runs-out-of-memory)).

## How big the object store is

By default Ray looks at how much memory is free on the node when it starts, gives 30% of that to the object store (up to 200 GB, and less when shared memory is small, see [Shared memory and the object store](#shared-memory-and-the-object-store)), and offers the rest to your tasks. You can read both from your session:

```python
import eugo.hpc

eugo.hpc.init()
gib = 1024**3
total = eugo.hpc.cluster_resources()
free = eugo.hpc.available_resources()
print(f"object store: {total['object_store_memory'] / gib:.2f} GiB, {free['object_store_memory'] / gib:.2f} GiB free")
print(f"memory for tasks: {total['memory'] / gib:.2f} GiB")
eugo.hpc.shutdown()
```

```text
object store: 2.04 GiB, 2.04 GiB free
memory for tasks: 4.76 GiB
```

The two add up to 6.80 GiB, the memory that was free in this 8 GiB session when Ray started, and 2.04 GiB is 30% of that. `memory for tasks` is the `memory` resource Ray uses when it schedules tasks (see [Tell Ray how much memory a task needs](#tell-ray-how-much-memory-a-task-needs)).

A bigger object store does not make Ray faster. It only raises the point at which Ray starts writing objects to disk.

### Changing the size when your session starts Ray

When you call `eugo.hpc.init()` with no address, in a session that is not connected to a cluster, Ray starts on your session's own machine. In that case you can set the store's size, in bytes, with `object_store_memory`:

```python
import eugo.hpc

eugo.hpc.init(object_store_memory=1 * 1024**3)
print(eugo.hpc.cluster_resources()["object_store_memory"] / 1024**3, "GiB")
eugo.hpc.shutdown()
```

```text
1.0 GiB
```

`object_store_memory` applies only when Ray starts on your session's own machine. Each node's object store is sized when that node starts, so a connection to a cluster that is already running cannot change it. If you add `object_store_memory` to a connection you write yourself (see [Connecting by hand](./hpc-cluster-manager.md#connecting-by-hand)), the connection fails with an error that includes `When connecting to an existing cluster, object_store_memory must not be provided.`

### Shared memory and the object store

Ray keeps each node's object store in shared memory (`/dev/shm`) when the store fits there. You can see how much shared memory your session has with:

```bash
df -h /dev/shm
```

When Ray starts on your session's own machine, the size of `/dev/shm` matters in two ways:

- **With no size given**, Ray keeps the 30% default no larger than 95% of the free shared memory, or 10 GB (about 9.3 GiB) if that is larger. If even that does not fit in `/dev/shm`, Ray keeps the store in files under `/tmp/ray` instead. The session behind this page's outputs had 64 MiB of shared memory, so its 2.04 GiB store was kept there.
- **With an `object_store_memory` that does not fit in `/dev/shm`**, Ray uses the size you asked for and keeps the store under `/tmp/ray` in the same way, as long as the size is 10 GB or less. Above 10 GB, Ray does not start, and `eugo.hpc.init` raises an error like this one (the sizes are replaced by `<size>`):

```text
ValueError: The configured object store size (<size> GB) exceeds /dev/shm size (<size> GB). This will harm performance. Consider deleting files in /dev/shm or increasing its size with --shm-size in Docker. To ignore this warning, set RAY_OBJECT_STORE_ALLOW_SLOW_STORAGE=1.
```

`--shm-size` is set when a container starts, so you cannot change it from inside your session. To start Ray with the larger store anyway (if your session has the memory for it), set the environment variable the message names before Ray starts, for example with `os.environ["RAY_OBJECT_STORE_ALLOW_SLOW_STORAGE"] = "1"` before `eugo.hpc.init`. Ray then keeps the store under `/tmp/ray`, as above.

Whenever Ray keeps the store under `/tmp/ray`, it logs a warning that this will harm performance. Your code still sees the same values.

### On a cluster from the EugoHPC Manager

The EugoHPC Manager starts Ray on each node of your cluster without fixing the object store's size, so Ray sizes each node's store from that node's memory, as above. There is no separate setting for it: to get a bigger object store, choose a larger **Instance Size** (more RAM) for that node type in the Cluster Launcher. This stops helping at the very largest sizes: once 30% of a node's free memory passes 200 GB (at about 620 GiB free), the store stays at 200 GB.

## When the object store fills up: spilling

When the object store fills up, Ray does not stop your job. Once the objects in it pass 90% of the store's size (Eugo images set Ray's `RAY_object_spilling_threshold` to `0.9`), Ray writes objects that are not in use to the node's local disk (it "spills" them) and reads them back when a task needs them again. Your code sees the same values; reading a spilled object back from disk takes longer than reading it from memory.

Here six 256 MiB arrays, 1.5 GiB in all, go into a 1 GiB store:

```python
import numpy as np
import eugo.hpc

eugo.hpc.init(object_store_memory=1 * 1024**3)

# Six 256 MiB arrays: 1.5 GiB of objects in a 1 GiB object store.
refs = [eugo.hpc.put(np.full(32 * 1024**2, i, dtype=np.float64)) for i in range(6)]
print([float(eugo.hpc.get(r)[0]) for r in refs])
eugo.hpc.shutdown()
```

```text
[0.0, 1.0, 2.0, 3.0, 4.0, 5.0]
```

All six arrays come back with the right values, although only 1 GiB of them fits in the store at a time: Ray wrote the ones that did not fit to disk and read them back when `eugo.hpc.get` asked for them.

If a job spills a lot, it takes longer than it needs to. Store fewer large objects at once (drop references you no longer need, so Ray can free them), or give the job more memory.

Spilling needs free space on the node's local disk. When the store is full and the disk is more than 95% used, Ray cannot spill, and storing a new object fails with an `OutOfDiskError` whose message begins `Local disk is full`.

## When a task runs out of memory

Ray watches each node's memory. When the node is nearly out of memory, Ray kills one or more worker processes to protect the node, and the tasks that were running in them fail with an `OutOfMemoryError`. This task keeps growing until that happens:

```python
import eugo.hpc

eugo.hpc.init()

@eugo.hpc.distribute
def grow():
    import time
    chunks = []
    while True:
        chunks.append(bytearray(256 * 1024**2))  # 256 MiB more each step
        time.sleep(0.5)

try:
    eugo.hpc.get(grow.remote())
except Exception as err:
    print(type(err).__name__)
    print(err)
eugo.hpc.shutdown()
```

The message is long; this is its start and end, with the node's address and Ray's internal IDs replaced by `<ip>` and `<id>`:

```text
OutOfMemoryError
1 worker(s) were killed due to the node running low on memory. Memory on the node (IP: <ip>, ID: <id>) was 7.63GB / 8.00GB (0.953372)
OOM kill reason: Memory usage 8189480960B exceeded threshold of 8160437760B (95.0% of 8589934592B total)
...
Ray killed 1 worker(s) based on the killing policy
Considered workers: [
Selected to kill: (Task: job ID=01000000, lease ID=<id>, task name=grow, required resources={CPU: 1}, pid=8935, actual memory used=5.09GB, worker ID=<id>)
]
...
Top 10 memory users: PID	MEM(GB)	COMMAND
8935	5.09	ray::grow
...
Refer to the documentation on how to address the out of memory issue: https://docs.ray.io/en/latest/ray-observability/user-guides/debug-apps/debug-memory.html. Consider provisioning more memory on this node or reducing task parallelism by requesting more CPUs per task. ... Since 2.56, Ray updated the oom killing policy to enabling killing multiple workers and selecting workers based on the time since the task start executing. ...
```

The node's memory use passed Ray's limit, 95% of the node's memory by default, so Ray killed the worker running `grow`, the only task running. Among running tasks, Ray does not choose the one using the most memory: it kills the one that started most recently first, and more after it if that does not free enough memory (the line that begins `Since 2.56` says so). So the task that failed is not always the one using the memory: the `Top 10 memory users` list in the full message shows which processes were, including Ray's own processes and idle workers, which count towards that limit too. Eugo images set Ray's `RAY_task_oom_retries` to `0`, so Ray does not retry a task it killed for running out of memory: the error reaches your code straight away.

What to do when you see this:

### Give each task less work

Split the input into more, smaller pieces, so each task holds less in memory at once. Ten tasks that each read a tenth of a file need far less memory per worker than one task that reads all of it.

### Put large inputs once, and pass the reference

When many tasks need the same large input, store it once with `eugo.hpc.put` and pass the reference to every task. Ray stores the array once, and each task on the node reads that one copy:

```python
import numpy as np
import eugo.hpc

eugo.hpc.init()

@eugo.hpc.distribute
def column_sum(table, col):
    return float(table[:, col].sum())

table = np.ones((5_000_000, 10))      # about 400 MB
table_ref = eugo.hpc.put(table)       # stored once

futures = [column_sum.remote(table_ref, c) for c in range(10)]
print(eugo.hpc.get(futures))
eugo.hpc.shutdown()
```

```text
[5000000.0, 5000000.0, 5000000.0, 5000000.0, 5000000.0, 5000000.0, 5000000.0, 5000000.0, 5000000.0, 5000000.0]
```

Passing `table` itself to each `.remote(...)` call instead makes Ray store a new copy of the array for every call (Ray's documentation describes this as [passing a large argument by value](https://docs.ray.io/en/latest/ray-core/patterns/pass-large-arg-by-value.html)).

### Tell Ray how much memory a task needs

By default Ray schedules tasks by CPU only, so it may start more memory-hungry tasks on a node at once than the node can hold. Declare what a task needs with `memory=` (in bytes), and Ray runs only as many of those tasks at once as the node's `memory` resource can fit:

```python
import time
import eugo.hpc

eugo.hpc.init(num_cpus=4)

@eugo.hpc.distribute(memory=2 * 1024**3)
def step(i):
    time.sleep(2)
    return i

start = time.time()
print(eugo.hpc.get([step.remote(i) for i in range(4)]))
print(f"{time.time() - start:.0f} s")
eugo.hpc.shutdown()
```

```text
[0, 1, 2, 3]
4 s
```

The session has 4 CPUs, enough to run all four tasks at once, but the `memory` Ray offers tasks here (under 5 GiB, as in the first example) holds only two 2 GiB reservations. So Ray ran the tasks two at a time, and four 2-second tasks took 4 seconds.

`num_cpus=` works the same way for CPUs: a task that declares `num_cpus=2` takes two of the node's CPUs, so fewer such tasks run at once. Both are reservations for scheduling, not limits: Ray does not stop a task from using more memory than it declared.

### Use larger nodes

If a single task genuinely needs more memory than a node has, choose a larger **Instance Size** for its node type in the Cluster Launcher.

## Learn more

These pages in Ray's own documentation go into depth:

- [Memory management](https://docs.ray.io/en/latest/ray-core/scheduling/memory-management.html): how Ray uses memory, and how to debug memory use.
- [Objects](https://docs.ray.io/en/latest/ray-core/objects.html): how objects and references work.
- [Object spilling](https://docs.ray.io/en/latest/ray-core/objects/object-spilling.html): spilling to disk.
- [Out-of-memory prevention](https://docs.ray.io/en/latest/ray-core/scheduling/ray-oom-prevention.html): how Ray decides which worker to kill, and how it retries.
- [Resources](https://docs.ray.io/en/latest/ray-core/scheduling/resources.html): `num_cpus`, `memory` and custom resources.

## Related

- [Installing Python packages](./installing-python-packages.md): packages and runtime environments on your cluster.
- [EugoHPC Manager](./hpc-cluster-manager.md): your clusters, the Cluster Launcher, and connecting your session to a cluster.
