Memory for distributed jobs
When your code runs on a Ray cluster through eugo.hpc, memory is used in two separate places on every node:
- The object store, a block of memory that holds the objects your tasks pass around: the values tasks return, and anything you store with
eugo.hpc.put. Ray keeps it in the node's shared memory when there is room for it (see Shared memory and the object store). - Your tasks' own memory, the ordinary memory each worker process uses while your function runs (the lists, arrays and DataFrames it builds as it goes).
This page explains both: how big the object store is and how to change it, what happens when it fills up, and what it looks like when a task runs out of its own memory and what to do about it. It builds on Installing Python packages and EugoHPC Manager, which cover what runs on your cluster and how to connect to it.
Note:
eugo.hpcis Ray, and the settings on this page are Ray's own. The outputs shown were captured in an Eugo image, in a session with 8 GiB of memory and 64 MiB of shared memory, with Ray's own log lines left out. Your numbers depend on the memory of the machine you run on.
Note: Run the examples on this page in a notebook that has not used Connect, or restart the kernel first. Each example starts Ray in your session with
eugo.hpc.init()and ends witheugo.hpc.shutdown(). In a notebook that is connected to a cluster, thateugo.hpc.shutdown()ends your connection to the cluster, and the code you run after it runs in your session, not on your cluster; to connect again, click Connect on the cluster (see The Connect button). To read your cluster's numbers while you are connected, run only thecluster_resources()andavailable_resources()lines of the first example, withouteugo.hpc.init()andeugo.hpc.shutdown().
What the object store is
Every node in a Ray cluster runs one object store. When a task returns a value, or when you call eugo.hpc.put(value), Ray stores the value there once and hands you a reference to it. Any task on that node can then read it without making its own copy; a large NumPy array is read in place. When a task on another node needs the object, Ray copies it across to that node's store.
The object store is not where your function's working memory lives. A task that builds a 3 GB list inside its own function body uses its worker process's memory, not the object store, and that is the memory that can run out (see When a task runs out of memory).
How big the object store is
By default Ray looks at how much memory is free on the node when it starts, gives 30% of that to the object store (up to 200 GB, and less when shared memory is small, see Shared memory and the object store), and offers the rest to your tasks. You can read both from your session:
import eugo.hpc
eugo.hpc.init()
gib = 1024**3
total = eugo.hpc.cluster_resources()
free = eugo.hpc.available_resources()
print(f"object store: {total['object_store_memory'] / gib:.2f} GiB, {free['object_store_memory'] / gib:.2f} GiB free")
print(f"memory for tasks: {total['memory'] / gib:.2f} GiB")
eugo.hpc.shutdown()
object store: 2.04 GiB, 2.04 GiB free
memory for tasks: 4.76 GiB
The two add up to 6.80 GiB, the memory that was free in this 8 GiB session when Ray started, and 2.04 GiB is 30% of that. memory for tasks is the memory resource Ray uses when it schedules tasks (see Tell Ray how much memory a task needs).
A bigger object store does not make Ray faster. It only raises the point at which Ray starts writing objects to disk.
Changing the size when your session starts Ray
When you call eugo.hpc.init() with no address, in a session that is not connected to a cluster, Ray starts on your session's own machine. In that case you can set the store's size, in bytes, with object_store_memory:
import eugo.hpc
eugo.hpc.init(object_store_memory=1 * 1024**3)
print(eugo.hpc.cluster_resources()["object_store_memory"] / 1024**3, "GiB")
eugo.hpc.shutdown()
1.0 GiB
object_store_memory applies only when Ray starts on your session's own machine. Each node's object store is sized when that node starts, so a connection to a cluster that is already running cannot change it. If you add object_store_memory to a connection you write yourself (see Connecting by hand), the connection fails with an error that includes When connecting to an existing cluster, object_store_memory must not be provided.
Shared memory and the object store
Ray keeps each node's object store in shared memory (/dev/shm) when the store fits there. You can see how much shared memory your session has with:
df -h /dev/shm
When Ray starts on your session's own machine, the size of /dev/shm matters in two ways:
- With no size given, Ray keeps the 30% default no larger than 95% of the free shared memory, or 10 GB (about 9.3 GiB) if that is larger. If even that does not fit in
/dev/shm, Ray keeps the store in files under/tmp/rayinstead. The session behind this page's outputs had 64 MiB of shared memory, so its 2.04 GiB store was kept there. - With an
object_store_memorythat does not fit in/dev/shm, Ray uses the size you asked for and keeps the store under/tmp/rayin the same way, as long as the size is 10 GB or less. Above 10 GB, Ray does not start, andeugo.hpc.initraises an error like this one (the sizes are replaced by<size>):
ValueError: The configured object store size (<size> GB) exceeds /dev/shm size (<size> GB). This will harm performance. Consider deleting files in /dev/shm or increasing its size with --shm-size in Docker. To ignore this warning, set RAY_OBJECT_STORE_ALLOW_SLOW_STORAGE=1.
--shm-size is set when a container starts, so you cannot change it from inside your session. To start Ray with the larger store anyway (if your session has the memory for it), set the environment variable the message names before Ray starts, for example with os.environ["RAY_OBJECT_STORE_ALLOW_SLOW_STORAGE"] = "1" before eugo.hpc.init. Ray then keeps the store under /tmp/ray, as above.
Whenever Ray keeps the store under /tmp/ray, it logs a warning that this will harm performance. Your code still sees the same values.
On a cluster from the EugoHPC Manager
The EugoHPC Manager starts Ray on each node of your cluster without fixing the object store's size, so Ray sizes each node's store from that node's memory, as above. There is no separate setting for it: to get a bigger object store, choose a larger Instance Size (more RAM) for that node type in the Cluster Launcher. This stops helping at the very largest sizes: once 30% of a node's free memory passes 200 GB (at about 620 GiB free), the store stays at 200 GB.
When the object store fills up: spilling
When the object store fills up, Ray does not stop your job. Once the objects in it pass 90% of the store's size (Eugo images set Ray's RAY_object_spilling_threshold to 0.9), Ray writes objects that are not in use to the node's local disk (it "spills" them) and reads them back when a task needs them again. Your code sees the same values; reading a spilled object back from disk takes longer than reading it from memory.
Here six 256 MiB arrays, 1.5 GiB in all, go into a 1 GiB store:
import numpy as np
import eugo.hpc
eugo.hpc.init(object_store_memory=1 * 1024**3)
# Six 256 MiB arrays: 1.5 GiB of objects in a 1 GiB object store.
refs = [eugo.hpc.put(np.full(32 * 1024**2, i, dtype=np.float64)) for i in range(6)]
print([float(eugo.hpc.get(r)[0]) for r in refs])
eugo.hpc.shutdown()
[0.0, 1.0, 2.0, 3.0, 4.0, 5.0]
All six arrays come back with the right values, although only 1 GiB of them fits in the store at a time: Ray wrote the ones that did not fit to disk and read them back when eugo.hpc.get asked for them.
If a job spills a lot, it takes longer than it needs to. Store fewer large objects at once (drop references you no longer need, so Ray can free them), or give the job more memory.
Spilling needs free space on the node's local disk. When the store is full and the disk is more than 95% used, Ray cannot spill, and storing a new object fails with an OutOfDiskError whose message begins Local disk is full.
When a task runs out of memory
Ray watches each node's memory. When the node is nearly out of memory, Ray kills one or more worker processes to protect the node, and the tasks that were running in them fail with an OutOfMemoryError. This task keeps growing until that happens:
import eugo.hpc
eugo.hpc.init()
@eugo.hpc.distribute
def grow():
import time
chunks = []
while True:
chunks.append(bytearray(256 * 1024**2)) # 256 MiB more each step
time.sleep(0.5)
try:
eugo.hpc.get(grow.remote())
except Exception as err:
print(type(err).__name__)
print(err)
eugo.hpc.shutdown()
The message is long; this is its start and end, with the node's address and Ray's internal IDs replaced by <ip> and <id>:
OutOfMemoryError
1 worker(s) were killed due to the node running low on memory. Memory on the node (IP: <ip>, ID: <id>) was 7.63GB / 8.00GB (0.953372)
OOM kill reason: Memory usage 8189480960B exceeded threshold of 8160437760B (95.0% of 8589934592B total)
...
Ray killed 1 worker(s) based on the killing policy
Considered workers: [
Selected to kill: (Task: job ID=01000000, lease ID=<id>, task name=grow, required resources={CPU: 1}, pid=8935, actual memory used=5.09GB, worker ID=<id>)
]
...
Top 10 memory users: PID MEM(GB) COMMAND
8935 5.09 ray::grow
...
Refer to the documentation on how to address the out of memory issue: https://docs.ray.io/en/latest/ray-observability/user-guides/debug-apps/debug-memory.html. Consider provisioning more memory on this node or reducing task parallelism by requesting more CPUs per task. ... Since 2.56, Ray updated the oom killing policy to enabling killing multiple workers and selecting workers based on the time since the task start executing. ...
The node's memory use passed Ray's limit, 95% of the node's memory by default, so Ray killed the worker running grow, the only task running. Among running tasks, Ray does not choose the one using the most memory: it kills the one that started most recently first, and more after it if that does not free enough memory (the line that begins Since 2.56 says so). So the task that failed is not always the one using the memory: the Top 10 memory users list in the full message shows which processes were, including Ray's own processes and idle workers, which count towards that limit too. Eugo images set Ray's RAY_task_oom_retries to 0, so Ray does not retry a task it killed for running out of memory: the error reaches your code straight away.
What to do when you see this:
Give each task less work
Split the input into more, smaller pieces, so each task holds less in memory at once. Ten tasks that each read a tenth of a file need far less memory per worker than one task that reads all of it.
Put large inputs once, and pass the reference
When many tasks need the same large input, store it once with eugo.hpc.put and pass the reference to every task. Ray stores the array once, and each task on the node reads that one copy:
import numpy as np
import eugo.hpc
eugo.hpc.init()
@eugo.hpc.distribute
def column_sum(table, col):
return float(table[:, col].sum())
table = np.ones((5_000_000, 10)) # about 400 MB
table_ref = eugo.hpc.put(table) # stored once
futures = [column_sum.remote(table_ref, c) for c in range(10)]
print(eugo.hpc.get(futures))
eugo.hpc.shutdown()
[5000000.0, 5000000.0, 5000000.0, 5000000.0, 5000000.0, 5000000.0, 5000000.0, 5000000.0, 5000000.0, 5000000.0]
Passing table itself to each .remote(...) call instead makes Ray store a new copy of the array for every call (Ray's documentation describes this as passing a large argument by value).
Tell Ray how much memory a task needs
By default Ray schedules tasks by CPU only, so it may start more memory-hungry tasks on a node at once than the node can hold. Declare what a task needs with memory= (in bytes), and Ray runs only as many of those tasks at once as the node's memory resource can fit:
import time
import eugo.hpc
eugo.hpc.init(num_cpus=4)
@eugo.hpc.distribute(memory=2 * 1024**3)
def step(i):
time.sleep(2)
return i
start = time.time()
print(eugo.hpc.get([step.remote(i) for i in range(4)]))
print(f"{time.time() - start:.0f} s")
eugo.hpc.shutdown()
[0, 1, 2, 3]
4 s
The session has 4 CPUs, enough to run all four tasks at once, but the memory Ray offers tasks here (under 5 GiB, as in the first example) holds only two 2 GiB reservations. So Ray ran the tasks two at a time, and four 2-second tasks took 4 seconds.
num_cpus= works the same way for CPUs: a task that declares num_cpus=2 takes two of the node's CPUs, so fewer such tasks run at once. Both are reservations for scheduling, not limits: Ray does not stop a task from using more memory than it declared.
Use larger nodes
If a single task genuinely needs more memory than a node has, choose a larger Instance Size for its node type in the Cluster Launcher.
Learn more
These pages in Ray's own documentation go into depth:
- Memory management: how Ray uses memory, and how to debug memory use.
- Objects: how objects and references work.
- Object spilling: spilling to disk.
- Out-of-memory prevention: how Ray decides which worker to kill, and how it retries.
- Resources:
num_cpus,memoryand custom resources.
Related
- Installing Python packages: packages and runtime environments on your cluster.
- EugoHPC Manager: your clusters, the Cluster Launcher, and connecting your session to a cluster.