The eugo.hpc API
eugo.hpc is the Python module you use to run work on your Eugo Ray cluster. It is Ray's Python API under Eugo's name: the same functions and the same .remote(...) call style, plus one function of Eugo's own, for the cluster's token. This page shows each part with an example and points to Ray's documentation for the details.
Two other pages cover the rest: EugoHPC Manager explains your clusters and how to connect a notebook to one, and Installing Python packages explains how to get your packages and code onto the cluster.
Note: Each example on this page that shows output was run as its own Python script in an Eugo image, with no cluster connected, so it ran on a local Ray started inside that process. The output shown is what that run printed. Ids, object names and timings will differ on your machine.
Importing
import eugo.hpc
Everything lives under eugo.hpc. The top-level eugo package is only a namespace and gives you nothing to call.
Connecting to a cluster
With the Connect button
In EugoIDE, click Connect on a cluster's card. Connect gives the notebook's kernel the cluster's token, without showing it, and adds a two-line cell that calls eugo.hpc.init(address="ray://<head>:10001"), so the notebook's work runs on that cluster. See The Connect button. This is the usual way to connect, and you do not need any of the calls below to use it.
Note: The examples on this page that call
eugo.hpc.init()oreugo.hpc.shutdown()are for a notebook that has not used Connect, or one whose kernel you have restarted. In a notebook connected with Connect,eugo.hpc.shutdown()ends your connection to the cluster, and the code you run after it runs in your own session, not on your cluster. To connect again, click Connect on the cluster's card (see The Connect button).
Without a cluster: a local Ray
eugo.hpc.init() with no address starts a private Ray inside your own process. It uses only the machine your code runs on, not your cluster.
import eugo.hpc
eugo.hpc.init() # no address: starts a local Ray in this process
print(eugo.hpc.is_initialized())
eugo.hpc.shutdown()
print(eugo.hpc.is_initialized())
True
False
You do not have to call init() first. If nothing is connected, the first call that runs work (a .remote(...), eugo.hpc.put or eugo.hpc.get) starts the local Ray itself:
import eugo.hpc
@eugo.hpc.distribute
def square(x):
return x * x
print(eugo.hpc.is_initialized())
print(eugo.hpc.get(square.remote(3))) # no init(): this call starts a local Ray
print(eugo.hpc.is_initialized())
False
9
True
Note: In EugoIDE this matters: if you forget to click Connect, your work still runs, but on a private Ray in your own session, not on your cluster. Click Connect before you run work you mean for the cluster.
Connecting with an address
eugo.hpc.init takes the same arguments as Ray's ray.init, such as address and runtime_env. Connect writes the address form for you. To write the connection yourself, for example to give the whole job a runtime environment, follow Connecting by hand.
import eugo.hpc
eugo.hpc.use_auth_token_file("/path/to/cluster_token") # before init
eugo.hpc.init(address="ray://<head>:10001")
print(eugo.hpc.is_initialized())
eugo.hpc.shutdown()
print(eugo.hpc.is_initialized())
Replace <head> with your cluster's head address, as in the cell Connect adds, and the file path with the file that holds your cluster's token. The token file is explained in Authenticating with a token file.
Calling init twice
While a local Ray you started is running in your process, a second eugo.hpc.init() fails. Run eugo.hpc.shutdown() first, then call eugo.hpc.init() again:
import eugo.hpc
eugo.hpc.init()
try:
eugo.hpc.init()
except RuntimeError as error:
print("RuntimeError:", error)
eugo.hpc.shutdown()
eugo.hpc.init() # works again after shutdown()
print(eugo.hpc.is_initialized())
RuntimeError: Maybe you called ray.init twice by accident? This error can be suppressed by passing in 'ignore_reinit_error=True' or by calling 'ray.shutdown()' prior to 'ray.init()'.
True
Authenticating with a token file
Every Eugo cluster runs with Ray's token authentication on, so a connection needs the cluster's token. eugo.hpc.use_auth_token_file(path) is the one function in eugo.hpc that is Eugo's own: it tells this process to read the token from the file at path (a string or a pathlib.Path). Connect calls it for you; call it yourself only when you have the cluster's token in a file of your own.
Three rules:
-
Call it before
eugo.hpc.init(...). Ray reads the token once per process and keeps it.use_auth_token_fileclears that saved token, so the next connection reads the file, even if the process connected to another cluster before. -
It refuses while the process is connected. Run
eugo.hpc.shutdown()first:import eugo.hpc
eugo.hpc.init()
try:
eugo.hpc.use_auth_token_file("/tmp/cluster_token")
except RuntimeError as error:
print("RuntimeError:", error)RuntimeError: eugo.hpc.use_auth_token_file: this process is connected to Ray; call eugo.hpc.shutdown() first -
A
RAY_AUTH_TOKENenvironment variable wins over the file. Ray looks at that variable first, so if it is set in your environment, the file is not used.
Running functions on the cluster
Decorate a function with @eugo.hpc.distribute, call .remote(...) on it to start it on the cluster, and pass what that returns to eugo.hpc.get to wait for the result. .remote(...) returns at once with a reference to the future result (an ObjectRef), so you can start many calls before you wait for any of them.
import eugo.hpc
@eugo.hpc.distribute
def square(x):
return x * x
future = square.remote(4)
print(type(future).__name__)
print(eugo.hpc.get(future))
print(eugo.hpc.get([square.remote(i) for i in range(5)]))
@eugo.hpc.distribute(num_cpus=2)
def double(x):
return x * 2
print(eugo.hpc.get(double.remote(10)))
print(eugo.hpc.get(double.options(num_cpus=1).remote(10)))
try:
square(4)
except TypeError as error:
print("TypeError:", error)
ObjectRef
16
[0, 1, 4, 9, 16]
20
20
TypeError: Remote functions cannot be called directly. Instead of running '__main__.square()', try '__main__.square.remote()'.
What the example shows:
eugo.hpc.gettakes one reference or a list. With a list it waits for all of them and returns the results in the same order.- The decorator takes Ray's options, such as
num_cpusandnum_gpus, to say what each call needs..options(...)changes them for one call. The options are Ray's: seeray.remoteand Tasks. - A decorated function cannot be called directly. Always call it with
.remote(...).
Actors
On a class, @eugo.hpc.distribute makes an actor: an object that lives on a cluster node and keeps its state between calls. Counter.remote() creates it, and each method call is .remote(...) too.
import eugo.hpc
@eugo.hpc.distribute
class Counter:
def __init__(self):
self.n = 0
def inc(self):
self.n += 1
return self.n
counter = Counter.remote()
print(eugo.hpc.get(counter.inc.remote()))
print(eugo.hpc.get([counter.inc.remote() for _ in range(3)]))
shared = Counter.options(name="shared-counter").remote()
same = eugo.hpc.get_actor("shared-counter")
eugo.hpc.get(shared.inc.remote())
print(eugo.hpc.get(same.inc.remote()))
eugo.hpc.kill(counter) # stops the actor
The first call prints 1 and the next three [2, 3, 4], because the actor keeps its count between calls. Giving an actor a name lets other code in the same job find it with eugo.hpc.get_actor, so the call through same reaches the same actor. eugo.hpc.kill stops an actor, interrupting any call it is running. See Ray's Actors and Named actors.
Objects: put, wait and cancel
import time
import eugo.hpc
numbers = eugo.hpc.put(list(range(1_000))) # stored once in the object store
@eugo.hpc.distribute
def total(values):
return sum(values)
print(eugo.hpc.get(total.remote(numbers))) # the task receives the list itself
@eugo.hpc.distribute
def sleep_for(seconds):
time.sleep(seconds)
return seconds
futures = [sleep_for.remote(s) for s in (5, 0, 10)]
ready, not_ready = eugo.hpc.wait(futures, num_returns=1, timeout=30)
print(eugo.hpc.get(ready), len(not_ready))
slow = sleep_for.remote(60)
eugo.hpc.cancel(slow)
try:
eugo.hpc.get(slow)
except Exception as error:
print(type(error).__name__)
499500
[0] 2
RayTaskError(TaskCancelledError)
eugo.hpc.putstores a value in the cluster's object store once and gives you a reference. Pass the reference to many calls instead of the value, so a large value is not sent again with every call.eugo.hpc.waitreturns as soon as some of the results are ready: a list of the ready references and a list of the rest.num_returnssays how many to wait for, andtimeout(in seconds) how long.eugo.hpc.cancelstops a call that has not finished. Getting its result afterwards raises an error.
See Ray's Objects, ray.wait and ray.cancel.
Disconnecting
eugo.hpc.shutdown() ends this process's connection to a cluster or, when there is no connection, stops the local Ray the process started. A local Ray and a connection are separate: if your process has both, one eugo.hpc.shutdown() ends only the connection, and the local Ray needs a second eugo.hpc.shutdown() of its own. eugo.hpc.is_initialized() tells you whether either is still running (see the first example on this page).
In EugoIDE, the Disconnect button on a cluster's card ends the connection Connect made. It does not stop a local Ray that the notebook also started; to stop that one, run eugo.hpc.shutdown() in a cell. To connect again, click Connect.
Ray's libraries
Ray Data is in eugo.hpc as eugo.hpc.data, the same object as Ray's ray.data. It is a name inside eugo.hpc, not a module of its own: write import eugo.hpc and use eugo.hpc.data, because import eugo.hpc.data fails with ModuleNotFoundError. See Ray Data.
Note: Ray Data does not work in the current Eugo image. Running a dataset, for example
eugo.hpc.data.range(8).take(3), fails withTypeError: cannot pass None into a C function argument that is declared 'not None'. Once it works, run Ray Data calls over a Connect session inside a function decorated with@eugo.hpc.distribute, not directly in the notebook.
Ray Train, Ray Tune and Ray Serve are not in eugo.hpc. Import them from Ray, which is installed in the same image:
from ray import train, tune, serve
eugo.hpc and Ray, name by name
eugo.hpc.distribute is Ray's remote under a second name, eugo.hpc.init takes the same arguments as ray.init, and eugo.hpc.use_auth_token_file is Eugo's own. Every other name is the Ray object of the same name, so Ray's documentation for it applies as written.
In eugo.hpc | In Ray | What it does |
|---|---|---|
eugo.hpc.init | ray.init | Connects to a cluster, or starts a local Ray. Takes ray.init's arguments. |
eugo.hpc.distribute | ray.remote | Decorator: makes a function a task, or a class an actor. |
eugo.hpc.remote | ray.remote | The same decorator under Ray's own name. |
eugo.hpc.get | ray.get | Waits for one or more results and returns them. |
eugo.hpc.put | ray.put | Stores a value in the object store and returns a reference. |
eugo.hpc.wait | ray.wait | Returns the references that are ready, and the rest. |
eugo.hpc.cancel | ray.cancel | Stops a call that has not finished. |
eugo.hpc.kill | ray.kill | Stops an actor. |
eugo.hpc.get_actor | ray.get_actor | Finds a named actor. |
eugo.hpc.method | ray.method | Sets options for one actor method. |
eugo.hpc.shutdown | ray.shutdown | Ends the connection to a cluster or, with no connection, stops the local Ray. A process with both needs one call for each. |
eugo.hpc.is_initialized | ray.is_initialized | Whether this process is connected or running a local Ray. |
eugo.hpc.cluster_resources | ray.cluster_resources | The CPUs, GPUs and memory the cluster has in total. |
eugo.hpc.available_resources | ray.available_resources | The part of those that is free right now. |
eugo.hpc.data | ray.data | Ray Data. See the note in Ray's libraries. |
eugo.hpc.use_auth_token_file | none: Eugo's own | Reads the cluster's token from a file. See Authenticating with a token file. |
Most other names Ray exports are in eugo.hpc too, as the same objects (for example eugo.hpc.nodes, eugo.hpc.get_runtime_context and eugo.hpc.timeline).
Related
- EugoHPC Manager: your clusters, the Connect button and connecting by hand.
- Installing Python packages: packages, code and environment variables on the cluster.
- Ray's own guides to tasks, actors and objects.