Skip to main content

The eugo.hpc API

eugo.hpc is the Python module you use to run work on your Eugo Ray cluster. It is Ray's Python API under Eugo's name: the same functions and the same .remote(...) call style, plus one function of Eugo's own, for the cluster's token. This page shows each part with an example and points to Ray's documentation for the details.

Two other pages cover the rest: EugoHPC Manager explains your clusters and how to connect a notebook to one, and Installing Python packages explains how to get your packages and code onto the cluster.

Note: Each example on this page that shows output was run as its own Python script in an Eugo image, with no cluster connected, so it ran on a local Ray started inside that process. The output shown is what that run printed. Ids, object names and timings will differ on your machine.

Importing​

import eugo.hpc

Everything lives under eugo.hpc. The top-level eugo package is only a namespace and gives you nothing to call.

Connecting to a cluster​

With the Connect button​

In EugoIDE, click Connect on a cluster's card. Connect gives the notebook's kernel the cluster's token, without showing it, and adds a two-line cell that calls eugo.hpc.init(address="ray://<head>:10001"), so the notebook's work runs on that cluster. See The Connect button. This is the usual way to connect, and you do not need any of the calls below to use it.

Note: The examples on this page that call eugo.hpc.init() or eugo.hpc.shutdown() are for a notebook that has not used Connect, or one whose kernel you have restarted. In a notebook connected with Connect, eugo.hpc.shutdown() ends your connection to the cluster, and the code you run after it runs in your own session, not on your cluster. To connect again, click Connect on the cluster's card (see The Connect button).

Without a cluster: a local Ray​

eugo.hpc.init() with no address starts a private Ray inside your own process. It uses only the machine your code runs on, not your cluster.

import eugo.hpc

eugo.hpc.init() # no address: starts a local Ray in this process
print(eugo.hpc.is_initialized())

eugo.hpc.shutdown()
print(eugo.hpc.is_initialized())
True
False

You do not have to call init() first. If nothing is connected, the first call that runs work (a .remote(...), eugo.hpc.put or eugo.hpc.get) starts the local Ray itself:

import eugo.hpc

@eugo.hpc.distribute
def square(x):
return x * x

print(eugo.hpc.is_initialized())
print(eugo.hpc.get(square.remote(3))) # no init(): this call starts a local Ray
print(eugo.hpc.is_initialized())
False
9
True

Note: In EugoIDE this matters: if you forget to click Connect, your work still runs, but on a private Ray in your own session, not on your cluster. Click Connect before you run work you mean for the cluster.

Connecting with an address​

eugo.hpc.init takes the same arguments as Ray's ray.init, such as address and runtime_env. Connect writes the address form for you. To write the connection yourself, for example to give the whole job a runtime environment, follow Connecting by hand.

import eugo.hpc

eugo.hpc.use_auth_token_file("/path/to/cluster_token") # before init
eugo.hpc.init(address="ray://<head>:10001")
print(eugo.hpc.is_initialized())

eugo.hpc.shutdown()
print(eugo.hpc.is_initialized())

Replace <head> with your cluster's head address, as in the cell Connect adds, and the file path with the file that holds your cluster's token. The token file is explained in Authenticating with a token file.

Calling init twice​

While a local Ray you started is running in your process, a second eugo.hpc.init() fails. Run eugo.hpc.shutdown() first, then call eugo.hpc.init() again:

import eugo.hpc

eugo.hpc.init()
try:
eugo.hpc.init()
except RuntimeError as error:
print("RuntimeError:", error)

eugo.hpc.shutdown()
eugo.hpc.init() # works again after shutdown()
print(eugo.hpc.is_initialized())
RuntimeError: Maybe you called ray.init twice by accident? This error can be suppressed by passing in 'ignore_reinit_error=True' or by calling 'ray.shutdown()' prior to 'ray.init()'.
True

Authenticating with a token file​

Every Eugo cluster runs with Ray's token authentication on, so a connection needs the cluster's token. eugo.hpc.use_auth_token_file(path) is the one function in eugo.hpc that is Eugo's own: it tells this process to read the token from the file at path (a string or a pathlib.Path). Connect calls it for you; call it yourself only when you have the cluster's token in a file of your own.

Three rules:

  1. Call it before eugo.hpc.init(...). Ray reads the token once per process and keeps it. use_auth_token_file clears that saved token, so the next connection reads the file, even if the process connected to another cluster before.

  2. It refuses while the process is connected. Run eugo.hpc.shutdown() first:

    import eugo.hpc

    eugo.hpc.init()
    try:
    eugo.hpc.use_auth_token_file("/tmp/cluster_token")
    except RuntimeError as error:
    print("RuntimeError:", error)
    RuntimeError: eugo.hpc.use_auth_token_file: this process is connected to Ray; call eugo.hpc.shutdown() first
  3. A RAY_AUTH_TOKEN environment variable wins over the file. Ray looks at that variable first, so if it is set in your environment, the file is not used.

Running functions on the cluster​

Decorate a function with @eugo.hpc.distribute, call .remote(...) on it to start it on the cluster, and pass what that returns to eugo.hpc.get to wait for the result. .remote(...) returns at once with a reference to the future result (an ObjectRef), so you can start many calls before you wait for any of them.

import eugo.hpc

@eugo.hpc.distribute
def square(x):
return x * x

future = square.remote(4)
print(type(future).__name__)
print(eugo.hpc.get(future))
print(eugo.hpc.get([square.remote(i) for i in range(5)]))

@eugo.hpc.distribute(num_cpus=2)
def double(x):
return x * 2

print(eugo.hpc.get(double.remote(10)))
print(eugo.hpc.get(double.options(num_cpus=1).remote(10)))

try:
square(4)
except TypeError as error:
print("TypeError:", error)
ObjectRef
16
[0, 1, 4, 9, 16]
20
20
TypeError: Remote functions cannot be called directly. Instead of running '__main__.square()', try '__main__.square.remote()'.

What the example shows:

  1. eugo.hpc.get takes one reference or a list. With a list it waits for all of them and returns the results in the same order.
  2. The decorator takes Ray's options, such as num_cpus and num_gpus, to say what each call needs. .options(...) changes them for one call. The options are Ray's: see ray.remote and Tasks.
  3. A decorated function cannot be called directly. Always call it with .remote(...).

Actors​

On a class, @eugo.hpc.distribute makes an actor: an object that lives on a cluster node and keeps its state between calls. Counter.remote() creates it, and each method call is .remote(...) too.

import eugo.hpc

@eugo.hpc.distribute
class Counter:
def __init__(self):
self.n = 0

def inc(self):
self.n += 1
return self.n

counter = Counter.remote()
print(eugo.hpc.get(counter.inc.remote()))
print(eugo.hpc.get([counter.inc.remote() for _ in range(3)]))

shared = Counter.options(name="shared-counter").remote()
same = eugo.hpc.get_actor("shared-counter")
eugo.hpc.get(shared.inc.remote())
print(eugo.hpc.get(same.inc.remote()))

eugo.hpc.kill(counter) # stops the actor

The first call prints 1 and the next three [2, 3, 4], because the actor keeps its count between calls. Giving an actor a name lets other code in the same job find it with eugo.hpc.get_actor, so the call through same reaches the same actor. eugo.hpc.kill stops an actor, interrupting any call it is running. See Ray's Actors and Named actors.

Objects: put, wait and cancel​

import time
import eugo.hpc

numbers = eugo.hpc.put(list(range(1_000))) # stored once in the object store

@eugo.hpc.distribute
def total(values):
return sum(values)

print(eugo.hpc.get(total.remote(numbers))) # the task receives the list itself

@eugo.hpc.distribute
def sleep_for(seconds):
time.sleep(seconds)
return seconds

futures = [sleep_for.remote(s) for s in (5, 0, 10)]
ready, not_ready = eugo.hpc.wait(futures, num_returns=1, timeout=30)
print(eugo.hpc.get(ready), len(not_ready))

slow = sleep_for.remote(60)
eugo.hpc.cancel(slow)
try:
eugo.hpc.get(slow)
except Exception as error:
print(type(error).__name__)
499500
[0] 2
RayTaskError(TaskCancelledError)
  1. eugo.hpc.put stores a value in the cluster's object store once and gives you a reference. Pass the reference to many calls instead of the value, so a large value is not sent again with every call.
  2. eugo.hpc.wait returns as soon as some of the results are ready: a list of the ready references and a list of the rest. num_returns says how many to wait for, and timeout (in seconds) how long.
  3. eugo.hpc.cancel stops a call that has not finished. Getting its result afterwards raises an error.

See Ray's Objects, ray.wait and ray.cancel.

Disconnecting​

eugo.hpc.shutdown() ends this process's connection to a cluster or, when there is no connection, stops the local Ray the process started. A local Ray and a connection are separate: if your process has both, one eugo.hpc.shutdown() ends only the connection, and the local Ray needs a second eugo.hpc.shutdown() of its own. eugo.hpc.is_initialized() tells you whether either is still running (see the first example on this page).

In EugoIDE, the Disconnect button on a cluster's card ends the connection Connect made. It does not stop a local Ray that the notebook also started; to stop that one, run eugo.hpc.shutdown() in a cell. To connect again, click Connect.

Ray's libraries​

Ray Data is in eugo.hpc as eugo.hpc.data, the same object as Ray's ray.data. It is a name inside eugo.hpc, not a module of its own: write import eugo.hpc and use eugo.hpc.data, because import eugo.hpc.data fails with ModuleNotFoundError. See Ray Data.

Note: Ray Data does not work in the current Eugo image. Running a dataset, for example eugo.hpc.data.range(8).take(3), fails with TypeError: cannot pass None into a C function argument that is declared 'not None'. Once it works, run Ray Data calls over a Connect session inside a function decorated with @eugo.hpc.distribute, not directly in the notebook.

Ray Train, Ray Tune and Ray Serve are not in eugo.hpc. Import them from Ray, which is installed in the same image:

from ray import train, tune, serve

eugo.hpc and Ray, name by name​

eugo.hpc.distribute is Ray's remote under a second name, eugo.hpc.init takes the same arguments as ray.init, and eugo.hpc.use_auth_token_file is Eugo's own. Every other name is the Ray object of the same name, so Ray's documentation for it applies as written.

In eugo.hpcIn RayWhat it does
eugo.hpc.initray.initConnects to a cluster, or starts a local Ray. Takes ray.init's arguments.
eugo.hpc.distributeray.remoteDecorator: makes a function a task, or a class an actor.
eugo.hpc.remoteray.remoteThe same decorator under Ray's own name.
eugo.hpc.getray.getWaits for one or more results and returns them.
eugo.hpc.putray.putStores a value in the object store and returns a reference.
eugo.hpc.waitray.waitReturns the references that are ready, and the rest.
eugo.hpc.cancelray.cancelStops a call that has not finished.
eugo.hpc.killray.killStops an actor.
eugo.hpc.get_actorray.get_actorFinds a named actor.
eugo.hpc.methodray.methodSets options for one actor method.
eugo.hpc.shutdownray.shutdownEnds the connection to a cluster or, with no connection, stops the local Ray. A process with both needs one call for each.
eugo.hpc.is_initializedray.is_initializedWhether this process is connected or running a local Ray.
eugo.hpc.cluster_resourcesray.cluster_resourcesThe CPUs, GPUs and memory the cluster has in total.
eugo.hpc.available_resourcesray.available_resourcesThe part of those that is free right now.
eugo.hpc.dataray.dataRay Data. See the note in Ray's libraries.
eugo.hpc.use_auth_token_filenone: Eugo's ownReads the cluster's token from a file. See Authenticating with a token file.

Most other names Ray exports are in eugo.hpc too, as the same objects (for example eugo.hpc.nodes, eugo.hpc.get_runtime_context and eugo.hpc.timeline).

  1. EugoHPC Manager: your clusters, the Connect button and connecting by hand.
  2. Installing Python packages: packages, code and environment variables on the cluster.
  3. Ray's own guides to tasks, actors and objects.