# The eugo.hpc API

The functions eugo.hpc gives you for running work on a Ray cluster: connecting, distributed functions and actors, objects, the token file, and how each name maps to Ray.

Source: https://docs.eugo.io/docs/guides/eugo-hpc-api/

---

`eugo.hpc` is the Python module you use to run work on your Eugo Ray cluster. It is Ray's Python API under Eugo's name: the same functions and the same `.remote(...)` call style, plus one function of Eugo's own, for the cluster's token. This page shows each part with an example and points to Ray's documentation for the details.

Two other pages cover the rest: [EugoHPC Manager](./hpc-cluster-manager.md) explains your clusters and how to connect a notebook to one, and [Installing Python packages](./installing-python-packages.md) explains how to get your packages and code onto the cluster.

> **Note:** Each example on this page that shows output was run as its own Python script in an Eugo image, with no cluster connected, so it ran on a local Ray started inside that process. The output shown is what that run printed. Ids, object names and timings will differ on your machine.

## Importing

```python
import eugo.hpc
```

Everything lives under `eugo.hpc`. The top-level `eugo` package is only a namespace and gives you nothing to call.

## Connecting to a cluster

### With the Connect button

In EugoIDE, click **Connect** on a cluster's card. Connect gives the notebook's kernel the cluster's token, without showing it, and adds a two-line cell that calls `eugo.hpc.init(address="ray://<head>:10001")`, so the notebook's work runs on that cluster. See [The Connect button](./hpc-cluster-manager.md#the-connect-button). This is the usual way to connect, and you do not need any of the calls below to use it.

> **Note:** The examples on this page that call `eugo.hpc.init()` or `eugo.hpc.shutdown()` are for a notebook that has not used **Connect**, or one whose kernel you have restarted. In a notebook connected with Connect, `eugo.hpc.shutdown()` ends your connection to the cluster, and the code you run after it runs in your own session, not on your cluster. To connect again, click **Connect** on the cluster's card (see [The Connect button](./hpc-cluster-manager.md#the-connect-button)).

### Without a cluster: a local Ray

`eugo.hpc.init()` with no address starts a private Ray inside your own process. It uses only the machine your code runs on, not your cluster.

```python
import eugo.hpc

eugo.hpc.init()                      # no address: starts a local Ray in this process
print(eugo.hpc.is_initialized())

eugo.hpc.shutdown()
print(eugo.hpc.is_initialized())
```

```text
True
False
```

You do not have to call `init()` first. If nothing is connected, the first call that runs work (a `.remote(...)`, `eugo.hpc.put` or `eugo.hpc.get`) starts the local Ray itself:

```python
import eugo.hpc

@eugo.hpc.distribute
def square(x):
    return x * x

print(eugo.hpc.is_initialized())
print(eugo.hpc.get(square.remote(3)))   # no init(): this call starts a local Ray
print(eugo.hpc.is_initialized())
```

```text
False
9
True
```

> **Note:** In EugoIDE this matters: if you forget to click Connect, your work still runs, but on a private Ray in your own session, not on your cluster. Click Connect before you run work you mean for the cluster.

### Connecting with an address

`eugo.hpc.init` takes the same arguments as Ray's [`ray.init`](https://docs.ray.io/en/latest/ray-core/api/doc/ray.init.html), such as `address` and `runtime_env`. Connect writes the address form for you. To write the connection yourself, for example to give the whole job a runtime environment, follow [Connecting by hand](./hpc-cluster-manager.md#connecting-by-hand).

```python
import eugo.hpc

eugo.hpc.use_auth_token_file("/path/to/cluster_token")   # before init
eugo.hpc.init(address="ray://<head>:10001")
print(eugo.hpc.is_initialized())

eugo.hpc.shutdown()
print(eugo.hpc.is_initialized())
```

Replace `<head>` with your cluster's head address, as in the cell Connect adds, and the file path with the file that holds your cluster's token. The token file is explained in [Authenticating with a token file](#authenticating-with-a-token-file).

### Calling init twice

While a local Ray you started is running in your process, a second `eugo.hpc.init()` fails. Run `eugo.hpc.shutdown()` first, then call `eugo.hpc.init()` again:

```python
import eugo.hpc

eugo.hpc.init()
try:
    eugo.hpc.init()
except RuntimeError as error:
    print("RuntimeError:", error)

eugo.hpc.shutdown()
eugo.hpc.init()                      # works again after shutdown()
print(eugo.hpc.is_initialized())
```

```text
RuntimeError: Maybe you called ray.init twice by accident? This error can be suppressed by passing in 'ignore_reinit_error=True' or by calling 'ray.shutdown()' prior to 'ray.init()'.
True
```

## Authenticating with a token file

Every Eugo cluster runs with Ray's [token authentication](https://docs.ray.io/en/latest/ray-security/token-auth.html) on, so a connection needs the cluster's token. `eugo.hpc.use_auth_token_file(path)` is the one function in `eugo.hpc` that is Eugo's own: it tells this process to read the token from the file at `path` (a string or a `pathlib.Path`). Connect calls it for you; call it yourself only when you have the cluster's token in a file of your own.

Three rules:

1. **Call it before `eugo.hpc.init(...)`.** Ray reads the token once per process and keeps it. `use_auth_token_file` clears that saved token, so the next connection reads the file, even if the process connected to another cluster before.
2. **It refuses while the process is connected.** Run `eugo.hpc.shutdown()` first:

   ```python
   import eugo.hpc

   eugo.hpc.init()
   try:
       eugo.hpc.use_auth_token_file("/tmp/cluster_token")
   except RuntimeError as error:
       print("RuntimeError:", error)
   ```

   ```text
   RuntimeError: eugo.hpc.use_auth_token_file: this process is connected to Ray; call eugo.hpc.shutdown() first
   ```

3. **A `RAY_AUTH_TOKEN` environment variable wins over the file.** Ray looks at that variable first, so if it is set in your environment, the file is not used.

## Running functions on the cluster

Decorate a function with `@eugo.hpc.distribute`, call `.remote(...)` on it to start it on the cluster, and pass what that returns to `eugo.hpc.get` to wait for the result. `.remote(...)` returns at once with a reference to the future result (an `ObjectRef`), so you can start many calls before you wait for any of them.

```python
import eugo.hpc

@eugo.hpc.distribute
def square(x):
    return x * x

future = square.remote(4)
print(type(future).__name__)
print(eugo.hpc.get(future))
print(eugo.hpc.get([square.remote(i) for i in range(5)]))

@eugo.hpc.distribute(num_cpus=2)
def double(x):
    return x * 2

print(eugo.hpc.get(double.remote(10)))
print(eugo.hpc.get(double.options(num_cpus=1).remote(10)))

try:
    square(4)
except TypeError as error:
    print("TypeError:", error)
```

```text
ObjectRef
16
[0, 1, 4, 9, 16]
20
20
TypeError: Remote functions cannot be called directly. Instead of running '__main__.square()', try '__main__.square.remote()'.
```

What the example shows:

1. **`eugo.hpc.get` takes one reference or a list.** With a list it waits for all of them and returns the results in the same order.
2. **The decorator takes Ray's options**, such as `num_cpus` and `num_gpus`, to say what each call needs. `.options(...)` changes them for one call. The options are Ray's: see [`ray.remote`](https://docs.ray.io/en/latest/ray-core/api/doc/ray.remote.html) and [Tasks](https://docs.ray.io/en/latest/ray-core/tasks.html).
3. **A decorated function cannot be called directly.** Always call it with `.remote(...)`.

## Actors

On a class, `@eugo.hpc.distribute` makes an actor: an object that lives on a cluster node and keeps its state between calls. `Counter.remote()` creates it, and each method call is `.remote(...)` too.

```python
import eugo.hpc

@eugo.hpc.distribute
class Counter:
    def __init__(self):
        self.n = 0

    def inc(self):
        self.n += 1
        return self.n

counter = Counter.remote()
print(eugo.hpc.get(counter.inc.remote()))
print(eugo.hpc.get([counter.inc.remote() for _ in range(3)]))

shared = Counter.options(name="shared-counter").remote()
same = eugo.hpc.get_actor("shared-counter")
eugo.hpc.get(shared.inc.remote())
print(eugo.hpc.get(same.inc.remote()))

eugo.hpc.kill(counter)                   # stops the actor
```

The first call prints `1` and the next three `[2, 3, 4]`, because the actor keeps its count between calls. Giving an actor a `name` lets other code in the same job find it with `eugo.hpc.get_actor`, so the call through `same` reaches the same actor. `eugo.hpc.kill` stops an actor, interrupting any call it is running. See Ray's [Actors](https://docs.ray.io/en/latest/ray-core/actors.html) and [Named actors](https://docs.ray.io/en/latest/ray-core/actors/named-actors.html).

## Objects: put, wait and cancel

```python
import time
import eugo.hpc

numbers = eugo.hpc.put(list(range(1_000)))   # stored once in the object store

@eugo.hpc.distribute
def total(values):
    return sum(values)

print(eugo.hpc.get(total.remote(numbers)))   # the task receives the list itself

@eugo.hpc.distribute
def sleep_for(seconds):
    time.sleep(seconds)
    return seconds

futures = [sleep_for.remote(s) for s in (5, 0, 10)]
ready, not_ready = eugo.hpc.wait(futures, num_returns=1, timeout=30)
print(eugo.hpc.get(ready), len(not_ready))

slow = sleep_for.remote(60)
eugo.hpc.cancel(slow)
try:
    eugo.hpc.get(slow)
except Exception as error:
    print(type(error).__name__)
```

```text
499500
[0] 2
RayTaskError(TaskCancelledError)
```

1. **`eugo.hpc.put`** stores a value in the cluster's object store once and gives you a reference. Pass the reference to many calls instead of the value, so a large value is not sent again with every call.
2. **`eugo.hpc.wait`** returns as soon as some of the results are ready: a list of the ready references and a list of the rest. `num_returns` says how many to wait for, and `timeout` (in seconds) how long.
3. **`eugo.hpc.cancel`** stops a call that has not finished. Getting its result afterwards raises an error.

See Ray's [Objects](https://docs.ray.io/en/latest/ray-core/objects.html), [`ray.wait`](https://docs.ray.io/en/latest/ray-core/api/doc/ray.wait.html) and [`ray.cancel`](https://docs.ray.io/en/latest/ray-core/api/doc/ray.cancel.html).

## Disconnecting

`eugo.hpc.shutdown()` ends this process's connection to a cluster or, when there is no connection, stops the local Ray the process started. A local Ray and a connection are separate: if your process has both, one `eugo.hpc.shutdown()` ends only the connection, and the local Ray needs a second `eugo.hpc.shutdown()` of its own. `eugo.hpc.is_initialized()` tells you whether either is still running (see the first example on this page).

In EugoIDE, the **Disconnect** button on a cluster's card ends the connection Connect made. It does not stop a local Ray that the notebook also started; to stop that one, run `eugo.hpc.shutdown()` in a cell. To connect again, click **Connect**.

## Ray's libraries

Ray Data is in `eugo.hpc` as `eugo.hpc.data`, the same object as Ray's `ray.data`. It is a name inside `eugo.hpc`, not a module of its own: write `import eugo.hpc` and use `eugo.hpc.data`, because `import eugo.hpc.data` fails with `ModuleNotFoundError`. See [Ray Data](https://docs.ray.io/en/latest/data/data.html).

> **Note:** Ray Data does not work in the current Eugo image. Running a dataset, for example `eugo.hpc.data.range(8).take(3)`, fails with `TypeError: cannot pass None into a C function argument that is declared 'not None'`. Once it works, run Ray Data calls over a Connect session inside a function decorated with `@eugo.hpc.distribute`, not directly in the notebook.

Ray Train, Ray Tune and Ray Serve are not in `eugo.hpc`. Import them from Ray, which is installed in the same image:

```python
from ray import train, tune, serve
```

## eugo.hpc and Ray, name by name

`eugo.hpc.distribute` is Ray's `remote` under a second name, `eugo.hpc.init` takes the same arguments as `ray.init`, and `eugo.hpc.use_auth_token_file` is Eugo's own. Every other name is the Ray object of the same name, so Ray's documentation for it applies as written.

| In `eugo.hpc` | In Ray | What it does |
|---|---|---|
| `eugo.hpc.init` | [`ray.init`](https://docs.ray.io/en/latest/ray-core/api/doc/ray.init.html) | Connects to a cluster, or starts a local Ray. Takes `ray.init`'s arguments. |
| `eugo.hpc.distribute` | [`ray.remote`](https://docs.ray.io/en/latest/ray-core/api/doc/ray.remote.html) | Decorator: makes a function a task, or a class an actor. |
| `eugo.hpc.remote` | [`ray.remote`](https://docs.ray.io/en/latest/ray-core/api/doc/ray.remote.html) | The same decorator under Ray's own name. |
| `eugo.hpc.get` | [`ray.get`](https://docs.ray.io/en/latest/ray-core/api/doc/ray.get.html) | Waits for one or more results and returns them. |
| `eugo.hpc.put` | [`ray.put`](https://docs.ray.io/en/latest/ray-core/api/doc/ray.put.html) | Stores a value in the object store and returns a reference. |
| `eugo.hpc.wait` | [`ray.wait`](https://docs.ray.io/en/latest/ray-core/api/doc/ray.wait.html) | Returns the references that are ready, and the rest. |
| `eugo.hpc.cancel` | [`ray.cancel`](https://docs.ray.io/en/latest/ray-core/api/doc/ray.cancel.html) | Stops a call that has not finished. |
| `eugo.hpc.kill` | [`ray.kill`](https://docs.ray.io/en/latest/ray-core/api/doc/ray.kill.html) | Stops an actor. |
| `eugo.hpc.get_actor` | [`ray.get_actor`](https://docs.ray.io/en/latest/ray-core/api/doc/ray.get_actor.html) | Finds a named actor. |
| `eugo.hpc.method` | [`ray.method`](https://docs.ray.io/en/latest/ray-core/api/doc/ray.method.html) | Sets options for one actor method. |
| `eugo.hpc.shutdown` | [`ray.shutdown`](https://docs.ray.io/en/latest/ray-core/api/doc/ray.shutdown.html) | Ends the connection to a cluster or, with no connection, stops the local Ray. A process with both needs one call for each. |
| `eugo.hpc.is_initialized` | [`ray.is_initialized`](https://docs.ray.io/en/latest/ray-core/api/doc/ray.is_initialized.html) | Whether this process is connected or running a local Ray. |
| `eugo.hpc.cluster_resources` | [`ray.cluster_resources`](https://docs.ray.io/en/latest/ray-core/api/doc/ray.cluster_resources.html) | The CPUs, GPUs and memory the cluster has in total. |
| `eugo.hpc.available_resources` | [`ray.available_resources`](https://docs.ray.io/en/latest/ray-core/api/doc/ray.available_resources.html) | The part of those that is free right now. |
| `eugo.hpc.data` | [`ray.data`](https://docs.ray.io/en/latest/data/data.html) | Ray Data. See the note in [Ray's libraries](#rays-libraries). |
| `eugo.hpc.use_auth_token_file` | none: Eugo's own | Reads the cluster's token from a file. See [Authenticating with a token file](#authenticating-with-a-token-file). |

Most other names Ray exports are in `eugo.hpc` too, as the same objects (for example `eugo.hpc.nodes`, `eugo.hpc.get_runtime_context` and `eugo.hpc.timeline`).

## Related

1. [EugoHPC Manager](./hpc-cluster-manager.md): your clusters, the Connect button and connecting by hand.
2. [Installing Python packages](./installing-python-packages.md): packages, code and environment variables on the cluster.
3. Ray's own guides to [tasks](https://docs.ray.io/en/latest/ray-core/tasks.html), [actors](https://docs.ray.io/en/latest/ray-core/actors.html) and [objects](https://docs.ray.io/en/latest/ray-core/objects.html).
