# EugoHPC Manager

How your workspaces and Ray clusters are organized, what the Allow Tasks on Head Node setting does, and how to connect to a cluster by hand with your own runtime environment.

Source: https://docs.eugo.io/docs/guides/hpc-cluster-manager/

---

The EugoHPC Manager is where you manage the Ray clusters in your Eugo workspaces. It shows the shape of your compute and controls cluster-level behaviour, most importantly whether a cluster's head node also runs your tasks. This page is the companion to [Installing Python packages](./installing-python-packages.md): that page covers what runs on your cluster (packages and runtime environments), and this one covers the clusters themselves.

> **Note:** The EugoHPC Manager's screens described here may differ in detail from what you see. The Ray behaviour on this page is Ray's own, and the outputs quoted were captured in an Eugo image.

## How your compute is organized

Eugo's topology has three levels:

```text
organization
└── workspace(s)
    └── one-or-more Ray cluster(s) per workspace
        └── each cluster has its own fleet running the Eugo image
```

An organization contains one or more workspaces, and each workspace owns one or more Ray clusters. Every cluster runs its own fleet of nodes, all started from the same Eugo image, so every node, head and workers alike, already carries Eugo's preinstalled Python packages. To see them in your own image, see [What is already installed](./installing-python-packages.md#what-is-already-installed).

The EugoHPC Manager is a panel inside [EugoIDE](../platform/ide.md), the same session you develop in. It shows your clusters as a list of cluster cards, each with the cluster's status and its actions (such as connect, edit and terminate; once connected, open the Ray dashboard), plus a Cluster Launcher dialog for creating a new cluster.

### Two environments, one image

A cluster is not a single machine. There are two places your Python code lives, and you manage them separately:

| Environment | What it is | Managed with |
|---|---|---|
| Your interactive session | The Python in your EugoIDE container, where you develop. It is separate from your cluster's machines. | `pip` and `venv` in your session (see [Installing a package for your own session](./installing-python-packages.md#installing-a-package-for-your-own-session)) |
| Your Ray cluster's workers | The Python on every node of your cluster. | A Ray runtime environment (see [Packages on the cluster](./installing-python-packages.md#packages-on-the-cluster)) |

The EugoHPC Manager works at the level of the cluster, the fleet of nodes that runs your work at scale, rather than the packages inside it.

### Cluster lifecycle

You drive a cluster's whole lifecycle from the EugoHPC Manager:

- **Create**: the Cluster Launcher dialog. You choose the head node's type and size, one or more worker pools, optional shared storage (EFS), and an idle timeout. Worker pools come in tiers (CPU, GPU, and capacity blocks for reserved capacity), and you add only the tiers you need. Launching starts a head node plus the worker pools, all running the Eugo image.
- **Scale**: each worker pool has a minimum and a maximum worker count. Ray's autoscaler grows the pool toward the maximum under load and shrinks it back toward the minimum when work drains; a worker that sits idle past the cluster's idle timeout is released. You don't manage individual nodes by hand.
- **Edit a running cluster**: you can change each pool's minimum and maximum workers, the idle timeout, and a pool's instance type. The autoscaler picks up the new limits without restarting the head node, so your running jobs keep going.
- **Terminate**: shuts the whole cluster down, head node, workers and shared storage. Terminating is final for that cluster.

## Allow Tasks on Head Node

This is the main control the EugoHPC Manager gives you over where your work runs. To use it well, start with how Ray behaves by default.

### How Ray schedules work across nodes

Your script runs in a driver process; the code you distribute runs in worker processes. Those worker processes run on every Ray node, not only the worker nodes but the head node too: by default a Ray head node schedules tasks and actors like any other node, unless it was started with `--num-cpus=0`. Two facts follow:

1. By default, the head node takes part in scheduling. It advertises CPUs and runs tasks and actors like a worker node, so your distributed work can land on it.
2. `--num-cpus=0` is the off switch. A head node started with zero advertised CPUs runs no tasks or actors; it only coordinates the cluster, and the worker nodes run your work.

The head node is not only a compute node. It also hosts the cluster's coordination services: Ray's Global Control Store (GCS) and raylet, the Ray Client server your session connects through, and the dashboard. Those services keep the whole cluster running, and they compete for the head node's CPU and memory with any tasks scheduled there.

### What the setting does

The EugoHPC Manager turns this Ray behaviour into one setting, **Allow Tasks on Head Node**, so you don't set `--num-cpus` yourself:

| Setting | Head node behaviour | Ray equivalent | When you want it |
|---|---|---|---|
| **On** | The head node runs your tasks and actors in addition to the cluster's coordination services. | Head started with its CPUs advertised (Ray's default). | You want the most compute and accept that tasks share the head node with the GCS, the Ray Client server and the dashboard. |
| **Off** | The head node only coordinates the cluster; only worker nodes run your tasks and actors. | Head started with `--num-cpus=0`. | You want the head node's resources kept for running the cluster, with no task competing with its services. |

The trade-off:

- **On** gives you more compute, because the head node's CPUs join the pool, at the cost of contention: a heavy task on the head node competes with the services that coordinate the cluster, which can make the whole cluster less responsive or less stable.
- **Off** gives you a dedicated, stable coordinator, at the cost of the head node's compute: your tasks run only on worker nodes.

### Where you set it

**Allow Tasks on Head Node** is a checkbox in the head-node section of the Cluster Launcher. Its help text in the product reads:

> When enabled, the Head Node can also run workloads in addition to management duties. Disabling makes the Head Node dedicated to management only and therefore makes the cluster more resilient.

Three practical facts:

- It is **on** by default for a new cluster.
- It is chosen when you create the cluster, and it is not among the settings you can change on a running cluster (those are each pool's minimum and maximum workers, the idle timeout and the instance type). To change it, launch a new cluster.
- If you turn it **off**, you must define at least one worker pool. With no tasks allowed on the head node, a cluster without workers would have nowhere to run your work, so the Launcher requires a worker pool.

## Connecting your session to a cluster

### The Connect button

To connect your interactive session to a cluster, click the **Connect** button (a play icon) on that cluster in the EugoHPC Manager. That attaches your notebook session to the cluster through Ray Client, and from then on your `eugo.hpc` code runs on that cluster (as in [Packages on the cluster](./installing-python-packages.md#packages-on-the-cluster)). The button adds a cell to your notebook and runs it; that cell holds the cluster's address, in the form `eugo.hpc.init(address="ray://<head>:10001")`, so you never have to look the address up. Clusters come and go, each with its own address, so never hard-code a `ray://` address in your code.

The Ray Client server your session connects through is one of the head node's coordination services, which is part of why **Allow Tasks on Head Node** matters: when it is on, your tasks share the head node with the server your session is talking to.

Once connected, a card also offers **Open Dashboard**, which opens that cluster's Ray dashboard, the main view of a running cluster's nodes, resources and jobs.

### Connecting by hand

The Connect button gives you a plain connection: it attaches your session, but it does not let you give that connection a runtime environment. When you need one for the whole job (your own `env_vars`, `pip`, `working_dir` or `py_modules`), let the button set the connection up first, then replace it with one you write yourself. `eugo.hpc.init` takes the cluster's `address` and a `runtime_env` in the same call, so everything [Installing Python packages](./installing-python-packages.md#packages-on-the-cluster) says about runtime environments works on a connection you write by hand (except `worker_process_setup_hook`; see point 3 below).

**Step 1: click Connect and copy the address.** Click **Connect** on the cluster, in the notebook you will work in. The cell it adds holds the address, in its `eugo.hpc.init(address="ray://<head>:10001")` line; copy the `ray://` address from there. Connect also gives this notebook's kernel the cluster's token, which no screen shows you. Clusters come and go, so get the address each session and never commit it to your code.

**Step 2: disconnect, in the same kernel.** Click **Disconnect** on the cluster's card, or run `eugo.hpc.shutdown()`. You need to end the button's connection before you start your own: while it is still connected, a second `eugo.hpc.init(address=...)` fails. Do not restart the kernel or switch notebooks: the token Connect set up lives in this kernel, and it stays there after you disconnect. In a restarted kernel or another notebook, click Connect again.

**About the token.** Every Eugo cluster runs with Ray's token authentication on, and that applies to a connection you write yourself just as it does to the Connect button, which supplies the token for you. Ray looks for the token in three places: the `RAY_AUTH_TOKEN` environment variable, the file named by `RAY_AUTH_TOKEN_PATH`, or `~/.ray/auth_token`. Connecting to a `host:6379` address with a wrong token logs `InvalidAuthToken: Authentication token is missing or incorrect` and then fails with `ConnectionError: Failed to connect to Ray cluster at <address>`; to a `ray://` address, a wrong token fails with `ConnectionError: ray client connection timeout`. Connecting with no token available fails at once with an `AuthenticationError` that names those same three places.

**Step 3: write the connection.** In the same kernel, all four settings go in one `runtime_env`:

```python
import eugo.hpc

eugo.hpc.init(
    address="<the ray:// address you copied in Step 1>",
    runtime_env={
        "env_vars": {"MY_FLAG": "1", "OMP_NUM_THREADS": "8"},
        "pip": ["cowsay==6.1"],
        "working_dir": "/home/eugo/my_project",
        "py_modules": ["/home/eugo/my_project/mypkg"],
    },
)
```

`eugo.hpc.init` is `ray.init`: `address=` and `runtime_env=` are Ray's own parameters, so the same connection written as `ray.init(...)` behaves identically, and that is how the outputs on this page were captured.

What each key does:

| Key | What it does on the cluster | More |
|---|---|---|
| `env_vars` | Sets environment variables in every worker process. | [Environment variables and setup code](./installing-python-packages.md#environment-variables-and-setup-code) |
| `pip` | Installs extra packages into a per-job venv created with `--system-site-packages`, so Eugo's packages stay visible. The package must be reachable from the cluster (your index, or a wheelhouse you point pip at). | [Installing packages on the workers with pip](./installing-python-packages.md#installing-packages-on-the-workers-with-pip) |
| `uv` | The same per-job venv as `pip`, installed by uv (faster for long lists); `uv_pip_install_options` in place of `pip_install_options`. | [Installing packages on the workers with uv](./installing-python-packages.md#installing-packages-on-the-workers-with-uv) |
| `working_dir` | Uploads a local directory and makes it the workers' working directory and import path. Pair it with `excludes` to keep large files out of the upload. | [Shipping your own code](./installing-python-packages.md#shipping-your-own-code) |
| `py_modules` | Uploads specific local modules or packages onto the workers' import path. | [Shipping your own code](./installing-python-packages.md#shipping-your-own-code) |
| `worker_process_setup_hook` | Runs setup code once per worker process, before any of your tasks. Takes a function or a string such as `"my_pkg.my_module.setup"`. | [Environment variables and setup code](./installing-python-packages.md#environment-variables-and-setup-code) |

Four things to know before you rely on this:

1. **Pass `runtime_env` as a plain dict.** `RuntimeEnv` is not among the names Ray exports in `ray.__all__`, so do not expect `eugo.hpc.RuntimeEnv`. Pass a dict, as above, or import the typed object explicitly with `from ray.runtime_env import RuntimeEnv`.
2. **The keys are Ray's, and a misspelled key fails silently.** The keys Ray acts on are `config`, `conda`, `container`, `env_vars`, `excludes`, `image_uri`, `java_jars`, `pip`, `py_executable`, `py_modules`, `uv`, `working_dir` and `worker_process_setup_hook` (plus a few internal ones that start with an underscore). Anything else is accepted with no error and does nothing: passing `{"env_var": {"TYPO_FLAG": "x"}, "env_vars": {"GOOD_FLAG": "y"}}` gave a worker `{'TYPO_FLAG': None, 'GOOD_FLAG': 'y'}`. Check your spelling, because nothing else will. `conda` is on Ray's list but is not usable in the Eugo image; `uv` is.
3. **`worker_process_setup_hook` depends on the form of the address.** Against a direct GCS address (`host:6379`) both forms work, a function and a `"my_pkg.my_module.setup"` string. Against a Ray Client address (`ray://host:port`) neither does: a function raises `TypeError: Object of type function is not JSON serializable … when serializing dict item 'worker_process_setup_hook'`, and the string form connects but the hook silently does not run. The other keys shown above (`env_vars`, `working_dir`, `py_modules`) apply normally over `ray://`. So if your address starts with `ray://`, do that setup inside your tasks, or in the code `py_modules` ships, rather than in the hook.
4. **This runtime environment applies to the whole connection.** To vary it for one task, use `@eugo.hpc.distribute(runtime_env=...)` instead.

> **Note:** The cards do not track a connection you write by hand. The EugoHPC Manager tracks the connections it makes itself, so a session you attached with `eugo.hpc.init(address=…)` does not show as connected on that cluster's card. Instead, once the manager sees that the notebook is running Ray that a cell started (when you next open or switch to the notebook, or click Connect), every card offers **Disconnect** in Connect's place, with the tooltip "This notebook is already running Ray, started by a cell rather than through Connect. Disconnect it before connecting a cluster." Clicking that Disconnect ends your connection and brings Connect back. If you detach with `eugo.hpc.shutdown()` in a cell instead, the cards keep showing Disconnect until you next open or switch to the notebook. Keep your own note of which cluster such a session is attached to, and do not read a card's connection state as the whole picture.

## Security

You don't configure cluster security yourself: the EugoHPC Manager sets up each cluster, its token and its storage when it creates the cluster.

- **Token authentication.** Eugo turns on Ray's built-in token authentication with a strong token for each cluster, so connecting to a cluster as a Ray client requires that cluster's token (see [Connecting by hand](#connecting-by-hand) for what a wrong or missing token does).
- **Treat a cluster's address and token as secrets**, and terminate a cluster you are no longer using. Terminating removes its nodes and its shared storage.

## Related

- [Installing Python packages](./installing-python-packages.md): what is preinstalled, `pip` and `venv` in your own session, and Ray runtime environments (`pip`, `uv`, `working_dir`, `py_modules`, `env_vars`) for the cluster's workers.
- [EugoIDE](../platform/ide.md): the development environment the EugoHPC Manager lives in.
