Skip to main content

EugoHPC Manager

The EugoHPC Manager is where you manage the Ray clusters in your Eugo workspaces. It shows the shape of your compute and controls cluster-level behaviour, most importantly whether a cluster's head node also runs your tasks. This page is the companion to Installing Python packages: that page covers what runs on your cluster (packages and runtime environments), and this one covers the clusters themselves.

Note: The EugoHPC Manager's screens described here may differ in detail from what you see. The Ray behaviour on this page is Ray's own, and the outputs quoted were captured in an Eugo image.

How your compute is organized​

Eugo's topology has three levels:

organization
└── workspace(s)
└── one-or-more Ray cluster(s) per workspace
└── each cluster has its own fleet running the Eugo image

An organization contains one or more workspaces, and each workspace owns one or more Ray clusters. Every cluster runs its own fleet of nodes, all started from the same Eugo image, so every node, head and workers alike, already carries Eugo's preinstalled Python packages. To see them in your own image, see What is already installed.

The EugoHPC Manager is a panel inside EugoIDE, the same session you develop in. It shows your clusters as a list of cluster cards, each with the cluster's status and its actions (such as connect, edit and terminate; once connected, open the Ray dashboard), plus a Cluster Launcher dialog for creating a new cluster.

Two environments, one image​

A cluster is not a single machine. There are two places your Python code lives, and you manage them separately:

EnvironmentWhat it isManaged with
Your interactive sessionThe Python in your EugoIDE container, where you develop. It is separate from your cluster's machines.pip and venv in your session (see Installing a package for your own session)
Your Ray cluster's workersThe Python on every node of your cluster.A Ray runtime environment (see Packages on the cluster)

The EugoHPC Manager works at the level of the cluster, the fleet of nodes that runs your work at scale, rather than the packages inside it.

Cluster lifecycle​

You drive a cluster's whole lifecycle from the EugoHPC Manager:

  • Create: the Cluster Launcher dialog. You choose the head node's type and size, one or more worker pools, optional shared storage (EFS), and an idle timeout. Worker pools come in tiers (CPU, GPU, and capacity blocks for reserved capacity), and you add only the tiers you need. Launching starts a head node plus the worker pools, all running the Eugo image.
  • Scale: each worker pool has a minimum and a maximum worker count. Ray's autoscaler grows the pool toward the maximum under load and shrinks it back toward the minimum when work drains; a worker that sits idle past the cluster's idle timeout is released. You don't manage individual nodes by hand.
  • Edit a running cluster: you can change each pool's minimum and maximum workers, the idle timeout, and a pool's instance type. The autoscaler picks up the new limits without restarting the head node, so your running jobs keep going.
  • Terminate: shuts the whole cluster down, head node, workers and shared storage. Terminating is final for that cluster.

Allow Tasks on Head Node​

This is the main control the EugoHPC Manager gives you over where your work runs. To use it well, start with how Ray behaves by default.

How Ray schedules work across nodes​

Your script runs in a driver process; the code you distribute runs in worker processes. Those worker processes run on every Ray node, not only the worker nodes but the head node too: by default a Ray head node schedules tasks and actors like any other node, unless it was started with --num-cpus=0. Two facts follow:

  1. By default, the head node takes part in scheduling. It advertises CPUs and runs tasks and actors like a worker node, so your distributed work can land on it.
  2. --num-cpus=0 is the off switch. A head node started with zero advertised CPUs runs no tasks or actors; it only coordinates the cluster, and the worker nodes run your work.

The head node is not only a compute node. It also hosts the cluster's coordination services: Ray's Global Control Store (GCS) and raylet, the Ray Client server your session connects through, and the dashboard. Those services keep the whole cluster running, and they compete for the head node's CPU and memory with any tasks scheduled there.

What the setting does​

The EugoHPC Manager turns this Ray behaviour into one setting, Allow Tasks on Head Node, so you don't set --num-cpus yourself:

SettingHead node behaviourRay equivalentWhen you want it
OnThe head node runs your tasks and actors in addition to the cluster's coordination services.Head started with its CPUs advertised (Ray's default).You want the most compute and accept that tasks share the head node with the GCS, the Ray Client server and the dashboard.
OffThe head node only coordinates the cluster; only worker nodes run your tasks and actors.Head started with --num-cpus=0.You want the head node's resources kept for running the cluster, with no task competing with its services.

The trade-off:

  • On gives you more compute, because the head node's CPUs join the pool, at the cost of contention: a heavy task on the head node competes with the services that coordinate the cluster, which can make the whole cluster less responsive or less stable.
  • Off gives you a dedicated, stable coordinator, at the cost of the head node's compute: your tasks run only on worker nodes.

Where you set it​

Allow Tasks on Head Node is a checkbox in the head-node section of the Cluster Launcher. Its help text in the product reads:

When enabled, the Head Node can also run workloads in addition to management duties. Disabling makes the Head Node dedicated to management only and therefore makes the cluster more resilient.

Three practical facts:

  • It is on by default for a new cluster.
  • It is chosen when you create the cluster, and it is not among the settings you can change on a running cluster (those are each pool's minimum and maximum workers, the idle timeout and the instance type). To change it, launch a new cluster.
  • If you turn it off, you must define at least one worker pool. With no tasks allowed on the head node, a cluster without workers would have nowhere to run your work, so the Launcher requires a worker pool.

Connecting your session to a cluster​

The Connect button​

To connect your interactive session to a cluster, click the Connect button (a play icon) on that cluster in the EugoHPC Manager. That attaches your notebook session to the cluster through Ray Client, and from then on your eugo.hpc code runs on that cluster (as in Packages on the cluster). The button adds a cell to your notebook and runs it; that cell holds the cluster's address, in the form eugo.hpc.init(address="ray://<head>:10001"), so you never have to look the address up. Clusters come and go, each with its own address, so never hard-code a ray:// address in your code.

The Ray Client server your session connects through is one of the head node's coordination services, which is part of why Allow Tasks on Head Node matters: when it is on, your tasks share the head node with the server your session is talking to.

Once connected, a card also offers Open Dashboard, which opens that cluster's Ray dashboard, the main view of a running cluster's nodes, resources and jobs.

Connecting by hand​

The Connect button gives you a plain connection: it attaches your session, but it does not let you give that connection a runtime environment. When you need one for the whole job (your own env_vars, pip, working_dir or py_modules), let the button set the connection up first, then replace it with one you write yourself. eugo.hpc.init takes the cluster's address and a runtime_env in the same call, so everything Installing Python packages says about runtime environments works on a connection you write by hand (except worker_process_setup_hook; see point 3 below).

Step 1: click Connect and copy the address. Click Connect on the cluster, in the notebook you will work in. The cell it adds holds the address, in its eugo.hpc.init(address="ray://<head>:10001") line; copy the ray:// address from there. Connect also gives this notebook's kernel the cluster's token, which no screen shows you. Clusters come and go, so get the address each session and never commit it to your code.

Step 2: disconnect, in the same kernel. Click Disconnect on the cluster's card, or run eugo.hpc.shutdown(). You need to end the button's connection before you start your own: while it is still connected, a second eugo.hpc.init(address=...) fails. Do not restart the kernel or switch notebooks: the token Connect set up lives in this kernel, and it stays there after you disconnect. In a restarted kernel or another notebook, click Connect again.

About the token. Every Eugo cluster runs with Ray's token authentication on, and that applies to a connection you write yourself just as it does to the Connect button, which supplies the token for you. Ray looks for the token in three places: the RAY_AUTH_TOKEN environment variable, the file named by RAY_AUTH_TOKEN_PATH, or ~/.ray/auth_token. Connecting to a host:6379 address with a wrong token logs InvalidAuthToken: Authentication token is missing or incorrect and then fails with ConnectionError: Failed to connect to Ray cluster at <address>; to a ray:// address, a wrong token fails with ConnectionError: ray client connection timeout. Connecting with no token available fails at once with an AuthenticationError that names those same three places.

Step 3: write the connection. In the same kernel, all four settings go in one runtime_env:

import eugo.hpc

eugo.hpc.init(
address="<the ray:// address you copied in Step 1>",
runtime_env={
"env_vars": {"MY_FLAG": "1", "OMP_NUM_THREADS": "8"},
"pip": ["cowsay==6.1"],
"working_dir": "/home/eugo/my_project",
"py_modules": ["/home/eugo/my_project/mypkg"],
},
)

eugo.hpc.init is ray.init: address= and runtime_env= are Ray's own parameters, so the same connection written as ray.init(...) behaves identically, and that is how the outputs on this page were captured.

What each key does:

KeyWhat it does on the clusterMore
env_varsSets environment variables in every worker process.Environment variables and setup code
pipInstalls extra packages into a per-job venv created with --system-site-packages, so Eugo's packages stay visible. The package must be reachable from the cluster (your index, or a wheelhouse you point pip at).Installing packages on the workers with pip
uvThe same per-job venv as pip, installed by uv (faster for long lists); uv_pip_install_options in place of pip_install_options.Installing packages on the workers with uv
working_dirUploads a local directory and makes it the workers' working directory and import path. Pair it with excludes to keep large files out of the upload.Shipping your own code
py_modulesUploads specific local modules or packages onto the workers' import path.Shipping your own code
worker_process_setup_hookRuns setup code once per worker process, before any of your tasks. Takes a function or a string such as "my_pkg.my_module.setup".Environment variables and setup code

Four things to know before you rely on this:

  1. Pass runtime_env as a plain dict. RuntimeEnv is not among the names Ray exports in ray.__all__, so do not expect eugo.hpc.RuntimeEnv. Pass a dict, as above, or import the typed object explicitly with from ray.runtime_env import RuntimeEnv.
  2. The keys are Ray's, and a misspelled key fails silently. The keys Ray acts on are config, conda, container, env_vars, excludes, image_uri, java_jars, pip, py_executable, py_modules, uv, working_dir and worker_process_setup_hook (plus a few internal ones that start with an underscore). Anything else is accepted with no error and does nothing: passing {"env_var": {"TYPO_FLAG": "x"}, "env_vars": {"GOOD_FLAG": "y"}} gave a worker {'TYPO_FLAG': None, 'GOOD_FLAG': 'y'}. Check your spelling, because nothing else will. conda is on Ray's list but is not usable in the Eugo image; uv is.
  3. worker_process_setup_hook depends on the form of the address. Against a direct GCS address (host:6379) both forms work, a function and a "my_pkg.my_module.setup" string. Against a Ray Client address (ray://host:port) neither does: a function raises TypeError: Object of type function is not JSON serializable … when serializing dict item 'worker_process_setup_hook', and the string form connects but the hook silently does not run. The other keys shown above (env_vars, working_dir, py_modules) apply normally over ray://. So if your address starts with ray://, do that setup inside your tasks, or in the code py_modules ships, rather than in the hook.
  4. This runtime environment applies to the whole connection. To vary it for one task, use @eugo.hpc.distribute(runtime_env=...) instead.

Note: The cards do not track a connection you write by hand. The EugoHPC Manager tracks the connections it makes itself, so a session you attached with eugo.hpc.init(address=…) does not show as connected on that cluster's card. Instead, once the manager sees that the notebook is running Ray that a cell started (when you next open or switch to the notebook, or click Connect), every card offers Disconnect in Connect's place, with the tooltip "This notebook is already running Ray, started by a cell rather than through Connect. Disconnect it before connecting a cluster." Clicking that Disconnect ends your connection and brings Connect back. If you detach with eugo.hpc.shutdown() in a cell instead, the cards keep showing Disconnect until you next open or switch to the notebook. Keep your own note of which cluster such a session is attached to, and do not read a card's connection state as the whole picture.

Security​

You don't configure cluster security yourself: the EugoHPC Manager sets up each cluster, its token and its storage when it creates the cluster.

  • Token authentication. Eugo turns on Ray's built-in token authentication with a strong token for each cluster, so connecting to a cluster as a Ray client requires that cluster's token (see Connecting by hand for what a wrong or missing token does).
  • Treat a cluster's address and token as secrets, and terminate a cluster you are no longer using. Terminating removes its nodes and its shared storage.
  • Installing Python packages: what is preinstalled, pip and venv in your own session, and Ray runtime environments (pip, uv, working_dir, py_modules, env_vars) for the cluster's workers.
  • EugoIDE: the development environment the EugoHPC Manager lives in.