Installing Python packages
Eugo comes with about 1,250 Python packages already installed, so most imports work with no install at all (see Libraries). When you need something else, you can add it yourself. This page covers:
- What is already installed, and how to check it.
- Installing a package for your own session, with
pip, avenvoruv. - Installing a package on your Ray cluster's workers, with a runtime environment.
Note: Every output on this page was captured in an Eugo image. Version strings and package counts (Python
3.14.6,numpy 2.6.0.dev0,ray 3.0.0.dev0, about 1,250 packages) change from one image to the next, so treat them as illustrative and run the same commands in your own session to see yours. Long resolver logs are trimmed and marked with….
What is already installed​
Eugo runs as a single container image that is also your runtime. Inside it there is one system Python interpreter, installed under /usr/local:
$ python3 --version
Python 3.14.6
$ python3 -c "import sys, site; print(sys.prefix); print(site.getsitepackages())"
/usr/local
['/usr/local/lib/python3.14/site-packages']
$ python3 -c "import sys; print('in venv:', sys.prefix != sys.base_prefix)"
in venv: False
There is no default virtual environment: python3, pip3, python and pip all use Eugo's own interpreter, and the preinstalled packages live in its system site-packages. That system location is writable, and there is no PEP 668 EXTERNALLY-MANAGED marker, so a plain pip install into it is not blocked. That also means an unguarded install can overwrite Eugo's packages in place (see When your requirement excludes Eugo's version). The only pip settings in the image are cosmetic:
$ pip3 config list
:env:.disable-pip-version-check='1'
:env:.root-user-action='ignore'
The preinstalled packages are ordinary distributions with .dist-info metadata, so pip's resolver sees them. To count them in your image:
$ ls -d /usr/local/lib/python3.14/site-packages/*.dist-info | wc -l
1255
$ python3 -c "import importlib.metadata as m; print(sum(1 for _ in m.distributions()))"
1255
That one fact drives everything below: when you pip install X, pip already knows about Eugo's copies of X and of its dependencies.
Two environments you manage separately​
Eugo gives you two places to put Python code, and installing into one does not install into the other:
| Environment | What it is | What it's for |
|---|---|---|
| Your interactive session | The Python in your EugoIDE container, where you run notebooks and terminals. It is separate from your cluster's machines. | Development, exploration and iteration. |
| Your Ray cluster's workers | The Python on every node of your Ray cluster. | Running your code at scale. |
A pip install you run in a terminal, or a %pip install in a notebook, changes only your interactive session. It does not reach the cluster's workers, not even when the package sits in your home folder: the workers run on separate machines and do not share it. To change what the workers see, you use a Ray runtime environment, with its pip or uv field (see Packages on the cluster). The two are separate on purpose: your session can carry half-finished experiments and extra debugging tools without forcing every cluster node to match, and you ship to the cluster deliberately. The EugoHPC Manager is where you create and manage those clusters.
What lasts between sessions​
EugoIDE runs in its own container. Only your home folder, /home/<you> (where the file browser opens), is kept from one session to the next. Everything else in the container, Python's site-packages folder included, goes back to the image when your session stops or restarts. So these last only until your session stops:
pip install Xand%pip install X, which install into/usr/local/lib/python3.14/site-packages.pip install --user X. In EugoIDE your$HOMEis/root, so the per-user site is under/root/.local(see The safe options at a glance): inside the container, not in your home folder.- A
venvcreated outside your home folder, such as the ones under/tmpon this page.
Restarting a notebook's kernel does not end your session, so your installs are still there after a kernel restart. The scratch_work folder in your home folder is temporary space too: it is emptied when your session stops.
To have a package in every session, keep its install line in your notebook's first cell, or ask for it to be added to the image.
Installing a package for your own session​
Using only your own packages​
Create a venv without --system-site-packages. That gives you a clean environment that sees none of Eugo's packages:
$ python3 -m venv /tmp/clean
$ /tmp/clean/bin/python -c "import numpy"
ModuleNotFoundError: No module named 'numpy'
$ /tmp/clean/bin/python -c "import ray"
ModuleNotFoundError: No module named 'ray'
$ /tmp/clean/bin/pip list
Package Version
------- -------
pip 26.1.2
A venv created with --system-site-packages inherits everything Eugo ships, and you add packages on top of it:
$ python3 -m venv --system-site-packages /tmp/inherit
$ /tmp/inherit/bin/python -c "import numpy, ray; print(numpy.__version__); print(ray.__version__)"
2.6.0.dev0+git20260704.c6dcb75
3.0.0.dev0
$ /tmp/inherit/bin/python -c "import importlib.metadata as m; print(sum(1 for _ in m.distributions()))"
1256
Rule of thumb: a plain venv holds only what you install; a venv --system-site-packages holds Eugo's packages plus what you install. (The inherited environment counts one distribution more than the system does, because the venv's own pip is counted too.)
Note: In the Eugo image,
--system-site-packagesis already the default, so you opt out rather than in:
- In an interactive
rootshell (containers run asrootby default),python3 -m venv <dir>passes--system-site-packagesfor you. To get the clean environment shown above in such a shell, runcommand python3 -m venv <dir>, which skips that default, or setinclude-system-site-packages = falsein an existing venv'spyvenv.cfg. Re-runningpython3 -m venvover an existing venv never changes this setting. Scripts and other non-interactive commands keep standard Python behaviour, so a plainpython3 -m venvin a script creates a clean environment. A container running as a different user keeps standard behaviour too.- The image sets
VIRTUALENV_SYSTEM_SITE_PACKAGES=true, so the third-partyvirtualenv <dir>tool inherits Eugo's packages everywhere, scripts included. Opt out for one run withVIRTUALENV_SYSTEM_SITE_PACKAGES=false virtualenv <dir>.
When a dependency is already installed​
pip reuses Eugo's copy. Because the preinstalled packages carry real metadata, pip reports them as already satisfied and does nothing, including for dependencies of what you ask for. (The pandas==3.0.3 pin below is the pandas version in the image these outputs came from. In your session, run pip3 show pandas first and put the version it prints in place of 3.0.3: pinning a version your image does not have replaces Eugo's build, as the next section shows.)
$ pip3 install numpy
Requirement already satisfied: numpy in /usr/local/lib/python3.14/site-packages (2.6.0.dev0)
$ pip3 install "pandas==3.0.3"
Requirement already satisfied: pandas==3.0.3 in /usr/local/lib/python3.14/site-packages (3.0.3)
Requirement already satisfied: numpy>=2.3.3 in /usr/local/lib/python3.14/site-packages (from pandas==3.0.3) (2.6.0.dev0)
Requirement already satisfied: python-dateutil>=2.8.2 in /usr/local/lib/python3.14/site-packages (from pandas==3.0.3) (2.9.0.post0)
Requirement already satisfied: six>=1.5 in /usr/local/lib/python3.14/site-packages (from python-dateutil>=2.8.2->pandas==3.0.3) (1.17.0)
So when you install a package whose dependencies Eugo already provides, pip pulls only the pieces that are genuinely missing and leaves Eugo's builds in place.
When your requirement excludes Eugo's version​
This is the case to understand well. Eugo ships numpy 2.6.0.dev0. Suppose a package you want needs numpy<2.5. pip ignores Eugo's copy, downloads the newest version that satisfies your constraint and installs it, and in the system location that means overwriting Eugo's build.
Check first with --dry-run, which changes nothing:
$ pip3 install --dry-run "numpy<2.5"
Collecting numpy<2.5
Downloading numpy-2.4.6-cp314-cp314-manylinux_2_27_aarch64.manylinux_2_28_aarch64.whl.metadata (6.6 kB)
Would install numpy-2.4.6
Note: The dry run is also where you find out that a constraint cannot be met. Pin far enough back and there is no wheel for the image's Python at all, so pip falls back to the source release and tries to build it.
pip3 install --dry-run "numpy<2"does exactly that: nocp314wheel exists for numpy 1.x, the source build runs, and it ends inerror: metadata-generation-failed. Nothing is installed and nothing is broken, but the error text comes from that package's own build, which is why it looks alarming. Building such old releases from source is not supported, so pick a release that publishes a wheel for the image's Python (cp314) instead; the dry run shows a…-cp314-…whldownload when one exists.
If you run the install against the system location, pip prints a list of dependency-conflict messages about the rest of the preinstalled set. It labels them ERROR:, but they are not fatal here: pip installs anyway, replaces Eugo's numpy, and exits 0:
$ pip3 install "numpy<2.5"
Collecting numpy<2.5
Downloading numpy-2.4.6-cp314-cp314-manylinux_2_27_aarch64.manylinux_2_28_aarch64.whl.metadata (6.6 kB)
Downloading numpy-2.4.6-cp314-cp314-manylinux_2_27_aarch64.manylinux_2_28_aarch64.whl (15.7 MB)
Installing collected packages: numpy
Attempting uninstall: numpy
Found existing installation: numpy 2.6.0.dev0
Uninstalling numpy-2.6.0.dev0:
Successfully uninstalled numpy-2.6.0.dev0
ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts.
segregation 2.5.5 requires numba>=0.60, which is not installed.
sktime 1.0.1 requires pandas<3.0.0,>=1.1, but you have pandas 3.0.3 which is incompatible.
awswrangler 3.17.0 requires pyarrow<25.0.0,>=8.0.0, but you have pyarrow 25.0.0a0 which is incompatible.
… (3 of the 27 conflict lines this image printed; the rest elided)
Successfully installed numpy-2.4.6
$ python3 -c "import numpy; print(numpy.__version__, numpy.__file__)"
2.4.6 /usr/local/lib/python3.14/site-packages/numpy/__init__.py
The Attempting uninstall and Uninstalling lines are the part to notice: pip did not put your version beside Eugo's, it removed Eugo's build first.
Note: This affects only your current session, but you cannot undo it in place. Eugo's
numpyis built from source for your hardware, and no PyPI wheel reproduces that build, so once it is overwritten you cannot simply reinstall it. The change lives only in your running session and does not alter the Eugo image: a fresh session (or a fresh Ray node) starts from the original image with Eugo'snumpyrestored. Even so, do not count on a restart to clean up. Keep the change inside avenvso your working session stays intact.
Recommendation: never let a conflicting install reach the system location. Use a venv so the override stays contained:
$ python3 -m venv --system-site-packages /tmp/v
$ /tmp/v/bin/pip install "numpy<2.5"
Successfully installed numpy-2.4.6
$ /tmp/v/bin/python -c "import numpy; print(numpy.__version__, numpy.__file__)"
2.4.6 /tmp/v/lib/python3.14/site-packages/numpy/__init__.py
# system prefix is untouched:
$ python3 -c "import numpy; print(numpy.__version__, numpy.__file__)"
2.6.0.dev0+git20260704.c6dcb75 /usr/local/lib/python3.14/site-packages/numpy/__init__.py
The venv's copy (2.4.6) takes precedence over Eugo's only inside that venv; the system build stays intact for everything else. (You still see the conflict ERROR: messages while pip resolves, because a --system-site-packages venv considers the inherited packages, but the install lands in the venv.)
Reinstalling a version Eugo already ships​
There are two outcomes, depending on whether the version differs:
- A different version behaves as in the previous section: pip installs the version you asked for, which takes precedence over Eugo's in a venv and replaces it in the system location.
- The same version, forced, still uninstalls Eugo's build and replaces it with the PyPI wheel, even though the version string matches. Eugo's copy is built from source for this hardware and the PyPI wheel is a generic build, so they are not interchangeable. (As above,
3.0.3is the version in the image these outputs came from; put in the versionpip3 show pandasprints in yours, or this example becomes the different-version case.)
$ pip3 install --force-reinstall --no-deps "pandas==3.0.3"
Downloading pandas-3.0.3-cp314-cp314-manylinux_2_24_aarch64.manylinux_2_28_aarch64.whl (10.5 MB)
Installing collected packages: pandas
Attempting uninstall: pandas
Found existing installation: pandas 3.0.3
Uninstalling pandas-3.0.3:
Successfully uninstalled pandas-3.0.3
Successfully installed pandas-3.0.3
The two flags do different jobs:
--force-reinstallis what makes anything happen at all. Without it, installingpandas==3.0.3when 3.0.3 is already installed prints "Requirement already satisfied" and changes nothing. (--upgradewould not do this either; it only moves to a strictly newer version.)--no-depslimits the change topandas. Without it,--force-reinstallapplies to every dependency too, and would also replace Eugo'snumpy,python-dateutilandsixwith generic PyPI wheels.
Only force a reinstall deliberately, and prefer a venv so it is reversible.
The safe options at a glance​
| You want | Do this | Touches Eugo's packages? |
|---|---|---|
| A clean environment without Eugo's packages | command python3 -m venv /path in an interactive shell (a plain python3 -m venv in a script is also clean) | No, it sees none of them |
| Eugo's packages plus your extras | python3 -m venv /path (interactive shells add --system-site-packages for you; pass the flag yourself in scripts) | No, your extras sit on top |
| A quick add that may conflict | pip install --user X (lasts until your session stops) | No, your user site takes precedence over the system |
| To change your whole session | pip install X (lasts until your session stops) | Yes, it can replace Eugo's builds |
The --user route keeps the system location intact and installs into your per-user site, $HOME/.local/lib/python3.14/site-packages, which Python searches first:
$ pip3 install --user "numpy<2.5"
Successfully installed numpy-2.4.6
$ python3 -c "import numpy; print(numpy.__version__, numpy.__file__)"
2.4.6 /root/.local/lib/python3.14/site-packages/numpy/__init__.py # user-site wins
$ ls -d /usr/local/lib/python3.14/site-packages/numpy-*.dist-info
/usr/local/lib/python3.14/site-packages/numpy-2.6.0.dev0.dist-info # system copy intact
Do not install a second Python with dnf​
There is exactly one Python in the image, and it is the one you get. Installing another with the system package manager (dnf install python3, or a dnf update that pulls in the python3 packages) does not give you a working second interpreter: the new python3 ends up loading Eugo's Python library instead of its own, and breaks. If you already did it, this restores the working state and leaves Eugo's Python untouched:
$ dnf remove python3 python3-libs
Use one of the options above instead: a venv for isolation, pip install --user for a quick add, or a plain pip install to change your whole session. All of them use the Python that is already there.
Installing from somewhere other than PyPI​
All of pip's usual sources work. Each example uses a venv to keep it tidy.
From an explicit index URL (your own index or a mirror):
$ python3 -m venv /tmp/idx
$ /tmp/idx/bin/pip install --index-url https://pypi.org/simple/ "wrapt==1.17.3"
Downloading wrapt-1.17.3-cp314-cp314-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl (87 kB)
Installing collected packages: wrapt
Successfully installed wrapt-1.17.3
For a private index, use --index-url https://<your-index>/simple/, or --extra-index-url to add one alongside PyPI. Keep credentials out of the URL: a URL on the command line is visible in the process list and in your shell history. Put them in a pip.conf with tight permissions (chmod 600), or supply them through keyring.
From a direct wheel URL, or from a wheel file on disk:
$ python3 -m venv /tmp/url
$ /tmp/url/bin/pip install "https://files.pythonhosted.org/packages/e4/0f/…/wrapt-1.17.3-cp314-cp314-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl"
Collecting wrapt==1.17.3
Downloading wrapt-1.17.3-cp314-cp314-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl (87 kB)
Installing collected packages: wrapt
Successfully installed wrapt-1.17.3
$ python3 -m venv /tmp/loc
$ /tmp/loc/bin/pip install /tmp/wh/wrapt-1.17.3-cp314-cp314-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl
Processing /tmp/wh/wrapt-1.17.3-cp314-cp314-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl
Installing collected packages: wrapt
Successfully installed wrapt-1.17.3
Installing from an arbitrary URL or file runs that package's code on import, and possibly while it builds. Only install from sources you trust.
Your own package while you develop it, as an editable install (-e): pip points at your source tree, and your edits take effect with no reinstall:
$ python3 -m venv /tmp/dev
$ /tmp/dev/bin/pip install -e /tmp/mypkg
Created wheel for mypkg: filename=mypkg-0.0.1-0.editable-py3-none-any.whl size=1110 …
Successfully built mypkg
Installing collected packages: mypkg
Successfully installed mypkg-0.0.1
$ /tmp/dev/bin/python -c "import mypkg, inspect, os; print(mypkg.hello()); print(os.path.dirname(inspect.getfile(mypkg)))"
hello from in-dev mypkg
/tmp/mypkg/src/mypkg # imported straight from your working tree
# edit the source, re-import — change is live, no reinstall:
$ sed -i 's/hello from in-dev mypkg/EDITED LIVE/' /tmp/mypkg/src/mypkg/__init__.py
$ /tmp/dev/bin/python -c "import mypkg; print(mypkg.hello())"
EDITED LIVE
An editable install changes only your interactive session. To run the same code on the cluster, ship it with working_dir or py_modules (see Shipping your own code).
How these sources interact with Eugo's copies:
- Index URLs (
--index-url,--extra-index-url) change only where a needed package is downloaded from. Eugo's copy is still kept if, and only if, its version satisfies your constraint. (--index-urlreplaces PyPI;--extra-index-urladds an index alongside it.) With--extra-index-url, pip pools candidates from all indexes and installs the highest matching version it finds anywhere, so a public package with the same name as your private one can be pulled in by accident. If you run a private index, point--index-urlonly at it rather than adding it as an extra. - Direct references are the exception: a wheel or source URL, a local file, a
git+…URL, or aname @ urlrequirement pins one exact file and does not honour "already satisfied". Against the system location it replaces Eugo's copy whether or not the version matches, which is why every example above uses avenv.
Using uv in your own session​
The image ships uv, Astral's package installer. uv venv and uv pip install work in a terminal or a notebook as usual; pass --python python3 so uv uses the image's interpreter rather than downloading one.
Packages on the cluster​
Everything above changes the Python in your own session. To change what runs on the nodes of your Ray cluster, you use a Ray runtime environment, passed either for one task (@eugo.hpc.distribute(runtime_env=...)) or for the whole job (eugo.hpc.init(runtime_env=...)).
Note:
eugo.hpcis Eugo's name for Ray's Python cluster API:eugo.hpc.initisray.init,@eugo.hpc.distributeis@ray.remote, andeugo.hpc.getisray.get. The same code written with theray.*names behaves identically, and that is how the outputs on this page were captured. Use whichever you prefer.
How you connect: click Connect on a cluster in the EugoHPC Manager to attach your notebook to it through Ray Client. eugo.hpc.init() with no address does not attach you: without Connect, it starts a small private Ray inside your notebook, and your code runs there, not on your cluster. That private Ray's workers run in your EugoIDE container, so they can import a package you installed there with pip or %pip, and a test like that does not show what your cluster's workers will see. After Connect, eugo.hpc.init(runtime_env=...) returns with no error, but your cluster's workers never get that runtime environment (Ray starts a second, private Ray in your notebook for it instead). To give the whole job a runtime environment, disconnect and connect by hand with the address and the runtime_env in one call (see Connecting by hand), or pass it per task with @eugo.hpc.distribute(runtime_env=...) (pip and env_vars work there; working_dir and py_modules work only for the whole job, see Shipping your own code).
Note: How to run the examples below. They ran against Ray inside a single container, with no cluster connected, so their output is reproducible; that is why they call
eugo.hpc.init(...)with no address. Run each one in a fresh kernel, or runeugo.hpc.shutdown()before the next one: with no cluster connected, a secondeugo.hpc.init(...)in the same session raises aRuntimeError. On your cluster, a whole-jobruntime_envfrom these examples goes into theeugo.hpc.init(address=..., runtime_env=...)call you write by hand.
"Worker" and "node": your script runs in a driver process, and the code you distribute runs in worker processes. Those worker processes run on every Ray node, the head node included unless the cluster reserves it for coordination (see Allow Tasks on Head Node). What you set in a runtime environment applies to the worker processes on all of them.
Workers already see the preinstalled packages​
Every Ray node runs the same image, so a remote task sees Eugo's preinstalled packages with no runtime environment at all:
import eugo.hpc, os
eugo.hpc.init()
@eugo.hpc.distribute
def what_worker_sees():
import numpy, ray as r, sys, os
return {"py": sys.version.split()[0], "numpy": numpy.__version__, "ray": r.__version__,
"numpy_file": numpy.__file__, "pid": os.getpid()}
print("DRIVER pid:", os.getpid(), "numpy:", __import__("numpy").__version__)
print("WORKER sees:", eugo.hpc.get(what_worker_sees.remote()))
DRIVER pid: 188 numpy: 2.6.0.dev0+git20260704.c6dcb75
WORKER sees: {'py': '3.14.6', 'numpy': '2.6.0.dev0+git20260704.c6dcb75',
'ray': '3.0.0.dev0', 'numpy_file': '/usr/local/lib/python3.14/site-packages/numpy/__init__.py', 'pid': 752}
The worker (pid 752) is a separate process from the driver (pid 188), and it finds numpy in Eugo's system build. You do not need to list preinstalled packages in a runtime environment.
How the cluster options compare​
| What you do | Where it takes effect | Effect on Eugo's packages |
|---|---|---|
pip install or a venv in your session | Your session only | Kept unless your version excludes Eugo's |
runtime_env with pip | The workers | Adds packages on top, in a per-job venv created with --system-site-packages; the package must be reachable |
runtime_env with uv | The workers | The same as pip, installed by the image's uv |
runtime_env with working_dir or py_modules | The workers | Your modules take precedence at import; nothing is uninstalled |
runtime_env with env_vars | The workers | None |
runtime_env with conda | The workers | Not available in the Eugo image |
Three things follow:
- You can override a package version on the cluster, but think twice for compiled packages. A version you pin with
piptakes precedence over Eugo's copy on the workers: pinningnumpy==2.4.6makes the worker import2.4.6from the job's venv rather than Eugo's2.6.0.dev0. That is fine for pure-Python packages. For a compiled package (numpy,pyarrowand the like) you would be replacing a build tuned for your hardware with a generic PyPI wheel. When you want a newer version of a compiled package, ask for it to be added to the image instead. - Code you ship takes precedence; it does not uninstall anything. Ray puts the directory you ship ahead of Eugo's
site-packageson each worker's import path, so a module you ship namedfoois found before an Eugofooof the same name. That is what you want for your own code, but do not hide a compiled Eugo package (numpy,pyarrow) behind a same-named pure-Python tree. - There is no environment on the workers that leaves out Eugo's packages. Locally a plain
venvgives you a clean environment. On the cluster, thepipanduvvenvs are created with--system-site-packages, so they always see Eugo's packages even while you override individual ones, andcondais not in the image.
Installing packages on the workers with pip​
Ray builds a fresh virtual environment for each job from a pip list. The venv is created with --system-site-packages, so it sees Eugo's packages, and your extra packages install on top:
import eugo.hpc
eugo.hpc.init(runtime_env={"pip": ["cowsay==6.1"]})
@eugo.hpc.distribute
def t():
import cowsay, sys
return {"cowsay": cowsay.__version__, "py": sys.version.split()[0]}
print(eugo.hpc.get(t.remote()))
{'cowsay': '6.1', 'py': '3.14.6'}
The package installs into the job's venv, and the worker imports it.
The one caveat is availability. The pip step still has to download what you ask for, so the package must be reachable from the cluster: through network access to PyPI or your own index, or from a folder of wheels (a wheelhouse) you point pip at:
eugo.hpc.init(runtime_env={"pip": {
"packages": ["cowsay==6.1"],
"pip_install_options": ["--no-index", "--find-links", "/path/to/wheelhouse"],
}})
Choosing between the options:
- A pure-Python package available on your index:
pip, as above. - A compiled package you want built for your hardware (
numpy,pyarrowand the like): ask for it to be added to the image, rather than pulling a generic PyPI wheel into the job's venv. Most of PyPI's common stack is already in the image. - Your own code:
working_dirorpy_modules(see Shipping your own code), with no venv to build. - Environment variables:
env_vars(see Environment variables and setup code).
Installing packages on the workers with uv​
The image ships uv. Ray's uv option works exactly like pip above, a per-job venv over Eugo's own interpreter and packages (--system-site-packages), but installs with uv, which is much faster for long package lists. The keys mirror pip's: packages, and uv_pip_install_options in place of pip_install_options. Ray finds the image's uv through python -m uv inside the job's venv. Offline, from a wheelhouse:
import eugo.hpc
eugo.hpc.init(runtime_env={"uv": {
"packages": ["cowsay==6.1"],
"uv_pip_install_options": ["--offline", "--no-index", "--find-links", "/path/to/wheelhouse"],
}})
@eugo.hpc.distribute
def t():
import cowsay, sys
return {"cowsay": cowsay.__version__, "py": sys.version.split()[0], "uv_venv": "/runtime_resources/uv/" in sys.prefix}
print(eugo.hpc.get(t.remote()))
{'cowsay': '6.1', 'py': '3.14.7', 'uv_venv': True}
As with pip, the packages you list must be reachable: PyPI, your index, or a wheelhouse.
Note:
condaruntime environments are not supported, becausecondais not in the image. Usepiporuvinstead.
Shipping your own code​
To put your own code on the workers, including a package you are still developing, use working_dir (upload a whole directory) or py_modules (upload specific modules). Both must be passed for the whole job, to eugo.hpc.init, not for a single task:
import eugo.hpc
eugo.hpc.init(runtime_env={"working_dir": "/tmp/usersrc"}) # uploads the dir
@eugo.hpc.distribute
def use_mod():
import mymod, os # importable because working_dir shipped it
return {"greet": mymod.greet(), "value": mymod.VALUE, "cwd_has_mod": os.path.isdir("mymod")}
print(eugo.hpc.get(use_mod.remote()))
{'greet': 'shipped via runtime_env', 'value': 42, 'cwd_has_mod': True}
py_modules works the same way but uploads specific modules instead of a whole directory. Point it at the module's directory, a .py file or a .whl:
import eugo.hpc
eugo.hpc.init(runtime_env={"py_modules": ["/tmp/usersrc/mymod"]}) # uploads the module
@eugo.hpc.distribute
def use_mod():
import mymod # importable because py_modules shipped it
return {"greet": mymod.greet(), "value": mymod.VALUE}
print("WORKER(job-level py_modules):", eugo.hpc.get(use_mod.remote()))
WORKER(job-level py_modules): {'greet': 'shipped via runtime_env', 'value': 42}
If you pass working_dir or py_modules for a single task instead, Ray raises:
ValueError: /tmp/usersrc is not a valid URI. Passing directories or modules to be dynamically
uploaded is only supported at the job level (i.e., passed to `ray.init`).
The message names ray.init because eugo.hpc.init is ray.init, and Ray knows only its own name.
This is the recommended way to take code you are developing in a notebook and run it across the cluster: develop locally with pip install -e, then ship the same tree to the workers with working_dir. Two more things to know:
- These options upload files from disk. Code that exists only in memory, such as a module you built with
types.ModuleType(...)or code typed into a notebook cell, cannot be shipped this way; write it to a.pyfile first. - If all you want is to run a function you wrote in a notebook cell, you may not need either option: decorate it with
@eugo.hpc.distribute, call it with.remote()and fetch the result witheugo.hpc.get, and Ray sends the function to the workers itself, as in Workers already see the preinstalled packages.working_dirandpy_modulesare for worker code that needs toimportyour module by name.
Environment variables and setup code​
To set environment variables in every worker process, use env_vars:
import eugo.hpc
eugo.hpc.init()
@eugo.hpc.distribute(runtime_env={"env_vars": {"MY_FLAG": "from-runtime-env"}})
def t():
import os
return {"MY_FLAG": os.environ.get("MY_FLAG"), "PATH_ok": "/usr/local/bin" in os.environ.get("PATH", "")}
print("WORKER(env_vars):", eugo.hpc.get(t.remote()))
WORKER(env_vars): {'MY_FLAG': 'from-runtime-env', 'PATH_ok': True}
To run a bit of setup code once in every worker process, right after it starts and before any of your tasks run (for example, to configure a package you installed), use worker_process_setup_hook. Pass it for the whole job, to eugo.hpc.init; it accepts a function, or a string naming an importable function:
import logging, warnings, eugo.hpc
def logging_setup_func():
logger = logging.getLogger("ray")
logger.setLevel(logging.DEBUG)
warnings.simplefilter("always")
eugo.hpc.init(runtime_env={"worker_process_setup_hook": logging_setup_func})
logging_setup_func() # the hook runs on workers; call it yourself for the driver
@eugo.hpc.distribute
def check_worker():
import logging
return {"ray_logger_level": logging.getLogger("ray").getEffectiveLevel()} # 10 == DEBUG
print("WORKER(setup hook ran):", eugo.hpc.get(check_worker.remote()))
WORKER(setup hook ran): {'ray_logger_level': 10}
The worker's ray logger comes back at DEBUG (10), which shows the hook ran on the worker before the task did. Good to know:
- It runs once per worker process, at startup, not once per task.
- It does not work over a Ray Client address (
ray://...), the kind Connect gives you: a function raises aTypeErrorand the string form never runs (see point 3 under Connecting by hand). On your cluster, do that setup inside your tasks or in the codepy_modulesships. - It is set for the whole job only. (If you submit jobs through Ray's Job API instead, pass the string form, for example
"my_pkg.my_module.setup", because a function object cannot be sent across that boundary.) - It is an experimental Ray API, so it may change in a future Ray release.
- It cannot install packages, because it runs inside a worker that has already started. Use
piporuvfor dependencies and the hook for setup.
Related​
- EugoHPC Manager: your clusters, the head-node setting, and connecting to a cluster by hand with your own runtime environment.
- Libraries: the packages that come preinstalled.