forked from SiVeGCS/dask_template
118 lines
4 KiB
Markdown
118 lines
4 KiB
Markdown
# Ray: How to launch a Ray Cluster on Hawk?
|
|
|
|
This guide shows you how to launch a Ray cluster on HLRS' Hawk system.
|
|
|
|
## Table of Contents
|
|
- [Ray: How to launch a Ray Cluster on Hawk?](#ray-how-to-launch-a-ray-cluster-on-hawk)
|
|
- [Table of Contents](#table-of-contents)
|
|
- [Prerequisites](#prerequisites)
|
|
- [Getting Started](#getting-started)
|
|
- [Launch a Ray Cluster in Interactive Mode](#launch-a-ray-cluster-in-interactive-mode)
|
|
- [Launch a Ray Cluster in Batch Mode](#launch-a-ray-cluster-in-batch-mode)
|
|
|
|
## Prerequisites
|
|
|
|
Before building the environment, make sure you have the following prerequisites:
|
|
- [Conda Installation](https://docs.conda.io/projects/conda/en/latest/user-guide/install/index.html): Ensure that Conda is installed on your local system.
|
|
- [Conda-Pack](https://conda.github.io/conda-pack/) installed in the base environment: Conda pack is used to package the Conda environment into a single tarball. This is used to transfer the environment to the target system.
|
|
- `linux-64` platform for installing the Conda packages because Conda/pip downloads and installs precompiled binaries suitable to the architecture and OS of the local environment.
|
|
|
|
For more information, look at the documentation for [Conda on HLRS HPC systems](https://kb.hlrs.de/platforms/index.php/How_to_move_local_conda_environments_to_the_clusters)
|
|
|
|
## Getting Started
|
|
|
|
Only the main and r channels are available using the conda module on the clusters. To use custom packages, we need to move the local conda environment to Hawk.
|
|
|
|
1. Clone this repository to your local machine:
|
|
|
|
```bash
|
|
git clone <repository_url>
|
|
```
|
|
|
|
2. Go into the directory and create an environment using Conda and environment.yaml.
|
|
|
|
Note: Be sure to add the necessary packages in `deployment_scripts/environment.yaml`:
|
|
|
|
```bash
|
|
cd deployment_scripts
|
|
./create-env.sh <your-env>
|
|
```
|
|
|
|
3. Package the environment and transfer the archive to the target system:
|
|
|
|
```bash
|
|
(base) $ conda pack -n <your-env> -o ray_env.tar.gz # conda-pack must be installed in the base environment
|
|
```
|
|
|
|
A workspace is suitable to store the compressed Conda environment archive on Hawk. Proceed to the next step if you have already configured your workspace. Use the following command to create a workspace on the high-performance filesystem, which will expire in 10 days. For more information, such as how to enable reminder emails, refer to the [workspace mechanism](https://kb.hlrs.de/platforms/index.php/Workspace_mechanism) guide.
|
|
|
|
```bash
|
|
ws_allocate hpda_project 10
|
|
ws_find hpda_project # find the path to workspace, which is the destination directory in the next step
|
|
```
|
|
|
|
You can send your data to an existing workspace using:
|
|
|
|
```bash
|
|
scp ray_env.tar.gz <username>@hawk.hww.hlrs.de:<workspace_directory>
|
|
rm ray_env.tar.gz # We don't need the archive locally anymore.
|
|
```
|
|
|
|
4. Clone the repository on Hawk to use the deployment scripts and project structure:
|
|
|
|
```bash
|
|
cd <workspace_directory>
|
|
git clone <repository_url>
|
|
```
|
|
|
|
## Launch a Ray Cluster in Interactive Mode
|
|
|
|
Using a single node interactively provides opportunities for faster code debugging.
|
|
|
|
1. On the Hawk login node, start an interactive job using:
|
|
|
|
```bash
|
|
qsub -I -l select=1:node_type=rome -l walltime=01:00:00
|
|
```
|
|
|
|
2. Go into the directory with all code:
|
|
|
|
```bash
|
|
cd <project_directory>/deployment_scripts
|
|
```
|
|
|
|
3. Deploy the conda environment to the ram disk:
|
|
|
|
```bash
|
|
source deploy-env.sh
|
|
```
|
|
Note: Make sure all permissions are set using `chmod +x`.
|
|
|
|
4. Initialize the Ray cluster.
|
|
|
|
You can use a Python interpreter to start a Ray cluster:
|
|
|
|
```python
|
|
import ray
|
|
|
|
ray.init(dashboard_host='127.0.0.1')
|
|
```
|
|
|
|
1. Connect to the dashboard.
|
|
|
|
Warning: Always use `127.0.0.1` as the dashboard host to make the Ray cluster reachable by only you.
|
|
|
|
## Launch a Ray Cluster in Batch Mode
|
|
|
|
1. Add execution permissions to `start-ray-worker.sh`
|
|
|
|
```bash
|
|
cd deployment_scripts
|
|
chmod +x ray-start-worker.sh
|
|
```
|
|
|
|
2. Submit a job to launch the head and worker nodes.
|
|
|
|
You must modify the following variables in `submit-ray-job.sh`:
|
|
- Line 3 changes the cluster size. The default configuration launches a 3 node cluster.
|
|
- `$PROJECT_DIR`
|