forked from SiVeGCS/dask_template
130 lines
No EOL
4.6 KiB
Markdown
130 lines
No EOL
4.6 KiB
Markdown
# Ray: How to launch a Ray Cluster on Hawk?
|
|
|
|
This guide shows you how to launch a Ray cluster on HLRS' Hawk system.
|
|
|
|
## Table of Contents
|
|
- [Ray: How to launch a Ray Cluster on Hawk?](#ray-how-to-launch-a-ray-cluster-on-hawk)
|
|
- [Table of Contents](#table-of-contents)
|
|
- [Getting Started](#getting-started)
|
|
- [Launch a local Ray Cluster in Interactive Mode](#launch-a-local-ray-cluster-in-interactive-mode)
|
|
- [Launch a Ray Cluster in Batch Mode](#launch-a-ray-cluster-in-batch-mode)
|
|
|
|
## Getting Started
|
|
|
|
**Step 1.** Build and transfer the Conda environment to Hawk:
|
|
|
|
Only the main and r channels are available using the Conda module on the clusters. To use custom packages, we need to move the local Conda environment to Hawk.
|
|
|
|
Follow the instructions in [the Conda environment builder repository](https://code.hlrs.de/SiVeGCS/conda-env-builder), which includes a YAML file for building a test environment to run Ray workflows.
|
|
|
|
**Step 2.** Allocate workspace on Hawk:
|
|
|
|
Proceed to the next step if you have already configured your workspace. Use the following command to create a workspace on the high-performance filesystem, which will expire in 10 days. For more information, such as how to enable reminder emails, refer to the [workspace mechanism](https://kb.hlrs.de/platforms/index.php/Workspace_mechanism) guide.
|
|
|
|
```bash
|
|
ws_allocate hpda_project 10
|
|
ws_find hpda_project # find the path to workspace, which is the destination directory in the next step
|
|
```
|
|
|
|
**Step 2.** Clone the repository on Hawk to use the deployment scripts and project structure:
|
|
|
|
```bash
|
|
cd <workspace_directory>
|
|
git clone <repository_url>
|
|
```
|
|
|
|
## Launch a local Ray Cluster in Interactive Mode
|
|
|
|
Using a single node interactively provides opportunities for faster code debugging.
|
|
|
|
**Step 1.** On the Hawk login node, start an interactive job using:
|
|
|
|
```bash
|
|
qsub -I -l select=1:node_type=rome -l walltime=01:00:00
|
|
```
|
|
|
|
**Step 2.** Go into the project directory:
|
|
|
|
```bash
|
|
cd <project_directory>/deployment_scripts
|
|
```
|
|
|
|
**Step 3.** Deploy the conda environment to the ram disk:
|
|
|
|
Change the following line by editing `deploy-env.sh`:
|
|
|
|
```bash
|
|
export WS_DIR=<workspace_dir>
|
|
```
|
|
|
|
Then, use the following command to deploy and activate the environment:
|
|
|
|
```bash
|
|
source deploy-env.sh
|
|
```
|
|
Note: Make sure all permissions are set using `chmod +x`.
|
|
|
|
**Step 4.** Initialize the Ray cluster.
|
|
|
|
You can use a Python interpreter to start a local Ray cluster:
|
|
|
|
```python
|
|
import ray
|
|
|
|
ray.init()
|
|
```
|
|
|
|
**Step 5.** Connect to the dashboard.
|
|
|
|
Warning: Do not change the default dashboard host `127.0.0.1` to keep Ray cluster reachable by only you.
|
|
|
|
Note: We recommend using a dedicated Firefox profile for accessing web-based services on HLRS Compute Platforms. If you haven't created a profile, check out our [guide](https://kb.hlrs.de/platforms/index.php/How_to_use_Web_Based_Services_on_HLRS_Compute_Platforms).
|
|
|
|
You need the job id and the hostname for your current job. You can obtain this information on the login node using:
|
|
|
|
```bash
|
|
qstat -anw # get the job id and the hostname
|
|
```
|
|
|
|
Then, on your local computer,
|
|
|
|
```bash
|
|
export PBS_JOBID=<job-id> # e.g., 2316419.hawk-pbs5
|
|
ssh <compute-host> # e.g., r38c3t8n3
|
|
```
|
|
|
|
Check your SSH config in the first step if this doesn't work.
|
|
|
|
Then, launch Firefox web browser using the configured profile. Open `localhost:8265` to access the Ray dashboard.
|
|
|
|
## Launch a Ray Cluster in Batch Mode
|
|
|
|
Let us [estimate the value of π](https://docs.ray.io/en/releases-2.8.0/ray-core/examples/monte_carlo_pi.html) as an example application.
|
|
|
|
**Step 1.** Add execution permissions to `start-ray-worker.sh`
|
|
|
|
```bash
|
|
cd deployment_scripts
|
|
chmod +x start-ray-worker.sh
|
|
```
|
|
|
|
**Step 2.** Submit a job to launch the head and worker nodes.
|
|
|
|
You must modify the following lines in `submit-ray-job.sh`:
|
|
- Line 3 changes the cluster size. The default configuration launches a 3 node cluster.
|
|
- `export WS_DIR=<workspace_dir>` - set the correct workspace directory.
|
|
- `export PROJECT_DIR=$WS_DIR/<project_name>` - set the correct project directory.
|
|
|
|
Note: The job script `src/monte-carlo-pi.py` waits for all nodes in the Ray cluster to become available. Preserve this pattern in your Python code while using a multiple node Ray cluster.
|
|
|
|
Launch the job and monitor the progress. As the job starts, its status (S) shifts from Q (Queued) to R (Running). Upon completion, the job will no longer appear in the `qstat -a` display.
|
|
|
|
```bash
|
|
qsub submit-ray-job.pbs
|
|
qstat -anw # Q: Queued, R: Running, E: Ending
|
|
ls -l # list files after the job finishes
|
|
cat ray-job.o... # inspect the output file
|
|
cat ray-job.e... # inspect the error file
|
|
```
|
|
|
|
If you need to delete the job, use `qdel <job-id>`. If this doesn't work, use the `-W force` option: `qdel -W force <job-id>` |