ray_template/README.md

# Ray: How to launch a Ray Cluster on Hawk?

This guide shows you how to launch a Ray cluster on HLRS' Hawk system.

## Table of Contents
- [Ray: How to launch a Ray Cluster on Hawk?](#ray-how-to-launch-a-ray-cluster-on-hawk)
  - [Table of Contents](#table-of-contents)
  - [Getting Started](#getting-started)
  - [Launch a local Ray Cluster in Interactive Mode](#launch-a-local-ray-cluster-in-interactive-mode)
  - [Launch a Ray Cluster in Batch Mode](#launch-a-ray-cluster-in-batch-mode)

## Getting Started

**Step 1.** Build and transfer the Conda environment to Hawk:

Only the main and r channels are available using the Conda module on the clusters. To use custom packages, we need to move the local Conda environment to Hawk. 

Follow the instructions in [the Conda environment builder repository](https://code.hlrs.de/SiVeGCS/conda-env-builder), which includes a YAML file for building a test environment to run Ray workflows.

**Step 2.** Allocate workspace on Hawk:

Proceed to the next step if you have already configured your workspace. Use the following command to create a workspace on the high-performance filesystem, which will expire in 10 days. For more information, such as how to enable reminder emails, refer to the [workspace mechanism](https://kb.hlrs.de/platforms/index.php/Workspace_mechanism) guide.

```bash
ws_allocate hpda_project 10
ws_find hpda_project # find the path to workspace, which is the destination directory in the next step
```

**Step 2.** Clone the repository on Hawk to use the deployment scripts and project structure:

```bash
cd <workspace_directory>
git clone <repository_url>
```

## Launch a local Ray Cluster in Interactive Mode

Using a single node interactively provides opportunities for faster code debugging.

**Step 1.** On the Hawk login node, start an interactive job using:

```bash
qsub -I -l select=1:node_type=rome -l walltime=01:00:00
```

**Step 2.** Go into the project directory:

```bash
cd <project_directory>/deployment_scripts
```

**Step 3.** Deploy the conda environment to the ram disk:

Change the following line by editing `deploy-env.sh`:

```bash
export WS_DIR=<workspace_dir>
```

Then, use the following command to deploy and activate the environment:

```bash
source deploy-env.sh
```
Note: Make sure all permissions are set using `chmod +x`.

**Step 4.** Initialize the Ray cluster.

You can use a Python interpreter to start a local Ray cluster:

```python
import ray

ray.init()
```

**Step 5.** Connect to the dashboard.

Warning: Do not change the default dashboard host `127.0.0.1` to keep Ray cluster reachable by only you.

Note: We recommend using a dedicated Firefox profile for accessing web-based services on HLRS Compute Platforms. If you haven't created a profile, check out our [guide](https://kb.hlrs.de/platforms/index.php/How_to_use_Web_Based_Services_on_HLRS_Compute_Platforms).

You need the job id and the hostname for your current job. You can obtain this information on the login node using:

```bash
qstat -anw # get the job id and the hostname
```

Then, on your local computer, 

```bash
export PBS_JOBID=<job-id> # e.g., 2316419.hawk-pbs5
ssh <compute-host> # e.g., r38c3t8n3
```

Check your SSH config in the first step if this doesn't work.

Then, launch Firefox web browser using the configured profile. Open `localhost:8265` to access the Ray dashboard.

## Launch a Ray Cluster in Batch Mode

Let us [estimate the value of π](https://docs.ray.io/en/releases-2.8.0/ray-core/examples/monte_carlo_pi.html) as an example application. 

**Step 1.** Add execution permissions to `start-ray-worker.sh`

```bash
cd deployment_scripts
chmod +x start-ray-worker.sh
```

**Step 2.** Submit a job to launch the head and worker nodes.

You must modify the following lines in `submit-ray-job.sh`:
- Line 3 changes the cluster size. The default configuration launches a 3 node cluster.
- `export WS_DIR=<workspace_dir>` - set the correct workspace directory.
- `export PROJECT_DIR=$WS_DIR/<project_name>` - set the correct project directory.

Note: The job script `src/monte-carlo-pi.py` waits for all nodes in the Ray cluster to become available. Preserve this pattern in your Python code while using a multiple node Ray cluster.

Launch the job and monitor the progress. As the job starts, its status (S) shifts from Q (Queued) to R (Running). Upon completion, the job will no longer appear in the `qstat -a` display.

```bash
qsub submit-ray-job.pbs
qstat -anw # Q: Queued, R: Running, E: Ending
ls -l # list files after the job finishes
cat ray-job.o... # inspect the output file
cat ray-job.e... # inspect the error file
```

If you need to delete the job, use `qdel <job-id>`. If this doesn't work, use the `-W force` option: `qdel -W force <job-id>`