finalized for documentation upload

This commit is contained in:
Rishabh Saxena 2024-01-03 09:23:41 +01:00
parent 7c37774ca5
commit f3b8da05d9
2 changed files with 13 additions and 6 deletions

View file

@ -13,6 +13,7 @@ Structure:
To do:
- [x] Made scripts for environment creation and deployment in the folder `local_scripts`
- [x] Changed scripts to `deployment_scripts`
- [x] Added step about sending python file
---
@ -28,19 +29,19 @@ This repository looks at a deployment of a Dask cluster on Vulcan, and executing
Before running the application, make sure you have the following prerequisites installed in a conda environment:
- [Python 3.8.18](https://www.python.org/downloads/release/python-3818/): This specific python version is used for all uses, you can select it using while creating the conda environment. For more information on, look at the documentation for Conda on [HLRS HPC systems](https://kb.hlrs.de/platforms/index.php/How_to_move_local_conda_environments_to_the_clusters).
- [Conda Installation](https://docs.conda.io/projects/conda/en/latest/user-guide/install/index.html): Ensure that Conda is installed on your local system. Follow the [official Conda installation guide] if not already installed.
- [Conda Installation](https://docs.conda.io/projects/conda/en/latest/user-guide/install/index.html): Ensure that Conda is installed on your local system. For more information on, look at the documentation for Conda on [HLRS HPC systems](https://kb.hlrs.de/platforms/index.php/How_to_move_local_conda_environments_to_the_clusters).
- [Dask](https://dask.org/): Install Dask using conda.
- [Conda Pack](https://conda.github.io/conda-pack/): Conda pack is used to package the Conda environment into a single tarball. This is used to transfer the environment to Vulcan.
## Getting Started
1. Clone this repository to your local machine:
1. Clone [this repository](https://code.hlrs.de/hpcrsaxe/spark_template) to your local machine:
```bash
git clone <repository_url>
```
2. Create an environment using Conda and enirvonment.yaml:
2. Go into the direcotry and create an environment using Conda and enirvonment.yaml. Note: Be sure to add the necessary packages in environemnt.yaml:
```bash
./deployment_scripts/create-env.sh <your-env>
@ -52,19 +53,25 @@ Before running the application, make sure you have the following prerequisites i
./deployment_scripts/deploy-env.sh <your-env> <destination_host>:<destination_directory>
```
4. SSH into Vulcan and start a job interatively using:
4. Send all the code to the appropriate directory on Vulcan using `scp`:
```bash
scp <your_script>.py <destination_host>:<destination_directory>
```
5. SSH into Vulcan and start a job interatively using:
```bash
qsub -I -N DaskJob -l select=4:node_type=clx-21 -l walltime=02:00:00
```
5. Go into the directory will all code:
6. Go into the directory with all code:
```bash
cd <destination_directory>
```
6. Initialize the Dask cluster:
7. Initialize the Dask cluster:
```bash
source deploy-dask.sh "$(pwd)"