diff --git a/README.md b/README.md index 0805de4..a24b0b1 100644 --- a/README.md +++ b/README.md @@ -13,6 +13,7 @@ Structure: To do: - [x] Made scripts for environment creation and deployment in the folder `local_scripts` - [x] Changed scripts to `deployment_scripts` +- [x] Added step about sending python file --- @@ -28,19 +29,19 @@ This repository looks at a deployment of a Dask cluster on Vulcan, and executing Before running the application, make sure you have the following prerequisites installed in a conda environment: - [Python 3.8.18](https://www.python.org/downloads/release/python-3818/): This specific python version is used for all uses, you can select it using while creating the conda environment. For more information on, look at the documentation for Conda on [HLRS HPC systems](https://kb.hlrs.de/platforms/index.php/How_to_move_local_conda_environments_to_the_clusters). -- [Conda Installation](https://docs.conda.io/projects/conda/en/latest/user-guide/install/index.html): Ensure that Conda is installed on your local system. Follow the [official Conda installation guide] if not already installed. +- [Conda Installation](https://docs.conda.io/projects/conda/en/latest/user-guide/install/index.html): Ensure that Conda is installed on your local system. For more information on, look at the documentation for Conda on [HLRS HPC systems](https://kb.hlrs.de/platforms/index.php/How_to_move_local_conda_environments_to_the_clusters). - [Dask](https://dask.org/): Install Dask using conda. - [Conda Pack](https://conda.github.io/conda-pack/): Conda pack is used to package the Conda environment into a single tarball. This is used to transfer the environment to Vulcan. ## Getting Started -1. Clone this repository to your local machine: +1. Clone [this repository](https://code.hlrs.de/hpcrsaxe/spark_template) to your local machine: ```bash git clone ``` -2. Create an environment using Conda and enirvonment.yaml: +2. Go into the direcotry and create an environment using Conda and enirvonment.yaml. Note: Be sure to add the necessary packages in environemnt.yaml: ```bash ./deployment_scripts/create-env.sh @@ -52,19 +53,25 @@ Before running the application, make sure you have the following prerequisites i ./deployment_scripts/deploy-env.sh : ``` -4. SSH into Vulcan and start a job interatively using: +4. Send all the code to the appropriate directory on Vulcan using `scp`: + + ```bash + scp .py : + ``` + +5. SSH into Vulcan and start a job interatively using: ```bash qsub -I -N DaskJob -l select=4:node_type=clx-21 -l walltime=02:00:00 ``` -5. Go into the directory will all code: +6. Go into the directory with all code: ```bash cd ``` -6. Initialize the Dask cluster: +7. Initialize the Dask cluster: ```bash source deploy-dask.sh "$(pwd)" diff --git a/dask_test.ipynb b/notebooks/dask_test.ipynb similarity index 100% rename from dask_test.ipynb rename to notebooks/dask_test.ipynb