diff --git a/.gitignore b/.gitignore new file mode 100644 index 0000000..ed8ebf5 --- /dev/null +++ b/.gitignore @@ -0,0 +1 @@ +__pycache__ \ No newline at end of file diff --git a/README.md b/README.md index a24b0b1..9a133a4 100644 --- a/README.md +++ b/README.md @@ -1,37 +1,22 @@ -# Dask: How to execute python workloads using a Dask cluster on Vulcan +# Ray: How to launch a Ray Cluster on Hawk? -Wiki link: - -Motivation: This document aims to show users how to launch a Dask cluster in our compute platforms and perform a simple workload using it. - -Structure: -- [ ] [Tutorial](https://diataxis.fr/tutorials/) -- [x] [How-to guide](https://diataxis.fr/how-to-guides/) -- [ ] [Reference](https://diataxis.fr/reference/) -- [ ] [Explanation](https://diataxis.fr/explanation/) - -To do: -- [x] Made scripts for environment creation and deployment in the folder `local_scripts` -- [x] Changed scripts to `deployment_scripts` -- [x] Added step about sending python file - ---- - -This repository looks at a deployment of a Dask cluster on Vulcan, and executing your programs using this cluster. +This guide shows you how to launch a Ray cluster on HLRS' Hawk system. ## Table of Contents -- [Prerequisites](#prerequisites) -- [Getting Started](#getting-started) -- [Usage](#usage) -- [Notes](#notes) +- [Ray: How to launch a Ray Cluster on Hawk?](#ray-how-to-launch-a-ray-cluster-on-hawk) + - [Table of Contents](#table-of-contents) + - [Prerequisites](#prerequisites) + - [Getting Started](#getting-started) + - [Usage](#usage) + - [Notes](#notes) ## Prerequisites Before running the application, make sure you have the following prerequisites installed in a conda environment: -- [Python 3.8.18](https://www.python.org/downloads/release/python-3818/): This specific python version is used for all uses, you can select it using while creating the conda environment. For more information on, look at the documentation for Conda on [HLRS HPC systems](https://kb.hlrs.de/platforms/index.php/How_to_move_local_conda_environments_to_the_clusters). -- [Conda Installation](https://docs.conda.io/projects/conda/en/latest/user-guide/install/index.html): Ensure that Conda is installed on your local system. For more information on, look at the documentation for Conda on [HLRS HPC systems](https://kb.hlrs.de/platforms/index.php/How_to_move_local_conda_environments_to_the_clusters). -- [Dask](https://dask.org/): Install Dask using conda. -- [Conda Pack](https://conda.github.io/conda-pack/): Conda pack is used to package the Conda environment into a single tarball. This is used to transfer the environment to Vulcan. +- [Python 3.9](https://www.python.org/downloads/release/python-3818/): This specific python version is used for all uses, you can select it using while creating the conda environment. For more information on, look at the documentation for Conda on [HLRS HPC systems](https://kb.hlrs.de/platforms/index.php/How_to_move_local_conda_environments_to_the_clusters). +- [Conda Installation](https://docs.conda.io/projects/conda/en/latest/user-guide/install/index.html): Ensure that Conda is installed on your local system. For more information, look at the documentation for Conda on [HLRS HPC systems](https://kb.hlrs.de/platforms/index.php/How_to_move_local_conda_environments_to_the_clusters). +- [Ray](https://dask.org/): You can install Ray inside +- [Conda Pack](https://conda.github.io/conda-pack/): Conda pack is used to package the Conda environment into a single tarball. This is used to transfer the environment to Vulcan. ## Getting Started @@ -41,7 +26,7 @@ Before running the application, make sure you have the following prerequisites i git clone ``` -2. Go into the direcotry and create an environment using Conda and enirvonment.yaml. Note: Be sure to add the necessary packages in environemnt.yaml: +2. Go into the directory and create an environment using Conda and environment.yaml. Note: Be sure to add the necessary packages in environment.yaml: ```bash ./deployment_scripts/create-env.sh diff --git a/__pycache__/daskdataset.cpython-38.pyc b/__pycache__/daskdataset.cpython-38.pyc deleted file mode 100644 index 162ec84..0000000 Binary files a/__pycache__/daskdataset.cpython-38.pyc and /dev/null differ diff --git a/deployment_scripts/README.md b/deployment_scripts/deployment_scripts_reference.md similarity index 98% rename from deployment_scripts/README.md rename to deployment_scripts/deployment_scripts_reference.md index 9dc3dc0..b4d7d30 100644 --- a/deployment_scripts/README.md +++ b/deployment_scripts/deployment_scripts_reference.md @@ -1,4 +1,4 @@ -# Reference Guide: Dask Cluster Deployment Scripts +# Reference: Cluster Deployment Scripts Wiki link: diff --git a/deployment_scripts/environment.yaml b/deployment_scripts/environment.yaml index 862e77c..09009e7 100644 --- a/deployment_scripts/environment.yaml +++ b/deployment_scripts/environment.yaml @@ -1,9 +1,21 @@ +name: ray channels: - defaults - - conda-forge dependencies: - - python=3.8.18 - - dask - - numpy - - scikit-learn - - conda-pack + - python=3.10 + - pip: + - ray==2.8.0 + - dask==2022.10.1 + - torch + - pydantic<2 + - six + - torch + - tqdm + - pandas<2 + - scikit-learn + - matplotlib + - optuna + - seaborn + - tabulate + - jupyterlab + - autopep8