From 212441c516974cda5bf588683d7c02c3c31d3a67 Mon Sep 17 00:00:00 2001 From: Kerem Kayabay Date: Wed, 3 Jan 2024 15:53:42 +0100 Subject: [PATCH] change environment.yaml to install Ray --- .gitignore | 1 + README.md | 41 ++++++------------ __pycache__/daskdataset.cpython-38.pyc | Bin 1657 -> 0 bytes ...DME.md => deployment_scripts_reference.md} | 2 +- deployment_scripts/environment.yaml | 24 +++++++--- 5 files changed, 33 insertions(+), 35 deletions(-) create mode 100644 .gitignore delete mode 100644 __pycache__/daskdataset.cpython-38.pyc rename deployment_scripts/{README.md => deployment_scripts_reference.md} (98%) diff --git a/.gitignore b/.gitignore new file mode 100644 index 0000000..ed8ebf5 --- /dev/null +++ b/.gitignore @@ -0,0 +1 @@ +__pycache__ \ No newline at end of file diff --git a/README.md b/README.md index a24b0b1..9a133a4 100644 --- a/README.md +++ b/README.md @@ -1,37 +1,22 @@ -# Dask: How to execute python workloads using a Dask cluster on Vulcan +# Ray: How to launch a Ray Cluster on Hawk? -Wiki link: - -Motivation: This document aims to show users how to launch a Dask cluster in our compute platforms and perform a simple workload using it. - -Structure: -- [ ] [Tutorial](https://diataxis.fr/tutorials/) -- [x] [How-to guide](https://diataxis.fr/how-to-guides/) -- [ ] [Reference](https://diataxis.fr/reference/) -- [ ] [Explanation](https://diataxis.fr/explanation/) - -To do: -- [x] Made scripts for environment creation and deployment in the folder `local_scripts` -- [x] Changed scripts to `deployment_scripts` -- [x] Added step about sending python file - ---- - -This repository looks at a deployment of a Dask cluster on Vulcan, and executing your programs using this cluster. +This guide shows you how to launch a Ray cluster on HLRS' Hawk system. ## Table of Contents -- [Prerequisites](#prerequisites) -- [Getting Started](#getting-started) -- [Usage](#usage) -- [Notes](#notes) +- [Ray: How to launch a Ray Cluster on Hawk?](#ray-how-to-launch-a-ray-cluster-on-hawk) + - [Table of Contents](#table-of-contents) + - [Prerequisites](#prerequisites) + - [Getting Started](#getting-started) + - [Usage](#usage) + - [Notes](#notes) ## Prerequisites Before running the application, make sure you have the following prerequisites installed in a conda environment: -- [Python 3.8.18](https://www.python.org/downloads/release/python-3818/): This specific python version is used for all uses, you can select it using while creating the conda environment. For more information on, look at the documentation for Conda on [HLRS HPC systems](https://kb.hlrs.de/platforms/index.php/How_to_move_local_conda_environments_to_the_clusters). -- [Conda Installation](https://docs.conda.io/projects/conda/en/latest/user-guide/install/index.html): Ensure that Conda is installed on your local system. For more information on, look at the documentation for Conda on [HLRS HPC systems](https://kb.hlrs.de/platforms/index.php/How_to_move_local_conda_environments_to_the_clusters). -- [Dask](https://dask.org/): Install Dask using conda. -- [Conda Pack](https://conda.github.io/conda-pack/): Conda pack is used to package the Conda environment into a single tarball. This is used to transfer the environment to Vulcan. +- [Python 3.9](https://www.python.org/downloads/release/python-3818/): This specific python version is used for all uses, you can select it using while creating the conda environment. For more information on, look at the documentation for Conda on [HLRS HPC systems](https://kb.hlrs.de/platforms/index.php/How_to_move_local_conda_environments_to_the_clusters). +- [Conda Installation](https://docs.conda.io/projects/conda/en/latest/user-guide/install/index.html): Ensure that Conda is installed on your local system. For more information, look at the documentation for Conda on [HLRS HPC systems](https://kb.hlrs.de/platforms/index.php/How_to_move_local_conda_environments_to_the_clusters). +- [Ray](https://dask.org/): You can install Ray inside +- [Conda Pack](https://conda.github.io/conda-pack/): Conda pack is used to package the Conda environment into a single tarball. This is used to transfer the environment to Vulcan. ## Getting Started @@ -41,7 +26,7 @@ Before running the application, make sure you have the following prerequisites i git clone ``` -2. Go into the direcotry and create an environment using Conda and enirvonment.yaml. Note: Be sure to add the necessary packages in environemnt.yaml: +2. Go into the directory and create an environment using Conda and environment.yaml. Note: Be sure to add the necessary packages in environment.yaml: ```bash ./deployment_scripts/create-env.sh diff --git a/__pycache__/daskdataset.cpython-38.pyc b/__pycache__/daskdataset.cpython-38.pyc deleted file mode 100644 index 162ec84df889d70fd4fa94933b23f6195a84043f..0000000000000000000000000000000000000000 GIT binary patch literal 0 HcmV?d00001 literal 1657 zcmZuxNpBlB6ec;VN3vtsX^XCCi(cj;p$pJUQ3SA&qJXPIQ(t%?rb8-{GmA=6NNlJ; zfdt4Y|3NzXFY(%wFZ}^M^*w5|I1K^5dE4-9A9;Crm=Rd?pRrm7g#3-d{t7{O1lvr2 zNFr%PIDT10YsOg}@PMxMVIA?f2hqqVXD&G}Y+;mj88R1bA(bxCxW5<(k6@dxfGEPLB%Dbq*(<_> zbwD1j1Mx^E5FmBE^TIsq4lkkd?Lr>+8Eo@8kOjShx+Qx<7evqn+mo0ia9A+i-jC|-rMICS7V?;*5Bd=>7D7T`wxA!9Ww1gyl=KaQUb7LjI+FH!rt-)|w$KZy;|>)df@e;_ZBULOfM=YE>=72{>(X{@{MjTc1%L zV{@CLRzkQ;h`N>Y3i-Yef6R+&ZPD0weS`#0-HVEI)z~VH6(eJ?sVPmvH1N6-#Ct2~ zq+#&gGP+^s4`0JNI7Gw!m zc}4#OXf5fM1CV(gU|jdXEf4g^@X?