forked from SiVeGCS/dask_template
change environment.yaml to install Ray
This commit is contained in:
parent
f3b8da05d9
commit
212441c516
5 changed files with 33 additions and 35 deletions
1
.gitignore
vendored
Normal file
1
.gitignore
vendored
Normal file
|
|
@ -0,0 +1 @@
|
|||
__pycache__
|
||||
31
README.md
31
README.md
|
|
@ -1,25 +1,10 @@
|
|||
# Dask: How to execute python workloads using a Dask cluster on Vulcan
|
||||
# Ray: How to launch a Ray Cluster on Hawk?
|
||||
|
||||
Wiki link:
|
||||
|
||||
Motivation: This document aims to show users how to launch a Dask cluster in our compute platforms and perform a simple workload using it.
|
||||
|
||||
Structure:
|
||||
- [ ] [Tutorial](https://diataxis.fr/tutorials/)
|
||||
- [x] [How-to guide](https://diataxis.fr/how-to-guides/)
|
||||
- [ ] [Reference](https://diataxis.fr/reference/)
|
||||
- [ ] [Explanation](https://diataxis.fr/explanation/)
|
||||
|
||||
To do:
|
||||
- [x] Made scripts for environment creation and deployment in the folder `local_scripts`
|
||||
- [x] Changed scripts to `deployment_scripts`
|
||||
- [x] Added step about sending python file
|
||||
|
||||
---
|
||||
|
||||
This repository looks at a deployment of a Dask cluster on Vulcan, and executing your programs using this cluster.
|
||||
This guide shows you how to launch a Ray cluster on HLRS' Hawk system.
|
||||
|
||||
## Table of Contents
|
||||
- [Ray: How to launch a Ray Cluster on Hawk?](#ray-how-to-launch-a-ray-cluster-on-hawk)
|
||||
- [Table of Contents](#table-of-contents)
|
||||
- [Prerequisites](#prerequisites)
|
||||
- [Getting Started](#getting-started)
|
||||
- [Usage](#usage)
|
||||
|
|
@ -28,9 +13,9 @@ This repository looks at a deployment of a Dask cluster on Vulcan, and executing
|
|||
## Prerequisites
|
||||
|
||||
Before running the application, make sure you have the following prerequisites installed in a conda environment:
|
||||
- [Python 3.8.18](https://www.python.org/downloads/release/python-3818/): This specific python version is used for all uses, you can select it using while creating the conda environment. For more information on, look at the documentation for Conda on [HLRS HPC systems](https://kb.hlrs.de/platforms/index.php/How_to_move_local_conda_environments_to_the_clusters).
|
||||
- [Conda Installation](https://docs.conda.io/projects/conda/en/latest/user-guide/install/index.html): Ensure that Conda is installed on your local system. For more information on, look at the documentation for Conda on [HLRS HPC systems](https://kb.hlrs.de/platforms/index.php/How_to_move_local_conda_environments_to_the_clusters).
|
||||
- [Dask](https://dask.org/): Install Dask using conda.
|
||||
- [Python 3.9](https://www.python.org/downloads/release/python-3818/): This specific python version is used for all uses, you can select it using while creating the conda environment. For more information on, look at the documentation for Conda on [HLRS HPC systems](https://kb.hlrs.de/platforms/index.php/How_to_move_local_conda_environments_to_the_clusters).
|
||||
- [Conda Installation](https://docs.conda.io/projects/conda/en/latest/user-guide/install/index.html): Ensure that Conda is installed on your local system. For more information, look at the documentation for Conda on [HLRS HPC systems](https://kb.hlrs.de/platforms/index.php/How_to_move_local_conda_environments_to_the_clusters).
|
||||
- [Ray](https://dask.org/): You can install Ray inside
|
||||
- [Conda Pack](https://conda.github.io/conda-pack/): Conda pack is used to package the Conda environment into a single tarball. This is used to transfer the environment to Vulcan.
|
||||
|
||||
## Getting Started
|
||||
|
|
@ -41,7 +26,7 @@ Before running the application, make sure you have the following prerequisites i
|
|||
git clone <repository_url>
|
||||
```
|
||||
|
||||
2. Go into the direcotry and create an environment using Conda and enirvonment.yaml. Note: Be sure to add the necessary packages in environemnt.yaml:
|
||||
2. Go into the directory and create an environment using Conda and environment.yaml. Note: Be sure to add the necessary packages in environment.yaml:
|
||||
|
||||
```bash
|
||||
./deployment_scripts/create-env.sh <your-env>
|
||||
|
|
|
|||
Binary file not shown.
|
|
@ -1,4 +1,4 @@
|
|||
# Reference Guide: Dask Cluster Deployment Scripts
|
||||
# Reference: Cluster Deployment Scripts
|
||||
|
||||
Wiki link:
|
||||
|
||||
|
|
@ -1,9 +1,21 @@
|
|||
name: ray
|
||||
channels:
|
||||
- defaults
|
||||
- conda-forge
|
||||
dependencies:
|
||||
- python=3.8.18
|
||||
- dask
|
||||
- numpy
|
||||
- python=3.10
|
||||
- pip:
|
||||
- ray==2.8.0
|
||||
- dask==2022.10.1
|
||||
- torch
|
||||
- pydantic<2
|
||||
- six
|
||||
- torch
|
||||
- tqdm
|
||||
- pandas<2
|
||||
- scikit-learn
|
||||
- conda-pack
|
||||
- matplotlib
|
||||
- optuna
|
||||
- seaborn
|
||||
- tabulate
|
||||
- jupyterlab
|
||||
- autopep8
|
||||
|
|
|
|||
Loading…
Reference in a new issue