deploy-env.sh deploys a custom conda env to the HOME directory

README.md

@@ -45,7 +45,7 @@ This Docker image includes a utility script, `build_and_pack_env.sh`, that autom
 2. **Run the Docker Container with Volume Mount**: Run the Docker container, mounting the directory containing your environment YAML file. Replace `<path-to-your-yaml-file>` with the actual path to your YAML file:
 
 ```bash
-docker run -it -v <path-to-your-yaml-file>:/envs --workdir /envs miniconda-rockylinux:latest
+docker run --rm -it -v <path-to-your-yaml-file>:/envs --workdir /envs miniconda-rockylinux:latest
 ```
 
 3. **Execute the Script Inside the Container**: Once inside the container, run the build_and_pack_env.sh script with your YAML file as an argument. Replace your_environment.yml with the name of your environment file:
@@ -58,4 +58,52 @@ The script will create the environment, pack it, and output a `.tar.gz` file tha
 
 **Notes**
 - The packed environment file will be saved in the same directory as the original YAML file.
-- Ensure the volume mount (-v) option correctly maps the local directory containing your YAML file to the `/envs` directory inside the container.
+- Ensure the volume mount (-v) option correctly maps the local directory containing your YAML file to the `/envs` directory inside the container.
+
+## Unpacking and Activating the Environment
+
+1. **Transfer the packed environment**: Use `scp` to transfer the packed environment file and `deploy-env.sh` to the remote server:
+
+```bash
+scp deploy-env.sh user@remote-server:~/deploy-env.sh
+scp envs/your_environment.tar.gz user@remote-server:~/your_environment.tar.gz
+```
+
+2. **SSH into the remote server**
+
+```bash
+ssh user@remote-server
+```
+
+3. **Deploy the environment**
+
+```bash
+chmod +x deploy-env.sh
+./deploy-env.sh your_environment.tar.gz
+rm your_environment.tar.gz
+```
+
+4. **Test the environment using a Hawk compute node**
+
+```bash
+# Request an interactive job allocation for 5 minutes on a CPU node
+qsub -I -l select=1:node_type=rome -l walltime=00:05:00
+# Load the Conda module
+module load bigdata/conda
+# List available Conda environments (for verification purposes)
+conda env list
+# Activate a specific Conda environment.
+# Replace '/path/to/your_environment' with the actual path to your Conda environment.
+# This path might look something like '$HOME/.conda/envs/your_environment'
+source /path/to/your_environment/bin/activate
+# Add any additional commands you want to run within the Conda environment below this line.
+```
+
+## Important Notice Regarding Filesystem Space
+
+Conda environments, especially those with many dependencies or large packages, can consume a significant amount of disk space. Given that the file system space on `HOME` is limited by a quota (50 GB per user, 200 GB per group), it's important to manage the space used by Conda environments carefully.
+
+Recommendations:
+- **Monitor Disk Usage**: Regularly check the disk space used by your Conda environments. You can use commands like `du -sh ~/.conda/envs/*` to see the space used by each environment.
+- **Clean Up Regularly**: Remove unused environments with `conda env remove --name <env-name>`. Also, consider using `conda clean --all` to remove unused packages and caches.
+- **Environment Management**: Be strategic about the packages included in your environments to minimize space usage. Only include necessary packages and consider lighter alternatives for heavy dependencies.

build_and_pack_env.sh

@@ -24,4 +24,9 @@ echo "Packing environment: $ENV_NAME"
 conda deactivate
 conda pack -n $ENV_NAME -o ${ENV_NAME}.tar.gz
 
-echo "Environment $ENV_NAME packed successfully into ${ENV_NAME}.tar.gz"
+# Check if the file exists
+if [ -f "${ENV_NAME}.tar.gz" ]; then
+  echo "Environment $ENV_NAME packed successfully into ${ENV_NAME}.tar.gz"
+else
+    echo "Failed to pack the environment $ENV_NAME."
+fi

deploy-env.sh

@@ -0,0 +1,53 @@
+#!/bin/bash
+
+# Check if an environment file was provided
+if [ "$#" -ne 1 ]; then
+  echo "Usage: $0 environment.tar.gz"
+  exit 0
+fi
+
+echo "Extracting the environment name from the provided archive..."
+
+extract_filename() {
+    local fullpath="$1"
+    local filename="${fullpath##*/}" # Remove the path, retain the filename
+    local name="${filename%%.*}" # Remove the extension
+    echo "$name"
+}
+
+# Set environment variables based on the provided archive path.
+export ENV_ARCHIVE=$1
+export ENV_NAME=$(extract_filename $ENV_ARCHIVE) # The name of the environment, derived from the archive file name.
+
+echo "Environment name extracted: $ENV_NAME"
+
+# Define the path where Conda environments are stored and the specific path for the new environment.
+export CONDA_ENVS="$HOME/.conda/envs"
+export ENV_PATH="$CONDA_ENVS/$ENV_NAME"
+
+echo "Designated path for the new environment: $ENV_PATH"
+
+# Create the directory structure for the new environment.
+mkdir -p $ENV_PATH
+
+echo "Unpacking the provided environment archive into the designated environment path..."
+
+# Unpack the provided environment archive into the designated environment path.
+tar -xzf $ENV_ARCHIVE -C $ENV_PATH
+
+echo "Changing into the environment directory..."
+
+# Change into the environment directory.
+cd $ENV_PATH
+
+module load bigdata/conda
+
+echo "Unpacking Conda environment..."
+./bin/conda-unpack
+
+# Check if the environment was deployed successfully by verifying the existence of the conda-meta directory.
+if [ -d "$ENV_PATH/conda-meta" ]; then
+    echo "Environment setup completed successfully."
+else
+    echo "Failed to deploy the Conda environment properly. Please check the archive and try again."
+fi