69f7a8f4d1
This patchset refactors our GitHub actions into a single top-level ci workflow that invokes a series of reusable actions. The main goal of this is to be able to easily control which tests run and in what order based on the success or failure of top-level prechecks. Our previous workflows ran in three sets: * nix tests: style and verification first, then linux and macos tests if successful * windows tests: style and verification first, then linux and macos tests if successful * bootstrap tests As a result, the bootstrap tests ran even if the style failed, and style and verification had to run on two different platforms despite running identical checks. I'm relatively sure that's because of the limitation on dependencies between steps in the jobs. Reusable workflows allow us to run the style, verification and now audit checks once, then depending on the results, and the files changed, run the appropriate nix, windows and bootstrap tests. While it saves only a few minutes by itself, this makes it easier to refactor checks to subset tests without having to replicate tests or other workflow components in the future. Co-authored-by: Massimiliano Culpo <massimiliano.culpo@gmail.com>
346 lines
12 KiB
YAML
346 lines
12 KiB
YAML
name: Bootstrapping
|
|
|
|
on:
|
|
# This Workflow can be triggered manually
|
|
workflow_dispatch:
|
|
workflow_call:
|
|
schedule:
|
|
# nightly at 2:16 AM
|
|
- cron: '16 2 * * *'
|
|
|
|
concurrency:
|
|
group: bootstrap-${{ github.workflow }}-${{ github.event.pull_request.number || github.run_number }}
|
|
cancel-in-progress: true
|
|
|
|
jobs:
|
|
fedora-clingo-sources:
|
|
runs-on: ubuntu-latest
|
|
container: "fedora:latest"
|
|
steps:
|
|
- name: Install dependencies
|
|
run: |
|
|
dnf install -y \
|
|
bzip2 curl file gcc-c++ gcc gcc-gfortran git gnupg2 gzip \
|
|
make patch unzip which xz python3 python3-devel tree \
|
|
cmake bison bison-devel libstdc++-static
|
|
- name: Checkout
|
|
uses: actions/checkout@2541b1294d2704b0964813337f33b291d3f8596b
|
|
with:
|
|
fetch-depth: 0
|
|
- name: Setup non-root user
|
|
run: |
|
|
# See [1] below
|
|
git config --global --add safe.directory /__w/spack/spack
|
|
useradd spack-test && mkdir -p ~spack-test
|
|
chown -R spack-test . ~spack-test
|
|
- name: Setup repo
|
|
shell: runuser -u spack-test -- bash {0}
|
|
run: |
|
|
git --version
|
|
. .github/workflows/setup_git.sh
|
|
- name: Bootstrap clingo
|
|
shell: runuser -u spack-test -- bash {0}
|
|
run: |
|
|
source share/spack/setup-env.sh
|
|
spack bootstrap untrust github-actions-v0.2
|
|
spack external find cmake bison
|
|
spack -d solve zlib
|
|
tree ~/.spack/bootstrap/store/
|
|
|
|
ubuntu-clingo-sources:
|
|
runs-on: ubuntu-latest
|
|
container: "ubuntu:latest"
|
|
steps:
|
|
- name: Install dependencies
|
|
env:
|
|
DEBIAN_FRONTEND: noninteractive
|
|
run: |
|
|
apt-get update -y && apt-get upgrade -y
|
|
apt-get install -y \
|
|
bzip2 curl file g++ gcc gfortran git gnupg2 gzip \
|
|
make patch unzip xz-utils python3 python3-dev tree \
|
|
cmake bison
|
|
- name: Checkout
|
|
uses: actions/checkout@2541b1294d2704b0964813337f33b291d3f8596b
|
|
with:
|
|
fetch-depth: 0
|
|
- name: Setup non-root user
|
|
run: |
|
|
# See [1] below
|
|
git config --global --add safe.directory /__w/spack/spack
|
|
useradd spack-test && mkdir -p ~spack-test
|
|
chown -R spack-test . ~spack-test
|
|
- name: Setup repo
|
|
shell: runuser -u spack-test -- bash {0}
|
|
run: |
|
|
git --version
|
|
. .github/workflows/setup_git.sh
|
|
- name: Bootstrap clingo
|
|
shell: runuser -u spack-test -- bash {0}
|
|
run: |
|
|
source share/spack/setup-env.sh
|
|
spack bootstrap untrust github-actions-v0.2
|
|
spack external find cmake bison
|
|
spack -d solve zlib
|
|
tree ~/.spack/bootstrap/store/
|
|
|
|
ubuntu-clingo-binaries-and-patchelf:
|
|
runs-on: ubuntu-latest
|
|
container: "ubuntu:latest"
|
|
steps:
|
|
- name: Install dependencies
|
|
env:
|
|
DEBIAN_FRONTEND: noninteractive
|
|
run: |
|
|
apt-get update -y && apt-get upgrade -y
|
|
apt-get install -y \
|
|
bzip2 curl file g++ gcc gfortran git gnupg2 gzip \
|
|
make patch unzip xz-utils python3 python3-dev tree
|
|
- name: Checkout
|
|
uses: actions/checkout@2541b1294d2704b0964813337f33b291d3f8596b
|
|
with:
|
|
fetch-depth: 0
|
|
- name: Setup non-root user
|
|
run: |
|
|
# See [1] below
|
|
git config --global --add safe.directory /__w/spack/spack
|
|
useradd spack-test && mkdir -p ~spack-test
|
|
chown -R spack-test . ~spack-test
|
|
- name: Setup repo
|
|
shell: runuser -u spack-test -- bash {0}
|
|
run: |
|
|
git --version
|
|
. .github/workflows/setup_git.sh
|
|
- name: Bootstrap clingo
|
|
shell: runuser -u spack-test -- bash {0}
|
|
run: |
|
|
source share/spack/setup-env.sh
|
|
spack -d solve zlib
|
|
tree ~/.spack/bootstrap/store/
|
|
|
|
opensuse-clingo-sources:
|
|
runs-on: ubuntu-latest
|
|
container: "opensuse/leap:latest"
|
|
steps:
|
|
- name: Install dependencies
|
|
run: |
|
|
# Harden CI by applying the workaround described here: https://www.suse.com/support/kb/doc/?id=000019505
|
|
zypper update -y || zypper update -y
|
|
zypper install -y \
|
|
bzip2 curl file gcc-c++ gcc gcc-fortran tar git gpg2 gzip \
|
|
make patch unzip which xz python3 python3-devel tree \
|
|
cmake bison
|
|
- name: Checkout
|
|
uses: actions/checkout@2541b1294d2704b0964813337f33b291d3f8596b
|
|
with:
|
|
fetch-depth: 0
|
|
- name: Setup repo
|
|
run: |
|
|
# See [1] below
|
|
git config --global --add safe.directory /__w/spack/spack
|
|
git --version
|
|
. .github/workflows/setup_git.sh
|
|
- name: Bootstrap clingo
|
|
run: |
|
|
source share/spack/setup-env.sh
|
|
spack bootstrap untrust github-actions-v0.2
|
|
spack external find cmake bison
|
|
spack -d solve zlib
|
|
tree ~/.spack/bootstrap/store/
|
|
|
|
macos-clingo-sources:
|
|
runs-on: macos-latest
|
|
steps:
|
|
- name: Install dependencies
|
|
run: |
|
|
brew install cmake bison@2.7 tree
|
|
- name: Checkout
|
|
uses: actions/checkout@2541b1294d2704b0964813337f33b291d3f8596b
|
|
- name: Bootstrap clingo
|
|
run: |
|
|
source share/spack/setup-env.sh
|
|
export PATH=/usr/local/opt/bison@2.7/bin:$PATH
|
|
spack bootstrap untrust github-actions-v0.2
|
|
spack external find --not-buildable cmake bison
|
|
spack -d solve zlib
|
|
tree ~/.spack/bootstrap/store/
|
|
|
|
macos-clingo-binaries:
|
|
runs-on: ${{ matrix.macos-version }}
|
|
strategy:
|
|
matrix:
|
|
macos-version: ['macos-11', 'macos-12']
|
|
steps:
|
|
- name: Install dependencies
|
|
run: |
|
|
brew install tree
|
|
- name: Checkout
|
|
uses: actions/checkout@2541b1294d2704b0964813337f33b291d3f8596b
|
|
- name: Bootstrap clingo
|
|
run: |
|
|
set -ex
|
|
for ver in '3.6' '3.7' '3.8' '3.9' '3.10' ; do
|
|
not_found=1
|
|
ver_dir="$(find $RUNNER_TOOL_CACHE/Python -wholename "*/${ver}.*/*/bin" | grep . || true)"
|
|
echo "Testing $ver_dir"
|
|
if [[ -d "$ver_dir" ]] ; then
|
|
if $ver_dir/python --version ; then
|
|
export PYTHON="$ver_dir/python"
|
|
not_found=0
|
|
old_path="$PATH"
|
|
export PATH="$ver_dir:$PATH"
|
|
./bin/spack-tmpconfig -b ./.github/workflows/bootstrap-test.sh
|
|
export PATH="$old_path"
|
|
fi
|
|
fi
|
|
# NOTE: test all pythons that exist, not all do on 12
|
|
done
|
|
|
|
ubuntu-clingo-binaries:
|
|
runs-on: ubuntu-20.04
|
|
steps:
|
|
- name: Checkout
|
|
uses: actions/checkout@2541b1294d2704b0964813337f33b291d3f8596b
|
|
with:
|
|
fetch-depth: 0
|
|
- name: Setup repo
|
|
run: |
|
|
git --version
|
|
. .github/workflows/setup_git.sh
|
|
- name: Bootstrap clingo
|
|
run: |
|
|
set -ex
|
|
for ver in '2.7' '3.6' '3.7' '3.8' '3.9' '3.10' ; do
|
|
not_found=1
|
|
ver_dir="$(find $RUNNER_TOOL_CACHE/Python -wholename "*/${ver}.*/*/bin" | grep . || true)"
|
|
echo "Testing $ver_dir"
|
|
if [[ -d "$ver_dir" ]] ; then
|
|
if $ver_dir/python --version ; then
|
|
export PYTHON="$ver_dir/python"
|
|
not_found=0
|
|
old_path="$PATH"
|
|
export PATH="$ver_dir:$PATH"
|
|
./bin/spack-tmpconfig -b ./.github/workflows/bootstrap-test.sh
|
|
export PATH="$old_path"
|
|
fi
|
|
fi
|
|
if (($not_found)) ; then
|
|
echo Required python version $ver not found in runner!
|
|
exit 1
|
|
fi
|
|
done
|
|
|
|
ubuntu-gnupg-binaries:
|
|
runs-on: ubuntu-latest
|
|
container: "ubuntu:latest"
|
|
steps:
|
|
- name: Install dependencies
|
|
env:
|
|
DEBIAN_FRONTEND: noninteractive
|
|
run: |
|
|
apt-get update -y && apt-get upgrade -y
|
|
apt-get install -y \
|
|
bzip2 curl file g++ gcc patchelf gfortran git gzip \
|
|
make patch unzip xz-utils python3 python3-dev tree
|
|
- name: Checkout
|
|
uses: actions/checkout@2541b1294d2704b0964813337f33b291d3f8596b
|
|
with:
|
|
fetch-depth: 0
|
|
- name: Setup non-root user
|
|
run: |
|
|
# See [1] below
|
|
git config --global --add safe.directory /__w/spack/spack
|
|
useradd spack-test && mkdir -p ~spack-test
|
|
chown -R spack-test . ~spack-test
|
|
- name: Setup repo
|
|
shell: runuser -u spack-test -- bash {0}
|
|
run: |
|
|
git --version
|
|
. .github/workflows/setup_git.sh
|
|
- name: Bootstrap GnuPG
|
|
shell: runuser -u spack-test -- bash {0}
|
|
run: |
|
|
source share/spack/setup-env.sh
|
|
spack bootstrap untrust spack-install
|
|
spack -d gpg list
|
|
tree ~/.spack/bootstrap/store/
|
|
|
|
ubuntu-gnupg-sources:
|
|
runs-on: ubuntu-latest
|
|
container: "ubuntu:latest"
|
|
steps:
|
|
- name: Install dependencies
|
|
env:
|
|
DEBIAN_FRONTEND: noninteractive
|
|
run: |
|
|
apt-get update -y && apt-get upgrade -y
|
|
apt-get install -y \
|
|
bzip2 curl file g++ gcc patchelf gfortran git gzip \
|
|
make patch unzip xz-utils python3 python3-dev tree \
|
|
gawk
|
|
- name: Checkout
|
|
uses: actions/checkout@2541b1294d2704b0964813337f33b291d3f8596b
|
|
with:
|
|
fetch-depth: 0
|
|
- name: Setup non-root user
|
|
run: |
|
|
# See [1] below
|
|
git config --global --add safe.directory /__w/spack/spack
|
|
useradd spack-test && mkdir -p ~spack-test
|
|
chown -R spack-test . ~spack-test
|
|
- name: Setup repo
|
|
shell: runuser -u spack-test -- bash {0}
|
|
run: |
|
|
git --version
|
|
. .github/workflows/setup_git.sh
|
|
- name: Bootstrap GnuPG
|
|
shell: runuser -u spack-test -- bash {0}
|
|
run: |
|
|
source share/spack/setup-env.sh
|
|
spack solve zlib
|
|
spack bootstrap untrust github-actions-v0.2
|
|
spack -d gpg list
|
|
tree ~/.spack/bootstrap/store/
|
|
|
|
macos-gnupg-binaries:
|
|
runs-on: macos-latest
|
|
steps:
|
|
- name: Install dependencies
|
|
run: |
|
|
brew install tree
|
|
# Remove GnuPG since we want to bootstrap it
|
|
sudo rm -rf /usr/local/bin/gpg
|
|
- name: Checkout
|
|
uses: actions/checkout@2541b1294d2704b0964813337f33b291d3f8596b
|
|
- name: Bootstrap GnuPG
|
|
run: |
|
|
source share/spack/setup-env.sh
|
|
spack bootstrap untrust spack-install
|
|
spack -d gpg list
|
|
tree ~/.spack/bootstrap/store/
|
|
|
|
macos-gnupg-sources:
|
|
runs-on: macos-latest
|
|
steps:
|
|
- name: Install dependencies
|
|
run: |
|
|
brew install gawk tree
|
|
# Remove GnuPG since we want to bootstrap it
|
|
sudo rm -rf /usr/local/bin/gpg
|
|
- name: Checkout
|
|
uses: actions/checkout@2541b1294d2704b0964813337f33b291d3f8596b
|
|
- name: Bootstrap GnuPG
|
|
run: |
|
|
source share/spack/setup-env.sh
|
|
spack solve zlib
|
|
spack bootstrap untrust github-actions-v0.2
|
|
spack -d gpg list
|
|
tree ~/.spack/bootstrap/store/
|
|
|
|
|
|
# [1] Distros that have patched git to resolve CVE-2022-24765 (e.g. Ubuntu patching v2.25.1)
|
|
# introduce breaking behaviorso we have to set `safe.directory` in gitconfig ourselves.
|
|
# See:
|
|
# - https://github.blog/2022-04-12-git-security-vulnerability-announced/
|
|
# - https://github.com/actions/checkout/issues/760
|
|
# - http://changelogs.ubuntu.com/changelogs/pool/main/g/git/git_2.25.1-1ubuntu3.3/changelog
|