hlrs-software-stack/spack
7
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity
spack/.github/workflows/ci.yaml

94 lines
3.2 KiB
YAML
Raw Normal View History

Reorder workflow execution in GHA (#32183) This patchset refactors our GitHub actions into a single top-level ci workflow that invokes a series of reusable actions. The main goal of this is to be able to easily control which tests run and in what order based on the success or failure of top-level prechecks. Our previous workflows ran in three sets: * nix tests: style and verification first, then linux and macos tests if successful * windows tests: style and verification first, then linux and macos tests if successful * bootstrap tests As a result, the bootstrap tests ran even if the style failed, and style and verification had to run on two different platforms despite running identical checks. I'm relatively sure that's because of the limitation on dependencies between steps in the jobs. Reusable workflows allow us to run the style, verification and now audit checks once, then depending on the results, and the files changed, run the appropriate nix, windows and bootstrap tests. While it saves only a few minutes by itself, this makes it easier to refactor checks to subset tests without having to replicate tests or other workflow components in the future. Co-authored-by: Massimiliano Culpo <massimiliano.culpo@gmail.com>
2022-09-02 21:09:23 +00:00
name: ci
on:
push:
branches:
- develop
- releases/**
pull_request:
branches:
- develop
- releases/**
concurrency:
Make GHA tests parallel by using xdist (#32361) * Add two no-op jobs named "all-prechecks" and "all" These are a suggestion from @tgamblin, they are stable named markers we can use from gitlab and possibly for required checks to make CI more resilient to refactors changing the names of specific checks. * Enable parallel testing using xdist for unit testing in CI * Normalize tmp paths to deal with macos * add -u flag compatibility to spack python As of now, it is accepted and ignored. The usage with xdist, where it is invoked specifically by `python -u spack python` which is then passed `-u` by xdist is the entire reason for doing this. It should never be used without explicitly passing -u to the executing python interpreter. * use spack python in xdist to support python 2 When running on python2, spack has many import cycles unless started through main. To allow that, this uses `spack python` as the interpreter, leveraging the `-u` support so xdist doesn't error out when it unconditionally requests unbuffered binary IO. * Use shutil.move to account for tmpdir being in a separate filesystem sometimes
2022-09-07 18:12:57 +00:00
group: ci-${{github.ref}}-${{github.event.pull_request.number || github.run_number}}
Reorder workflow execution in GHA (#32183) This patchset refactors our GitHub actions into a single top-level ci workflow that invokes a series of reusable actions. The main goal of this is to be able to easily control which tests run and in what order based on the success or failure of top-level prechecks. Our previous workflows ran in three sets: * nix tests: style and verification first, then linux and macos tests if successful * windows tests: style and verification first, then linux and macos tests if successful * bootstrap tests As a result, the bootstrap tests ran even if the style failed, and style and verification had to run on two different platforms despite running identical checks. I'm relatively sure that's because of the limitation on dependencies between steps in the jobs. Reusable workflows allow us to run the style, verification and now audit checks once, then depending on the results, and the files changed, run the appropriate nix, windows and bootstrap tests. While it saves only a few minutes by itself, this makes it easier to refactor checks to subset tests without having to replicate tests or other workflow components in the future. Co-authored-by: Massimiliano Culpo <massimiliano.culpo@gmail.com>
2022-09-02 21:09:23 +00:00
cancel-in-progress: true
jobs:
prechecks:
needs: [ changes ]
uses: ./.github/workflows/valid-style.yml
with:
ci: avoid running coverage on package only PRs (#32573) * ci: remove !docs from "core" filters Written like it is now it causes package only PRs to run with coverage. * Try to skip job under condition, see if the workflow proceed * Try to cancel a running CI job * Simplify linux unit-tests, skip windows unit-tests on package PRs * Reduce the inputs to unit-tests workflow * Move control logic to main workflow, remove inputs * Revert "Move control logic to main workflow, remove inputs" This reverts commit 0c46fece4c49eb7a37585ec3ba651a31d7f958af. * Do not compute "with_coverage" since it's always == to "core" * Remove workflow dispatch from unit tests * Revert "Revert "Move control logic to main workflow, remove inputs"" This reverts commit dd4e4a4e61a825901e736348fd044d37e88c90b5. * Try to skip all from the main workflow * Add back bootstrap to needed checks for "all" * Restore the correct logic for conditionals
2022-09-08 17:58:53 +00:00
with_coverage: ${{ needs.changes.outputs.core }}
Make GHA tests parallel by using xdist (#32361) * Add two no-op jobs named "all-prechecks" and "all" These are a suggestion from @tgamblin, they are stable named markers we can use from gitlab and possibly for required checks to make CI more resilient to refactors changing the names of specific checks. * Enable parallel testing using xdist for unit testing in CI * Normalize tmp paths to deal with macos * add -u flag compatibility to spack python As of now, it is accepted and ignored. The usage with xdist, where it is invoked specifically by `python -u spack python` which is then passed `-u` by xdist is the entire reason for doing this. It should never be used without explicitly passing -u to the executing python interpreter. * use spack python in xdist to support python 2 When running on python2, spack has many import cycles unless started through main. To allow that, this uses `spack python` as the interpreter, leveraging the `-u` support so xdist doesn't error out when it unconditionally requests unbuffered binary IO. * Use shutil.move to account for tmpdir being in a separate filesystem sometimes
2022-09-07 18:12:57 +00:00
audit-ancient-python:
uses: ./.github/workflows/audit.yaml
needs: [ changes ]
with:
ci: avoid running coverage on package only PRs (#32573) * ci: remove !docs from "core" filters Written like it is now it causes package only PRs to run with coverage. * Try to skip job under condition, see if the workflow proceed * Try to cancel a running CI job * Simplify linux unit-tests, skip windows unit-tests on package PRs * Reduce the inputs to unit-tests workflow * Move control logic to main workflow, remove inputs * Revert "Move control logic to main workflow, remove inputs" This reverts commit 0c46fece4c49eb7a37585ec3ba651a31d7f958af. * Do not compute "with_coverage" since it's always == to "core" * Remove workflow dispatch from unit tests * Revert "Revert "Move control logic to main workflow, remove inputs"" This reverts commit dd4e4a4e61a825901e736348fd044d37e88c90b5. * Try to skip all from the main workflow * Add back bootstrap to needed checks for "all" * Restore the correct logic for conditionals
2022-09-08 17:58:53 +00:00
with_coverage: ${{ needs.changes.outputs.core }}
Make GHA tests parallel by using xdist (#32361) * Add two no-op jobs named "all-prechecks" and "all" These are a suggestion from @tgamblin, they are stable named markers we can use from gitlab and possibly for required checks to make CI more resilient to refactors changing the names of specific checks. * Enable parallel testing using xdist for unit testing in CI * Normalize tmp paths to deal with macos * add -u flag compatibility to spack python As of now, it is accepted and ignored. The usage with xdist, where it is invoked specifically by `python -u spack python` which is then passed `-u` by xdist is the entire reason for doing this. It should never be used without explicitly passing -u to the executing python interpreter. * use spack python in xdist to support python 2 When running on python2, spack has many import cycles unless started through main. To allow that, this uses `spack python` as the interpreter, leveraging the `-u` support so xdist doesn't error out when it unconditionally requests unbuffered binary IO. * Use shutil.move to account for tmpdir being in a separate filesystem sometimes
2022-09-07 18:12:57 +00:00
python_version: 2.7
all-prechecks:
needs: [ prechecks ]
runs-on: ubuntu-latest
steps:
- name: Success
run: "true"
Reorder workflow execution in GHA (#32183) This patchset refactors our GitHub actions into a single top-level ci workflow that invokes a series of reusable actions. The main goal of this is to be able to easily control which tests run and in what order based on the success or failure of top-level prechecks. Our previous workflows ran in three sets: * nix tests: style and verification first, then linux and macos tests if successful * windows tests: style and verification first, then linux and macos tests if successful * bootstrap tests As a result, the bootstrap tests ran even if the style failed, and style and verification had to run on two different platforms despite running identical checks. I'm relatively sure that's because of the limitation on dependencies between steps in the jobs. Reusable workflows allow us to run the style, verification and now audit checks once, then depending on the results, and the files changed, run the appropriate nix, windows and bootstrap tests. While it saves only a few minutes by itself, this makes it easier to refactor checks to subset tests without having to replicate tests or other workflow components in the future. Co-authored-by: Massimiliano Culpo <massimiliano.culpo@gmail.com>
2022-09-02 21:09:23 +00:00
# Check which files have been updated by the PR
changes:
runs-on: ubuntu-latest
# Set job outputs to values from filter step
outputs:
bootstrap: ${{ steps.filter.outputs.bootstrap }}
core: ${{ steps.filter.outputs.core }}
packages: ${{ steps.filter.outputs.packages }}
steps:
- uses: actions/checkout@2541b1294d2704b0964813337f33b291d3f8596b # @v2
if: ${{ github.event_name == 'push' }}
with:
fetch-depth: 0
# For pull requests it's not necessary to checkout the code
- uses: dorny/paths-filter@b2feaf19c27470162a626bd6fa8438ae5b263721
id: filter
with:
# See https://github.com/dorny/paths-filter/issues/56 for the syntax used below
# Don't run if we only modified packages in the
# built-in repository or documentation
filters: |
bootstrap:
- '!var/spack/repos/builtin/**'
- 'var/spack/repos/builtin/packages/clingo-bootstrap/**'
- 'var/spack/repos/builtin/packages/clingo/**'
- 'var/spack/repos/builtin/packages/python/**'
- 'var/spack/repos/builtin/packages/re2c/**'
- '!lib/spack/docs/**'
- 'lib/spack/**'
- 'share/spack/**'
- '.github/workflows/bootstrap.yml'
core:
- './!(var/**)/**'
packages:
- 'var/**'
# Some links for easier reference:
#
# "github" context: https://docs.github.com/en/actions/reference/context-and-expression-syntax-for-github-actions#github-context
# job outputs: https://docs.github.com/en/actions/reference/workflow-syntax-for-github-actions#jobsjob_idoutputs
# setting environment variables from earlier steps: https://docs.github.com/en/actions/reference/workflow-commands-for-github-actions#setting-an-environment-variable
#
bootstrap:
if: ${{ github.repository == 'spack/spack' && needs.changes.outputs.bootstrap == 'true' }}
needs: [ prechecks, changes ]
uses: ./.github/workflows/bootstrap.yml
unit-tests:
Make GHA tests parallel by using xdist (#32361) * Add two no-op jobs named "all-prechecks" and "all" These are a suggestion from @tgamblin, they are stable named markers we can use from gitlab and possibly for required checks to make CI more resilient to refactors changing the names of specific checks. * Enable parallel testing using xdist for unit testing in CI * Normalize tmp paths to deal with macos * add -u flag compatibility to spack python As of now, it is accepted and ignored. The usage with xdist, where it is invoked specifically by `python -u spack python` which is then passed `-u` by xdist is the entire reason for doing this. It should never be used without explicitly passing -u to the executing python interpreter. * use spack python in xdist to support python 2 When running on python2, spack has many import cycles unless started through main. To allow that, this uses `spack python` as the interpreter, leveraging the `-u` support so xdist doesn't error out when it unconditionally requests unbuffered binary IO. * Use shutil.move to account for tmpdir being in a separate filesystem sometimes
2022-09-07 18:12:57 +00:00
if: ${{ github.repository == 'spack/spack' && needs.changes.outputs.core == 'true' }}
Reorder workflow execution in GHA (#32183) This patchset refactors our GitHub actions into a single top-level ci workflow that invokes a series of reusable actions. The main goal of this is to be able to easily control which tests run and in what order based on the success or failure of top-level prechecks. Our previous workflows ran in three sets: * nix tests: style and verification first, then linux and macos tests if successful * windows tests: style and verification first, then linux and macos tests if successful * bootstrap tests As a result, the bootstrap tests ran even if the style failed, and style and verification had to run on two different platforms despite running identical checks. I'm relatively sure that's because of the limitation on dependencies between steps in the jobs. Reusable workflows allow us to run the style, verification and now audit checks once, then depending on the results, and the files changed, run the appropriate nix, windows and bootstrap tests. While it saves only a few minutes by itself, this makes it easier to refactor checks to subset tests without having to replicate tests or other workflow components in the future. Co-authored-by: Massimiliano Culpo <massimiliano.culpo@gmail.com>
2022-09-02 21:09:23 +00:00
needs: [ prechecks, changes ]
uses: ./.github/workflows/unit_tests.yaml
windows:
Make GHA tests parallel by using xdist (#32361) * Add two no-op jobs named "all-prechecks" and "all" These are a suggestion from @tgamblin, they are stable named markers we can use from gitlab and possibly for required checks to make CI more resilient to refactors changing the names of specific checks. * Enable parallel testing using xdist for unit testing in CI * Normalize tmp paths to deal with macos * add -u flag compatibility to spack python As of now, it is accepted and ignored. The usage with xdist, where it is invoked specifically by `python -u spack python` which is then passed `-u` by xdist is the entire reason for doing this. It should never be used without explicitly passing -u to the executing python interpreter. * use spack python in xdist to support python 2 When running on python2, spack has many import cycles unless started through main. To allow that, this uses `spack python` as the interpreter, leveraging the `-u` support so xdist doesn't error out when it unconditionally requests unbuffered binary IO. * Use shutil.move to account for tmpdir being in a separate filesystem sometimes
2022-09-07 18:12:57 +00:00
if: ${{ github.repository == 'spack/spack' && needs.changes.outputs.core == 'true' }}
Reorder workflow execution in GHA (#32183) This patchset refactors our GitHub actions into a single top-level ci workflow that invokes a series of reusable actions. The main goal of this is to be able to easily control which tests run and in what order based on the success or failure of top-level prechecks. Our previous workflows ran in three sets: * nix tests: style and verification first, then linux and macos tests if successful * windows tests: style and verification first, then linux and macos tests if successful * bootstrap tests As a result, the bootstrap tests ran even if the style failed, and style and verification had to run on two different platforms despite running identical checks. I'm relatively sure that's because of the limitation on dependencies between steps in the jobs. Reusable workflows allow us to run the style, verification and now audit checks once, then depending on the results, and the files changed, run the appropriate nix, windows and bootstrap tests. While it saves only a few minutes by itself, this makes it easier to refactor checks to subset tests without having to replicate tests or other workflow components in the future. Co-authored-by: Massimiliano Culpo <massimiliano.culpo@gmail.com>
2022-09-02 21:09:23 +00:00
needs: [ prechecks ]
uses: ./.github/workflows/windows_python.yml
Make GHA tests parallel by using xdist (#32361) * Add two no-op jobs named "all-prechecks" and "all" These are a suggestion from @tgamblin, they are stable named markers we can use from gitlab and possibly for required checks to make CI more resilient to refactors changing the names of specific checks. * Enable parallel testing using xdist for unit testing in CI * Normalize tmp paths to deal with macos * add -u flag compatibility to spack python As of now, it is accepted and ignored. The usage with xdist, where it is invoked specifically by `python -u spack python` which is then passed `-u` by xdist is the entire reason for doing this. It should never be used without explicitly passing -u to the executing python interpreter. * use spack python in xdist to support python 2 When running on python2, spack has many import cycles unless started through main. To allow that, this uses `spack python` as the interpreter, leveraging the `-u` support so xdist doesn't error out when it unconditionally requests unbuffered binary IO. * Use shutil.move to account for tmpdir being in a separate filesystem sometimes
2022-09-07 18:12:57 +00:00
all:
needs: [ windows, unit-tests, bootstrap, audit-ancient-python ]
runs-on: ubuntu-latest
steps:
- name: Success
run: "true"
Reference in a new issue Copy permalink