* Add template creation test
* Added --skip-editor option to "spack create": normally
"spack create" opens an editor for the user after generating a
package file; when the --skip-editor option is used, "spack create"
only generates the package file and does not open an editor
* Added --skip-editor option to bash completion
- `setup-env.sh` was not properly detecting a bash shell when bash was run
as /bin/sh.
- Detection routine now always reports bash when bash is run as sh, and
no longer parses the path to the executable indicated in `$BASH`.
- Add set -u to the setup-env.sh test script
- Refactor lines in setup-env.sh that tested potentially undefined
variables to use the `[ -z ${var+x} ]` construct
- tests use a shell-script harness and test all Spack commands that
require special shell support.
- tests work in bash, zsh, and dash
- run setup-env.sh tests on macos and linux builds.
- we run them on macos and linux
- replace use of [[ with [
- replace function foo { .. } with foo() { .. }
- wrap some long lines
- add lsof and /proc/fd magic so that we can find the sourced file even in dash
- only do the complicated shell checks in one place; test $_sp_shell
elsewhere.
- We've seen this a few times now where users have set up `cd` to echo
the new directory, and it screws up `setup-env.sh`
- In the past we've said this is user error.
- Here, we just fix it by sending `cd` output to /dev/null where needed.
- this works in bash, zsh, and dash
- Codecov cannot handle as many coverage reports as we are generating
- as a result, our PR coverage pages have been broken for a while, and
it's hard to tell people where to enhance their testing in PR reviews.
- Scale back to only running coverage for 3.7 and 2.7 unit tests
- This is *probably* better. We run the build tests for good measure,
but we do not need to evaluate them for coverage. The coverage reports
are about unit tests.
* Added a function that concretizes specs together
* Specs concretized together are copied instead of being referenced
This makes the specs different objects and removes any reference to the
fake root package that is needed currently for concretization.
* Factored creating a repository for concretization into its own function
* Added a test on overlapping dependencies
Usage of double quotes was preventing word-splitting when parsing
module roots in setup-env.sh, which lead to an error when multiple
module roots are used (in particular when Spack is pointed to use
an upstream module root in addition to its own).
Still look for BASH_SOURCE[0] first, but if it's not set,
_sp_source_file is initialized to an empty value addressing the
unset parameter error (line 217).
* initial work to make use of an 'upstream' spack installation: this uses the DB of the upstream installation to check if a package is installed
* need to query upstream dbs when adding new record to local db
* prevent reindexing upstream DBs
* set prefix on specs read from DB based on path stored in install record
* check that Spack does not install packages that are recorded as installed in an upstream db
* externals do not add their path to install records - need to use 'external_path' to get path of upstream externals
* views need to check for upstream installations when linking metadata
* package and spec now calculate upstream installation properties on-demand themselves rather than depending on concretization to set these properties up-front. The added tests for upstream installations don't work with this new strategy so they need to be updated
* only refresh modules for local specs (not those in upstream packages); optionally generate local module files for packages installed upstream
* when a user tries to locate a module file for a package installed upstream, tell them to use the upstream spack instance to locate it
* support recursive upstream databases (allow upstream databases to use their own upstream databases)
* separate upstream config into separate file with its own schema; each entry now also includes a name
* metadata_dir is no longer customizable on a per-instance basis for YamlDirectoryLayout
* treat metadata_dir as an instance variable but dont set it from kwargs; this follows several other hardcoded variables which must be consistent between upstream and downstream DBs. Also update DirectoryLayout.metadata_path to work entirely with Spec.prefix, since Spec.prefix is set from the DB when available (so metadata_path was duplicating that logic)
This spack command adds a new schema for a file which describes the
builder containers available, along with the compilers availabe on
each builder. The release-jobs command then generates the .gitlab-ci.yml
file by first expanding the release spec set, concretizing each spec
(in an appropriate docker container if --this-machine-only argument is
not provided on command line), and then combining and staging all the
concrete specs as jobs to be run by gitlab.
The built images are set up with fairly recent versions of gcc and
clang:
- centos_7: [ gcc@5.5.0 (built from src), clang@6.0.0 (spack-built from src) ]
- ubuntu_18.04: [ gcc@5.5.0 (system), clang@6.0.0-1ubuntu2 (system) ]
Adds four new sub-commands to the buildcache command:
1. save-yaml: Takes a root spec and a list of dependent spec names,
along with a directory in which to save yaml files, and writes out
the full spec.yaml for each of the dependent specs. This only needs
to concretize the root spec once, then indexes it with the names of
the dependent specs.
2. check: Checks a spec (via either an abstract spec or via a full
spec.yaml) against remote mirror to see if it needs to be rebuilt.
Comparies full_hash stored on remote mirror with full_hash computed
locally to determine whether spec needs to be rebuilt. Can also
generate list of specs to check against remote mirror by expanding
the set of release specs expressed in etc/spack/defaults/release.yaml.
3. get-buildcache-name: Makes it possible to attempt to read directly
the spec.yaml file on a remote or local mirror by providing the path
where the file should live based on concretizing the spec.
4. download: Downloads all buildcache files associated with a spec
on a remote mirror, including any .spack, .spec, and .cdashid files
that might exist. Puts the files into the local path provided on
the command line, and organizes them in the same hierarchy found on
the remote mirror
This commit also refactors lib/spack/spack/util/web.py to expose
functionality allowing other modules to read data from a url.
Spack shell detection in setup-env.sh was originally based on
examining the executable name of $$ (from "ps"). In some cases this
does not actually give the name of the shell used, for example when
setup-env.sh was invoked from a script using "#!". To make shell
detection more robust, this adds a preliminary check for shell
variables which indicate that the shell is bash or zsh; the
executable name of $$ is retained as a fallback if those variables
are not defined.
- currently just looks at patches
- allows you to find out which package applied a patch to a spec
- intended to work with tarballs and resources in the future.
- add tab completion for `spack resource` and subcommands
- fixed an issue where some undesirable parts of
the spack source tree were being copied into
the image context.
- added a workaround for a tty ioctl warning on
ubuntu
- adjusted how the main images are built so that
`RUN spack ...` works automatically for child
images that base themselves on them.
Lately many CI runs for PRs are failing due to the `mpich` build that
times out on Travis (10 mins. without output). As the timeout seems to
happen consistently during the build phase, increasing the verbosity of
that test can help working around the issue.
- `spack env create <name>` works as before
- `spack env create <path>` now works as well -- environments can be
created in their own directories outside of Spack.
- `spack install` will look for a `spack.yaml` file in the current
directory, and will install the entire project from the environment
- The Environment class has been refactored so that it does not depend on
the internal Spack environment root; it just takes a path and operates
on an environment in that path (so internal and external envs are
handled the same)
- The named environment interface has been hoisted to the
spack.environment module level.
- env.yaml is now spack.yaml in all places. It was easier to go with one
name for these files than to try to handle logic for both env.yaml and
spack.yaml.
- `spack env activate foo`: sets SPACK_ENV to the current active env name
- `spack env deactivate`: unsets SPACK_ENV, deactivates the environment
- added support to setup_env.sh and setup_env.csh
- other env commands work properly with SPACK_ENV, as with an environment
arguments.
- command-line --env arguments take precedence over the active
environment, if given.
setup-env includes a call to 'ps' to determine what shell is being
used. 'ps' can be instructed to use a different default output format
via the 'PS_FORMAT' env variable. Thus unset this variable before
calling 'ps'.
* Unite Dockerfiles - add build/run/push scripts
* update docker documentation
* update .travis.yml
* switch to using a preprocessor on Dockerfiles
* skip building docker images on pull requests
* update files with copyright info
* tweak when travis builds for docker files are done
- `spack license list-files`: list all files that should have license headers
- `spack license list-lgpl`: list files still under LGPL-2.1
- `spack license verify`: check that license headers are correct
- Added `spack license verify` to style tests
- remove the old LGPL license headers from all files in Spack
- add SPDX headers to all files
- core and most packages are (Apache-2.0 OR MIT)
- a very small number of remaining packages are LGPL-2.1-only
- Many container builds are timing out frequently during Spack tests in
Travis CI.
- Travis recommends to try `sudo: required` to see whether this is an
infrastructure issue or something else.
- added `sudo: required` to all Linux builds.
- added --verbose to `spack test` invocation so that we can see more
easily what tests it's timing out on.
Signed-off-by: Todd Gamblin <tgamblin@llnl.gov>
As requested in the review all the commands meant to manage module
files have been grouped under the `spack module` command.
Unit tests have been refactored to match the new command structure.
fixes#2215fixes#2570fixes#6676fixes#7281closes#3827
This PR reverts the use of `spack module loads` in favor of
`spack module find` when loading module files via Spack. After this PR
`spack load` will accept a single spec at a time, and will be able
to interpret correctly the `--dependencies` option.
- The setup-env.sh script currently makes two calls to spack, but it
should only need to make one.
- Add a fast-path shell setup routine in `main.py` to allow the shell
setup to happen in a single, fast call that doesn't load more than it
needs to.
- This simplifies setup code, as it has to eval what Spack prints
- TODO: consider eventually making the whole setup script the output of a
spack command
- replace `spack.config.get_configuration()` with `spack.config.config()`
- replace `get_config`/`update_config` with `get`, `set`
- add a path syntax that can be used to refer to specific config options
without firt getting the entire configuration dict
- update usages of `get_config` and `update_config` to use `get` and `set`
fixes#7593
Unit tests on OSX are trying to concretize mpileaks, and they fail due
to a conflict in the package:
"%gcc@7.2.0:" conflicts with "elfutils@0.163"
This solves the issue asking explicitly to concretize against
elfutils@1.170
Fixes#7593
By default MacOS concretizes using the clang compiler. The unit tests
include a call to "spack spec mpileaks", which has elfutils as a
dependency; #7096 added a conflict in elfutils to avoid building
with clang, which lead to the MacOS unit tests to start failing.
This updates the concretization to force using gcc when concretizing
mpileaks.
* Revert "Travis: use --concurrency=multiprocessing only on build tests (#6872)"
This reverts commit 596d463714.
* Removing 'coverage combine' in test script
According to what was discovered in #6887, one of the problems is
calling 'coverage combine' twice without the '-a' flag. This removes
the first call within our test scripts.
On a local workstation, it seems that tracking multiple processes during
coverage may result in malformed coverage reports for unit tests and not
for build tests.
Given that multiple processes make a difference in coverage mainly for
build tests, try to disable the tracking for unit tests to see if we get
more stable coverage results.
* Reduce the calls to the python interpreter during initialization
This should reduce the delay the users experience when sourcing the
setup file to activate shell support. It works by generating at once
all the commands that needs to evaluated (they are stored in
a string and later `eval`ed by the shell).
* setup_env.sh: changed `read` with an equivalent magic
For some reason `read` breaks when sourced from a running script.
Change the incantation we use to construct the unique python command
that will be evaluated.
* setup_env.sh: python command now constructed with `printf` for portability
This recovers the support for `zsh` that was broken in previous commits.
* Reworked module file tutorial section
First draft for the SC17 update. This includes:
- adding an introduction on module files + Spack's module
generation blueprints
- adding a set-up section and provide a docker image for easy set-up
- updating all the relevant snippets
- extending a bit some of the concepts that were already touched
* Added reference to #5582 + committed Dockerfiles
Also fixed a couple of typos spotted by Denis.
* module file tutorial: added section on template customization
* module file tutorial: fixed minor typos + rephrased a sentence
* module file tutorial: made explicit that Docker image comes with software
* module file tutorial: improved phrasing and layout.
Thanks Hartzell!
* module file tutorial: added vim and nano to editors
* module file tutorial: fixed typo
* Fixed typos
Thanks Adam!
* module file tutorial: updated Dockerfile + minor changes in introduction
- This isn't one of those autogenerated SVGs from a drawing program!
- This is a completely re-traced, minimalist SVG file with clearly
delineated pieces so that your favorite renderer can draw a Spack logo
at whatever resolution you want.
- Included versions with text, as well.
setup-env.sh adds the 'module' command to the user's environment
if it is not defined and if there is a Spack installation of
environment-modules available. This commit updates that logic to
perform these checks and updates quietly.
These were discovered with bash 4.1.2.
Add quotations around a variable to prevent the destruction of a
newline. Without this fix a conditional doesn't work properly.
Remove square brackets around a conditional meant to be evaluated based
on the return code of a command. This wasn't working properly with an
old bash.
Fix a typo.
Renames the existing bootstrap command to 'clone'. Repurposes
'spack bootstrap' to install packages that are useful to the
operation of Spack (for now this is just environment-modules).
For bash and ksh users running setup-env.sh, if a Spack-installed
instance of environment-modules is detected and environment modules
and dotkit are not externally available, Spack will define the
'module' command in the user's shell to use the environment-modules
built by Spack.
- This should speed-up Travis CI tests and refers to #5049
- Travis uses build-stages to group tests together
- The idea is to let fast tests fail first, then move to longer ones.
- Added external perl to avoid download failure from CPAN and reduce build time
- Disabling perl-dbi: continues to fail with (504 Gateway Time-out) on Travis
- We now cover all the build systems in tests:
- Add back `openblas` to Travis as a separate package.
- Switched `openblas` for `astyle` to build a simpler MakefilePackage.
- Added 'tut' (WafPackage)
- Added 'py-setuptools' (PythonPackage)
- Added 'perl-dbi' (PerlPackage)
- Added 'build_systems' directory to the ones for which we get a summary
- Added 'openjpeg' (CMakePackage)
- Added 'r-rcpp' (RPackage)
- Added comments to build tests to show the covered build system
* Merged 'purge' command with 'clean'. Deleted 'purge'. fixes#2942
'spack purge' has been merged with 'spack clean'. Documentation has been
updated accordingly. The 'clean' and 'purge' behavior are not mutually
exclusive, and they log brief information to tty while they go.
* Fixed a wrong reference to spack clean in the docs
* Added tests for 'spack clean'. Updated bash completion.
* Disable spec colorization when redirecting stdout and add command line flag to re-enable
* Add command line `--color` flag to control output colorization
* Add options to `llnl.util.tty.color` to allow color to be auto/always/never
* Add `Spec.cformat()` function to be used when `format()` should have auto-coloring
