Commit graph

323 commits

Author SHA1 Message Date
Patrick Gartung
8c41173678
Buildcache: bindist test without invoking spack compiler wrappers. (#15687)
* Buildcache:
   * Try mocking an install of quux, corge and garply using prebuilt binaries
   * Put patchelf install after ccache restore
   * Add script to install patchelf from source so it can be used on Ubuntu:Trusty which does not have a patchelf pat package. The script will skip building on macOS
   * Remove mirror at end of bindist test
   * Add patchelf to Ubuntu build env
   * Revert mock patchelf package to allow other tests to run.
   * Remove depends_on('patchelf', type='build') relying instead on
   * Test fixture to ensure patchelf is available.

* Call g++ command to build libraries directly during test build

* Flake8

* Install patchelf in before_install stage using apt unless on Trusty where a build is done.

* Add some symbolic links between packages

* Flake8

* Flake8:

* Update mock packages to write their own source files

* Create the stage because spec search does not create it any longer

* updates after change of list command arguments

* cleanup after merge

* flake8
2020-07-08 15:05:58 -05:00
Adam J. Stewart
207e496162
spack create: ask how many to download (#17373) 2020-07-08 09:38:42 +02:00
Todd Gamblin
c00a05bfba bugfix: no infinite recursion in setup-env.sh on Cray
On Cray platforms, we rely heavily on the module system to figure out
what targets, compilers, etc. are available. This unfortunately means
that we shell out to the `module` command as part of platform
initialization.

Because we run subcommands in a shell, we can get infinite recursion if
`setup-env.sh` and friends are in some init script like `.bashrc`.

This fixes the infinite loop by adding guards around `setup-env.sh`,
`setup-env.csh`, and `setup-env.fish`, to prevent recursive
initializations of Spack. This is safe because Spack never shells out to
itself, so we do not need it to be initialized in subshells.

- [x] add recursion guard around `setup-env.sh`
- [x] add recursion guard around `setup-env.csh`
- [x] add recursion guard around `setup-env.fish`
2020-07-06 13:55:14 -07:00
Massimiliano Culpo
a5eabfad91
Moved flake8, shell and documentation tests to Github Action (#17328)
* Move flake8 tests on Github Actions

* Move shell test to Github Actions

* Moved documentation build to Github Action

* Don't run coverage on Python 2.6

Since we get connection errors consistently on Travis
when trying to upload coverage results for Python 2.6,
avoid computing coverage entirely to speed-up tests.
2020-07-01 11:58:53 -05:00
Glenn Johnson
37d7b5b199
Activate environment in container file (#17316)
* Activate environment in container file

This PR will ensure that the container recipes will build the spack
environment by first activating the environment.

* Deactivate environment before environment collection

For Singularity, the environment must be deactivated before running the
command to collect the environment variables. This is because the
environment collection uses `spack env activate`.
2020-06-30 18:17:58 -05:00
Johannes Blaschke
1d55adfd2b
Add fish shell support (#9279)
* share/spack/setup-env.fish file to setup environment in fish shell

* setup-env.fish testing script

* Update share/spack/setup-env.fish

Co-Authored-By: Elsa Gonsiorowski, PhD <gonsie@me.com>

* Update share/spack/qa/setup-env-test.fish

Co-Authored-By: Adam J. Stewart <ajstewart426@gmail.com>

* updates completions using `spack commands --update-completion`

* added stderr-nocaret warning

* added fish shell tests to CI system


Co-authored-by: becker33 <becker33@llnl.gov>
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
Co-authored-by: Elsa Gonsiorowski, PhD <gonsie@me.com>
2020-06-30 14:26:27 -05:00
Greg Becker
d71fdc9719
remove three commands that have been deprecated since v0.13.0 (#17291)
* remove three commands that have been deprecated since v0.13.0
2020-06-29 11:15:56 -05:00
Scott Wittenburg
dfac09eade
Use json for buildcache index (#15002)
* Start moving toward a json buildcache index

* Add spec and database index schemas

* Add a schema for buildcache spec.yaml files

* Provide a mode for database class to generate buildcache index

* Update db and ci tests to validate object w/ new schema

* Remove unused temporary upload-s3 command

* Use database class to generate buildcache index

* Do not generate index with each buildcache creation

* Make buildcache index mode into a couple of constructor args to Database class

* Use keyword args for  _createtarball 

* Parse new json index when we get specs from buildcache

Now that only one index file per mirror needs to be fetched in
order to have all the concrete specs for binaries available on the
mirror, we can just fetch and refresh the cached specs every time
instead of needing to use the '-f' flag to force re-reading.
2020-06-26 17:05:56 -05:00
Omar Padron
7c54aa2eb0
add workaround for gitlab ci needs limit (#17219)
* add workaround for gitlab ci needs limit

* fix style/address review comments

* convert filter obj to list

* update command completion

* remove dict comprehension

* add workaround tests

* fix sorting issue between disparate types

* add indeces to format
2020-06-25 14:27:20 -04:00
Greg Becker
b26e93af3d
spack config: new subcommands add/remove (#13920)
spack config add <value>: add nested value value to the configuration scope specified
spack config remove/rm: remove specified configuration from the relevant scope
2020-06-25 09:38:01 +02:00
Tamara Dahlgren
48d3e8d350
features: Add install failure tracking removal through spack clean (#15314)
* Add ability to force removal of install failure tracking data through spack clean

* Add clean failures option to packaging guide
2020-06-24 20:28:53 -05:00
Tamara Dahlgren
96932d65a8 Added support for --fail-fast install option to terminate on first failure 2020-06-23 10:22:41 -07:00
Massimiliano Culpo
239b709f97
Added unit tests to Github Actions (#16610)
* Added unit tests to Github Actions

* Set user e-mail and name for git tests to succeed

* Simplify setup.sh logic

* Replicate Travis script on Github Actions

* Update flags since '.' is not allowed

* Added badge, simplified workflow

* Remove pinning of coverage

* Remove unit tests run on Github Actions from Travis
2020-06-23 08:24:02 -05:00
Omar Padron
224dc95159
Pre ci optimization (#16372)
* add initial optimization script

* integrate optimization in spack ci

* make optimization opt-in

* fix import error

* flake8 fixes

* update command completion

* work around vermin errors

* fix sphynx errors
2020-06-22 13:19:47 -04:00
Omar Padron
82eb71cb95
Explicitly install setuptools in docker images (#17143) 2020-06-18 11:16:24 +02:00
Omar Padron
e816b66c94
fix docker image entrypoints (#17105)
Also removes extraneous prompt and ssh handling logic.
2020-06-16 14:11:13 -04:00
Massimiliano Culpo
5b272e3ff3
commands: use a single ThreadPool for spack versions (#16749)
This fixes a fork bomb in `spack versions`. Recursive generation of pools
to scrape URLs in `_spider` was creating large numbers of processes.
Instead of recursively creating process pools, we now use a single
`ThreadPool` with a concurrency limit.

More on the issue: having ~10 users running at the same time spack
versions on front-end nodes caused kernel lockup due to the high number
of sockets opened (sys-admin reports ~210k distributed over 3 nodes).
Users were internal, so they had ulimit -n set to ~70k.

The forking behavior could be observed by just running:

    $ spack versions boost

and checking the number of processes spawned. Number of processes
per se was not the issue, but each one of them opens a socket
which can stress `iptables`.

In the original issue the kernel watchdog was reporting:

    Message from syslogd@login03 at May 19 12:01:30 ...
    kernel:Watchdog CPU:110 Hard LOCKUP
    Message from syslogd@login03 at May 19 12:01:31 ...
    kernel:watchdog: BUG: soft lockup - CPU#110 stuck for 23s! [python3:2756]
    Message from syslogd@login03 at May 19 12:01:31 ...
    kernel:watchdog: BUG: soft lockup - CPU#94 stuck for 22s! [iptables:5603]
2020-06-05 00:08:32 -07:00
Peter Scheibel
24775697f5
Mirrors: add option to exclude packages from "mirror create" (#14154)
* add an --exclude-file option to 'spack mirror create' which allows a user to specify a file of specs to exclude when creating a mirror. this is anticipated to be useful especially when using the '--all' option

* allow specifying number of versions when mirroring all packages

* when mirroring all specs within an environment, include dependencies of root specs

* add '--exclude-specs' option to allow user to specify that specs should be excluded on the command line

* add test for excluding specs
2020-06-03 17:43:51 -07:00
Greg Becker
3347ef2de4
Feature: add option to create view by copying/relocating files (#16480)
* add subcommand `spack view copy/relocate`

* update bash completions

* add copy/relocate commands to view tests

* allow copied views to be removed
2020-06-03 09:45:13 -07:00
Scott Wittenburg
e0572a7d96 Pipelines: Support DAG scheduling and dynamic child pipelines
This change also adds a code path through the spack ci pipelines
infrastructure which supports PR testing on the Spack repository.
Gitlab pipelines run as a result of a PR (either creation or pushing
to a PR branch) will only verify that the packages in the environment
build without error.  When the PR branch is merged to develop,
another pipeline will run which results in the generated binaries
getting pushed to the binary mirror.
2020-05-14 21:11:07 -07:00
Ben Bergen
37e307e8cd
Added alias and bash completion for spacktivate (#16472) 2020-05-13 12:02:38 -06:00
Massimiliano Culpo
11fa61665f
travis: use bionic as default for Linux (#16521)
Modifications:

- [x] Travis now uses `bionic` as a default (`xenial` used for Python 3.5, `trusty` for Python 2.6)
- [x] Shell unit tests have been factored into their own run
- [x] `kcov` is built only for tests that upload coverage results

Overall with this we shave 3-4 mins. on each run and add an additional run of about 3 min. For some reason `kcov` 38 fails forwarding output when used with Python unit tests, so I used v34 for that and v38 (latest) for shell testing. Previously we were using v25.
2020-05-12 13:56:01 -07:00
Massimiliano Culpo
43c9ad3421
Remove 'spack bootstrap' and associated docs (#15179)
fixes #15145

This commit removes the outdated `spack bootstrap`
command and any reference to it in the documentation
and unit tests.
2020-05-11 10:55:18 -07:00
iarspider
08f449ae9a
"spack checksum" QoL (#14311)
* Non-interactive mode for spack checksum; allow passing 'package@version' to spack checksum

* Flake8 fixes

* Update checksum.py

Fix typo

* Update spack-completion script

* Automatically set non-interactive mode if more than one version passed

* Update lib/spack/spack/cmd/checksum.py

Co-Authored-By: Adam J. Stewart <ajstewart426@gmail.com>

* Add documentation and update spack-completion

* Flake8

* Rename option

* Update spack-completion

* Update lib/spack/spack/cmd/checksum.py

Co-Authored-By: Adam J. Stewart <ajstewart426@gmail.com>

* Update checksum.py

* Update stage.py

* Update create.py

Use batch mode when adding a new package

Co-authored-by: Ivan Razumov <ivan.razumov@cern.ch>
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
2020-05-07 18:34:36 -05:00
Peter Scheibel
b030a81a5f
Automatically find externals (#15158)
Add a `spack external find` command that tries to populate
`packages.yaml` with external packages from the user's `$PATH`. This
focuses on finding build dependencies. Currently, support has only been
added for `cmake`.

For a package to be discoverable with `spack external find`, it must define:
  * an `executables` class attribute containing a list of
    regular expressions that match executable names.
  * a `determine_spec_details(prefix, specs_in_prefix)` method

Spack will call `determine_spec_details()` once for each prefix where
executables are found, passing in the path to the prefix and the path to
all found executables. The package is responsible for invoking the
executables and figuring out what type of installation(s) are in the
prefix, and returning one or more specs (each with version, variants or
whatever else the user decides to include in the spec).

The found specs and prefixes will be added to the user's `packages.yaml`
file. Providing the `--not-buildable` option will mark all generated
entries in `packages.yaml` as `buildable: False`
2020-05-05 17:37:34 -07:00
Axel Huebl
d0dfa1ea4d
dev-build: --drop-in <shell> (#14887)
* dev-build: --drop-in <shell>

Add a `--drop-in <shell>` option to `spack dev-build`.
This option will automatically run a
`spack build-env <spec> -- <shell>` at the end of a `dev-build`, e.g.
to quickly drop-and-devel into a build phase of a package.

Example usage:
```
spack dev-build --before cmake --drop-in bash openpmd-api@develop
```

* build_env: drop in unit test

Co-authored-by: Greg Becker <becker33@llnl.gov>
2020-05-01 09:37:21 -07:00
Axel Huebl
00d83cd79d
dev-build: stop before phase (#14699)
Add `-b,--before` option to dev-build command to stop before the phase in question.
2020-04-28 09:55:57 -07:00
Massimiliano Culpo
19c5843871
Travis: change the spec we concretize on MacOS (#16256)
Since CMake can't build with GCC on MacOS, choose a
spec that doesn't have CMake in the DAG.
2020-04-23 11:40:23 +02:00
Todd Gamblin
de13ea2284
autotools: remove FORCE_UNSAFE_CONFIGURE from docs, commands, code (#16145)
Since #16132, we've consolidated the setting of FORCE_UNSAFE_CONFIGURE to
`autotools.py`, so we don't need to use it in packages like `coreutils`,
in our commands, or in our container recipes.

- [x] Remove FORCE_UNSAFE_CONFIGURE from packages
- [x] Remove FORCE_UNSAFE_CONFIGURE from container recipes
- [x] Remove FORCE_UNSAFE_CONFIGURE from `spack ci` command
2020-04-20 08:32:03 -07:00
Todd Gamblin
f6d26db939 macos: use DYLD_FALLBACK_LIBRARY_PATH instead of DYLD_LIBRARY_PATH
`DYLD_LIBRARY_PATH` can frequently break builtin macOS software when
pointed at Spack libraries.  This is because it takes *higher* precedence
than the default library search paths, which are used by system software.

`DYLD_FALLBACK_LIBRARY_PATH`, on the other hand, takes lower precedence.
At first glance, this might seem bad, because the software installed by
Spack in an environment needs to find *its* libraries, and it should not
use the defaults.  However, Spack's isntallations are always `RPATH`'d,
so they do not have this problem.

`DYLD_FALLBACK_LIBRARY_PATH` is thus useful for things built in an
environment that need to use Spack's libraries, that don't set *their*
RPATHs correctly for whatever reason. We now prefer it to
`DYLD_LIBRARY_PATH` in modules and in environments because it helps a
little bit, and it is much less intrusive.
2020-04-16 17:23:33 -07:00
G-Ragghianti
bae4f91bfe
Add option "--first" for "spack load" (#15622)
* Implemented --first option for "spack load"

* added test for "spack load --first"

Co-authored-by: gragghia <gragghia@localhost.localdomain>
2020-04-03 13:33:20 -07:00
Adam J. Stewart
75a2f8046a
Add commands to facilitate Spack/Python/OS reporting (#15834)
* Add --version arg to spack python command
* Add `spack debug report` command
2020-04-02 23:12:03 -07:00
Adam J. Stewart
a4b3edd68a
Allow Spack Environments with '-h' in the name (#15429)
If a user invoked "spack env activate example-henv", Spack would
mistakenly interpret the "-h" from "example-henv" as the "-h" option.
This commit allows users to create and activate environments with
"-h" in the name.

This issue existed for bash shell support as well as csh support, and
this commit addresses both, along with some other unrelated csh
support issues.
2020-03-31 16:57:14 -07:00
Peter Scheibel
697719c181
Only use stable versions for public mirror (#15100)
* add --skip-unstable-versions option to 'spack mirror create' which skips sources/resource for packages if their version is not stable (i.e. if they are the head of a git branch rather than a fixed commit)

* '--skip-unstable-versions' should skip all VCS sources/resources, not just those which are not cachable
2020-03-07 13:38:08 +01:00
Tamara Dahlgren
b2e7e7edaa
Recover coverage from subprocesses during unit tests (#15354)
* Recover coverage from subprocesses during unit tests
2020-03-05 16:54:29 -08:00
Patrick Gartung
ae87828520
buildcache cmd: add explicit message with default output dir for buildcaches. (#15090)
* Make -d directory a required option. Print messages about where buildcaches will be written.

* Add mutually exclusive required options

* spack commands --update-completion

* Apply @opadron's patch

* Update share/spack/spack-completion.bash

* Incorporate @opadron's suggestions
2020-02-25 17:49:25 -06:00
Omar Padron
00090f8f97
add --only option to buildcache create cmd (#14921)
* add --only option to buildcache create cmd

replaces the --no-deps option
2020-02-25 17:32:20 -05:00
Patrick Gartung
676eb56ab2
Buildcache cmd: add install -o/--otherarch option for installing macOS buildcaches on linux (#15192)
* Buildcache command: add install option -o/--otherarch
This will allow matching specs from other archs, for example
installing macOS buildcaches on linux hosts.

* spack commands --update-completion
2020-02-25 11:01:59 -06:00
Adam J. Stewart
342200774b
spack extensions prints list of extendable packages (#14473)
* spack extensions prints list of extendable packages

* Update tab completion scripts
2020-02-17 17:41:47 -06:00
Adam J. Stewart
e0dfc3ddbf
Fix shell detection: zsh5 -> zsh (#14858) 2020-02-13 20:07:09 -06:00
Todd Gamblin
a7b43f1015 spack python: add -m option to run modules as scripts
It's often useful to run a module with `python -m`, e.g.:

    python -m pyinstrument script.py

Running a python script this way was hard, though, as `spack python` did
not have a similar `-m` option.  This PR adds a `-m` option to `spack
python` so that we can do things like this:

    spack python -m pyinstrument ./test.py

This makes it easy to write a script that uses a small part of Spack and
then profile it.  Previously thee easiest way to do this was to write a
custom Spack command, which is often overkill.
2020-02-12 16:45:41 -08:00
Oliver Breitwieser
22c9f5cbd8
Allow installing unsigned binary packages (#11107)
This commit introduces a `--no-check-signature` option for
`spack install` so that unsigned packages can be installed. It is
off by default (signatures required).
2020-02-06 18:59:16 -08:00
Patrick Gartung
5ad44477b2
buildcache list: restore original behavior of allowing constraints like @version. (#14732) 2020-02-03 13:40:14 -06:00
Massimiliano Culpo
9635ff3d20
spack containerize generates containers from envs (#14202)
This PR adds a new command to Spack:
```console
$ spack containerize -h
usage: spack containerize [-h] [--config CONFIG]

creates recipes to build images for different container runtimes

optional arguments:
  -h, --help       show this help message and exit
  --config CONFIG  configuration for the container recipe that will be generated
```
which takes an environment with an additional `container` section:
```yaml
spack:
  specs:
  - gromacs build_type=Release 
  - mpich
  - fftw precision=float
  packages:
    all:
      target: [broadwell]

  container:
    # Select the format of the recipe e.g. docker,
    # singularity or anything else that is currently supported
    format: docker
    
    # Select from a valid list of images
    base:
      image: "ubuntu:18.04"
      spack: prerelease

    # Additional system packages that are needed at runtime
    os_packages:
    - libgomp1
```
and turns it into a `Dockerfile` or a Singularity definition file, for instance:
```Dockerfile
# Build stage with Spack pre-installed and ready to be used
FROM spack/ubuntu-bionic:prerelease as builder

# What we want to install and how we want to install it
# is specified in a manifest file (spack.yaml)
RUN mkdir /opt/spack-environment \
&&  (echo "spack:" \
&&   echo "  specs:" \
&&   echo "  - gromacs build_type=Release" \
&&   echo "  - mpich" \
&&   echo "  - fftw precision=float" \
&&   echo "  packages:" \
&&   echo "    all:" \
&&   echo "      target:" \
&&   echo "      - broadwell" \
&&   echo "  config:" \
&&   echo "    install_tree: /opt/software" \
&&   echo "  concretization: together" \
&&   echo "  view: /opt/view") > /opt/spack-environment/spack.yaml

# Install the software, remove unecessary deps and strip executables
RUN cd /opt/spack-environment && spack install && spack autoremove -y
RUN find -L /opt/view/* -type f -exec readlink -f '{}' \; | \
    xargs file -i | \
    grep 'charset=binary' | \
    grep 'x-executable\|x-archive\|x-sharedlib' | \
    awk -F: '{print $1}' | xargs strip -s


# Modifications to the environment that are necessary to run
RUN cd /opt/spack-environment && \
    spack env activate --sh -d . >> /etc/profile.d/z10_spack_environment.sh

# Bare OS image to run the installed executables
FROM ubuntu:18.04

COPY --from=builder /opt/spack-environment /opt/spack-environment
COPY --from=builder /opt/software /opt/software
COPY --from=builder /opt/view /opt/view
COPY --from=builder /etc/profile.d/z10_spack_environment.sh /etc/profile.d/z10_spack_environment.sh

RUN apt-get -yqq update && apt-get -yqq upgrade                                   \
 && apt-get -yqq install libgomp1 \
 && rm -rf /var/lib/apt/lists/*

ENTRYPOINT ["/bin/bash", "--rcfile", "/etc/profile", "-l"]
```
2020-01-30 17:19:55 -08:00
Patrick Gartung
23a7feb917
Limit the number of spec files downloaded to find matches for buildcaches (#14659)
* Limit the number of spec flies downloaded to find matches
2020-01-30 10:56:10 -06:00
Adam J. Stewart
dcd8d7a620 Add spack config list command for tab completion (#14474)
* Add spack config list command for tab completion
* Update tab completion scripts
2020-01-24 17:28:20 -08:00
Todd Gamblin
04a6a55cf8
commands: add simple spack commands --update-completion argument (#14607)
Instead of another script, this adds a simple argument to `spack
commands` that updates the completion script.  Developers can now just
run:

    spack commands --update-completion

This should make it simpler for developers to remember to run this
*before* the tests fail.  Also, this version tab-completes.
2020-01-23 14:48:06 -08:00
Greg Becker
c9e01ff9d7 shell support: spack load no longer needs modules (#14062)
Previously the `spack load` command was a wrapper around `module load`. This required some bootstrapping of modules to make `spack load` work properly.

With this PR, the `spack` shell function handles the environment modifications necessary to add packages to your user environment. This removes the dependence on environment modules or lmod and removes the requirement to bootstrap spack (beyond using the setup-env scripts).

Included in this PR is support for MacOS when using Apple's System Integrity Protection (SIP), which is enabled by default in modern MacOS versions. SIP clears the `LD_LIBRARY_PATH` and `DYLD_LIBRARY_PATH` variables on process startup for executables that live in `/usr` (but not '/usr/local', `/System`, `/bin`, and `/sbin` among other system locations. Spack cannot know the `LD_LIBRARY_PATH` of the calling process when executed using `/bin/sh` and `/usr/bin/python`. The `spack` shell function now manually forwards these two variables, if they are present, as `SPACK_<VAR>` and recovers those values on startup.

- [x] spack load/unload no longer delegate to modules
- [x] refactor user_environment modification calculations
- [x] update documentation for spack load/unload

Co-authored-by: Todd Gamblin <tgamblin@llnl.gov>
2020-01-22 22:36:02 -08:00
Adam J. Stewart
11f2b61261 Use spack commands --format=bash to generate shell completion (#14393)
This PR adds a `--format=bash` option to `spack commands` to
auto-generate the Bash programmable tab completion script. It can be
extended to work for other shells.

Progress:

- [x] Fix bug in superclass initialization in `ArgparseWriter`
- [x] Refactor `ArgparseWriter` (see below)
- [x] Ensure that output of old `--format` options remains the same
- [x] Add `ArgparseCompletionWriter` and `BashCompletionWriter`
- [x] Add `--aliases` option to add command aliases
- [x] Standardize positional argument names
- [x] Tests for `spack commands --format=bash` coverage
- [x] Tests to make sure `spack-completion.bash` stays up-to-date
- [x] Tests for `spack-completion.bash` coverage
- [x] Speed up `spack-completion.bash` by caching subroutine calls

This PR also necessitates a significant refactoring of
`ArgparseWriter`. Previously, `ArgparseWriter` was mostly a single
`_write` method which handled everything from extracting the information
we care about from the parser to formatting the output. Now, `_write`
only handles recursion, while the information extraction is split into a
separate `parse` method, and the formatting is handled by `format`. This
allows subclasses to completely redefine how the format will appear
without overriding all of `_write`.

Co-Authored-by: Todd Gamblin <tgamblin@llnl.gov>
2020-01-22 21:31:12 -08:00
Tom Scogland
df8ee438e5 stop word splitting from leaking out of setup-env (#14472)
The pathadd function was using setopt to configure zsh for word
splitting, which leaks out of the function and breaks default
functionality in a number of external zsh plugins and packages.  This
switches to emulate -L, just as the spack function uses, to keep the
setting local to the function.
2020-01-14 15:32:57 -08:00
Adam J. Stewart
257e71d87a
Reformat Bash tab completion script (#14456) 2020-01-10 11:32:50 -06:00
Adam J. Stewart
eddb42ed43
Fix outdated bash tab completion (#14392) 2020-01-06 23:18:14 -06:00
Todd Gamblin
4beb9fc5d3 tests: improved spack test command line options
Previously, `spack test` automatically passed all of its arguments to
`pytest -k` if no options were provided, and to `pytest` if they were.
`spack test -l` also provided a list of test filenames, but they didn't
really let you completely narrow down which tests you wanted to run.

Instead of trying to do our own weird thing, this passes `spack test`
args directly to `pytest`, and omits the implicit `-k`.  This means we
can now run, e.g.:

```console
$ spack test spec_syntax.py::TestSpecSyntax::test_ambiguous
```

This wasn't possible before, because we'd pass the fully qualified name
to `pytest -k` and get an error.

Because `pytest` doesn't have the greatest ability to list tests, I've
tweaked the `-l`/`--list`, `-L`/`--list-long`, and `-N`/`--list-names`
options to `spack test` so that they help you understand the names
better.  you can combine these options with `-k` or other arguments to do
pretty powerful searches.

This one makes it easy to get a list of names so you can run tests in
different orders (something I find useful for debugging `pytest` issues):

```console
$ spack test --list-names -k "spec and concretize"
cmd/env.py::test_concretize_user_specs_together
concretize.py::TestConcretize::test_conflicts_in_spec
concretize.py::TestConcretize::test_find_spec_children
concretize.py::TestConcretize::test_find_spec_none
concretize.py::TestConcretize::test_find_spec_parents
concretize.py::TestConcretize::test_find_spec_self
concretize.py::TestConcretize::test_find_spec_sibling
concretize.py::TestConcretize::test_no_matching_compiler_specs
concretize.py::TestConcretize::test_simultaneous_concretization_of_specs
spec_dag.py::TestSpecDag::test_concretize_deptypes
spec_dag.py::TestSpecDag::test_copy_concretized
```

You can combine any list option with keywords:

```console
$ spack test --list -k microarchitecture
llnl/util/cpu.py  modules/lmod.py
```

```console
$ spack test --list-long -k microarchitecture
llnl/util/cpu.py::
    test_generic_microarchitecture

modules/lmod.py::TestLmod::
    test_only_generic_microarchitectures_in_root
```

Or just list specific files:

```console
$ spack test --list-long cmd/test.py
cmd/test.py::
    test_list                       test_list_names_with_pytest_arg
    test_list_long                  test_list_with_keywords
    test_list_long_with_pytest_arg  test_list_with_pytest_arg
    test_list_names
```

Hopefully this stuff will help with debugging test issues.

- [x] make `spack test` send args directly to `pytest` instead of trying
  to do fancy things.
- [x] rework `--list`, `--list-long`, and add `--list-names` to make
  searching for tests easier.
- [x] make it possible to mix Spack's list args with `pytest` args
  (they're just fancy parsing around `pytest --collect-only`)
- [x] add docs
- [x] add tests
- [x] update spack completion
2020-01-01 21:37:02 -08:00
Todd Gamblin
4af6303086
copyright: update copyright dates for 2020 (#14328) 2019-12-30 22:36:56 -08:00
Johannes Blaschke
c0d5c360d5 setup-env-test: fix pipe redirect (#14306) 2019-12-27 22:37:47 -08:00
Massimiliano Culpo
2aa8132afd Migrate build tests from Travis to Github Actions (#13967)
This PR moves build smoke tests from TravisCI and migrates them to Github Actions. The result is that build tests are performed in parallel with unit tests and they don't hog additional resources on Travis. The workflow will not run if a PR only changes packages in the built-in repository, but will always run on pushes to develop or master.

* Removed build tests from Travis and passed them to Github Actions
* Store ~/.ccache in Github Actions cache
* Add filters on paths and make sure this workflow don't run
* Use paths-ignore and exclude only files in the built-in repo
* Added a badge to README.md
2019-12-25 00:06:48 -08:00
Massimiliano Culpo
9ceec7e219 Harden shell detection when procfs is available (#13950) 2019-12-16 14:51:58 -08:00
Massimiliano Culpo
f80491826b
Travis exits at the first failing test, pin codecov at v4.5.4 (#14179)
Before this commit we used to run the entire unit test suite
in the presence of a failure. Since we currently rely a lot
on the state of the filesystem etc. the end report was most
of the time showing spurious failures that were a consequence
of the first failing test.

This PR makes unit tests exit at the first failing test

Also, pin codecov at v4.5.4 (last one supporting Python 2.6)
2019-12-16 10:56:54 +01:00
Scott Wittenburg
2520806df2 docker: add file, adjust locale, and use python3 for ubuntu (#13508)
* docker: add missing module to ubuntu images
* docker: fix issue with missing locale
* docker: one package per line + rm python2 support
* docker: ubuntu image also needs 'file' for buildcache creation
2019-12-13 10:22:20 -08:00
Zack Galbreath
0f5724e908 Split out CDash options to a separate help document (#13704)
Prevent `spack help install` from getting too cluttered with CDash-specific documentation.
2019-12-13 10:15:22 -08:00
Axel Huebl
7a81c37bde
Package Index: Build in Dockerhub (#13810)
* Package Index: Build in Dockerhub

Prepare to build the package index service, packages.spack.io,
on Dockerhub.

Local build (in spack root dir):
```
docker build -t spack/packages.spack.io:latest -f share/spack/docker/package-index/Dockerfile .
```

Local test:
```
docker run -p 8080:80 spack/packages.spack.io:latest
```

* Travis-CI: Remove Docker

Remove leftover docker stages from Travis-CI.

* Simplify Split Call
2019-11-26 10:11:29 -07:00
Greg Becker
6c55a7c85f
cmd/install: remove unused install_status option (#13751)
* cmd/install: remove unused install_status option

* update bash completions for spack install
2019-11-22 11:17:37 -07:00
Todd Gamblin
e0b94dba14
completion: add bash completion for spack spec --json (#13433) 2019-10-25 11:02:52 -07:00
Massimiliano Culpo
3d77ecd92e Bootstrap environment modules optimizing for generic architectures (#13105)
fixes #13073

Since #3206 was merged bootstrapping environment-modules was using the architecture of the current host or the best match supported by the default compiler. The former case is an issue since shell integration was looking for a spec targeted at the host microarchitecture.

1. Bootstrap an env modules targeted at generic architectures
2. Look for generic targets in shell integration scripts
3. Add a new entry in Travis to test shell integration
2019-10-21 11:20:05 -07:00
Massimiliano Culpo
d33b0ffc50 lmod: module files are written in a root folder named by target family (#13121)
fixes #13005

This commit fixes an issue with the name of the root directory for
module file hierarchies. Since #3206 the root folder was named after
the microarchitecture used for the spec, which is too specific and
not backward compatible for lmod hierarchies. Here we compute the
root folder name using the target family instead of the target name
itself and we add target information in the 'whatis' portion of the
module file.
2019-10-15 11:20:49 -07:00
Massimiliano Culpo
76b9c56110 Remove support for generating dotkit files (#11986)
Dotkit is being used only at a few sites and has been deprecated on new
machines. This commit removes all the code that provide support for the
generation of dotkit module files.

A new validator named "deprecatedProperties" has been added to the
jsonschema validators. It permits to prompt a warning message or exit
with an error if a property that has been marked as deprecated is
encountered.

* Removed references to dotkit in the docs
* Removed references to dotkit in setup-env-test.sh
* Added a unit test for the 'deprecatedProperties' schema validator
2019-10-02 22:15:01 -07:00
Massimiliano Culpo
1b18ec90ab Add all compatible system types directory to module paths
fixes #12915
closes #12916

Since Spack has support for specific targets it might happen that
software is built for targets that are not exactly the host because
it was either an explicit user request or the compiler being used is
too old to support the host.

Modules for different targets are written into different directories
and by default Spack was adding to MODULEPATH only the directory
corresponding to the current host. This PR modifies this behavior to
add all the directories that are **compatible** with the current host.
2019-10-01 19:18:27 -07:00
Glenn Johnson
5397606b34 Add --known-targets to bash completion for arch command (#12887)
This PR adds the new --known-targets flag to the `spack arch` command.
2019-09-20 20:09:44 +02:00
Scott Wittenburg
fabbb3d58a Refactor release-jobs cmd based on use of environments (no docker either) 2019-09-13 22:57:15 -07:00
Tyler Reddy
25dbc9fb8e BUG: recursively load modules with tcsh (#12664)
* for tcsh and csh, spack load -r package
should now correctly load recursively instead
of only loading the target package without any
dependencies
2019-09-12 11:11:27 -06:00
Axel Huebl
113365744e
packages service: fix docker build again (#12774)
CD is hard.
2019-09-09 18:24:16 -07:00
Axel Huebl
0d270e0d2a
packages service: fix docker build (#12773)
The build instructions I cloned from did not work ;)
2019-09-09 16:41:04 -07:00
Axel Huebl
221acadc7d
packages build: fix path to spack (#12772)
Fix docker build for packages.spack.io
2019-09-09 14:52:36 -07:00
Axel Huebl
8ea358d849
Fix CD: Packages Service First (#12764)
* Fix CD: Packages Service First

Build the packages.spack.io service images first, so they are
guaranteed to be pushed even if further images fail to build.

Fix the query to the `spack` script executed in later builds.

* CD: Remove Spack Images

Now done on Dockerhub.
2019-09-09 13:22:47 -07:00
Axel Huebl
02931a82fd Add CORS for spack.packages.io
Add the HTTP header `Access-Control-Allow-Origin: *` for our NGINX
service that is serving static JSON content on
https://spack.packages.io .
2019-09-08 17:19:37 -07:00
Todd Gamblin
1b877e8e0f tests and completions for spack find --json and spack find --format 2019-09-02 19:24:48 -07:00
Todd Gamblin
b0abbfecb8 new command: spack maintainers queries package maintainers
- We don't currently make enough use of the maintainers field on
  packages, though we could use it to assign reviews.

- add a command that allows maintainers to be queried
  - can ask who is maintaining a package or packages
  - can ask what packages users are maintaining
  - can list all maintained or unmaintained packages

- add tests for the command
2019-07-24 14:10:08 -07:00
Omar Padron
1d5ab13be8 docker: fix builds, remove extra distros, add ci builds (#11621)
* fix docker builds/remove extra builds/add ci builds
* preinstall vim in CI builder images
* simplify & streamline docker resources
* restore os-container-mapping.yaml file
2019-07-20 14:46:06 -07:00
Todd Gamblin
993ee7f199 environments: add activate/deactivate tests, work wtih set -u
- [x] Add shell tests to ensure that `spack env activate`, `spack env
  deactivate`, and `despacktivate` continue to work.

- [x] Also ensure that activate and deactivate both work with `set -u`
2019-07-20 00:36:56 -07:00
Greg Becker
5cf8878185 feature: Allow developers to use Spack for partial builds (#12006)
Added new diy option.
2019-07-17 11:46:56 -07:00
Todd Gamblin
3f47c5967d
bugfix: fix module() function in setup-env.sh (#12030)
Modify `setup-env.sh` to export fewer variables but to still set the module() function correctly.
2019-07-15 22:43:53 -07:00
Adam J. Stewart
41f5b09a98
Fix typo in module template (#12028) 2019-07-15 20:54:20 -05:00
Oliver Breitwieser
d1af7ae5ab Copy --{use,no}-cache from install to bootstrap (#11786) 2019-07-12 16:30:12 -07:00
Tamara Dahlgren
d615d0a1ec Add template creation test, --skip-editor option (#11970)
* Add template creation test
* Added --skip-editor option to "spack create": normally
  "spack create" opens an editor for the user after generating a
  package file; when the --skip-editor option is used, "spack create"
  only generates the package file and does not open an editor
* Added --skip-editor option to bash completion
2019-07-12 11:22:09 -07:00
Todd Gamblin
e8e030a518
bugfix: setup-env.sh detects bash even when bash is run as sh (#11994)
- `setup-env.sh` was not properly detecting a bash shell when bash was run
  as /bin/sh.

- Detection routine now always reports bash when bash is run as sh, and
  no longer parses the path to the executable indicated in `$BASH`.
2019-07-12 08:33:23 -07:00
Todd Gamblin
5686c2a544 setup-env.sh: make setup scripts work with set -u
- Add set -u to the setup-env.sh test script

- Refactor lines in setup-env.sh that tested potentially undefined
  variables to use the `[ -z ${var+x} ]` construct
2019-07-05 12:54:17 -07:00
Todd Gamblin
47e9f7aac9 tests: add tests for setup-env.sh
- tests use a shell-script harness and test all Spack commands that
  require special shell support.

- tests work in bash, zsh, and dash

- run setup-env.sh tests on macos and linux builds.
  - we run them on macos and linux
2019-07-05 12:54:17 -07:00
Todd Gamblin
e625ee3ffd setup-env.sh: now compatible with bash, zsh, and dash
- replace use of [[ with [

- replace function foo { .. } with foo() { .. }

- wrap some long lines

- add lsof and /proc/fd magic so that we can find the sourced file even in dash

- only do the complicated shell checks in one place; test $_sp_shell
  elsewhere.
2019-07-05 12:54:17 -07:00
Todd Gamblin
600f3c1104 setup-env.sh: send cd output to /dev/null when it affects Spack
- We've seen this a few times now where users have set up `cd` to echo
  the new directory, and it screws up `setup-env.sh`

- In the past we've said this is user error.

- Here, we just fix it by sending `cd` output to /dev/null where needed.
  - this works in bash, zsh, and dash
2019-06-29 19:01:44 -07:00
Axel Huebl
d4008db75d
CD: Package Index (#11665)
Build the docker image for the new package index REST API on each
merge/push to develop and master.
2019-06-10 18:21:33 +02:00
Todd Gamblin
20e5f81653
tests: restore spack spec mpileaks in test output for Python 2 (#11664)
- also remove coverage calls of these commands
- we JUST want coverage to tell us what is in the test suite.
2019-06-08 13:04:44 -07:00
Todd Gamblin
3bd854f991 travis: disable spack spec for python 2.7 as well as 2.6
- keep this disabled until the `compiler find` fork bomb is fixed.
2019-06-05 22:41:28 -07:00
Todd Gamblin
2a51e07fde
commands: completion for new --update and --header options (#11566) 2019-05-27 14:13:58 -07:00
Matthias Wolf
7fcf534b25 junit: escape remaining inputs. (#11382) 2019-05-16 14:40:07 +02:00
Todd Gamblin
dda90a5666 bugfix: Do not run spack spec in the test environment for Python 2.6
- Trying to get our Python 2.6 tests working again.
- This seems to crash travis in the 2.6 environment, for whatever reason.
2019-05-10 13:13:44 -07:00
Todd Gamblin
3903658bfc coverage: only run coverage for subset of unit tests
- Codecov cannot handle as many coverage reports as we are generating

- as a result, our PR coverage pages have been broken for a while, and
  it's hard to tell people where to enhance their testing in PR reviews.

- Scale back to only running coverage for 3.7 and 2.7 unit tests

- This is *probably* better.  We run the build tests for good measure,
  but we do not need to evaluate them for coverage.  The coverage reports
  are about unit tests.
2019-05-11 02:23:22 +09:00
Massimiliano Culpo
5ffb270714 Added a function that concretizes specs together (#11158)
* Added a function that concretizes specs together

* Specs concretized together are copied instead of being referenced

This makes the specs different objects and removes any reference to the
fake root package that is needed currently for concretization.

* Factored creating a repository for concretization into its own function

* Added a test on overlapping dependencies
2019-05-04 03:04:38 +09:00
Omar Padron
0bbd41c7f7
install spack under /opt/spack in docker images (#10969)
* install spack under /opt/spack
2019-05-02 12:05:42 -04:00
Toyohisa Kameyama
ac7f9ef93d Fix handling of MODULEPATH in setup-env.sh (#11274)
Usage of double quotes was preventing word-splitting when parsing
module roots in setup-env.sh, which lead to an error when multiple
module roots are used (in particular when Spack is pointed to use
an upstream module root in addition to its own).
2019-04-24 11:53:25 +09:00
paulanda
88cc895a70 setup-env.sh: fix zsh compatibility error (#11153)
Still look for BASH_SOURCE[0] first, but if it's not set,
_sp_source_file is initialized to an empty value addressing the
unset parameter error (line 217).
2019-04-16 12:33:02 -07:00