Commit graph

3016 commits

Author SHA1 Message Date
Todd Gamblin
3fd9fc8994 env: spack env install automatically concretizes specs 2018-11-09 00:31:24 -08:00
Todd Gamblin
d14f7b82bb env: add spack env activate/deactivate and shell support
- `spack env activate foo`: sets SPACK_ENV to the current active env name

- `spack env deactivate`: unsets SPACK_ENV, deactivates the environment

- added support to setup_env.sh and setup_env.csh

- other env commands work properly with SPACK_ENV, as with an environment
  arguments.

- command-line --env arguments take precedence over the active
  environment, if given.
2018-11-09 00:31:24 -08:00
Todd Gamblin
15c5c36eaf env: bugfix: spack env list won't fail if var/spack/environments doesn't exist 2018-11-09 00:31:24 -08:00
Todd Gamblin
ce230fa3f4 env: rework environments
- env.yaml is now meaningful; it contains authoritative user specs

  - concretize diffs user specs in env.yaml and env.json to allow user to
    add/remove by simply updating env.yaml

  - comments are preserved when env.yaml is updated by add/unadd

  - env.yaml can contain configuration and include external configuration
    either from merged files or from config scopes

  - there is only one file format to remember (env.yaml, no separate init
    format)

- env.json is now env.lock, and it stores the *last* user specs to be
  concretized, along with full provenance.
  - internal structure was modified slightly for readability
  - env.lock contains a _meta section with metadata, in case needed

- added more tests for environments

- env commands follow Spack conventions; no more `spack env foo install`
2018-11-09 00:31:24 -08:00
Todd Gamblin
6af5dfbbc2 config: allow env.yaml to contain configuration in a single file
- add `SingleFileScope` to configuration, which allows us to pull config
  sections from a single file.

- update `env.yaml` and tests to ensure that the env.yaml schema works
  when pulling configurtion from the env file.
2018-11-09 00:31:24 -08:00
Todd Gamblin
9ee2623486 env: rename spack env list to spack env status 2018-11-09 00:31:24 -08:00
Todd Gamblin
c19000038b schemas: rework schemas so that they can be included from other files
- Each schema now has a top-level `properties` and `schema` attribute.

- The `properties` is a fragment that can be included in other
  jsonschemas, via Python, not via '$ref'

- Th `schema` is a complete `jsonschema` with `title` and `$schema`
  properties.
2018-11-09 00:31:24 -08:00
Todd Gamblin
83323f4e71 bugfix: Avoid KeyError in compilers.yaml version check 2018-11-09 00:31:24 -08:00
Todd Gamblin
84140c6cd3 env: add -e as global spack argument, make spack -e <env> spec work
- add -E/--exact-env instead of --use-env-repo
- simplify env handling in `spack find`
2018-11-09 00:31:24 -08:00
Todd Gamblin
180d804615 env: add --env argument to spack location 2018-11-09 00:31:24 -08:00
Todd Gamblin
47e60d5ef8 env: add --env argument to spack find
- add a common argument for `-e/--env`
- modify the database to support queries on subsets of hashes
- allow `spack find` to be filtered by hashes in an environment
2018-11-09 00:31:24 -08:00
Todd Gamblin
ea7648ff84 bugfix: identical specs with different DAG hashes don't shadow each other
- logic used in `spack find` was hiding duplicate installations if their
  hashes were different

- short hash doesn't work in this scenario, since specs are structurally
  identical

- ConstraintAction always works on a DB query, so use the DAG hash to
  ensure uniqueness
2018-11-09 00:31:24 -08:00
Todd Gamblin
3e94c4d573 env: move main Environment class and logic to spack.environment
- `spack.environment` is now the home for most of the infrastructure
   around Spack environments

- refactor `cmd/env.py` to use everything from spack.environment

- refactor the cmd/env test to use pytest and fixtures
2018-11-09 00:31:24 -08:00
Todd Gamblin
0e60fcccfb utils: merge spack.environment into spack.util.environment
- `spack.util.environment` is the new home for routines that modify
  environment variables.

- This is to make room for `spack.environment` to contain new routines
  for dealing with spack environments
2018-11-09 00:31:24 -08:00
Todd Gamblin
cd075b04d2 env: refactor imports in cmd/env.py and tests 2018-11-09 00:31:24 -08:00
Todd Gamblin
b738b7b05b env: refactor subparsers in env.py
- Instead of one method with all parsers, each subcommand gets two
  functions: `setup_<cmd>_parser()` and `environment_<cmd>()`

- the `setup_parser()` and `env()` functions now generate the parser
  based on these and a list of subcommands.

- it is now easier to associate the arguments with the subcommand.
2018-11-09 00:31:24 -08:00
Peter Josef Scheibel
31cb2041c3 env: add spack env command, along with env.yaml schema and tests
Co-authored-by: Elizabeth Fischer <rpf2116@columbia.edu>
2018-11-09 00:31:24 -08:00
Peter Josef Scheibel
4b2f51d063 env: preserve command_line as the scope of highest precedence
Co-authored-by: Elizabeth Fischer <rpf2116@columbia.edu>
2018-11-09 00:31:24 -08:00
Elizabeth Fischer
d1cce990cd env: refactor common arguments 2018-11-09 00:31:24 -08:00
Elizabeth Fischer
037457adc9 specs: save/restore concrete & patches when exporting/importing Specs 2018-11-09 00:31:24 -08:00
Peter Josef Scheibel
7f224b616d specs: _concrete should never be cleared when copying Specs 2018-11-09 00:31:24 -08:00
Peter Josef Scheibel
acdb391931 specs: add convenience function for reading spec dependencies from node dict 2018-11-09 00:31:24 -08:00
Peter Josef Scheibel
4daa164fbf specs: allow writing full spec (including build deps) to dict 2018-11-09 00:31:24 -08:00
Peter Josef Scheibel
e6c6ab64b8 env: rename 'spack env' command to 'spack build-env' 2018-11-09 00:31:24 -08:00
Peter Josef Scheibel
bb8da72f0c env: renamed 'environment' test 2018-11-09 00:31:24 -08:00
Peter Scheibel
7d97e8b067 New repo for advanced packaging tutorial (#9711)
* modified tutorial packages

* update hint in hdf5 tutorial file (typo for suggested argument)

* add repo.yaml to tutorial repository

* update tutorial docs to refer user to tutorial package repository

* flake edits

* recommend site scope vs. defaults

* you don't specify the repo's name when adding a repo, just the path
2018-11-06 19:57:32 -08:00
Peter Scheibel
058cf81312 Binary caching: remove symlinks, copy files instead (#9747)
* omit symlinks and create file copies when making a binary cache of a package

* unrelated flake edits involving regexes that recent flake is now angry about
2018-11-06 19:56:03 -08:00
Zack Galbreath
30eda8b837 cdash: report clean results to CDash server (#9564)
* Record stdout for packages without errors

  Previously our reporter only stored stdout if something went wrong
  while installing a package.  This prevented us from properly reporting
  on steps where everything went as expected.

* More robustly report all phases to CDash

  Previously if a phase generated no output it would not be reported to CDash.
  For example, consider the following output:

    ==> Executing phase: 'configure'
    ==> Executing phase: 'build'

  This would not generate a report for the configure phase. Now it does.

* Add test case for CDash reporting clean builds

* Fix default directory for CDash reports

  The default 'cdash_report' directory name was getting overwritten
  by 'junit-report'.

* Upload the build phase first to CDash

  Older versions of CDash expect Build.xml to be the first file uploaded
  for any given build.

* Define cdash_phase before referring to it
2018-11-06 16:09:47 -08:00
Massimiliano Culpo
05779d911f Adapted the code of the non-daemonic pool to recent python versions
fixes #9739

The non-daemonic pool relies heavily on implementation details of the
multiprocessing package. In this commit we provide an implementation
that fits recent python versions.
2018-11-06 16:02:37 -08:00
Nick Forrington
9f5865a68d compilers: add arm compiler detection to Spack
- added arm.py with support for detecting `armclang` and `armflang`

Co-authored-by: Srinath Vadlamani <srinath.vadlamani@arm.com>
2018-11-02 12:53:50 -07:00
Tim Fuller
fc25ba1b22 Parse the ${NAMESPACE} format string in a spec's format method. (#9686)
This allows installing software on a namespace basis by including ${NAMESPACE} in `install_path_scheme`. e.g.,

```
cat ~/.spack/config.yaml
config:
  install_path_scheme:
      "${ARCHITECTURE}/${NAMESPACE}/${COMPILERNAME}-${COMPILERVER}/${PACKAGE}-${VERSION}-${HASH}"
```
2018-10-31 20:07:07 -07:00
Satish Balay
fb849a4b7f static_to_shared_library: separate options from option values (#9690)
The 'static_to_shared_library' function takes a compiler Executable,
which is intended to be invoked with a list of arguments; the
arguments must be separated from their values in the list, given
the way that 'Executable.__call__' invokes the underlying executable.
'static_to_shared_library' was not doing this, which this commit fixes.
2018-10-31 12:49:04 -07:00
Todd Gamblin
2912cf3e17
compilers: update clang fortran compiler wrapper selection (#9678)
Clang has support for using different fortran compilers with the Clang executable.

Spack includes logic to select a compiler wrapper symlink which refers to the fortran executable (since some build systems depend on the name of the compiler, e.g. 'gfortran' or 'flang').

This selection was previously based on the architecture, and chose incorrectly in some situations (e.g. for clang/gfortran on Linux). This replaces architecture-based wrapper selection with a selection that is based on the name of the Fortran compiler executable.
2018-10-30 23:00:43 -07:00
Levi Baber
72fa122101 tests: add test for FIXME boilerplate to package_sanity (#9285)
* package_sanity: add test_no_fixme
* cleanup & better assert message
2018-10-30 13:44:25 -07:00
Todd Gamblin
b27fbfb379
config: spack config blame now colors filenames in config output (#9656)
- it was hard to distinguish all-gray filenames
- added rotating colors to `spack config blame`
2018-10-26 22:19:11 -07:00
Omar Padron
aa1c814c75 docker: unite Dockerfiles; auto-deploy images to DockerHub (#9329)
* Unite Dockerfiles - add build/run/push scripts
* update docker documentation
* update .travis.yml
* switch to using a preprocessor on Dockerfiles
* skip building docker images on pull requests
* update files with copyright info
* tweak when travis builds for docker files are done
2018-10-26 10:15:05 -07:00
Massimiliano Culpo
da537d2211
Fix autoload of direct dependencies for python (#9630)
fixes #9624

merge_config_rules was using `strict=False` to check if a spec
satisfies a constraint, which loosely translates to "this spec has
no conflict with the constraint, so I can potentially add it to the
spec". We want instead `strict=True` which means "the spec satisfies
the constraint right now".
2018-10-25 23:56:03 +02:00
Todd Gamblin
a1f90d5b8c
bugfix: use OS default permissions for mkdirp when mode is not provided (#9604)
- #8773 made the default mode 0o777, which is what's documented but
   mkdirp actually takes the OS default or umask by default

- revert to the Python default by default, and only set the mode when
  asked explicitly.
2018-10-24 11:36:41 -07:00
Javier Cervantes
02f67b5c16 Fix typo in help message (#9599) 2018-10-24 12:40:35 +02:00
Satish Balay
79e25032e8 Bug fix: Module PATH check (#9574)
#9100 added a warning message when a path extracted from a module file
did not appear to be a valid filesystem path. This check was applied
to a variable which could be a list of paths, which would erroneously
trigger the warning. This commit updates the check to run at the
actual point where the path has been extracted.
2018-10-22 18:10:49 -07:00
Todd Gamblin
ccd5a5fb7b license: exclude ignored files from the license check (#9578)
Previously, if you built the docs, you'd get license errors for
generated .rst files. This removes them from the list of licensed
files.
2018-10-22 11:13:30 -07:00
Michael Kuhn
7070d9fcd4 Allow setting language of compiler messages (#9486)
* Add a build_language config.yaml option which controls the language
  of compiler messages
* build_language defaults to "C", in which case the compiler messages
  will be in English. This allows Spack log parsing to detect and
  highlight error messages (since the regular expressions to find
  error messages are in English)
* The user can use the default language in their environment by setting
  the build_language config variable to null or ''
2018-10-18 12:55:58 -07:00
Todd Gamblin
e2e0b5df1c relicense: add spack license command
- `spack license list-files`: list all files that should have license headers
- `spack license list-lgpl`:  list files still under LGPL-2.1
- `spack license verify`:     check that license headers are correct

- Added `spack license verify` to style tests
2018-10-17 14:42:06 -07:00
Todd Gamblin
7cb5638516 relicense: add Apache2/MIT header to spack create 2018-10-17 14:42:06 -07:00
Todd Gamblin
eea786f4e8 relicense: replace LGPL headers with Apache-2.0/MIT SPDX headers
- remove the old LGPL license headers from all files in Spack
- add SPDX headers to all files
  - core and most packages are (Apache-2.0 OR MIT)
  - a very small number of remaining packages are LGPL-2.1-only
2018-10-17 14:42:06 -07:00
Mark W. Krentel
e5e8d89ad9 docs: add note about compilers with modules (#9520)
compilers.yaml can track a module that is needed for a compiler, but
Spack does not fill this in automatically. This adds a note to the
documentation informing the user how to do this.
2018-10-16 15:34:22 -07:00
Michael Kuhn
cc6e765d28 meson: explicitly add --libdir (#9504)
If we do not specify libdir explicitly, Meson chooses something like
lib/x86_64-linux-gnu, which causes problems when trying to find libraries
and pkg-config files.
2018-10-15 11:14:59 -07:00
Peter Scheibel
6977d933c8
buildcache install: generate modules (#9509)
Spack packages installed using spack buildcache were not running
post-install hooks, which create module files and manage licenses
(if necessary).

This was already occurring for Spack packages installed with
spack install --use-cache
2018-10-12 10:43:06 -07:00
Greg Becker
d1a5113cfe permissions: add permission configuration to packages.yaml (#8773)
Spack can now be configured to assign permissions to the files installed by a package.

In the `packages.yaml` file under `permissions`, the attributes `read`, `write`, and `group` control the package permissions. These attributes can be set per-package, or for all packages under `all`. If permissions are set under `all` and for a specific package, the package-specific settings take precedence.  The `read` and `write` attributes take one of `user`, `group`, and `world`.

   packages:
    all:
      permissions:
        write: group
        group: spack
    my_app:
      permissions:
        read: group
        group: my_team
2018-10-11 14:29:07 -07:00
Zack Galbreath
bc8bb9dfaf Make CDash reporting easier to use (#9357)
* Better default CLI arguments for CDash reporting

--log-format=cdash is now implied if you specify the --cdash-upload-url
option to spack install.

We also now default to writing CTest XML files to cdash_report/ when using
the CDash reporter if no --log-file argument was specified.

* Improved documentation on how to use the CDash reporter
2018-10-10 22:29:14 -07:00