Commit graph

134 commits

Author SHA1 Message Date
Harmen Stoppels
80389911cc
Update bootstrap buildcache to v0.3 (#32262)
This release allow to bootstrap patchelf from binaries.
2022-09-02 12:48:46 +02:00
Harmen Stoppels
8af3743e91
Relative paths in default prefix_inspections start with ./ (#31867) 2022-08-23 12:54:12 -07:00
Harmen Stoppels
ceda5fb46c
Don't set LD_LIBRARY_PATH by default on Linux (#28354)
`LD_LIBRARY_PATH` can break system executables (e.g., when an enviornment is loaded) and isn't necessary thanks to `RPATH`s.  Packages that require `LD_LIBRARY_PATH` can set this in `setup_run_environment`.

- [x] Prefix inspections no longer set `LD_LIBRARY_PATH` by default
- [x] Document changes and workarounds for people who want `LD_LIBRARY_PATH`
2022-08-11 09:33:08 -05:00
Chuck Atkins
57af891cb1
OpenGL restructure (#31748)
* OpenGL: Restructures the OpenGL packages

This provides concrete glx and osmesa packages which delegate to
virtual libglx and libosmesa packages provided by mesa. This was
necessary because GLX and OSMesa both provide gl implementations but
with mesa providing the girtual gl package there was no way to properly
distinguish which of the two opengl implementations was beiing requested
when querying the spec['gl'] dependency.  This additional level of
indirection allows for that.

* OpenGL: Adjust downstream dependents of OpenGL for the restructure

This implements the necessary fixes in the packages that depend on
OpenGL to work with the restructuring.  This also attempts to create a
consistent variant for specifying glx or osmesa.
2022-07-29 12:04:42 -06:00
Massimiliano Culpo
ba907defca
Add a command to generate a local mirror for bootstrapping (#28556)
This PR builds on #28392 by adding a convenience command to create a local mirror that can be used to bootstrap Spack. This is to overcome the inconvenience in setting up this mirror manually, which has been reported when trying to setup Spack on air-gapped systems.

Using this PR the user can create a bootstrapping mirror, on a machine with internet access, by:

% spack bootstrap mirror --binary-packages /opt/bootstrap
==> Adding "clingo-bootstrap@spack+python %apple-clang target=x86_64" and dependencies to the mirror at /opt/bootstrap/local-mirror
==> Adding "gnupg@2.3: %apple-clang target=x86_64" and dependencies to the mirror at /opt/bootstrap/local-mirror
==> Adding "patchelf@0.13.1:0.13.99 %apple-clang target=x86_64" and dependencies to the mirror at /opt/bootstrap/local-mirror
==> Adding binary packages from "https://github.com/alalazo/spack-bootstrap-mirrors/releases/download/v0.1-rc.2/bootstrap-buildcache.tar.gz" to the mirror at /opt/bootstrap/local-mirror

To register the mirror on the platform where it's supposed to be used run the following command(s):
  % spack bootstrap add --trust local-sources /opt/bootstrap/metadata/sources
  % spack bootstrap add --trust local-binaries /opt/bootstrap/metadata/binaries
The mirror has to be moved over to the air-gapped system, and registered using the commands shown at prompt. The command has options to:

1. Add pre-built binaries downloaded from Github (default is not to add them)
2. Add development dependencies for Spack (currently the Python packages needed to use spack style)

* bootstrap: refactor bootstrap.yaml to move sources metadata out

* bootstrap: allow adding/removing custom bootstrapping sources

This operation can be performed from the command line since
new subcommands have been added to `spack bootstrap`

* Add --trust argument to spack bootstrap add

* Add a command to generate a local mirror for bootstrapping

* Add a unit test for mirror creation
2022-05-24 21:33:52 +00:00
Harmen Stoppels
f7258e246f
Deprecate spack:concretization over concretizer:unify (#30038)
* Introduce concretizer:unify option to replace spack:concretization

* Deprecate concretization

* Make spack:concretization overrule concretize:unify for now

* Add environment update logic to move from spack:concretization to spack:concretizer:reuse

* Migrate spack:concretization to spack:concretize:unify in all locations

* For new environments make concretizer:unify explicit, so that defaults can be changed in 0.19
2022-05-23 13:20:34 -07:00
robgics
1f6b880fff
Add license dir to config (#30135)
* Change license dir from hard-coded to a configurable item

* Change config item to be a string not an array

Co-authored-by: Todd Gamblin <tgamblin@llnl.gov>
2022-05-18 18:26:42 -07:00
Massimiliano Culpo
d900ac2003
Reuse concretization by default (#30396)
* Enable reuse by default in Spack
* Update documentation to match new default
* Configure pipelines not to reuse software
2022-05-13 09:11:10 -07:00
Tom Scogland
7f1659786b
Add a Lua build-system (#28854)
Reworking lua to allow easier substitution of the base lua implementation.

Also adding in a maintained version of luajit and re-factoring the entire stack 
to use a custom build-system to centralize functionality like environment 
variable management and luarocks installation.

The `lua-lang` virtual is now versioned so that a package that requires 
Lua 5.1 semantics can get any lua, but one that requires 5.2 will only 
get upstream lua.

The luaposix package requires lua-bit32, but only when built with a 
lua conforming to version 5.1.  This adds the package, and the 
dependencies, but exposed a problem with luarocks dependency 
detection.  Since we're  installing each package in its own "tree" and 
there's no environment  variable to list extra trees, spack now 
generates a luarocks config  file that lists all the trees of all the 
dependencies, and references  it by setting `LUAROCKS_CONFIG` 
in the build environment of every LuaPackage.  This allows luarocks 
to find the spack installed  dependencies correctly rather than 
trying (and failing) to download them.

Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
Co-authored-by: Tom Scogland <tscogland@llnl.gov>
Co-authored-by: Massimiliano Culpo <massimiliano.culpo@gmail.com>
2022-05-10 06:54:38 +02:00
Massimiliano Culpo
b9d6a5103d
ASP-based solver: allow configuring target selection (#29835)
* ASP-based solver: allow configuring target selection

This commit adds a new "concretizer:targets" configuration
section, and two options under it.

 - "concretizer:targets:granularity" allows switching from
considering only generic targets to consider all possible
microarchitectures.

 - "concretizer:targets:host_compatible" instead controls
whether we can concretize for microarchitectures that
are incompatible with the current host.

* Add documentation

* Add unit-tests
2022-04-25 17:19:51 -07:00
Massimiliano Culpo
c846b5149d
Add support for Python 3.10 (#29581)
* Add support for Python 3.10

* Update unit-tests to use 3.10

* Update Getting started section of the docs

* Update bootstrap action
2022-04-13 14:32:23 -07:00
Harmen Stoppels
99083f1706
Deprecate top-level module config (#28659)
* Ignore top-level module config; add auto-update

In Spack 0.17 we got module sets (modules:[name]:[prop]), and for
backwards compat modules:[prop] was short for modules:default:[prop].

But this makes it awkward to define default config for the "default"
module set.

Since 0.17 is branched off, we can now deprecate top-level module config
(that is, just ignore it with a warning).

This PR does that, and it implements `spack config update modules` to
make upgrading easy (we should have added that to 0.17 already...)

It also removes references to  `dotkit` stuff which was already
deprecated in 0.13 and could have been removed in 0.14.

Prefix inspections are the only exception, since the top-level prefix inspections
used for `spack load` and `spack env activate`.
2022-04-08 19:00:35 +00:00
John Parent
df4129d395 Expand external find for Windows (#27588)
* Incorporate new search location

* Add external user option

* proper doc string

* Explicit commands in getting started

* raise during chgrp on Win

recover installer changes

Notate admin privleges

Windows phase install hooks

Find external python and install ninja (#23496)

Allow external find python to find windows python and spack install ninja

Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
Co-authored-by: Betsy McPhail <betsy.mcphail@kitware.com>
2022-03-17 09:01:01 -07:00
Jared Popelar
15ef85e161 Packaging of netlib-lapack for windows (#24993)
MSVC's internal CMake and Ninja now detected by spack external find and added to packages.yaml

Saving progress on packaging zlib for Windows

Fixing the shared CMake flag

* Loading Intel's ifx Fortran compiler into MSVC; if there are multiple
versions of MSVC installed and detected, ifx will only be placed into
the first block written in compilers.yaml. The version number of ifx can
be detected using MSVC's version flag (instead of /QV) by using
ignore_version_errors. This commit also provides support for detection
of Intel compilers in their own compiler block by adding ifx.exe to the
fc/f77_name blocks inside intel.py

* Giving CMake a Fortran compiler argument

* Adding patch file for removing duplicated mangling header for versions 3.9.1 and older; static and shared now successfully building on Windows

* Have netlib-lapack depend  on ninja@1.10

Co-authored-by: John R. Cary <cary@txcorp.com>
Co-authored-by: Jared Popelar <jpopelar@txcorp.com>

Making a default config.yaml for Windows

Small path length for build_stage

Provide more prerequisite details, mention default config.yaml

Killing an unnecessary setvars call

Replacing some lost changes, proofreading, updating windows-supported package list

Co-authored-by: John Parent <john.parent@kitware.com>
2022-03-17 09:01:01 -07:00
Todd Gamblin
800ed16e7a config: add a new concretizer config section
The concretizer is going to grow to have many more configuration,
and we really need some structured config for that.

* We have the `config:concretizer` option that chooses the solver,
  but extending that is awkward (we'd need to replace a string with
  a `dict`) and the solver choice will be deprecated eventually.

* We have the `concretization` option in environments, but it's
  not a top-level config section -- it's just for environments,
  and it also only admits a string right now.

To avoid overlapping with either of these and to allow the most
extensibility in the future, this adds a new `concretizer` config
section that can be used in and outside of environments. There
is only one option right now: `reuse`.  This can expand to include
other options later.

Likely, we will soon deprecate `config:concretizer` and warn when
the user doesn't use `clingo`, and we will eventually (sometime later)
move the `together` / `separately` options from `concretization` into
the top-level `concretizer` section.

This commit just adds the new section and schema. Fully wiring it
up is TBD.
2022-02-16 10:17:18 -08:00
Harmen Stoppels
e72f87ec64
Switch lmod default all:autoload from none to direct (#28357)
* Switch lmod module all autoload default from none to direct

* Fix the docs
2022-01-18 09:06:41 -08:00
kwryankrattiger
363a565ce3
Packaging: Virtual package for libllvm (#27200)
Add an abstraction around libllvm to allow libllvm
providers to be specified for all packages.

This is targeting allowing mesa to build against
llvm-amdgpu or intel-llvm or llvm or any other
custom llvm variant that arises for specific GPU
toolchains
2022-01-11 13:28:13 -08:00
Todd Gamblin
a1216138f6
config: fix SPACK_DISABLE_LOCAL_CONFIG, remove $user_config_path (#27022)
There were some loose ends left in ##26735 that cause errors when
using `SPACK_DISABLE_LOCAL_CONFIG`.

- [x] Fix hard-coded `~/.spack` references in `install_test.py` and `monitor.py`

Also, if `SPACK_DISABLE_LOCAL_CONFIG` is used, there is the issue that
`$user_config_path`, when used in configuration files, makes no sense,
because there is no user config scope.

Since we already have `$user_cache_path` in configuration files, and since there
really shouldn't be *any* data stored in a configuration scope (which is what
you'd configure in `config.yaml`/`bootstrap.yaml`/etc., this just removes
`$user_config_path`.

There will *always* be a `$user_cache_path`, as Spack needs to write files, but
we shouldn't rely on the existence of a particular configuration scope in the
Spack code, as scopes are configurable, both in number and location.

- [x] Remove `$user_config_path` substitution.
- [x] Fix reference to `$user_config_path` in `etc/spack/deaults/bootstrap.yaml`
      to refer to `$user_cache_path`, which is where it was intended to be.
2021-10-28 21:33:44 +00:00
Harmen Stoppels
ae6e83b1d5 config: overrides for caches and system and user scopes (#26735)
Spack's `system` and `user` scopes provide ways for administrators and
users to set global defaults for all Spack instances, but for use cases
where one wants a clean Spack installation, these scopes can be undesirable.
For example, users may want to opt out of global system configuration, or
they may want to ignore their own home directory settings when running in
a continuous integration environment.

Spack also, by default, keeps various caches and user data in `~/.spack`,
but users may want to override these locations.

Spack provides three environment variables that allow you to override or
opt out of configuration locations:

 * `SPACK_USER_CONFIG_PATH`: Override the path to use for the
   `user` (`~/.spack`) scope.

 * `SPACK_SYSTEM_CONFIG_PATH`: Override the path to use for the
   `system` (`/etc/spack`) scope.

 * `SPACK_DISABLE_LOCAL_CONFIG`: set this environment variable to completely
   disable *both* the system and user configuration directories. Spack will
   only consider its own defaults and `site` configuration locations.

And one that allows you to move the default cache location:

 * `SPACK_USER_CACHE_PATH`: Override the default path to use for user data
   (misc_cache, tests, reports, etc.)

With these settings, if you want to isolate Spack in a CI environment, you can do this:

   export SPACK_DISABLE_LOCAL_CONFIG=true
   export SPACK_USER_CACHE_PATH=/tmp/spack

This is a stop-gap approach until we have figured out how to deal with
the system and user config scopes more generally, as there are plans to
potentially / eventually get rid of them.

**User config**

Spack is a bit of a pain when you have:

- a shared $HOME folder across different systems.
- multiple Spack versions on the same system.

**System config**

- On shared systems with a versioned programming environment / toolkit,
  system administrators want to provide config for each version (e.g.
  21.09, 21.10) of the programming environment, and the user Spack
  instance should be able to pick this up without a steep learning
  curve.
- On shared systems the user should be able to opt out of the
  hard-coded config scope in /etc/spack, since it may be incompatible
  with their particular instance. Currently Spack can only opt out of all
  config scopes through overrides with `"config:":`, `"packages:":`, but that
  also drops the defaults config, which would have to be repeated, which
  is undesirable, especially the lengthy packages.yaml.

An example use case is: having config in this folder:

```
/path/to/programming/environment/{version}/{compilers,packages}.yaml
```

and have `module load spack-system-config` set the variable

```
SPACK_SYSTEM_CONFIG_PATH=/path/to/programming/environment/{version}
```

where the user no longer has to worry about what `{version}` they are
on.

**Continuous integration**

Finally, there is the use case of continuous integration, which may
clone an arbitrary Spack version, which optimally should not pick up
system or user config from the previous run (like may happen in
classical bare metal non-containerized filesystem side effect ridden
jenkins pipelines). In fact this is very similar to how spack itself
tries to avoid picking up system dependencies during builds...

**But environments solve this?**

- You could do `include`s in environment files to get similar behavior
  to the spack_system_config_path example, but environments require you
  to:
  1) require paths to individual config files, not directories.
  2) fail if the listed config file does not exist
- They allow you to override config scopes, but this is generally too
  rigurous, as it requires you to repeat the default config, in
  particular packages.yaml, and just defies the point of layered config.

Co-authored-by: Tom Scogland <tscogland@llnl.gov>
Co-authored-by: Tim Fuller <tjfulle@sandia.gov>
Co-authored-by: Steve Leak <sleak@lbl.gov>
Co-authored-by: Todd Gamblin <tgamblin@llnl.gov>
2021-10-26 18:08:25 -07:00
Michael Kuhn
d1f3279607
installer: Support showing status information in terminal title (#16259)
Installing packages with a lot of dependencies does not have an easy way
of judging the current progress (apart from running `spack spec -I pkg`
in another terminal). This change allows Spack to update the terminal's
title with status information, including its current progress as well as
information about the current and total number of packages.
2021-10-11 17:54:59 +02:00
Massimiliano Culpo
c52426ea7a
Make clingo the default solver (#25502)
Modifications:
- [x] Change `defaults/config.yaml`
- [x] Add a fix for bootstrapping patchelf from sources if `compilers.yaml` is empty
- [x] Make `SPACK_TEST_SOLVER=clingo` the default for unit-tests
- [x] Fix package failures in the e4s pipeline

Caveats:
1. CentOS 6 still uses the original concretizer as it can't connect to the buildcache due to issues with `ssl` (bootstrapping from sources requires a C++14 capable compiler)
1. I had to update the image tag for GitlabCI in e699f14.  
1. libtool v2.4.2 has been deprecated and other packages received some update
2021-09-14 22:44:16 -07:00
Massimiliano Culpo
4318ceb2b3
Bootstrap clingo from binaries (#22720)
* Bootstrap clingo from binaries

* Move information on clingo binaries to a JSON file

* Add support to bootstrap on Cray

Bootstrapping on Cray requires, at the moment, to
swap the platform when looking for binaries - due
to #22800.

* Add SHA256 verification for bootstrapped software

Use sha256 verification for binaries necessary to bootstrap
the concretizer and gpg for signature verification

* patchelf: use Spec._old_concretize() to bootstrap

As noted in #24450 we may happen to need the
concretizer when bootstrapping clingo. In that case
only the old concretizer is available.

* Add a schema for bootstrapping methods

Two fields have been added to bootstrap.yaml:
  "sources" which lists the methods available for
       bootstrapping software
  "trusted" which records if a source is trusted or not

A subcommand has been added to "spack bootstrap" to list
the sources currently available.

* Methods used for bootstrapping are configurable from bootstrap:sources

The function that tries to ensure a given Python module
is importable now tries bootstrapping methods in the same
order as they are defined in `bootstrap.yaml`

* Permit to trust/untrust bootstrapping methods

* Add binary tests for MacOS, Ubuntu

* Add documentation

* Add a note on bash
2021-08-18 11:14:02 -07:00
loulawrence
be3e6a0e9b
document config option "url_fetch_method" (#24638)
- Change config from the undocumented `use_curl: true/false` to `url_fetch_method: urllib/curl`.
- Documentation of `url_fetch_method` in `defaults/config.yaml`
- Default fetch option explicitly set to `urllib` for users who may not have curl on their system

To upgrade from `use_curl` to `url_fetch_method`, run `spack config update config`
2021-08-02 10:30:25 +02:00
Seth R. Johnson
0a41d4ebb8
pbs: new virtual package (#24568)
* pbs: new virtual package

Some of our clusters have an older installation of
libtorque and tm.h that are *not* from OpenPBS. Using the current
openpbs dependency for openmpi causes concretization errors due to
restrictions on older python and hwloc requirements that don't apply,
even with an external non-buildable installation.
The new 'torque' bundle package allows users to point to that external
installation without problems.

Detailed description of torque by Sergey Kosukhin <skosukhin@gmail.com>
2021-07-27 16:37:00 +00:00
Massimiliano Culpo
3228c35df6
Enable/disable bootstrapping and customize store location (#23677)
* Permit to enable/disable bootstrapping and customize store location

This PR adds configuration handles to allow enabling
and disabling bootstrapping, and to customize the store
location.

* Move bootstrap related configuration into its own YAML file

* Add a bootstrap command to manage configuration
2021-07-12 19:00:37 -04:00
Larry Knox
cc20dbf645
Hdf5 cmake (#18937)
* Switch hdf5 package from autotools to cmake.

* Add variant for building with zlib, default to ON.

* Update for format requirements.

* Format change.

* Fix breakage from last merge from develop.
Switch szip to use libaec (unrestricted encryption).
Remove 'static' variant:  static libs will only be installed when
~shared.

* Improve args based on suggestions from pull request.

* Update code URL to github.com
Add/modify 4 depends_on lines to fix running "spack graph --deptype=link hdf5".

* Remove trailing whitespace.

* Remove dependencies added solely to make "spack greph --type=link" work.

* Add new version HDF5 1.8.22.

* Remove unnecessary java_check.

* Fix whitespace for style checks.

* Reverted zlib version dependency to 1.1.2:.
zlib variant removed.
api version default renamed "default".

* Remove blank line.

* Whitespace corrections.

* iRemoved unnecessary 'debug' variant.

* Fix typo in version number in conflict for '+szip'.

* Set default for tools variant to True.
Remove patch functions dependent on 'libtool' file that cmake doesn't
produce.

* Remove line to set ONLY_SHARED_LIBS to true.
Add post_install code to install only one version of tools with shared
linkage and original tool names.

* Remove trailing white space and import of glob package not used.

* Leave BUILD_TESTING set to default which is ON.

* Remove  post_install code to install only one version of tools because
some dependent packages running tests in e4s testing are using
h5diff-shared.  Keep both tools versions for now.

* No longer need to import os.
2021-06-28 13:42:54 -06:00
Richarda Butler
1c44912f9b
add irep and lua-lang virtual dependency (#22492)
This adds a package for `irep`, a tool for reading `lua` input decks from 
Fortran, C, and C++.

`irep` can be built with either `lua` or `luajit`.  To address this, we also add
a virtual package for lua called `lua-lang`.  `luajit` isn't, by default, a drop-in
replacement for `lua`, but we add a `+lualinks` variant to it that adds symlinks
that make it behave like `lua@5.1`.  With this variant enabled, it provides the
`lua-lang` virtual.  `lua` always provides `lua-lang`.

- [x] add `irep` package
- [x] add `+lualinks` variant to `lua-luajit`
- [x] create `lua-lang` virtual, provided by `lua` and `luajit+lualinks`

Co-authored-by: Kayla Richarda Butler <butler59@quartz1148.llnl.gov>
Co-authored-by: Todd Gamblin <tgamblin@llnl.gov>
2021-06-15 17:50:04 -07:00
Nikolay Petrov
ec2d8a1571
Fixing provides directive for intel-oneapi-onedal package (#24108) 2021-06-11 09:47:27 +02:00
Greg Becker
af3ebeeea1
apply default linux prefix inspections to all module sets (#24151) 2021-06-04 21:37:20 -06:00
Greg Becker
7490d63c38
Separable module configuration -- without the bugs this time (#23703)
Currently, module configurations are inconsistent because modulefiles are generated with the configs for the active environment, but are shared among all environments (and spack outside any environment).

This PR fixes that by allowing Spack environments (or other spack config scopes) to define additional sets of modules to generate. Each set of modules can enable either lmod or tcl modules, and contains all of the previously available module configuration. The user defines the name of each module set -- the set configured in Spack by default is named "default", and is the one returned by module manipulation commands in the absence of user intervention.

As part of this change, the module roots configuration moved from the config section to inside each module configuration.

Additionally, it adds a feature that the modulefiles for an environment can be configured to be relative to an environment view rather than the underlying prefix. This will not be enabled by default, as it should only be enabled within an environment and for non-default views constructed with separate projections per-spec.
2021-05-28 14:12:05 -07:00
Todd Gamblin
6fc0e8dc9b
Use AWS CloudFront for source mirror (#23978)
Spack's source mirror was previously in a plain old S3 bucket. That will still
work, but we can do better. This switches to AWS's CloudFront CDN for hosting
the mirror.

CloudFront is 16x faster (or more) than the old bucket.

- [x] change mirror to https://mirror.spack.io
2021-05-28 00:18:30 -07:00
Adam J. Stewart
ef56462867
Add fuse virtual dependency, new macfuse package (#23904) 2021-05-27 23:21:11 +02:00
eugeneswalker
35f7959f54
defaults/cray: use modules.yaml from defaults/linux (#23932) 2021-05-26 20:16:13 +00:00
Harmen Stoppels
f2ce57bf77
Add xxd for hsa-rocr-dev build script (#23855) 2021-05-25 20:58:30 +00:00
Harmen Stoppels
8446bebdd9
Revert "Separable module configurations (#22588)" (#23674)
This reverts commit cefbe48c89.
2021-05-17 06:42:48 -07:00
Greg Becker
cefbe48c89
Separable module configurations (#22588)
Currently, module configurations are inconsistent because modulefiles are generated with the configs for the active environment, but are shared among all environments (and spack outside any environment).

This PR fixes that by allowing Spack environments (or other spack config scopes) to define additional sets of modules to generate. Each set of modules can enable either lmod or tcl modules, and contains all of the previously available module configuration. The user defines the name of each module set -- the set configured in Spack by default is named "default", and is the one returned by module manipulation commands in the absence of user intervention.

As part of this change, the module roots configuration moved from the `config` section to inside each module configuration.

Additionally, it adds a feature that the modulefiles for an environment can be configured to be relative to an environment view rather than the underlying prefix. This will not be enabled by default, as it should only be enabled within an environment and for non-default views constructed with separate projections per-spec.

TODO:
- [x] code changes to support multiple module sets
- [x] code changes to support modules relative to a view
- [x] Tests for multiple module configurations
- [x] Tests for modules relative to a view
- [x] Backwards compatibility for module roots from config section
- [x] Backwards compatibility for default module set without the name specified
- [x] Tests for backwards compatibility
2021-05-14 15:03:28 -07:00
Harmen Stoppels
1db6cd5d16
Make -j flag less exceptional (#22360)
* Make -j flag less exceptional

The -j flag in spack behaves differently from make, ctest, ninja, etc,
because it caps the number of jobs to an arbitrary number 16.
Spack will behave like other tools if `spack install` uses a reasonable
default, and `spack install -j <num>` *overrides* that default.

This will be particularly useful for Spack usage outside of a traditional
HPC context and for HPC centers that encourage users to compile on
login nodes with many cores instead of on compute nodes, which has
become increasingly common as individual nodes have more cores.

This maintains the existing default value of min(num_cpus, 16). However, 
as it is right now, Spack does a poor job at determining the number of 
cpus on linux, since it doesn't take cgroups into account. This is
particularly problematic when using distributed builds with slurm. This PR
also introduces `spack.util.cpus.cpus_available()` to consolidate
knowledge on determining the number of available cores, and improves
core detection for linux. This should also improve core detection for Docker/
Kubernetes, which also use cgroups.
2021-03-30 12:03:50 -07:00
Massimiliano Culpo
796588b2fd
zig: add new package at v0.7.1 (#22046) 2021-03-03 13:30:41 +01:00
Harmen Stoppels
0664b90751
Drop compiler variables from spack load (#21699)
Drops:

* C_INCLUDE_PATH
* CPLUS_INCLUDE_PATH
* LIBRARY_PATH
* INCLUDE

We already decided to use C_INCLUDE_PATH, CPLUS_INCLUDE_PATH, INCLUDE over CPATH here:

https://github.com/spack/spack/pull/14749

However, none of these flags apply to Fortran on Linux. So for consistency it seems better to make the user use -I and -L flags by hand or through pkgconfig.
2021-02-23 13:35:19 +01:00
Adam J. Stewart
7ccb9992a6
Procedure to deprecate old versions of software (#20767)
* Procedure to deprecate old versions of software

* Add documentation

* Fix bug in logic

* Update tab completion

* Deprecate legacy packages

* Deprecate old mxnet as well

* More explicit docs
2021-02-09 13:51:18 -05:00
Laurent Aphecetche
e0b7ca47d8
config: (darwin only) change prefix of external libuuid (#21480) 2021-02-04 14:01:40 -06:00
Adam J. Stewart
5964cb49d9
Use system libuuid on macOS (#20608) 2020-12-30 10:13:34 -06:00
Michael Kuhn
c22141f444
Introduce virtual provider uuid (#18322)
libuuid is currently contained in util-linux, libuuid and uuid. This
change introduces a new virtual provider `uuid` and renames the existing
`uuid` package to `ossp-uuid`.

util-linux's libuuid is provided in the form of a separate package
util-linux-uuid to make sure that packages depending on uuid and
util-linux can use a separate uuid implementation, which the concretizer
does not allow if libuuid is contained in util-linux.
2020-12-29 10:12:26 -06:00
Chuck Atkins
7d79db2de7
Fix Mesa GLES conflicts (#20184) 2020-11-30 23:54:24 +01:00
Greg Becker
77b2e578ec
spack test (#15702)
Users can add test() methods to their packages to run smoke tests on
installations with the new `spack test` command (the old `spack test` is
now `spack unit-test`). spack test is environment-aware, so you can
`spack install` an environment and then run `spack test run` to run smoke
tests on all of its packages. Historical test logs can be perused with
`spack test results`. Generic smoke tests for MPI implementations, C,
C++, and Fortran compilers as well as specific smoke tests for 18
packages.

Inside the test method, individual tests can be run separately (and
continue to run best-effort after a test failure) using the `run_test`
method. The `run_test` method encapsulates finding test executables,
running and checking return codes, checking output, and error handling.

This handles the following trickier aspects of testing with direct
support in Spack's package API:

- [x] Caching source or intermediate build files at build time for
      use at test time.
- [x] Test dependencies,
- [x] packages that require a compiler for testing (such as library only
      packages).

See the packaging guide for more details on using Spack testing support.
Included is support for package.py files for virtual packages. This does
not change the Spack interface, but is a major change in internals.

Co-authored-by: Tamara Dahlgren <dahlgren1@llnl.gov>
Co-authored-by: wspear <wjspear@gmail.com>
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
2020-11-18 02:39:02 -08:00
Todd Gamblin
14ab63f97c concretizer: add a configuration option to use new or old concretizer
- [x] spec.py can call out to the new concretizer
- [x] config.yaml now has an option to choose a concretizer (original, clingo)
2020-11-17 10:04:13 -08:00
Greg Becker
fafff0c6c0
move sbang to unpadded install tree root (#19640)
Since #11598 sbang has been installed within the install_tree. This doesn’t play
nicely with install_tree padding, since sbang can’t do its job if it is installed in a
long path (this is the whole point of sbang).

This PR changes the padding specification.  Instead of $padding inside paths,
we now have a separate `padding:` field in the `install_tree` configuration.

Previously, the `install_tree` looked like this:

```
    /path/to/opt/spack_padding_padding_padding_padding_padding/
        bin/
            sbang
        .spack-db/
            ...
        linux-rhel7-x86_64/
            ...
```

```
This PR updates things to look like this:

    /path/to/opt/
        bin/
            sbang
        spack_padding_padding_padding_padding_padding/
            .spack-db/
                ...
            linux-rhel7-x86_64/
                ...

So padding is added at the start of all install prefixes *within* the unpadded
root.  The database and all installations still go under the padded root.

This ensures that `sbang` is in the shorted possible path while also allowing
us to make long paths for relocatable binaries.
2020-11-12 16:08:55 -08:00
vijay kallesh
51a9473bf8
[NEW] Added amdfftw, amdlibflame and amdscalapack recipes (#19457)
* [NEW] Added amdfftw, amdlibflame and amdscalapack recipes

Updated base fftw, libflame and netlib-scalapack recipes
to accommodate the above listed AMD Optimizing CPU Libraries
which are a set of numerical routines optimized for AMD platforms.

Updated amdblis spack recipe

amdblis:
	1. updated with amdblis 2.2 release
amdfftw:
	1. "--enable-single" now work as synonym for "--enable-float"
amdlibflame:
	1. Added enable_or_disable_threads() to set value for "--enable-multithreading" flag
Libflame:
	1. Added enable_or_disable_threads() to set value for "--enable-multithreading" flag
	2. Corrected invocation of "enable_or_disable('threads')"

Change-Id: I9da0a2c2c4e2075b7fa2776e7cfe6548a2e0b32f

* Added amd-toolchain-support as maintainers

Added team github account amd-toolchain-support
as maintainers for all the recipes owned by
AMD Optimizing CPU Libraries (AOCL) team

Change-Id: I9a7969bd48fc42cfbb88dd7bd93e0802c6138582

* Incorporated review comments

Updated packages.yaml with aocl components
Handled Flake8 test failures

Change-Id: I0a03f02d8c9f326b2434ec907958c3de3a8e18eb

* Readded accidental removal of stream recipe

amdfftw:
	1. Updated the aocc clang selection as per spack standards
fftw:
	1. Currently apple-clang section is redundant,
	already it is handled in the conflict checks.

Change-Id: Idef4a3f61717eb81f321e0cd16e7ba9619eac846

* Fix for style and docs/validate (pull_request) test

unnumbered format placeholders from {} to {0}

Change-Id: If67a3374177ec067573e5504462d257712fafc05

* changed compiler references to Spack's compiler wrapper:spack_cc, spack_cxx, spack_fc

Change-Id: I7ae29c978fff16e37773913f14c84df232499763

* Removed 'single' variant from amdfftw recipe

Instead of conflict for apple-clang + openmp, handled this senario
via below available feature:
depends_on('llvm-openmp', when='%apple-clang +openmp')

Change-Id: I701b23d83e822a500ca3aaf2b60cc9ace09e13dc

* Added relevant info for users who prefers to use single precision

Change-Id: I3506e21da428ddef5fb7895b5aaed32c2a061ef6

* Minor changes on fftw, amdfftw and libflame

amdfftw:
	1. Removed escape symbol to the single quotes
	2. Rewording the conflict line from Recommended
	to Required

fftw:
	1. Reorded to following recommended sections:
	versions, variants, dependencies, providers,
	patches

libflame:
	1. Added provides entry for 5.1.0 version

Change-Id: I21ebff99b6dfde031763154693ecb3f1fa47b476

* Removed single quote from amdfftw docstring to fix style failures

Change-Id: Ife939a5a2f5ccbc8879b730c7bebfe2fcfef9332
2020-10-31 11:57:17 -05:00
Chuck Atkins
aaa1afe9ec
mesa: Retire older autotools mesa as mesa18 and create current meson mesa (#19528) 2020-10-30 11:16:17 -04:00
GaneshPrasadMA
0253f0af29
Adding AOCC compiler to SPACK community (#19345)
* Adding AOCC compiler to SPACK community

The AOCC compiler system offers a high level of advanced optimizations, multi-threading and processor support that includes global optimization, vectorization, inter-procedural analyses, loop transformations, and code generation. AMD also provides highly optimized libraries, which extract the optimal performance from each x86 processor core when utilized. The AOCC Compiler Suite simplifies and accelerates development and tuning for x86 applications.

* Added unit tests for detection and flags for AOCC

* Addressed reviewers comments w.r.t version checks and url,checksum related line lengths

Co-authored-by: Test User <spack@example.com>
2020-10-20 10:50:09 -05:00