This PR adds a new command to Spack:
```console
$ spack containerize -h
usage: spack containerize [-h] [--config CONFIG]
creates recipes to build images for different container runtimes
optional arguments:
-h, --help show this help message and exit
--config CONFIG configuration for the container recipe that will be generated
```
which takes an environment with an additional `container` section:
```yaml
spack:
specs:
- gromacs build_type=Release
- mpich
- fftw precision=float
packages:
all:
target: [broadwell]
container:
# Select the format of the recipe e.g. docker,
# singularity or anything else that is currently supported
format: docker
# Select from a valid list of images
base:
image: "ubuntu:18.04"
spack: prerelease
# Additional system packages that are needed at runtime
os_packages:
- libgomp1
```
and turns it into a `Dockerfile` or a Singularity definition file, for instance:
```Dockerfile
# Build stage with Spack pre-installed and ready to be used
FROM spack/ubuntu-bionic:prerelease as builder
# What we want to install and how we want to install it
# is specified in a manifest file (spack.yaml)
RUN mkdir /opt/spack-environment \
&& (echo "spack:" \
&& echo " specs:" \
&& echo " - gromacs build_type=Release" \
&& echo " - mpich" \
&& echo " - fftw precision=float" \
&& echo " packages:" \
&& echo " all:" \
&& echo " target:" \
&& echo " - broadwell" \
&& echo " config:" \
&& echo " install_tree: /opt/software" \
&& echo " concretization: together" \
&& echo " view: /opt/view") > /opt/spack-environment/spack.yaml
# Install the software, remove unecessary deps and strip executables
RUN cd /opt/spack-environment && spack install && spack autoremove -y
RUN find -L /opt/view/* -type f -exec readlink -f '{}' \; | \
xargs file -i | \
grep 'charset=binary' | \
grep 'x-executable\|x-archive\|x-sharedlib' | \
awk -F: '{print $1}' | xargs strip -s
# Modifications to the environment that are necessary to run
RUN cd /opt/spack-environment && \
spack env activate --sh -d . >> /etc/profile.d/z10_spack_environment.sh
# Bare OS image to run the installed executables
FROM ubuntu:18.04
COPY --from=builder /opt/spack-environment /opt/spack-environment
COPY --from=builder /opt/software /opt/software
COPY --from=builder /opt/view /opt/view
COPY --from=builder /etc/profile.d/z10_spack_environment.sh /etc/profile.d/z10_spack_environment.sh
RUN apt-get -yqq update && apt-get -yqq upgrade \
&& apt-get -yqq install libgomp1 \
&& rm -rf /var/lib/apt/lists/*
ENTRYPOINT ["/bin/bash", "--rcfile", "/etc/profile", "-l"]
```
* Add binary_distribution::get_spec which takes concretized spec
Add binary_distribution::try_download_specs for downloading of spec.yaml files to cache
get_spec is used by package::try_install_from_binary_cache to download only the spec.yaml
for the concretized spec if it exists.
Previously, the install stage would compile in things that were
disabled during the build_ext phase. This would also result in the
build pulling in locally installed versions of libraries that were
disabled. The install process doesn't honor the same command-line
flags that build_ext does, but does call build_ext again. Avoid the
whole issue by just writing the options to setup.cfg
Also, add the Imagemagick dependency for tests.
The Spec parser currently calls `spec.traverse()` after every parse, in
order to set the platform if it's not set. We don't need to do a full
traverse -- we can just check the platforrm as new specs are parsed.
This takes about a second off the time required to import all packages in
Spack (from 8s to 7s).
- [x] simplify platform-setting logic in `SpecParser`.
`filename_for_package_name()` and `dirname_for_package_name()`
automatically construct a Spec from their arguments, which adds a fair
amount of overhead to importing lots of packages. Removing this removes
about 11% of the runtime of importing all packages in Spack (9s -> 8s).
- [x] `filename_for_package_name()` and `dirname_for_package_name()` now
take a string `pkg_name` arguments instead of specs.
* `Environment.__init__` is now synchronized with all writing operations
* `spack uninstall` now synchronizes its updates to any associated environment
* A side effect of this is that the environment is no longer updated piecemeal as specs are uninstalled - all specs are removed from the environment before they are uninstalled
* pumi: sim version check, meshes via submodule, ctest
* Apply suggestions from code review
Co-Authored-By: Adam J. Stewart <ajstewart426@gmail.com>
* pumi: update comment on master version string
* pumi: description of simmodsuite_version_check variant
* pumi: add white space to variant description
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
This commit makes two fundamental corrections to tests:
1) Changes 'matches' to the correct 'match' argument for 'pytest.raises' (for all affected tests except those checking for 'SystemExit');
2) Replaces the 'match' argument for tests expecting 'SystemExit' (since the exit code is retained instead) with 'capsys' error message capture.
Both changes are needed to ensure the associated exception message is actually checked.
* git: add version 2.25.0 and fixup pcre dependency
pcre2 became optional in 2.14 and the default in 2.18. I noticed this
as git was compiling against the system pcre2 (spack was
specifying pcre as the dependency).
* missed a chunk from my internal repo
Updates to environments were not multi-process safe, which prevented them from taking advantage of parallel builds as implemented in #13100. This is a minimal set of changes to enable `spack install` in an environment to be parallelized:
- [x] add an internal lock, stored in the `.spack-env` directory,
to synchronize updates to `spack.yaml` and `spack.lock`
- [x] add `Environment.write_transaction` interface for this lock
- [x] makes use of `Environment.write_transaction` in `install`,
`add`, and `remove` commands
- `uninstall` is not synchronized yet; that is left for a future PR.
* Set netcdf-fortran to build serially with Intel compiler
This PR turns off parallel builds when the Intel compiler is used.
Builds with the Intel compiler will fail otherwise.
* Change how parallel build is handled
Use patch from netcdf-fortran project to turn off parallel buildi for
version 4.5.2.
* diffutils: Changed the handling of undeclared functions from warning to error.
* diffutils: Change the handling of warnings or error
* Delete '-Werror=implicit-function-declaration'
* Add '-Qunused-arguments'
Replace the deprecated ADIOS1 backend default with ADIOS2 default.
Disable sz since we do not need it and it conflicts with supported
version ranges between ADIOS2 and ADIOS1 if someone enables both.
* intel-tbb: Fix install names on Darwin
Intel-TBB's libraries on Darwin are installed with "@rpath" prefixed
to their install names. This was found to cause issues building the root
package on Darwin due to libtbb not being found when running some of the
generated tools linking to it.
Follow example from other packages with the same issue and fixup up install
names for intel-tbb post install.
* intel-tbb: fix flake8 errors
* Dirty hack to fix#14148
* A better way of checking if a package is taken from system
* Update var/spack/repos/builtin/packages/qt/package.py
Co-Authored-By: Adam J. Stewart <ajstewart426@gmail.com>
* Update qt/package.py
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
Spack commands referring to upstream-installed specs by hash have
been broken since 6b619da (merged September 2019), which added a new
Database function specifically for parsing hashes from command-line
specs; this function was inappropriately attempting to acquire locks
on upstream databases.
This PR updates the offending function to avoid locking upstream
databases and also updates associated tests to catch regression
errors: the upstream database created for these tests was not
explicitly set as an upstream (i.e. initialized with upstream=True)
so it was not guarding against inappropriate accesses.
