hlrs-software-stack/spack
7
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity

python: always use a venv (#40773)

Browse source 
This commit adds a layer of indirection to improve build isolation with 
and without external Python, as well as usability of environment views.

It adds `python-venv` as a dependency to all packages that `extends("python")`, 
which has the following advantages:

1. Build isolation: only `PYTHONPATH` is considered in builds, not 
   user / system packages
2. Stable install layout: fixes the problem on Debian, RHEL and Fedora where 
   external / system python produces `bin/local` subdirs in Spack install prefixes. 
3. Environment views are Python virtual environments (and if you add 
   `py-pip` things like `pip list` work)

Views work whether they're symlink, hardlink or copy type.

This commit additionally makes `spec["python"].command` return 
`spec["python-venv"].command`. The rationale is that packages in repos we do 
not own do not pass the underlying python to the build system, which could still 
result in incorrectly computed install layouts.

Other attributes like `libs`, `headers` should be on `python` anyways and need no change.
This commit is contained in:
Harmen Stoppels 2024-05-06 16:17:35 +02:00 committed by GitHub
parent a081b875b4
commit 125206d44d
No known key found for this signature in database
GPG key ID: B5690EEEBB952194
55 changed files with 430 additions and 497 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

242
lib/spack/docs/basic_usage.rst
View file

@ -865,7 +865,7 @@ There are several different ways to use Spack packages once you have
installed them. As you've seen, spack packages are installed into long
paths with hashes, and you need a way to get them into your path. The
easiest way is to use :ref:`spack load <cmd-spack-load>`, which is
described in the next section.
described in this section.
Some more advanced ways to use Spack packages include:
@ -959,7 +959,86 @@ use ``spack find --loaded``.
You can also use ``spack load --list`` to get the same output, but it
does not have the full set of query options that ``spack find`` offers.
We'll learn more about Spack's spec syntax in the next section.
We'll learn more about Spack's spec syntax in :ref:`a later section <sec-specs>`.
.. _extensions:
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Python packages and virtual environments
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Spack can install a large number of Python packages. Their names are
typically prefixed with ``py-``. Installing and using them is no
different from any other package:
.. code-block:: console
$ spack install py-numpy
$ spack load py-numpy
$ python3
>>> import numpy
The ``spack load`` command sets the ``PATH`` variable so that the right Python
executable is used, and makes sure that ``numpy`` and its dependencies can be
located in the ``PYTHONPATH``.
Spack is different from other Python package managers in that it installs
every package into its *own* prefix. This is in contrast to ``pip``, which
installs all packages into the same prefix, be it in a virtual environment
or not.
For many users, **virtual environments** are more convenient than repeated
``spack load`` commands, particularly when working with multiple Python
packages. Fortunately Spack supports environments itself, which together
with a view are no different from Python virtual environments.
The recommended way of working with Python extensions such as ``py-numpy``
is through :ref:`Environments <environments>`. The following example creates
a Spack environment with ``numpy`` in the current working directory. It also
puts a filesystem view in ``./view``, which is a more traditional combined
prefix for all packages in the environment.
.. code-block:: console
$ spack env create --with-view view --dir .
$ spack -e . add py-numpy
$ spack -e . concretize
$ spack -e . install
Now you can activate the environment and start using the packages:
.. code-block:: console
$ spack env activate .
$ python3
>>> import numpy
The environment view is also a virtual environment, which is useful if you are
sharing the environment with others who are unfamiliar with Spack. They can
either use the Python executable directly:
.. code-block:: console
$ ./view/bin/python3
>>> import numpy
or use the activation script:
.. code-block:: console
$ source ./view/bin/activate
$ python3
>>> import numpy
In general, there should not be much difference between ``spack env activate``
and using the virtual environment. The main advantage of ``spack env activate``
is that it knows about more packages than just Python packages, and it may set
additional runtime variables that are not covered by the virtual environment
activation script.
See :ref:`environments` for a more in-depth description of Spack
environments and customizations to views.
.. _sec-specs:
@ -1705,165 +1784,6 @@ check only local packages (as opposed to those used transparently from
``upstream`` spack instances) and the ``-j,--json`` option to output
machine-readable json data for any errors.
.. _extensions:
---------------------------
Extensions & Python support
---------------------------
Spack's installation model assumes that each package will live in its
own install prefix. However, certain packages are typically installed
*within* the directory hierarchy of other packages. For example,
`Python <https://www.python.org>`_ packages are typically installed in the
``$prefix/lib/python-2.7/site-packages`` directory.
In Spack, installation prefixes are immutable, so this type of installation
is not directly supported. However, it is possible to create views that
allow you to merge install prefixes of multiple packages into a single new prefix.
Views are a convenient way to get a more traditional filesystem structure.
Using *extensions*, you can ensure that Python packages always share the
same prefix in the view as Python itself. Suppose you have
Python installed like so:
.. code-block:: console
$ spack find python
==> 1 installed packages.
-- linux-debian7-x86_64 / gcc@4.4.7 --------------------------------
python@2.7.8
.. _cmd-spack-extensions:
^^^^^^^^^^^^^^^^^^^^
``spack extensions``
^^^^^^^^^^^^^^^^^^^^
You can find extensions for your Python installation like this:
.. code-block:: console
$ spack extensions python
==> python@2.7.8%gcc@4.4.7 arch=linux-debian7-x86_64-703c7a96
==> 36 extensions:
geos py-ipython py-pexpect py-pyside py-sip
py-basemap py-libxml2 py-pil py-pytz py-six
py-biopython py-mako py-pmw py-rpy2 py-sympy
py-cython py-matplotlib py-pychecker py-scientificpython py-virtualenv
py-dateutil py-mpi4py py-pygments py-scikit-learn
py-epydoc py-mx py-pylint py-scipy
py-gnuplot py-nose py-pyparsing py-setuptools
py-h5py py-numpy py-pyqt py-shiboken
==> 12 installed:
-- linux-debian7-x86_64 / gcc@4.4.7 --------------------------------
py-dateutil@2.4.0 py-nose@1.3.4 py-pyside@1.2.2
py-dateutil@2.4.0 py-numpy@1.9.1 py-pytz@2014.10
py-ipython@2.3.1 py-pygments@2.0.1 py-setuptools@11.3.1
py-matplotlib@1.4.2 py-pyparsing@2.0.3 py-six@1.9.0
The extensions are a subset of what's returned by ``spack list``, and
they are packages like any other. They are installed into their own
prefixes, and you can see this with ``spack find --paths``:
.. code-block:: console
$ spack find --paths py-numpy
==> 1 installed packages.
-- linux-debian7-x86_64 / gcc@4.4.7 --------------------------------
py-numpy@1.9.1 ~/spack/opt/linux-debian7-x86_64/gcc@4.4.7/py-numpy@1.9.1-66733244
However, even though this package is installed, you cannot use it
directly when you run ``python``:
.. code-block:: console
$ spack load python
$ python
Python 2.7.8 (default, Feb 17 2015, 01:35:25)
[GCC 4.4.7 20120313 (Red Hat 4.4.7-11)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import numpy
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
ImportError: No module named numpy
>>>
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Using Extensions in Environments
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
The recommended way of working with extensions such as ``py-numpy``
above is through :ref:`Environments <environments>`. For example,
the following creates an environment in the current working directory
with a filesystem view in the ``./view`` directory:
.. code-block:: console
$ spack env create --with-view view --dir .
$ spack -e . add py-numpy
$ spack -e . concretize
$ spack -e . install
We recommend environments for two reasons. Firstly, environments
can be activated (requires :ref:`shell-support`):
.. code-block:: console
$ spack env activate .
which sets all the right environment variables such as ``PATH`` and
``PYTHONPATH``. This ensures that
.. code-block:: console
$ python
>>> import numpy
works. Secondly, even without shell support, the view ensures
that Python can locate its extensions:
.. code-block:: console
$ ./view/bin/python
>>> import numpy
See :ref:`environments` for a more in-depth description of Spack
environments and customizations to views.
^^^^^^^^^^^^^^^^^^^^
Using ``spack load``
^^^^^^^^^^^^^^^^^^^^
A more traditional way of using Spack and extensions is ``spack load``
(requires :ref:`shell-support`). This will add the extension to ``PYTHONPATH``
in your current shell, and Python itself will be available in the ``PATH``:
.. code-block:: console
$ spack load py-numpy
$ python
>>> import numpy
The loaded packages can be checked using ``spack find --loaded``
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Loading Extensions via Modules
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Apart from ``spack env activate`` and ``spack load``, you can load numpy
through your environment modules (using ``environment-modules`` or
``lmod``). This will also add the extension to the ``PYTHONPATH`` in
your current shell.
.. code-block:: console
$ module load <name of numpy module>
If you do not know the name of the specific numpy module you wish to
load, you can use the ``spack module tcl|lmod loads`` command to get
the name of the module from the Spack spec.
-----------------------
Filesystem requirements
-----------------------

42
lib/spack/docs/build_systems/pythonpackage.rst
View file

@ -718,23 +718,45 @@ command-line tool, or C/C++/Fortran program with optional Python
modules? The former should be prepended with ``py-``, while the
latter should not.
""""""""""""""""""""""
extends vs. depends_on
""""""""""""""""""""""
""""""""""""""""""""""""""""""
``extends`` vs. ``depends_on``
""""""""""""""""""""""""""""""
This is very similar to the naming dilemma above, with a slight twist.
As mentioned in the :ref:`Packaging Guide <packaging_extensions>`,
``extends`` and ``depends_on`` are very similar, but ``extends`` ensures
that the extension and extendee share the same prefix in views.
This allows the user to import a Python module without
having to add that module to ``PYTHONPATH``.
When deciding between ``extends`` and ``depends_on``, the best rule of
thumb is to check the installation prefix. If Python libraries are
installed to ``<prefix>/lib/pythonX.Y/site-packages``, then you
should use ``extends``. If Python libraries are installed elsewhere
or the only files that get installed reside in ``<prefix>/bin``, then
don't use ``extends``.
Additionally, ``extends("python")`` adds a dependency on the package
``python-venv``. This improves isolation from the system, whether
it's during the build or at runtime: user and system site packages
cannot accidentally be used by any package that ``extends("python")``.
As a rule of thumb: if a package does not install any Python modules
of its own, and merely puts a Python script in the ``bin`` directory,
then there is no need for ``extends``. If the package installs modules
in the ``site-packages`` directory, it requires ``extends``.
"""""""""""""""""""""""""""""""""""""
Executing ``python`` during the build
"""""""""""""""""""""""""""""""""""""
Whenever you need to execute a Python command or pass the path of the
Python interpreter to the build system, it is best to use the global
variable ``python`` directly. For example:
.. code-block:: python
@run_before("install")
def recythonize(self):
python("setup.py", "clean") # use the `python` global
As mentioned in the previous section, ``extends("python")`` adds an
automatic dependency on ``python-venv``, which is a virtual environment
that guarantees build isolation. The ``python`` global always refers to
the correct Python interpreter, whether the package uses ``extends("python")``
or ``depends_on("python")``.
^^^^^^^^^^^^^^^^^^^^^
Alternatives to Spack

10
lib/spack/spack/bootstrap/_common.py
View file

@ -54,10 +54,14 @@ def _try_import_from_store(
installed_specs = spack.store.STORE.db.query(query_spec, installed=True)
for candidate_spec in installed_specs:
pkg = candidate_spec["python"].package
# previously bootstrapped specs may not have a python-venv dependency.
if candidate_spec.dependencies("python-venv"):
python, *_ = candidate_spec.dependencies("python-venv")
else:
python, *_ = candidate_spec.dependencies("python")
module_paths = [
os.path.join(candidate_spec.prefix, pkg.purelib),
os.path.join(candidate_spec.prefix, pkg.platlib),
os.path.join(candidate_spec.prefix, python.package.purelib),
os.path.join(candidate_spec.prefix, python.package.platlib),
]
path_before = list(sys.path)

62
lib/spack/spack/bootstrap/environment.py
View file

@ -3,13 +3,11 @@
#
# SPDX-License-Identifier: (Apache-2.0 OR MIT)
"""Bootstrap non-core Spack dependencies from an environment."""
import glob
import hashlib
import os
import pathlib
import sys
import warnings
from typing import List
from typing import Iterable, List
import archspec.cpu
@ -28,6 +26,16 @@
class BootstrapEnvironment(spack.environment.Environment):
"""Environment to install dependencies of Spack for a given interpreter and architecture"""
def __init__(self) -> None:
if not self.spack_yaml().exists():
self._write_spack_yaml_file()
super().__init__(self.environment_root())
# Remove python package roots created before python-venv was introduced
for s in self.concrete_roots():
if "python" in s.package.extendees and not s.dependencies("python-venv"):
self.deconcretize(s)
@classmethod
def spack_dev_requirements(cls) -> List[str]:
"""Spack development requirements"""
@ -59,31 +67,19 @@ def view_root(cls) -> pathlib.Path:
return cls.environment_root().joinpath("view")
@classmethod
def pythonpaths(cls) -> List[str]:
"""Paths to be added to sys.path or PYTHONPATH"""
python_dir_part = f"python{'.'.join(str(x) for x in sys.version_info[:2])}"
glob_expr = str(cls.view_root().joinpath("**", python_dir_part, "**"))
result = glob.glob(glob_expr)
if not result:
msg = f"Cannot find any Python path in {cls.view_root()}"
warnings.warn(msg)
return result
@classmethod
def bin_dirs(cls) -> List[pathlib.Path]:
def bin_dir(cls) -> pathlib.Path:
"""Paths to be added to PATH"""
return [cls.view_root().joinpath("bin")]
return cls.view_root().joinpath("bin")
def python_dirs(self) -> Iterable[pathlib.Path]:
python = next(s for s in self.all_specs_generator() if s.name == "python-venv").package
return {self.view_root().joinpath(p) for p in (python.platlib, python.purelib)}
@classmethod
def spack_yaml(cls) -> pathlib.Path:
"""Environment spack.yaml file"""
return cls.environment_root().joinpath("spack.yaml")
def __init__(self) -> None:
if not self.spack_yaml().exists():
self._write_spack_yaml_file()
super().__init__(self.environment_root())
def update_installations(self) -> None:
"""Update the installations of this environment."""
log_enabled = tty.is_debug() or tty.is_verbose()
@ -100,21 +96,13 @@ def update_installations(self) -> None:
self.install_all()
self.write(regenerate=True)
def update_syspath_and_environ(self) -> None:
"""Update ``sys.path`` and the PATH, PYTHONPATH environment variables to point to
the environment view.
"""
# Do minimal modifications to sys.path and environment variables. In particular, pay
# attention to have the smallest PYTHONPATH / sys.path possible, since that may impact
# the performance of the current interpreter
sys.path.extend(self.pythonpaths())
os.environ["PATH"] = os.pathsep.join(
[str(x) for x in self.bin_dirs()] + os.environ.get("PATH", "").split(os.pathsep)
)
os.environ["PYTHONPATH"] = os.pathsep.join(
os.environ.get("PYTHONPATH", "").split(os.pathsep)
+ [str(x) for x in self.pythonpaths()]
)
def load(self) -> None:
"""Update PATH and sys.path."""
# Make executables available (shouldn't need PYTHONPATH)
os.environ["PATH"] = f"{self.bin_dir()}{os.pathsep}{os.environ.get('PATH', '')}"
# Spack itself imports pytest
sys.path.extend(str(p) for p in self.python_dirs())
def _write_spack_yaml_file(self) -> None:
tty.msg(
@ -164,4 +152,4 @@ def ensure_environment_dependencies() -> None:
_add_externals_if_missing()
with BootstrapEnvironment() as env:
env.update_installations()
env.update_syspath_and_environ()
env.load()

13
lib/spack/spack/build_systems/cmake.py
View file

@ -39,16 +39,11 @@ def _maybe_set_python_hints(pkg: spack.package_base.PackageBase, args: List[str]
"""Set the PYTHON_EXECUTABLE, Python_EXECUTABLE, and Python3_EXECUTABLE CMake variables
if the package has Python as build or link dep and ``find_python_hints`` is set to True. See
``find_python_hints`` for context."""
if not getattr(pkg, "find_python_hints", False):
if not getattr(pkg, "find_python_hints", False) or not pkg.spec.dependencies(
"python", dt.BUILD | dt.LINK
):
return
pythons = pkg.spec.dependencies("python", dt.BUILD | dt.LINK)
if len(pythons) != 1:
return
try:
python_executable = pythons[0].package.command.path
except RuntimeError:
return
python_executable = pkg.spec["python"].command.path
args.extend(
[
CMakeBuilder.define("PYTHON_EXECUTABLE", python_executable),

40
lib/spack/spack/build_systems/python.py
View file

@ -120,6 +120,12 @@ def skip_modules(self) -> Iterable[str]:
"""
return []
@property
def python_spec(self):
"""Get python-venv if it exists or python otherwise."""
python, *_ = self.spec.dependencies("python-venv") or self.spec.dependencies("python")
return python
def view_file_conflicts(self, view, merge_map):
"""Report all file conflicts, excepting special cases for python.
Specifically, this does not report errors for duplicate
@ -138,16 +144,17 @@ def view_file_conflicts(self, view, merge_map):
return conflicts
def add_files_to_view(self, view, merge_map, skip_if_exists=True):
# Patch up shebangs to the python linked in the view only if python is built by Spack.
if not self.extendee_spec or self.extendee_spec.external:
# Patch up shebangs if the package extends Python and we put a Python interpreter in the
# view.
python = self.python_spec
if not self.extendee_spec or python.external:
return super().add_files_to_view(view, merge_map, skip_if_exists)
# We only patch shebangs in the bin directory.
copied_files: Dict[Tuple[int, int], str] = {} # File identifier -> source
delayed_links: List[Tuple[str, str]] = [] # List of symlinks from merge map
bin_dir = self.spec.prefix.bin
python_prefix = self.extendee_spec.prefix
for src, dst in merge_map.items():
if skip_if_exists and os.path.lexists(dst):
continue
@ -168,7 +175,7 @@ def add_files_to_view(self, view, merge_map, skip_if_exists=True):
copied_files[(s.st_dev, s.st_ino)] = dst
shutil.copy2(src, dst)
fs.filter_file(
python_prefix, os.path.abspath(view.get_projection_for_spec(self.spec)), dst
python.prefix, os.path.abspath(view.get_projection_for_spec(self.spec)), dst
)
else:
view.link(src, dst)
@ -199,14 +206,13 @@ def remove_files_from_view(self, view, merge_map):
ignore_namespace = True
bin_dir = self.spec.prefix.bin
global_view = self.extendee_spec.prefix == view.get_projection_for_spec(self.spec)
to_remove = []
for src, dst in merge_map.items():
if ignore_namespace and namespace_init(dst):
continue
if global_view or not fs.path_contains_subdirectory(src, bin_dir):
if not fs.path_contains_subdirectory(src, bin_dir):
to_remove.append(dst)
else:
os.remove(dst)
@ -371,8 +377,9 @@ def headers(self) -> HeaderList:
# Headers should only be in include or platlib, but no harm in checking purelib too
include = self.prefix.join(self.spec["python"].package.include).join(name)
platlib = self.prefix.join(self.spec["python"].package.platlib).join(name)
purelib = self.prefix.join(self.spec["python"].package.purelib).join(name)
python = self.python_spec
platlib = self.prefix.join(python.package.platlib).join(name)
purelib = self.prefix.join(python.package.purelib).join(name)
headers_list = map(fs.find_all_headers, [include, platlib, purelib])
headers = functools.reduce(operator.add, headers_list)
@ -391,8 +398,9 @@ def libs(self) -> LibraryList:
name = self.spec.name[3:]
# Libraries should only be in platlib, but no harm in checking purelib too
platlib = self.prefix.join(self.spec["python"].package.platlib).join(name)
purelib = self.prefix.join(self.spec["python"].package.purelib).join(name)
python = self.python_spec
platlib = self.prefix.join(python.package.platlib).join(name)
purelib = self.prefix.join(python.package.purelib).join(name)
find_all_libraries = functools.partial(fs.find_all_libraries, recursive=True)
libs_list = map(find_all_libraries, [platlib, purelib])
@ -504,6 +512,8 @@ def global_options(self, spec: Spec, prefix: Prefix) -> Iterable[str]:
def install(self, pkg: PythonPackage, spec: Spec, prefix: Prefix) -> None:
"""Install everything from build directory."""
pip = spec["python"].command
pip.add_default_arg("-m", "pip")
args = PythonPipBuilder.std_args(pkg) + [f"--prefix={prefix}"]
@ -519,14 +529,6 @@ def install(self, pkg: PythonPackage, spec: Spec, prefix: Prefix) -> None:
else:
args.append(".")
pip = spec["python"].command
# Hide user packages, since we don't have build isolation. This is
# necessary because pip / setuptools may run hooks from arbitrary
# packages during the build. There is no equivalent variable to hide
# system packages, so this is not reliable for external Python.
pip.add_default_env("PYTHONNOUSERSITE", "1")
pip.add_default_arg("-m")
pip.add_default_arg("pip")
with fs.working_dir(self.build_directory):
pip(*args)

6
lib/spack/spack/directives.py
View file

@ -662,6 +662,7 @@ def _execute_redistribute(
@directive(("extendees", "dependencies"))
def extends(spec, when=None, type=("build", "run"), patches=None):
"""Same as depends_on, but also adds this package to the extendee list.
In case of Python, also adds a dependency on python-venv.
keyword arguments can be passed to extends() so that extension
packages can pass parameters to the extendee's extension
@ -677,6 +678,11 @@ def _execute_extends(pkg):
_depends_on(pkg, spec, when=when, type=type, patches=patches)
spec_obj = spack.spec.Spec(spec)
# When extending python, also add a dependency on python-venv. This is done so that
# Spack environment views are Python virtual environments.
if spec_obj.name == "python" and not pkg.name == "python-venv":
_depends_on(pkg, "python-venv", when=when, type=("build", "run"))
# TODO: the values of the extendees dictionary are not used. Remove in next refactor.
pkg.extendees[spec_obj.name] = (spec_obj, None)

8
lib/spack/spack/hooks/windows_runtime_linkage.py Normal file
View file

@ -0,0 +1,8 @@
# Copyright 2013-2024 Lawrence Livermore National Security, LLC and other
# Spack Project Developers. See the top-level COPYRIGHT file for details.
#
# SPDX-License-Identifier: (Apache-2.0 OR MIT)
def post_install(spec, explicit=None):
spec.package.windows_establish_runtime_linkage()

4
lib/spack/spack/installer.py
View file

@ -1698,10 +1698,6 @@ def _install_task(self, task: BuildTask, install_status: InstallStatus) -> None:
spack.package_base.PackageBase._verbose = spack.build_environment.start_build_process(
pkg, build_process, install_args
)
# Currently this is how RPATH-like behavior is achieved on Windows, after install
# establish runtime linkage via Windows Runtime link object
# Note: this is a no-op on non Windows platforms
pkg.windows_establish_runtime_linkage()
# Note: PARENT of the build process adds the new package to
# the database, so that we don't need to re-read from file.
spack.store.STORE.db.add(pkg.spec, spack.store.STORE.layout, explicit=explicit)

95
lib/spack/spack/spec.py
View file

@ -1030,16 +1030,13 @@ def clear(self):
self.edges.clear()
def _command_default_handler(descriptor, spec, cls):
def _command_default_handler(spec: "Spec"):
"""Default handler when looking for the 'command' attribute.
Tries to search for ``spec.name`` in the ``spec.home.bin`` directory.
Parameters:
descriptor (ForwardQueryToPackage): descriptor that triggered the call
spec (Spec): spec that is being queried
cls (type(spec)): type of spec, to match the signature of the
descriptor ``__get__`` method
spec: spec that is being queried
Returns:
Executable: An executable of the command
@ -1052,22 +1049,17 @@ def _command_default_handler(descriptor, spec, cls):
if fs.is_exe(path):
return spack.util.executable.Executable(path)
else:
msg = "Unable to locate {0} command in {1}"
raise RuntimeError(msg.format(spec.name, home.bin))
raise RuntimeError(f"Unable to locate {spec.name} command in {home.bin}")
def _headers_default_handler(descriptor, spec, cls):
def _headers_default_handler(spec: "Spec"):
"""Default handler when looking for the 'headers' attribute.
Tries to search for ``*.h`` files recursively starting from
``spec.package.home.include``.
Parameters:
descriptor (ForwardQueryToPackage): descriptor that triggered the call
spec (Spec): spec that is being queried
cls (type(spec)): type of spec, to match the signature of the
descriptor ``__get__`` method
spec: spec that is being queried
Returns:
HeaderList: The headers in ``prefix.include``
@ -1080,12 +1072,10 @@ def _headers_default_handler(descriptor, spec, cls):
if headers:
return headers
else:
msg = "Unable to locate {0} headers in {1}"
raise spack.error.NoHeadersError(msg.format(spec.name, home))
raise spack.error.NoHeadersError(f"Unable to locate {spec.name} headers in {home}")
def _libs_default_handler(descriptor, spec, cls):
def _libs_default_handler(spec: "Spec"):
"""Default handler when looking for the 'libs' attribute.
Tries to search for ``lib{spec.name}`` recursively starting from
@ -1093,10 +1083,7 @@ def _libs_default_handler(descriptor, spec, cls):
``{spec.name}`` instead.
Parameters:
descriptor (ForwardQueryToPackage): descriptor that triggered the call
spec (Spec): spec that is being queried
cls (type(spec)): type of spec, to match the signature of the
descriptor ``__get__`` method
spec: spec that is being queried
Returns:
LibraryList: The libraries found
@ -1135,27 +1122,33 @@ def _libs_default_handler(descriptor, spec, cls):
if libs:
return libs
msg = "Unable to recursively locate {0} libraries in {1}"
raise spack.error.NoLibrariesError(msg.format(spec.name, home))
raise spack.error.NoLibrariesError(
f"Unable to recursively locate {spec.name} libraries in {home}"
)
class ForwardQueryToPackage:
"""Descriptor used to forward queries from Spec to Package"""
def __init__(self, attribute_name, default_handler=None):
def __init__(
self,
attribute_name: str,
default_handler: Optional[Callable[["Spec"], Any]] = None,
_indirect: bool = False,
) -> None:
"""Create a new descriptor.
Parameters:
attribute_name (str): name of the attribute to be
searched for in the Package instance
default_handler (callable, optional): default function to be
called if the attribute was not found in the Package
instance
attribute_name: name of the attribute to be searched for in the Package instance
default_handler: default function to be called if the attribute was not found in the
Package instance
_indirect: temporarily added to redirect a query to another package.
"""
self.attribute_name = attribute_name
self.default = default_handler
self.indirect = _indirect
def __get__(self, instance, cls):
def __get__(self, instance: "SpecBuildInterface", cls):
"""Retrieves the property from Package using a well defined chain
of responsibility.
@ -1177,13 +1170,18 @@ def __get__(self, instance, cls):
indicating a query failure, e.g. that library files were not found in a
'libs' query.
"""
pkg = instance.package
# TODO: this indirection exist solely for `spec["python"].command` to actually return
# spec["python-venv"].command. It should be removed when `python` is a virtual.
if self.indirect and instance.indirect_spec:
pkg = instance.indirect_spec.package
else:
pkg = instance.wrapped_obj.package
try:
query = instance.last_query
except AttributeError:
# There has been no query yet: this means
# a spec is trying to access its own attributes
_ = instance[instance.name] # NOQA: ignore=F841
_ = instance.wrapped_obj[instance.wrapped_obj.name] # NOQA: ignore=F841
query = instance.last_query
callbacks_chain = []
@ -1195,7 +1193,8 @@ def __get__(self, instance, cls):
callbacks_chain.append(lambda: getattr(pkg, self.attribute_name))
# Final resort : default callback
if self.default is not None:
callbacks_chain.append(lambda: self.default(self, instance, cls))
_default = self.default # make mypy happy
callbacks_chain.append(lambda: _default(instance.wrapped_obj))
# Trigger the callbacks in order, the first one producing a
# value wins
@ -1254,25 +1253,33 @@ def __set__(self, instance, value):
class SpecBuildInterface(lang.ObjectWrapper):
# home is available in the base Package so no default is needed
home = ForwardQueryToPackage("home", default_handler=None)
command = ForwardQueryToPackage("command", default_handler=_command_default_handler)
headers = ForwardQueryToPackage("headers", default_handler=_headers_default_handler)
libs = ForwardQueryToPackage("libs", default_handler=_libs_default_handler)
command = ForwardQueryToPackage(
"command", default_handler=_command_default_handler, _indirect=True
)
def __init__(self, spec, name, query_parameters):
def __init__(self, spec: "Spec", name: str, query_parameters: List[str], _parent: "Spec"):
super().__init__(spec)
# Adding new attributes goes after super() call since the ObjectWrapper
# resets __dict__ to behave like the passed object
original_spec = getattr(spec, "wrapped_obj", spec)
self.wrapped_obj = original_spec
self.token = original_spec, name, query_parameters
self.token = original_spec, name, query_parameters, _parent
is_virtual = spack.repo.PATH.is_virtual(name)
self.last_query = QueryState(
name=name, extra_parameters=query_parameters, isvirtual=is_virtual
)
# TODO: this ad-hoc logic makes `spec["python"].command` return
# `spec["python-venv"].command` and should be removed when `python` is a virtual.
self.indirect_spec = None
if spec.name == "python":
python_venvs = _parent.dependencies("python-venv")
if not python_venvs:
return
self.indirect_spec = python_venvs[0]
def __reduce__(self):
return SpecBuildInterface, self.token
@ -4137,7 +4144,7 @@ def version(self):
raise spack.error.SpecError("Spec version is not concrete: " + str(self))
return self.versions[0]
def __getitem__(self, name):
def __getitem__(self, name: str):
"""Get a dependency from the spec by its name. This call implicitly
sets a query state in the package being retrieved. The behavior of
packages may be influenced by additional query parameters that are
@ -4146,7 +4153,7 @@ def __getitem__(self, name):
Note that if a virtual package is queried a copy of the Spec is
returned while for non-virtual a reference is returned.
"""
query_parameters = name.split(":")
query_parameters: List[str] = name.split(":")
if len(query_parameters) > 2:
raise KeyError("key has more than one ':' symbol. At most one is admitted.")
@ -4169,7 +4176,7 @@ def __getitem__(self, name):
)
try:
value = next(
child: Spec = next(
itertools.chain(
# Regular specs
(x for x in order() if x.name == name),
@ -4186,9 +4193,9 @@ def __getitem__(self, name):
raise KeyError(f"No spec with name {name} in {self}")
if self._concrete:
return SpecBuildInterface(value, name, query_parameters)
return SpecBuildInterface(child, name, query_parameters, _parent=self)
return value
return child
def __contains__(self, spec):
"""True if this spec or some dependency satisfies the spec.

14
lib/spack/spack/test/cmd/extensions.py
View file

@ -33,21 +33,23 @@ def check_output(ni):
packages = extensions("-s", "packages", "python")
installed = extensions("-s", "installed", "python")
assert "==> python@2.7.11" in output
assert "==> 2 extensions" in output
assert "==> 3 extensions" in output
assert "py-extension1" in output
assert "py-extension2" in output
assert "python-venv" in output
assert "==> 2 extensions" in packages
assert "==> 3 extensions" in packages
assert "py-extension1" in packages
assert "py-extension2" in packages
assert "python-venv" in packages
assert "installed" not in packages
assert ("%s installed" % (ni if ni else "None")) in output
assert ("%s installed" % (ni if ni else "None")) in installed
assert f"{ni if ni else 'None'} installed" in output
assert f"{ni if ni else 'None'} installed" in installed
check_output(2)
check_output(3)
ext2.package.do_uninstall(force=True)
check_output(1)
check_output(2)
def test_extensions_no_arguments(mock_packages):

14
lib/spack/spack/util/executable.py
View file

@ -39,6 +39,20 @@ def add_default_arg(self, *args):
"""Add default argument(s) to the command."""
self.exe.extend(args)
def with_default_args(self, *args):
"""Same as add_default_arg, but returns a copy of the executable."""
new = self.copy()
new.add_default_arg(*args)
return new
def copy(self):
"""Return a copy of this Executable."""
new = Executable(self.exe[0])
new.exe[:] = self.exe
new.default_env.update(self.default_env)
new.default_envmod.extend(self.default_envmod)
return new
def add_default_env(self, key, value):
"""Set an environment variable when the command is run.

21
var/spack/repos/builtin.mock/packages/python-venv/package.py Normal file
View file

@ -0,0 +1,21 @@
# Copyright 2013-2023 Lawrence Livermore National Security, LLC and other
# Spack Project Developers. See the top-level COPYRIGHT file for details.
#
# SPDX-License-Identifier: (Apache-2.0 OR MIT)
from spack.package import *
class PythonVenv(Package):
"""A Spack managed Python virtual environment"""
homepage = "https://docs.python.org/3/library/venv.html"
has_code = False
version("1.0")
extends("python")
def install(self, spec, prefix):
pass

2
var/spack/repos/builtin/packages/ascent/package.py
View file

@ -495,7 +495,7 @@ def hostconfig(self):
cfg.write("# Enable python module builds\n")
cfg.write(cmake_cache_entry("ENABLE_PYTHON", "ON"))
cfg.write("# python from spack \n")
cfg.write(cmake_cache_entry("PYTHON_EXECUTABLE", spec["python"].command.path))
cfg.write(cmake_cache_entry("PYTHON_EXECUTABLE", python.path))
try:
cfg.write("# python module install dir\n")
cfg.write(cmake_cache_entry("PYTHON_MODULE_INSTALL_PREFIX", python_platlib))

7
var/spack/repos/builtin/packages/bohrium/package.py
View file

@ -256,10 +256,6 @@ def check_install(self):
cxx("-o", "test_cxxadd", file_cxxadd, *cxx_flags)
test_cxxadd = Executable("./test_cxxadd")
# Build python test commandline
file_pyadd = join_path(os.path.dirname(self.module.__file__), "pyadd.py")
test_pyadd = Executable(spec["python"].command.path + " " + file_pyadd)
# Run tests for each available stack
for bh_stack in stacks:
tty.info("Testing with bohrium stack '" + bh_stack + "'")
@ -270,5 +266,6 @@ def check_install(self):
# Python test (if +python)
if "+python" in spec:
py_output = test_pyadd(output=str, env=test_env)
file_pyadd = join_path(os.path.dirname(self.module.__file__), "pyadd.py")
py_output = python(file_pyadd, output=str, env=test_env)
compare_output(py_output, "Success!\n")

8
var/spack/repos/builtin/packages/cantera/package.py
View file

@ -134,13 +134,9 @@ def build_args(self, spec, prefix):
# Python module
if "+python" in spec:
args.extend(
["python_package=full", "python_cmd={0}".format(spec["python"].command.path)]
)
args.extend(["python_package=full", "python_cmd={0}".format(python.path)])
if spec["python"].satisfies("@3:"):
args.extend(
["python3_package=y", "python3_cmd={0}".format(spec["python"].command.path)]
)
args.extend(["python3_package=y", "python3_cmd={0}".format(python.path)])
else:
args.append("python3_package=n")
else:

4
var/spack/repos/builtin/packages/clingo-bootstrap/package.py
View file

@ -120,9 +120,7 @@ def pgo_train(self):
)
python_runtime_env.unset("SPACK_ENV")
python_runtime_env.unset("SPACK_PYTHON")
self.spec["python"].command(
spack.paths.spack_script, "solve", "--fresh", "hdf5", extra_env=python_runtime_env
)
python(spack.paths.spack_script, "solve", "--fresh", "hdf5", extra_env=python_runtime_env)
# Clean the build dir.
rmtree(self.build_directory, ignore_errors=True)

7
var/spack/repos/builtin/packages/clingo/package.py
View file

@ -3,8 +3,6 @@
#
# SPDX-License-Identifier: (Apache-2.0 OR MIT)
import os
from spack.compiler import UnsupportedCompilerFlag
from spack.package import *
@ -123,7 +121,4 @@ def cmake_args(self):
return args
def win_add_library_dependent(self):
if "+python" in self.spec:
return [os.path.join(self.prefix, self.spec["python"].package.platlib)]
else:
return []
return [python_platlib] if "+python" in self.spec else []

2
var/spack/repos/builtin/packages/conduit/package.py
View file

@ -443,7 +443,7 @@ def hostconfig(self):
cfg.write("# Enable python module builds\n")
cfg.write(cmake_cache_entry("ENABLE_PYTHON", "ON"))
cfg.write("# python from spack \n")
cfg.write(cmake_cache_entry("PYTHON_EXECUTABLE", spec["python"].command.path))
cfg.write(cmake_cache_entry("PYTHON_EXECUTABLE", python.path))
try:
cfg.write("# python module install dir\n")
cfg.write(cmake_cache_entry("PYTHON_MODULE_INSTALL_PREFIX", python_platlib))

1
var/spack/repos/builtin/packages/dsqss/package.py
View file

@ -58,7 +58,6 @@ def test_dla(self):
copy(join_path(test01, "std.toml"), ".")
# prepare
python = self.spec["python"].command
opts = [self.spec.prefix.bin.dla_pre, "std.toml"]
with test_part(self, "test_dla_pre", purpose="prepare dla"):
python(*opts)

7
var/spack/repos/builtin/packages/fenics/package.py
View file

@ -156,7 +156,7 @@ class Fenics(CMakePackage):
depends_on("py-sphinx@1.0.1:", when="+doc", type="build")
def cmake_args(self):
args = [
return [
self.define_from_variant("BUILD_SHARED_LIBS", "shared"),
self.define("DOLFIN_SKIP_BUILD_TESTS", True),
self.define_from_variant("DOLFIN_ENABLE_OPENMP", "openmp"),
@ -180,11 +180,6 @@ def cmake_args(self):
self.define_from_variant("DOLFIN_ENABLE_ZLIB", "zlib"),
]
if "+python" in self.spec:
args.append(self.define("PYTHON_EXECUTABLE", self.spec["python"].command.path))
return args
# set environment for bulding python interface
def setup_build_environment(self, env):
env.set("DOLFIN_DIR", self.prefix)

1
var/spack/repos/builtin/packages/gurobi/package.py
View file

@ -57,5 +57,4 @@ def install(self, spec, prefix):
@run_after("install")
def gurobipy(self):
with working_dir("linux64"):
python = which("python")
python("setup.py", "install", "--prefix={0}".format(self.prefix))

8
var/spack/repos/builtin/packages/lbann/package.py
View file

@ -402,12 +402,8 @@ def initconfig_package_entries(self):
)
entries.append(cmake_cache_option("protobuf_MODULE_COMPATIBLE", True))
if spec.satisfies("^python") and "+pfe" in spec:
entries.append(
cmake_cache_path(
"LBANN_PFE_PYTHON_EXECUTABLE", "{0}/python3".format(spec["python"].prefix.bin)
)
)
if spec.satisfies("+pfe ^python"):
entries.append(cmake_cache_path("LBANN_PFE_PYTHON_EXECUTABLE", python.path))
entries.append(
cmake_cache_string("LBANN_PFE_PYTHONPATH", env["PYTHONPATH"])
) # do NOT need to sub ; for : because

2
var/spack/repos/builtin/packages/libcap-ng/package.py
View file

@ -33,7 +33,7 @@ class LibcapNg(AutotoolsPackage):
def setup_build_environment(self, env):
if self.spec.satisfies("+python"):
env.set("PYTHON", self.spec["python"].command.path)
env.set("PYTHON", python.path)
def configure_args(self):
args = []

6
var/spack/repos/builtin/packages/mapserver/package.py
View file

@ -3,8 +3,6 @@
#
# SPDX-License-Identifier: (Apache-2.0 OR MIT)
import os
from spack.package import *
@ -60,9 +58,7 @@ def patch(self):
# prefix. This hack patches the CMakeLists.txt for the Python
# bindings and hard-wires in the right destination. A bit ugly,
# sorry, but I don't speak cmake.
pyversiondir = "python{0}".format(self.spec["python"].version.up_to(2))
sitepackages = os.path.join(self.spec.prefix.lib, pyversiondir, "site-packages")
filter_file(r"\${PYTHON_SITE_PACKAGES}", sitepackages, "mapscript/python/CMakeLists.txt")
filter_file(r"\${PYTHON_SITE_PACKAGES}", python_platlib, "mapscript/python/CMakeLists.txt")
def cmake_args(self):
args = []

4
var/spack/repos/builtin/packages/omnitrace/package.py
View file

@ -141,7 +141,3 @@ def setup_build_environment(self, env):
files = glob.glob(pattern)
if files:
env.set("TAU_MAKEFILE", files[0])
def setup_run_environment(self, env):
if "+python" in self.spec:
env.prepend_path("PYTHONPATH", join_path(self.prefix.lib, "python", "site-packages"))

2
var/spack/repos/builtin/packages/open3d/package.py
View file

@ -119,7 +119,7 @@ def install(self, spec, prefix):
def test(self):
if "+python" in self.spec:
self.run_test(
self.spec["python"].command.path,
python.path,
["-c", "import open3d"],
purpose="checking import of open3d",
work_dir="spack-test",

3
var/spack/repos/builtin/packages/opencv/package.py
View file

@ -1027,14 +1027,13 @@ def cmake_args(self):
)
# Python
python_exe = spec["python"].command.path
python_lib = spec["python"].libs[0]
python_include_dir = spec["python"].headers.directories[0]
if "+python3" in spec:
args.extend(
[
self.define("PYTHON3_EXECUTABLE", python_exe),
self.define("PYTHON3_EXECUTABLE", python.path),
self.define("PYTHON3_LIBRARY", python_lib),
self.define("PYTHON3_INCLUDE_DIR", python_include_dir),
self.define("PYTHON2_EXECUTABLE", ""),

6
var/spack/repos/builtin/packages/openwsman/package.py
View file

@ -31,13 +31,9 @@ class Openwsman(CMakePackage):
def patch(self):
"""Change python install directory."""
if self.spec.satisfies("+python"):
python_spec = self.spec["python"]
python_libdir = join_path(
self.spec.prefix.lib, "python" + str(python_spec.version.up_to(2)), "site-packages"
)
filter_file(
"DESTINATION .*",
"DESTINATION {0} )".format(python_libdir),
"DESTINATION {0} )".format(python_platlib),
join_path("bindings", "python", "CMakeLists.txt"),
)

2
var/spack/repos/builtin/packages/precice/package.py
View file

@ -191,7 +191,7 @@ def cmake_args(self):
python_library = spec["python"].libs[0]
python_include = spec["python"].headers.directories[0]
numpy_include = join_path(
spec["py-numpy"].prefix, spec["python"].package.platlib, "numpy", "core", "include"
spec["py-numpy"].package.module.python_platlib, "numpy", "core", "include"
)
cmake_args.extend(
[

2
var/spack/repos/builtin/packages/py-alphafold/package.py
View file

@ -77,7 +77,7 @@ class PyAlphafold(PythonPackage, CudaPackage):
@run_after("install")
def install_scripts(self):
mkdirp(self.prefix.bin)
shebang = "#!{0}\n".format(self.spec["python"].command)
shebang = f"#!{python.path}\n"
for fname in glob.glob("run_alphafold*.py"):
destfile = join_path(self.prefix.bin, fname)
with open(fname, "r") as src:

2
var/spack/repos/builtin/packages/py-chainer/package.py
View file

@ -59,7 +59,7 @@ def test_chainermn(self):
mnist_file = join_path(self.install_test_root.examples.chainermn.mnist, "train_mnist.py")
mpirun = which(self.spec["mpi"].prefix.bin.mpirun)
opts = ["-n", "4", self.spec["python"].command.path, mnist_file, "-o", "."]
opts = ["-n", "4", python.path, mnist_file, "-o", "."]
env["OMP_NUM_THREADS"] = "4"
mpirun(*opts)

1
var/spack/repos/builtin/packages/py-eccodes/package.py
View file

@ -43,5 +43,4 @@ def setup_dependent_run_environment(self, env, dependent_spec):
def test_selfcheck(self):
"""checking system setup"""
python = self.spec["python"].command
python("-m", "eccodes", "selfcheck")

1
var/spack/repos/builtin/packages/py-gmxapi/package.py
View file

@ -49,5 +49,4 @@ def setup_build_environment(self, env):
def install_test(self):
with working_dir("spack-test", create=True):
# test include helper points to right location
python = self.spec["python"].command
python("-m", "pytest", "-x", os.path.join(self.build_directory, "test"))

2
var/spack/repos/builtin/packages/py-gpaw/package.py
View file

@ -59,7 +59,7 @@ def patch(self):
python_include = spec["python"].headers.directories[0]
numpy_include = join_path(
spec["py-numpy"].prefix, spec["python"].package.platlib, "numpy", "core", "include"
spec["py-numpy"].package.module.python_platlib, "numpy", "core", "include"
)
libs = blas.libs + lapack.libs + libxc.libs

4
var/spack/repos/builtin/packages/py-installer/package.py
View file

@ -36,6 +36,4 @@ def install(self, spec, prefix):
python(*args)
def setup_dependent_package(self, module, dependent_spec):
installer = dependent_spec["python"].command
installer.add_default_arg("-m", "installer")
setattr(module, "installer", installer)
setattr(module, "installer", python.with_default_args("-m", "installer"))

2
var/spack/repos/builtin/packages/py-ipykernel/package.py
View file

@ -89,4 +89,4 @@ class PyIpykernel(PythonPackage):
@run_after("install")
def install_data(self):
"""install the Jupyter kernel spec"""
self.spec["python"].command("-m", "ipykernel", "install", "--prefix=" + self.prefix)
python("-m", "ipykernel", "install", "--prefix=" + self.prefix)

1
var/spack/repos/builtin/packages/py-libensemble/package.py
View file

@ -100,7 +100,6 @@ def run_tutorial_script(self, script):
if not os.path.isfile(exe):
raise SkipTest(f"{script} is missing")
python = self.spec["python"].command
python(exe, "--comms", "local", "--nworkers", "2")
def test_uniform_sampling(self):

2
var/spack/repos/builtin/packages/py-nanobind/package.py
View file

@ -60,5 +60,5 @@ class PyNanobind(PythonPackage):
@property
def cmake_prefix_paths(self):
paths = [join_path(self.prefix, self.spec["python"].package.platlib, "nanobind", "cmake")]
paths = [join_path(python_platlib, "nanobind", "cmake")]
return paths

6
var/spack/repos/builtin/packages/py-pip/package.py
View file

@ -77,7 +77,5 @@ def install(self, spec, prefix):
args.insert(0, os.path.join(whl, "pip"))
python(*args)
def setup_dependent_package(self, module, dependent_spec):
pip = dependent_spec["python"].command
pip.add_default_arg("-m", "pip")
setattr(module, "pip", pip)
def setup_dependent_package(self, module, dependent_spec: Spec):
setattr(module, "pip", python.with_default_args("-m", "pip"))

1
var/spack/repos/builtin/packages/py-pybind11/package.py
View file

@ -102,7 +102,6 @@ def install_test(self):
with working_dir("spack-test", create=True):
# test include helper points to right location
python = self.spec["python"].command
py_inc = python(
"-c", "import pybind11 as py; print(py.get_include())", output=str
).strip()

2
var/spack/repos/builtin/packages/py-pymol/package.py
View file

@ -58,7 +58,7 @@ def install_launcher(self):
script = join_path(python_platlib, "pymol", "__init__.py")
shebang = "#!/bin/sh\n"
fdata = 'exec {0} {1} "$@"'.format(self.spec["python"].command, script)
fdata = f'exec {python.path} {script} "$@"'
with open(fname, "w") as new:
new.write(shebang + fdata)
set_executable(fname)

2
var/spack/repos/builtin/packages/py-pyqt4/package.py
View file

@ -48,7 +48,7 @@ def configure_args(self):
"--destdir",
python_platlib,
"--pyuic4-interpreter",
self.spec["python"].command.path,
python.path,
"--sipdir",
self.prefix.share.sip.PyQt4,
"--stubsdir",

4
var/spack/repos/builtin/packages/py-pyspark/package.py
View file

@ -25,5 +25,5 @@ class PyPyspark(PythonPackage):
depends_on("py-py4j@0.10.9", when="@3.0.1:3.1.3", type=("build", "run"))
def setup_run_environment(self, env):
env.set("PYSPARK_PYTHON", self.spec["python"].command.path)
env.set("PYSPARK_DRIVER_PYTHON", self.spec["python"].command.path)
env.set("PYSPARK_PYTHON", python.path)
env.set("PYSPARK_DRIVER_PYTHON", python.path)

7
var/spack/repos/builtin/packages/py-pythonsollya/package.py
View file

@ -34,9 +34,4 @@ class PyPythonsollya(PythonPackage):
@run_before("install")
def patch(self):
filter_file(
"PYTHON ?= python2",
"PYTHON ?= " + self.spec["python"].command.path,
"GNUmakefile",
string=True,
)
filter_file("PYTHON ?= python2", f"PYTHON ?= {python.path}", "GNUmakefile", string=True)

2
var/spack/repos/builtin/packages/py-tensorflow/package.py
View file

@ -410,7 +410,7 @@ def setup_build_environment(self, env):
spec = self.spec
# Please specify the location of python
env.set("PYTHON_BIN_PATH", spec["python"].command.path)
env.set("PYTHON_BIN_PATH", python.path)
# Please input the desired Python library path to use
env.set("PYTHON_LIB_PATH", python_platlib)

4
var/spack/repos/builtin/packages/py-torch/package.py
View file

@ -688,7 +688,5 @@ def install_test(self):
@property
def cmake_prefix_paths(self):
cmake_prefix_paths = [
join_path(self.prefix, self.spec["python"].package.platlib, "torch", "share", "cmake")
]
cmake_prefix_paths = [join_path(python_platlib, "torch", "share", "cmake")]
return cmake_prefix_paths

3
var/spack/repos/builtin/packages/py-xdot/package.py
View file

@ -50,8 +50,7 @@ def post_install(self):
dst,
)
# regenerate the byte-compiled __init__.py
python3 = spec["python"].command
python3("-m", "compileall", dst)
python("-m", "compileall", dst)
def setup_run_environment(self, env):
spec = self.spec

100
var/spack/repos/builtin/packages/python-venv/package.py Normal file
View file

@ -0,0 +1,100 @@
# Copyright 2013-2023 Lawrence Livermore National Security, LLC and other
# Spack Project Developers. See the top-level COPYRIGHT file for details.
#
# SPDX-License-Identifier: (Apache-2.0 OR MIT)
import os
import shutil
import llnl.util.filesystem as fs
from spack.package import *
class PythonVenv(Package):
"""A Spack managed Python virtual environment"""
homepage = "https://docs.python.org/3/library/venv.html"
has_code = False
maintainers("haampie")
version("1.0")
extends("python")
def install(self, spec, prefix):
# Create a virtual environment
python("-m", "venv", "--without-pip", prefix)
def add_files_to_view(self, view, merge_map: Dict[str, str], skip_if_exists=True):
for src, dst in merge_map.items():
if skip_if_exists and os.path.lexists(dst):
continue
name = os.path.basename(dst)
# Replace the VIRTUAL_ENV variable in the activate scripts after copying
if name.lower().startswith("activate"):
shutil.copy(src, dst)
fs.filter_file(
self.spec.prefix,
os.path.abspath(view.get_projection_for_spec(self.spec)),
dst,
string=True,
)
else:
view.link(src, dst)
@property
def bindir(self):
windows = self.spec.satisfies("platform=windows")
return join_path(self.prefix, "Scripts" if windows else "bin")
@property
def command(self):
"""Returns a python Executable instance"""
return which("python3", path=self.bindir)
def _get_path(self, name) -> str:
return self.command(
"-Ec", f"import sysconfig; print(sysconfig.get_path('{name}'))", output=str
).strip()
@property
def platlib(self) -> str:
"""Directory for site-specific, platform-specific files."""
relative_platlib = os.path.relpath(self._get_path("platlib"), self.prefix)
assert not relative_platlib.startswith("..")
return relative_platlib
@property
def purelib(self) -> str:
"""Directory for site-specific, non-platform-specific files."""
relative_purelib = os.path.relpath(self._get_path("purelib"), self.prefix)
assert not relative_purelib.startswith("..")
return relative_purelib
@property
def headers(self):
return HeaderList([])
@property
def libs(self):
return LibraryList([])
def setup_dependent_run_environment(self, env, dependent_spec):
"""Set PYTHONPATH to include the site-packages directory for the
extension and any other python extensions it depends on."""
# Packages may be installed in platform-specific or platform-independent site-packages
# directories
for directory in {self.platlib, self.purelib}:
path = os.path.join(dependent_spec.prefix, directory)
if os.path.isdir(path):
env.prepend_path("PYTHONPATH", path)
def setup_dependent_package(self, module, dependent_spec):
"""Called before python modules' install() methods."""
module.python = self.command
module.python_platlib = join_path(dependent_spec.prefix, self.platlib)
module.python_purelib = join_path(dependent_spec.prefix, self.purelib)

107
var/spack/repos/builtin/packages/python/package.py
View file

@ -8,19 +8,16 @@
import os
import platform
import re
import stat
import subprocess
import sys
from shutil import copy
from typing import Dict, List, Tuple
from typing import Dict, List
import llnl.util.tty as tty
from llnl.util.filesystem import is_nonsymlink_exe_with_shebang, path_contains_subdirectory
from llnl.util.lang import dedupe
from spack.build_environment import dso_suffix, stat_suffix
from spack.package import *
from spack.util.environment import is_system_path
from spack.util.prefix import Prefix
@ -1115,7 +1112,7 @@ def platlib(self):
path = self.config_vars["platlib"]
if path.startswith(prefix):
return path.replace(prefix, "")
return os.path.join("lib64", "python{}".format(self.version.up_to(2)), "site-packages")
return os.path.join("lib64", f"python{self.version.up_to(2)}", "site-packages")
@property
def purelib(self):
@ -1135,7 +1132,7 @@ def purelib(self):
path = self.config_vars["purelib"]
if path.startswith(prefix):
return path.replace(prefix, "")
return os.path.join("lib", "python{}".format(self.version.up_to(2)), "site-packages")
return os.path.join("lib", f"python{self.version.up_to(2)}", "site-packages")
@property
def include(self):
@ -1163,39 +1160,6 @@ def setup_dependent_build_environment(self, env, dependent_spec):
"""Set PYTHONPATH to include the site-packages directory for the
extension and any other python extensions it depends on.
"""
# Ensure the current Python is first in the PATH
path = os.path.dirname(self.command.path)
if not is_system_path(path):
env.prepend_path("PATH", path)
# Add installation prefix to PYTHONPATH, needed to run import tests
prefixes = set()
if dependent_spec.package.extends(self.spec):
prefixes.add(dependent_spec.prefix)
# Add direct build/run/test dependencies to PYTHONPATH,
# needed to build the package and to run import tests
for direct_dep in dependent_spec.dependencies(deptype=("build", "run", "test")):
if direct_dep.package.extends(self.spec):
prefixes.add(direct_dep.prefix)
# Add recursive run dependencies of all direct dependencies,
# needed by direct dependencies at run-time
for indirect_dep in direct_dep.traverse(deptype="run"):
if indirect_dep.package.extends(self.spec):
prefixes.add(indirect_dep.prefix)
for prefix in prefixes:
# Packages may be installed in platform-specific or platform-independent
# site-packages directories
for directory in {self.platlib, self.purelib}:
env.prepend_path("PYTHONPATH", os.path.join(prefix, directory))
if self.spec.satisfies("platform=windows"):
prefix_scripts_dir = prefix.Scripts
if os.path.exists(prefix_scripts_dir):
env.prepend_path("PATH", prefix_scripts_dir)
# We need to make sure that the extensions are compiled and linked with
# the Spack wrapper. Paths to the executables that are used for these
# operations are normally taken from the sysconfigdata file, which we
@ -1241,9 +1205,7 @@ def setup_dependent_build_environment(self, env, dependent_spec):
# invoked directly (no change would be required in that case
# because Spack arranges for the Spack ld wrapper to be the
# first instance of "ld" in PATH).
new_link = config_link.replace(
" {0} ".format(config_compile), " {0} ".format(new_compile)
)
new_link = config_link.replace(f" {config_compile} ", f" {new_compile} ")
# There is logic in the sysconfig module that is sensitive to the
# fact that LDSHARED is set in the environment, therefore we export
@ -1256,66 +1218,23 @@ def setup_dependent_run_environment(self, env, dependent_spec):
"""Set PYTHONPATH to include the site-packages directory for the
extension and any other python extensions it depends on.
"""
if dependent_spec.package.extends(self.spec):
# Packages may be installed in platform-specific or platform-independent
# site-packages directories
for directory in {self.platlib, self.purelib}:
env.prepend_path("PYTHONPATH", os.path.join(dependent_spec.prefix, directory))
if not dependent_spec.package.extends(self.spec) or dependent_spec.dependencies(
"python-venv"
):
return
# Packages may be installed in platform-specific or platform-independent site-packages
# directories
for directory in {self.platlib, self.purelib}:
env.prepend_path("PYTHONPATH", os.path.join(dependent_spec.prefix, directory))
def setup_dependent_package(self, module, dependent_spec):
"""Called before python modules' install() methods."""
module.python = self.command
module.python_include = join_path(dependent_spec.prefix, self.include)
module.python_platlib = join_path(dependent_spec.prefix, self.platlib)
module.python_purelib = join_path(dependent_spec.prefix, self.purelib)
def add_files_to_view(self, view, merge_map, skip_if_exists=True):
# The goal is to copy the `python` executable, so that its search paths are relative to the
# view instead of the install prefix. This is an obsolete way of creating something that
# resembles a virtual environnent. Also we copy scripts with shebang lines. Finally we need
# to re-target symlinks pointing to copied files.
bin_dir = self.spec.prefix.bin if sys.platform != "win32" else self.spec.prefix
copied_files: Dict[Tuple[int, int], str] = {} # File identifier -> source
delayed_links: List[Tuple[str, str]] = [] # List of symlinks from merge map
for src, dst in merge_map.items():
if skip_if_exists and os.path.lexists(dst):
continue
# Files not in the bin dir are linked the default way.
if not path_contains_subdirectory(src, bin_dir):
view.link(src, dst, spec=self.spec)
continue
s = os.lstat(src)
# Symlink is delayed because we may need to re-target if its target is copied in view
if stat.S_ISLNK(s.st_mode):
delayed_links.append((src, dst))
continue
# Anything that's not a symlink gets copied. Scripts with shebangs are immediately
# updated when necessary.
copied_files[(s.st_dev, s.st_ino)] = dst
copy(src, dst)
if is_nonsymlink_exe_with_shebang(src):
filter_file(
self.spec.prefix, os.path.abspath(view.get_projection_for_spec(self.spec)), dst
)
# Finally re-target the symlinks that point to copied files.
for src, dst in delayed_links:
try:
s = os.stat(src)
target = copied_files[(s.st_dev, s.st_ino)]
except (OSError, KeyError):
target = None
if target:
os.symlink(os.path.relpath(target, os.path.dirname(dst)), dst)
else:
view.link(src, dst, spec=self.spec)
def test_hello_world(self):
"""run simple hello world program"""
# do not use self.command because we are also testing the run env

6
var/spack/repos/builtin/packages/qgis/package.py
View file

@ -188,9 +188,7 @@ class Qgis(CMakePackage):
@run_before("cmake", when="^py-pyqt5")
def fix_pyqt5_cmake(self):
cmfile = FileFilter(join_path("cmake", "FindPyQt5.cmake"))
pyqtpath = join_path(
self.spec["py-pyqt5"].prefix, self.spec["python"].package.platlib, "PyQt5"
)
pyqtpath = join_path(self.spec["py-pyqt5"].package.module.python_platlib, "PyQt5")
cmfile.filter(
'SET(PYQT5_MOD_DIR "${Python_SITEARCH}/PyQt5")',
'SET(PYQT5_MOD_DIR "' + pyqtpath + '")',
@ -210,7 +208,7 @@ def fix_qsci_sip(self):
pyqtx = "PyQt6"
sip_inc_dir = join_path(
self.spec["qscintilla"].prefix, self.spec["python"].package.platlib, pyqtx, "bindings"
self.spec["qscintilla"].package.module.python_platlib, pyqtx, "bindings"
)
with open(join_path("python", "gui", "pyproject.toml.in"), "a") as tomlfile:
tomlfile.write(f'\n[tool.sip.project]\nsip-include-dirs = ["{sip_inc_dir}"]\n')

2
var/spack/repos/builtin/packages/qscintilla/package.py
View file

@ -101,7 +101,7 @@ def make_qsci_python(self):
with working_dir(join_path(self.stage.source_path, "Python")):
copy(ftoml, "pyproject.toml")
sip_inc_dir = join_path(
self.spec[py_pyqtx].prefix, self.spec["python"].package.platlib, pyqtx, "bindings"
self.spec[py_pyqtx].package.module.python_platlib, pyqtx, "bindings"
)
with open("pyproject.toml", "a") as tomlfile:

11
var/spack/repos/builtin/packages/redland-bindings/package.py
View file

@ -3,8 +3,6 @@
#
# SPDX-License-Identifier: (Apache-2.0 OR MIT)
import os
from spack.package import *
@ -27,11 +25,4 @@ class RedlandBindings(AutotoolsPackage):
extends("python")
def configure_args(self):
plib = self.spec["python"].prefix.lib
plib64 = self.spec["python"].prefix.lib64
mybase = self.prefix.lib
if os.path.isdir(plib64) and not os.path.isdir(plib):
mybase = self.prefix.lib64
pver = "python{0}".format(self.spec["python"].version.up_to(2))
myplib = join_path(mybase, pver, "site-packages")
return ["--with-python", "PYTHON_LIB={0}".format(myplib)]
return ["--with-python", f"PYTHON_LIB={python_platlib}"]

9
var/spack/repos/builtin/packages/z3/package.py
View file

@ -53,13 +53,6 @@ def cmake_args(self):
]
if spec.satisfies("+python"):
args.append(
self.define(
"CMAKE_INSTALL_PYTHON_PKG_DIR",
join_path(
prefix.lib, "python%s" % spec["python"].version.up_to(2), "site-packages"
),
)
)
args.append(self.define("CMAKE_INSTALL_PYTHON_PKG_DIR", python_platlib))
return args

1
var/spack/repos/duplicates.test/packages/python-venv Symbolic link
View file

@ -0,0 +1 @@
../../builtin.mock/packages/python-venv