diff --git a/lib/spack/docs/basic_usage.rst b/lib/spack/docs/basic_usage.rst index f26c6e5683..e49ca3073e 100644 --- a/lib/spack/docs/basic_usage.rst +++ b/lib/spack/docs/basic_usage.rst @@ -865,7 +865,7 @@ There are several different ways to use Spack packages once you have installed them. As you've seen, spack packages are installed into long paths with hashes, and you need a way to get them into your path. The easiest way is to use :ref:`spack load `, which is -described in the next section. +described in this section. Some more advanced ways to use Spack packages include: @@ -959,7 +959,86 @@ use ``spack find --loaded``. You can also use ``spack load --list`` to get the same output, but it does not have the full set of query options that ``spack find`` offers. -We'll learn more about Spack's spec syntax in the next section. +We'll learn more about Spack's spec syntax in :ref:`a later section `. + + +.. _extensions: + +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +Python packages and virtual environments +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +Spack can install a large number of Python packages. Their names are +typically prefixed with ``py-``. Installing and using them is no +different from any other package: + +.. code-block:: console + + $ spack install py-numpy + $ spack load py-numpy + $ python3 + >>> import numpy + +The ``spack load`` command sets the ``PATH`` variable so that the right Python +executable is used, and makes sure that ``numpy`` and its dependencies can be +located in the ``PYTHONPATH``. + +Spack is different from other Python package managers in that it installs +every package into its *own* prefix. This is in contrast to ``pip``, which +installs all packages into the same prefix, be it in a virtual environment +or not. + +For many users, **virtual environments** are more convenient than repeated +``spack load`` commands, particularly when working with multiple Python +packages. Fortunately Spack supports environments itself, which together +with a view are no different from Python virtual environments. + +The recommended way of working with Python extensions such as ``py-numpy`` +is through :ref:`Environments `. The following example creates +a Spack environment with ``numpy`` in the current working directory. It also +puts a filesystem view in ``./view``, which is a more traditional combined +prefix for all packages in the environment. + +.. code-block:: console + + $ spack env create --with-view view --dir . + $ spack -e . add py-numpy + $ spack -e . concretize + $ spack -e . install + +Now you can activate the environment and start using the packages: + +.. code-block:: console + + $ spack env activate . + $ python3 + >>> import numpy + +The environment view is also a virtual environment, which is useful if you are +sharing the environment with others who are unfamiliar with Spack. They can +either use the Python executable directly: + +.. code-block:: console + + $ ./view/bin/python3 + >>> import numpy + +or use the activation script: + +.. code-block:: console + + $ source ./view/bin/activate + $ python3 + >>> import numpy + +In general, there should not be much difference between ``spack env activate`` +and using the virtual environment. The main advantage of ``spack env activate`` +is that it knows about more packages than just Python packages, and it may set +additional runtime variables that are not covered by the virtual environment +activation script. + +See :ref:`environments` for a more in-depth description of Spack +environments and customizations to views. .. _sec-specs: @@ -1705,165 +1784,6 @@ check only local packages (as opposed to those used transparently from ``upstream`` spack instances) and the ``-j,--json`` option to output machine-readable json data for any errors. - -.. _extensions: - ---------------------------- -Extensions & Python support ---------------------------- - -Spack's installation model assumes that each package will live in its -own install prefix. However, certain packages are typically installed -*within* the directory hierarchy of other packages. For example, -`Python `_ packages are typically installed in the -``$prefix/lib/python-2.7/site-packages`` directory. - -In Spack, installation prefixes are immutable, so this type of installation -is not directly supported. However, it is possible to create views that -allow you to merge install prefixes of multiple packages into a single new prefix. -Views are a convenient way to get a more traditional filesystem structure. -Using *extensions*, you can ensure that Python packages always share the -same prefix in the view as Python itself. Suppose you have -Python installed like so: - -.. code-block:: console - - $ spack find python - ==> 1 installed packages. - -- linux-debian7-x86_64 / gcc@4.4.7 -------------------------------- - python@2.7.8 - -.. _cmd-spack-extensions: - -^^^^^^^^^^^^^^^^^^^^ -``spack extensions`` -^^^^^^^^^^^^^^^^^^^^ - -You can find extensions for your Python installation like this: - -.. code-block:: console - - $ spack extensions python - ==> python@2.7.8%gcc@4.4.7 arch=linux-debian7-x86_64-703c7a96 - ==> 36 extensions: - geos py-ipython py-pexpect py-pyside py-sip - py-basemap py-libxml2 py-pil py-pytz py-six - py-biopython py-mako py-pmw py-rpy2 py-sympy - py-cython py-matplotlib py-pychecker py-scientificpython py-virtualenv - py-dateutil py-mpi4py py-pygments py-scikit-learn - py-epydoc py-mx py-pylint py-scipy - py-gnuplot py-nose py-pyparsing py-setuptools - py-h5py py-numpy py-pyqt py-shiboken - - ==> 12 installed: - -- linux-debian7-x86_64 / gcc@4.4.7 -------------------------------- - py-dateutil@2.4.0 py-nose@1.3.4 py-pyside@1.2.2 - py-dateutil@2.4.0 py-numpy@1.9.1 py-pytz@2014.10 - py-ipython@2.3.1 py-pygments@2.0.1 py-setuptools@11.3.1 - py-matplotlib@1.4.2 py-pyparsing@2.0.3 py-six@1.9.0 - -The extensions are a subset of what's returned by ``spack list``, and -they are packages like any other. They are installed into their own -prefixes, and you can see this with ``spack find --paths``: - -.. code-block:: console - - $ spack find --paths py-numpy - ==> 1 installed packages. - -- linux-debian7-x86_64 / gcc@4.4.7 -------------------------------- - py-numpy@1.9.1 ~/spack/opt/linux-debian7-x86_64/gcc@4.4.7/py-numpy@1.9.1-66733244 - -However, even though this package is installed, you cannot use it -directly when you run ``python``: - -.. code-block:: console - - $ spack load python - $ python - Python 2.7.8 (default, Feb 17 2015, 01:35:25) - [GCC 4.4.7 20120313 (Red Hat 4.4.7-11)] on linux2 - Type "help", "copyright", "credits" or "license" for more information. - >>> import numpy - Traceback (most recent call last): - File "", line 1, in - ImportError: No module named numpy - >>> - -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ -Using Extensions in Environments -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - -The recommended way of working with extensions such as ``py-numpy`` -above is through :ref:`Environments `. For example, -the following creates an environment in the current working directory -with a filesystem view in the ``./view`` directory: - -.. code-block:: console - - $ spack env create --with-view view --dir . - $ spack -e . add py-numpy - $ spack -e . concretize - $ spack -e . install - -We recommend environments for two reasons. Firstly, environments -can be activated (requires :ref:`shell-support`): - -.. code-block:: console - - $ spack env activate . - -which sets all the right environment variables such as ``PATH`` and -``PYTHONPATH``. This ensures that - -.. code-block:: console - - $ python - >>> import numpy - -works. Secondly, even without shell support, the view ensures -that Python can locate its extensions: - -.. code-block:: console - - $ ./view/bin/python - >>> import numpy - -See :ref:`environments` for a more in-depth description of Spack -environments and customizations to views. - -^^^^^^^^^^^^^^^^^^^^ -Using ``spack load`` -^^^^^^^^^^^^^^^^^^^^ - -A more traditional way of using Spack and extensions is ``spack load`` -(requires :ref:`shell-support`). This will add the extension to ``PYTHONPATH`` -in your current shell, and Python itself will be available in the ``PATH``: - -.. code-block:: console - - $ spack load py-numpy - $ python - >>> import numpy - -The loaded packages can be checked using ``spack find --loaded`` - -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ -Loading Extensions via Modules -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - -Apart from ``spack env activate`` and ``spack load``, you can load numpy -through your environment modules (using ``environment-modules`` or -``lmod``). This will also add the extension to the ``PYTHONPATH`` in -your current shell. - -.. code-block:: console - - $ module load - -If you do not know the name of the specific numpy module you wish to -load, you can use the ``spack module tcl|lmod loads`` command to get -the name of the module from the Spack spec. - ----------------------- Filesystem requirements ----------------------- diff --git a/lib/spack/docs/build_systems/pythonpackage.rst b/lib/spack/docs/build_systems/pythonpackage.rst index 372d4ad47c..9512b08885 100644 --- a/lib/spack/docs/build_systems/pythonpackage.rst +++ b/lib/spack/docs/build_systems/pythonpackage.rst @@ -718,23 +718,45 @@ command-line tool, or C/C++/Fortran program with optional Python modules? The former should be prepended with ``py-``, while the latter should not. -"""""""""""""""""""""" -extends vs. depends_on -"""""""""""""""""""""" +"""""""""""""""""""""""""""""" +``extends`` vs. ``depends_on`` +"""""""""""""""""""""""""""""" -This is very similar to the naming dilemma above, with a slight twist. As mentioned in the :ref:`Packaging Guide `, ``extends`` and ``depends_on`` are very similar, but ``extends`` ensures that the extension and extendee share the same prefix in views. This allows the user to import a Python module without having to add that module to ``PYTHONPATH``. -When deciding between ``extends`` and ``depends_on``, the best rule of -thumb is to check the installation prefix. If Python libraries are -installed to ``/lib/pythonX.Y/site-packages``, then you -should use ``extends``. If Python libraries are installed elsewhere -or the only files that get installed reside in ``/bin``, then -don't use ``extends``. +Additionally, ``extends("python")`` adds a dependency on the package +``python-venv``. This improves isolation from the system, whether +it's during the build or at runtime: user and system site packages +cannot accidentally be used by any package that ``extends("python")``. + +As a rule of thumb: if a package does not install any Python modules +of its own, and merely puts a Python script in the ``bin`` directory, +then there is no need for ``extends``. If the package installs modules +in the ``site-packages`` directory, it requires ``extends``. + +""""""""""""""""""""""""""""""""""""" +Executing ``python`` during the build +""""""""""""""""""""""""""""""""""""" + +Whenever you need to execute a Python command or pass the path of the +Python interpreter to the build system, it is best to use the global +variable ``python`` directly. For example: + +.. code-block:: python + + @run_before("install") + def recythonize(self): + python("setup.py", "clean") # use the `python` global + +As mentioned in the previous section, ``extends("python")`` adds an +automatic dependency on ``python-venv``, which is a virtual environment +that guarantees build isolation. The ``python`` global always refers to +the correct Python interpreter, whether the package uses ``extends("python")`` +or ``depends_on("python")``. ^^^^^^^^^^^^^^^^^^^^^ Alternatives to Spack diff --git a/lib/spack/spack/bootstrap/_common.py b/lib/spack/spack/bootstrap/_common.py index 2ce53d3165..5c3ca93e94 100644 --- a/lib/spack/spack/bootstrap/_common.py +++ b/lib/spack/spack/bootstrap/_common.py @@ -54,10 +54,14 @@ def _try_import_from_store( installed_specs = spack.store.STORE.db.query(query_spec, installed=True) for candidate_spec in installed_specs: - pkg = candidate_spec["python"].package + # previously bootstrapped specs may not have a python-venv dependency. + if candidate_spec.dependencies("python-venv"): + python, *_ = candidate_spec.dependencies("python-venv") + else: + python, *_ = candidate_spec.dependencies("python") module_paths = [ - os.path.join(candidate_spec.prefix, pkg.purelib), - os.path.join(candidate_spec.prefix, pkg.platlib), + os.path.join(candidate_spec.prefix, python.package.purelib), + os.path.join(candidate_spec.prefix, python.package.platlib), ] path_before = list(sys.path) diff --git a/lib/spack/spack/bootstrap/environment.py b/lib/spack/spack/bootstrap/environment.py index f1af8990e8..13942ba86f 100644 --- a/lib/spack/spack/bootstrap/environment.py +++ b/lib/spack/spack/bootstrap/environment.py @@ -3,13 +3,11 @@ # # SPDX-License-Identifier: (Apache-2.0 OR MIT) """Bootstrap non-core Spack dependencies from an environment.""" -import glob import hashlib import os import pathlib import sys -import warnings -from typing import List +from typing import Iterable, List import archspec.cpu @@ -28,6 +26,16 @@ class BootstrapEnvironment(spack.environment.Environment): """Environment to install dependencies of Spack for a given interpreter and architecture""" + def __init__(self) -> None: + if not self.spack_yaml().exists(): + self._write_spack_yaml_file() + super().__init__(self.environment_root()) + + # Remove python package roots created before python-venv was introduced + for s in self.concrete_roots(): + if "python" in s.package.extendees and not s.dependencies("python-venv"): + self.deconcretize(s) + @classmethod def spack_dev_requirements(cls) -> List[str]: """Spack development requirements""" @@ -59,31 +67,19 @@ def view_root(cls) -> pathlib.Path: return cls.environment_root().joinpath("view") @classmethod - def pythonpaths(cls) -> List[str]: - """Paths to be added to sys.path or PYTHONPATH""" - python_dir_part = f"python{'.'.join(str(x) for x in sys.version_info[:2])}" - glob_expr = str(cls.view_root().joinpath("**", python_dir_part, "**")) - result = glob.glob(glob_expr) - if not result: - msg = f"Cannot find any Python path in {cls.view_root()}" - warnings.warn(msg) - return result - - @classmethod - def bin_dirs(cls) -> List[pathlib.Path]: + def bin_dir(cls) -> pathlib.Path: """Paths to be added to PATH""" - return [cls.view_root().joinpath("bin")] + return cls.view_root().joinpath("bin") + + def python_dirs(self) -> Iterable[pathlib.Path]: + python = next(s for s in self.all_specs_generator() if s.name == "python-venv").package + return {self.view_root().joinpath(p) for p in (python.platlib, python.purelib)} @classmethod def spack_yaml(cls) -> pathlib.Path: """Environment spack.yaml file""" return cls.environment_root().joinpath("spack.yaml") - def __init__(self) -> None: - if not self.spack_yaml().exists(): - self._write_spack_yaml_file() - super().__init__(self.environment_root()) - def update_installations(self) -> None: """Update the installations of this environment.""" log_enabled = tty.is_debug() or tty.is_verbose() @@ -100,21 +96,13 @@ def update_installations(self) -> None: self.install_all() self.write(regenerate=True) - def update_syspath_and_environ(self) -> None: - """Update ``sys.path`` and the PATH, PYTHONPATH environment variables to point to - the environment view. - """ - # Do minimal modifications to sys.path and environment variables. In particular, pay - # attention to have the smallest PYTHONPATH / sys.path possible, since that may impact - # the performance of the current interpreter - sys.path.extend(self.pythonpaths()) - os.environ["PATH"] = os.pathsep.join( - [str(x) for x in self.bin_dirs()] + os.environ.get("PATH", "").split(os.pathsep) - ) - os.environ["PYTHONPATH"] = os.pathsep.join( - os.environ.get("PYTHONPATH", "").split(os.pathsep) - + [str(x) for x in self.pythonpaths()] - ) + def load(self) -> None: + """Update PATH and sys.path.""" + # Make executables available (shouldn't need PYTHONPATH) + os.environ["PATH"] = f"{self.bin_dir()}{os.pathsep}{os.environ.get('PATH', '')}" + + # Spack itself imports pytest + sys.path.extend(str(p) for p in self.python_dirs()) def _write_spack_yaml_file(self) -> None: tty.msg( @@ -164,4 +152,4 @@ def ensure_environment_dependencies() -> None: _add_externals_if_missing() with BootstrapEnvironment() as env: env.update_installations() - env.update_syspath_and_environ() + env.load() diff --git a/lib/spack/spack/build_systems/cmake.py b/lib/spack/spack/build_systems/cmake.py index b6e66e136c..a64904715e 100644 --- a/lib/spack/spack/build_systems/cmake.py +++ b/lib/spack/spack/build_systems/cmake.py @@ -39,16 +39,11 @@ def _maybe_set_python_hints(pkg: spack.package_base.PackageBase, args: List[str] """Set the PYTHON_EXECUTABLE, Python_EXECUTABLE, and Python3_EXECUTABLE CMake variables if the package has Python as build or link dep and ``find_python_hints`` is set to True. See ``find_python_hints`` for context.""" - if not getattr(pkg, "find_python_hints", False): + if not getattr(pkg, "find_python_hints", False) or not pkg.spec.dependencies( + "python", dt.BUILD | dt.LINK + ): return - pythons = pkg.spec.dependencies("python", dt.BUILD | dt.LINK) - if len(pythons) != 1: - return - try: - python_executable = pythons[0].package.command.path - except RuntimeError: - return - + python_executable = pkg.spec["python"].command.path args.extend( [ CMakeBuilder.define("PYTHON_EXECUTABLE", python_executable), diff --git a/lib/spack/spack/build_systems/python.py b/lib/spack/spack/build_systems/python.py index 1f650be98a..c94e2db700 100644 --- a/lib/spack/spack/build_systems/python.py +++ b/lib/spack/spack/build_systems/python.py @@ -120,6 +120,12 @@ def skip_modules(self) -> Iterable[str]: """ return [] + @property + def python_spec(self): + """Get python-venv if it exists or python otherwise.""" + python, *_ = self.spec.dependencies("python-venv") or self.spec.dependencies("python") + return python + def view_file_conflicts(self, view, merge_map): """Report all file conflicts, excepting special cases for python. Specifically, this does not report errors for duplicate @@ -138,16 +144,17 @@ def view_file_conflicts(self, view, merge_map): return conflicts def add_files_to_view(self, view, merge_map, skip_if_exists=True): - # Patch up shebangs to the python linked in the view only if python is built by Spack. - if not self.extendee_spec or self.extendee_spec.external: + # Patch up shebangs if the package extends Python and we put a Python interpreter in the + # view. + python = self.python_spec + if not self.extendee_spec or python.external: return super().add_files_to_view(view, merge_map, skip_if_exists) # We only patch shebangs in the bin directory. copied_files: Dict[Tuple[int, int], str] = {} # File identifier -> source delayed_links: List[Tuple[str, str]] = [] # List of symlinks from merge map - bin_dir = self.spec.prefix.bin - python_prefix = self.extendee_spec.prefix + for src, dst in merge_map.items(): if skip_if_exists and os.path.lexists(dst): continue @@ -168,7 +175,7 @@ def add_files_to_view(self, view, merge_map, skip_if_exists=True): copied_files[(s.st_dev, s.st_ino)] = dst shutil.copy2(src, dst) fs.filter_file( - python_prefix, os.path.abspath(view.get_projection_for_spec(self.spec)), dst + python.prefix, os.path.abspath(view.get_projection_for_spec(self.spec)), dst ) else: view.link(src, dst) @@ -199,14 +206,13 @@ def remove_files_from_view(self, view, merge_map): ignore_namespace = True bin_dir = self.spec.prefix.bin - global_view = self.extendee_spec.prefix == view.get_projection_for_spec(self.spec) to_remove = [] for src, dst in merge_map.items(): if ignore_namespace and namespace_init(dst): continue - if global_view or not fs.path_contains_subdirectory(src, bin_dir): + if not fs.path_contains_subdirectory(src, bin_dir): to_remove.append(dst) else: os.remove(dst) @@ -371,8 +377,9 @@ def headers(self) -> HeaderList: # Headers should only be in include or platlib, but no harm in checking purelib too include = self.prefix.join(self.spec["python"].package.include).join(name) - platlib = self.prefix.join(self.spec["python"].package.platlib).join(name) - purelib = self.prefix.join(self.spec["python"].package.purelib).join(name) + python = self.python_spec + platlib = self.prefix.join(python.package.platlib).join(name) + purelib = self.prefix.join(python.package.purelib).join(name) headers_list = map(fs.find_all_headers, [include, platlib, purelib]) headers = functools.reduce(operator.add, headers_list) @@ -391,8 +398,9 @@ def libs(self) -> LibraryList: name = self.spec.name[3:] # Libraries should only be in platlib, but no harm in checking purelib too - platlib = self.prefix.join(self.spec["python"].package.platlib).join(name) - purelib = self.prefix.join(self.spec["python"].package.purelib).join(name) + python = self.python_spec + platlib = self.prefix.join(python.package.platlib).join(name) + purelib = self.prefix.join(python.package.purelib).join(name) find_all_libraries = functools.partial(fs.find_all_libraries, recursive=True) libs_list = map(find_all_libraries, [platlib, purelib]) @@ -504,6 +512,8 @@ def global_options(self, spec: Spec, prefix: Prefix) -> Iterable[str]: def install(self, pkg: PythonPackage, spec: Spec, prefix: Prefix) -> None: """Install everything from build directory.""" + pip = spec["python"].command + pip.add_default_arg("-m", "pip") args = PythonPipBuilder.std_args(pkg) + [f"--prefix={prefix}"] @@ -519,14 +529,6 @@ def install(self, pkg: PythonPackage, spec: Spec, prefix: Prefix) -> None: else: args.append(".") - pip = spec["python"].command - # Hide user packages, since we don't have build isolation. This is - # necessary because pip / setuptools may run hooks from arbitrary - # packages during the build. There is no equivalent variable to hide - # system packages, so this is not reliable for external Python. - pip.add_default_env("PYTHONNOUSERSITE", "1") - pip.add_default_arg("-m") - pip.add_default_arg("pip") with fs.working_dir(self.build_directory): pip(*args) diff --git a/lib/spack/spack/directives.py b/lib/spack/spack/directives.py index 4991040142..b69f83a75d 100644 --- a/lib/spack/spack/directives.py +++ b/lib/spack/spack/directives.py @@ -662,6 +662,7 @@ def _execute_redistribute( @directive(("extendees", "dependencies")) def extends(spec, when=None, type=("build", "run"), patches=None): """Same as depends_on, but also adds this package to the extendee list. + In case of Python, also adds a dependency on python-venv. keyword arguments can be passed to extends() so that extension packages can pass parameters to the extendee's extension @@ -677,6 +678,11 @@ def _execute_extends(pkg): _depends_on(pkg, spec, when=when, type=type, patches=patches) spec_obj = spack.spec.Spec(spec) + # When extending python, also add a dependency on python-venv. This is done so that + # Spack environment views are Python virtual environments. + if spec_obj.name == "python" and not pkg.name == "python-venv": + _depends_on(pkg, "python-venv", when=when, type=("build", "run")) + # TODO: the values of the extendees dictionary are not used. Remove in next refactor. pkg.extendees[spec_obj.name] = (spec_obj, None) diff --git a/lib/spack/spack/hooks/windows_runtime_linkage.py b/lib/spack/spack/hooks/windows_runtime_linkage.py new file mode 100644 index 0000000000..5bb3744910 --- /dev/null +++ b/lib/spack/spack/hooks/windows_runtime_linkage.py @@ -0,0 +1,8 @@ +# Copyright 2013-2024 Lawrence Livermore National Security, LLC and other +# Spack Project Developers. See the top-level COPYRIGHT file for details. +# +# SPDX-License-Identifier: (Apache-2.0 OR MIT) + + +def post_install(spec, explicit=None): + spec.package.windows_establish_runtime_linkage() diff --git a/lib/spack/spack/installer.py b/lib/spack/spack/installer.py index 1f33a7c6b0..289a48568d 100644 --- a/lib/spack/spack/installer.py +++ b/lib/spack/spack/installer.py @@ -1698,10 +1698,6 @@ def _install_task(self, task: BuildTask, install_status: InstallStatus) -> None: spack.package_base.PackageBase._verbose = spack.build_environment.start_build_process( pkg, build_process, install_args ) - # Currently this is how RPATH-like behavior is achieved on Windows, after install - # establish runtime linkage via Windows Runtime link object - # Note: this is a no-op on non Windows platforms - pkg.windows_establish_runtime_linkage() # Note: PARENT of the build process adds the new package to # the database, so that we don't need to re-read from file. spack.store.STORE.db.add(pkg.spec, spack.store.STORE.layout, explicit=explicit) diff --git a/lib/spack/spack/spec.py b/lib/spack/spack/spec.py index 1f30f7e923..27c762bd40 100644 --- a/lib/spack/spack/spec.py +++ b/lib/spack/spack/spec.py @@ -1030,16 +1030,13 @@ def clear(self): self.edges.clear() -def _command_default_handler(descriptor, spec, cls): +def _command_default_handler(spec: "Spec"): """Default handler when looking for the 'command' attribute. Tries to search for ``spec.name`` in the ``spec.home.bin`` directory. Parameters: - descriptor (ForwardQueryToPackage): descriptor that triggered the call - spec (Spec): spec that is being queried - cls (type(spec)): type of spec, to match the signature of the - descriptor ``__get__`` method + spec: spec that is being queried Returns: Executable: An executable of the command @@ -1052,22 +1049,17 @@ def _command_default_handler(descriptor, spec, cls): if fs.is_exe(path): return spack.util.executable.Executable(path) - else: - msg = "Unable to locate {0} command in {1}" - raise RuntimeError(msg.format(spec.name, home.bin)) + raise RuntimeError(f"Unable to locate {spec.name} command in {home.bin}") -def _headers_default_handler(descriptor, spec, cls): +def _headers_default_handler(spec: "Spec"): """Default handler when looking for the 'headers' attribute. Tries to search for ``*.h`` files recursively starting from ``spec.package.home.include``. Parameters: - descriptor (ForwardQueryToPackage): descriptor that triggered the call - spec (Spec): spec that is being queried - cls (type(spec)): type of spec, to match the signature of the - descriptor ``__get__`` method + spec: spec that is being queried Returns: HeaderList: The headers in ``prefix.include`` @@ -1080,12 +1072,10 @@ def _headers_default_handler(descriptor, spec, cls): if headers: return headers - else: - msg = "Unable to locate {0} headers in {1}" - raise spack.error.NoHeadersError(msg.format(spec.name, home)) + raise spack.error.NoHeadersError(f"Unable to locate {spec.name} headers in {home}") -def _libs_default_handler(descriptor, spec, cls): +def _libs_default_handler(spec: "Spec"): """Default handler when looking for the 'libs' attribute. Tries to search for ``lib{spec.name}`` recursively starting from @@ -1093,10 +1083,7 @@ def _libs_default_handler(descriptor, spec, cls): ``{spec.name}`` instead. Parameters: - descriptor (ForwardQueryToPackage): descriptor that triggered the call - spec (Spec): spec that is being queried - cls (type(spec)): type of spec, to match the signature of the - descriptor ``__get__`` method + spec: spec that is being queried Returns: LibraryList: The libraries found @@ -1135,27 +1122,33 @@ def _libs_default_handler(descriptor, spec, cls): if libs: return libs - msg = "Unable to recursively locate {0} libraries in {1}" - raise spack.error.NoLibrariesError(msg.format(spec.name, home)) + raise spack.error.NoLibrariesError( + f"Unable to recursively locate {spec.name} libraries in {home}" + ) class ForwardQueryToPackage: """Descriptor used to forward queries from Spec to Package""" - def __init__(self, attribute_name, default_handler=None): + def __init__( + self, + attribute_name: str, + default_handler: Optional[Callable[["Spec"], Any]] = None, + _indirect: bool = False, + ) -> None: """Create a new descriptor. Parameters: - attribute_name (str): name of the attribute to be - searched for in the Package instance - default_handler (callable, optional): default function to be - called if the attribute was not found in the Package - instance + attribute_name: name of the attribute to be searched for in the Package instance + default_handler: default function to be called if the attribute was not found in the + Package instance + _indirect: temporarily added to redirect a query to another package. """ self.attribute_name = attribute_name self.default = default_handler + self.indirect = _indirect - def __get__(self, instance, cls): + def __get__(self, instance: "SpecBuildInterface", cls): """Retrieves the property from Package using a well defined chain of responsibility. @@ -1177,13 +1170,18 @@ def __get__(self, instance, cls): indicating a query failure, e.g. that library files were not found in a 'libs' query. """ - pkg = instance.package + # TODO: this indirection exist solely for `spec["python"].command` to actually return + # spec["python-venv"].command. It should be removed when `python` is a virtual. + if self.indirect and instance.indirect_spec: + pkg = instance.indirect_spec.package + else: + pkg = instance.wrapped_obj.package try: query = instance.last_query except AttributeError: # There has been no query yet: this means # a spec is trying to access its own attributes - _ = instance[instance.name] # NOQA: ignore=F841 + _ = instance.wrapped_obj[instance.wrapped_obj.name] # NOQA: ignore=F841 query = instance.last_query callbacks_chain = [] @@ -1195,7 +1193,8 @@ def __get__(self, instance, cls): callbacks_chain.append(lambda: getattr(pkg, self.attribute_name)) # Final resort : default callback if self.default is not None: - callbacks_chain.append(lambda: self.default(self, instance, cls)) + _default = self.default # make mypy happy + callbacks_chain.append(lambda: _default(instance.wrapped_obj)) # Trigger the callbacks in order, the first one producing a # value wins @@ -1254,25 +1253,33 @@ def __set__(self, instance, value): class SpecBuildInterface(lang.ObjectWrapper): # home is available in the base Package so no default is needed home = ForwardQueryToPackage("home", default_handler=None) - - command = ForwardQueryToPackage("command", default_handler=_command_default_handler) - headers = ForwardQueryToPackage("headers", default_handler=_headers_default_handler) - libs = ForwardQueryToPackage("libs", default_handler=_libs_default_handler) + command = ForwardQueryToPackage( + "command", default_handler=_command_default_handler, _indirect=True + ) - def __init__(self, spec, name, query_parameters): + def __init__(self, spec: "Spec", name: str, query_parameters: List[str], _parent: "Spec"): super().__init__(spec) # Adding new attributes goes after super() call since the ObjectWrapper # resets __dict__ to behave like the passed object original_spec = getattr(spec, "wrapped_obj", spec) self.wrapped_obj = original_spec - self.token = original_spec, name, query_parameters + self.token = original_spec, name, query_parameters, _parent is_virtual = spack.repo.PATH.is_virtual(name) self.last_query = QueryState( name=name, extra_parameters=query_parameters, isvirtual=is_virtual ) + # TODO: this ad-hoc logic makes `spec["python"].command` return + # `spec["python-venv"].command` and should be removed when `python` is a virtual. + self.indirect_spec = None + if spec.name == "python": + python_venvs = _parent.dependencies("python-venv") + if not python_venvs: + return + self.indirect_spec = python_venvs[0] + def __reduce__(self): return SpecBuildInterface, self.token @@ -4137,7 +4144,7 @@ def version(self): raise spack.error.SpecError("Spec version is not concrete: " + str(self)) return self.versions[0] - def __getitem__(self, name): + def __getitem__(self, name: str): """Get a dependency from the spec by its name. This call implicitly sets a query state in the package being retrieved. The behavior of packages may be influenced by additional query parameters that are @@ -4146,7 +4153,7 @@ def __getitem__(self, name): Note that if a virtual package is queried a copy of the Spec is returned while for non-virtual a reference is returned. """ - query_parameters = name.split(":") + query_parameters: List[str] = name.split(":") if len(query_parameters) > 2: raise KeyError("key has more than one ':' symbol. At most one is admitted.") @@ -4169,7 +4176,7 @@ def __getitem__(self, name): ) try: - value = next( + child: Spec = next( itertools.chain( # Regular specs (x for x in order() if x.name == name), @@ -4186,9 +4193,9 @@ def __getitem__(self, name): raise KeyError(f"No spec with name {name} in {self}") if self._concrete: - return SpecBuildInterface(value, name, query_parameters) + return SpecBuildInterface(child, name, query_parameters, _parent=self) - return value + return child def __contains__(self, spec): """True if this spec or some dependency satisfies the spec. diff --git a/lib/spack/spack/test/cmd/extensions.py b/lib/spack/spack/test/cmd/extensions.py index 1f6ed95b56..5869e46642 100644 --- a/lib/spack/spack/test/cmd/extensions.py +++ b/lib/spack/spack/test/cmd/extensions.py @@ -33,21 +33,23 @@ def check_output(ni): packages = extensions("-s", "packages", "python") installed = extensions("-s", "installed", "python") assert "==> python@2.7.11" in output - assert "==> 2 extensions" in output + assert "==> 3 extensions" in output assert "py-extension1" in output assert "py-extension2" in output + assert "python-venv" in output - assert "==> 2 extensions" in packages + assert "==> 3 extensions" in packages assert "py-extension1" in packages assert "py-extension2" in packages + assert "python-venv" in packages assert "installed" not in packages - assert ("%s installed" % (ni if ni else "None")) in output - assert ("%s installed" % (ni if ni else "None")) in installed + assert f"{ni if ni else 'None'} installed" in output + assert f"{ni if ni else 'None'} installed" in installed - check_output(2) + check_output(3) ext2.package.do_uninstall(force=True) - check_output(1) + check_output(2) def test_extensions_no_arguments(mock_packages): diff --git a/lib/spack/spack/util/executable.py b/lib/spack/spack/util/executable.py index f160051674..afb8bcaa39 100644 --- a/lib/spack/spack/util/executable.py +++ b/lib/spack/spack/util/executable.py @@ -39,6 +39,20 @@ def add_default_arg(self, *args): """Add default argument(s) to the command.""" self.exe.extend(args) + def with_default_args(self, *args): + """Same as add_default_arg, but returns a copy of the executable.""" + new = self.copy() + new.add_default_arg(*args) + return new + + def copy(self): + """Return a copy of this Executable.""" + new = Executable(self.exe[0]) + new.exe[:] = self.exe + new.default_env.update(self.default_env) + new.default_envmod.extend(self.default_envmod) + return new + def add_default_env(self, key, value): """Set an environment variable when the command is run. diff --git a/var/spack/repos/builtin.mock/packages/python-venv/package.py b/var/spack/repos/builtin.mock/packages/python-venv/package.py new file mode 100644 index 0000000000..741fc3c627 --- /dev/null +++ b/var/spack/repos/builtin.mock/packages/python-venv/package.py @@ -0,0 +1,21 @@ +# Copyright 2013-2023 Lawrence Livermore National Security, LLC and other +# Spack Project Developers. See the top-level COPYRIGHT file for details. +# +# SPDX-License-Identifier: (Apache-2.0 OR MIT) + + +from spack.package import * + + +class PythonVenv(Package): + """A Spack managed Python virtual environment""" + + homepage = "https://docs.python.org/3/library/venv.html" + has_code = False + + version("1.0") + + extends("python") + + def install(self, spec, prefix): + pass diff --git a/var/spack/repos/builtin/packages/ascent/package.py b/var/spack/repos/builtin/packages/ascent/package.py index 5fb06916f3..17c5604033 100644 --- a/var/spack/repos/builtin/packages/ascent/package.py +++ b/var/spack/repos/builtin/packages/ascent/package.py @@ -495,7 +495,7 @@ def hostconfig(self): cfg.write("# Enable python module builds\n") cfg.write(cmake_cache_entry("ENABLE_PYTHON", "ON")) cfg.write("# python from spack \n") - cfg.write(cmake_cache_entry("PYTHON_EXECUTABLE", spec["python"].command.path)) + cfg.write(cmake_cache_entry("PYTHON_EXECUTABLE", python.path)) try: cfg.write("# python module install dir\n") cfg.write(cmake_cache_entry("PYTHON_MODULE_INSTALL_PREFIX", python_platlib)) diff --git a/var/spack/repos/builtin/packages/bohrium/package.py b/var/spack/repos/builtin/packages/bohrium/package.py index 64ad8c100e..fd9e665663 100644 --- a/var/spack/repos/builtin/packages/bohrium/package.py +++ b/var/spack/repos/builtin/packages/bohrium/package.py @@ -256,10 +256,6 @@ def check_install(self): cxx("-o", "test_cxxadd", file_cxxadd, *cxx_flags) test_cxxadd = Executable("./test_cxxadd") - # Build python test commandline - file_pyadd = join_path(os.path.dirname(self.module.__file__), "pyadd.py") - test_pyadd = Executable(spec["python"].command.path + " " + file_pyadd) - # Run tests for each available stack for bh_stack in stacks: tty.info("Testing with bohrium stack '" + bh_stack + "'") @@ -270,5 +266,6 @@ def check_install(self): # Python test (if +python) if "+python" in spec: - py_output = test_pyadd(output=str, env=test_env) + file_pyadd = join_path(os.path.dirname(self.module.__file__), "pyadd.py") + py_output = python(file_pyadd, output=str, env=test_env) compare_output(py_output, "Success!\n") diff --git a/var/spack/repos/builtin/packages/cantera/package.py b/var/spack/repos/builtin/packages/cantera/package.py index efb9d7fcbe..ef057f955b 100644 --- a/var/spack/repos/builtin/packages/cantera/package.py +++ b/var/spack/repos/builtin/packages/cantera/package.py @@ -134,13 +134,9 @@ def build_args(self, spec, prefix): # Python module if "+python" in spec: - args.extend( - ["python_package=full", "python_cmd={0}".format(spec["python"].command.path)] - ) + args.extend(["python_package=full", "python_cmd={0}".format(python.path)]) if spec["python"].satisfies("@3:"): - args.extend( - ["python3_package=y", "python3_cmd={0}".format(spec["python"].command.path)] - ) + args.extend(["python3_package=y", "python3_cmd={0}".format(python.path)]) else: args.append("python3_package=n") else: diff --git a/var/spack/repos/builtin/packages/clingo-bootstrap/package.py b/var/spack/repos/builtin/packages/clingo-bootstrap/package.py index 022cb7e6e3..abb5c53b3c 100644 --- a/var/spack/repos/builtin/packages/clingo-bootstrap/package.py +++ b/var/spack/repos/builtin/packages/clingo-bootstrap/package.py @@ -120,9 +120,7 @@ def pgo_train(self): ) python_runtime_env.unset("SPACK_ENV") python_runtime_env.unset("SPACK_PYTHON") - self.spec["python"].command( - spack.paths.spack_script, "solve", "--fresh", "hdf5", extra_env=python_runtime_env - ) + python(spack.paths.spack_script, "solve", "--fresh", "hdf5", extra_env=python_runtime_env) # Clean the build dir. rmtree(self.build_directory, ignore_errors=True) diff --git a/var/spack/repos/builtin/packages/clingo/package.py b/var/spack/repos/builtin/packages/clingo/package.py index fe1a6f859f..7af9b213fb 100644 --- a/var/spack/repos/builtin/packages/clingo/package.py +++ b/var/spack/repos/builtin/packages/clingo/package.py @@ -3,8 +3,6 @@ # # SPDX-License-Identifier: (Apache-2.0 OR MIT) -import os - from spack.compiler import UnsupportedCompilerFlag from spack.package import * @@ -123,7 +121,4 @@ def cmake_args(self): return args def win_add_library_dependent(self): - if "+python" in self.spec: - return [os.path.join(self.prefix, self.spec["python"].package.platlib)] - else: - return [] + return [python_platlib] if "+python" in self.spec else [] diff --git a/var/spack/repos/builtin/packages/conduit/package.py b/var/spack/repos/builtin/packages/conduit/package.py index c1066ce752..001b04953d 100644 --- a/var/spack/repos/builtin/packages/conduit/package.py +++ b/var/spack/repos/builtin/packages/conduit/package.py @@ -443,7 +443,7 @@ def hostconfig(self): cfg.write("# Enable python module builds\n") cfg.write(cmake_cache_entry("ENABLE_PYTHON", "ON")) cfg.write("# python from spack \n") - cfg.write(cmake_cache_entry("PYTHON_EXECUTABLE", spec["python"].command.path)) + cfg.write(cmake_cache_entry("PYTHON_EXECUTABLE", python.path)) try: cfg.write("# python module install dir\n") cfg.write(cmake_cache_entry("PYTHON_MODULE_INSTALL_PREFIX", python_platlib)) diff --git a/var/spack/repos/builtin/packages/dsqss/package.py b/var/spack/repos/builtin/packages/dsqss/package.py index 72e5f5961f..44f76ab619 100644 --- a/var/spack/repos/builtin/packages/dsqss/package.py +++ b/var/spack/repos/builtin/packages/dsqss/package.py @@ -58,7 +58,6 @@ def test_dla(self): copy(join_path(test01, "std.toml"), ".") # prepare - python = self.spec["python"].command opts = [self.spec.prefix.bin.dla_pre, "std.toml"] with test_part(self, "test_dla_pre", purpose="prepare dla"): python(*opts) diff --git a/var/spack/repos/builtin/packages/fenics/package.py b/var/spack/repos/builtin/packages/fenics/package.py index ae8336aa39..cf56c14fd0 100644 --- a/var/spack/repos/builtin/packages/fenics/package.py +++ b/var/spack/repos/builtin/packages/fenics/package.py @@ -156,7 +156,7 @@ class Fenics(CMakePackage): depends_on("py-sphinx@1.0.1:", when="+doc", type="build") def cmake_args(self): - args = [ + return [ self.define_from_variant("BUILD_SHARED_LIBS", "shared"), self.define("DOLFIN_SKIP_BUILD_TESTS", True), self.define_from_variant("DOLFIN_ENABLE_OPENMP", "openmp"), @@ -180,11 +180,6 @@ def cmake_args(self): self.define_from_variant("DOLFIN_ENABLE_ZLIB", "zlib"), ] - if "+python" in self.spec: - args.append(self.define("PYTHON_EXECUTABLE", self.spec["python"].command.path)) - - return args - # set environment for bulding python interface def setup_build_environment(self, env): env.set("DOLFIN_DIR", self.prefix) diff --git a/var/spack/repos/builtin/packages/gurobi/package.py b/var/spack/repos/builtin/packages/gurobi/package.py index c0fa33639d..d9dd35c2ed 100644 --- a/var/spack/repos/builtin/packages/gurobi/package.py +++ b/var/spack/repos/builtin/packages/gurobi/package.py @@ -57,5 +57,4 @@ def install(self, spec, prefix): @run_after("install") def gurobipy(self): with working_dir("linux64"): - python = which("python") python("setup.py", "install", "--prefix={0}".format(self.prefix)) diff --git a/var/spack/repos/builtin/packages/lbann/package.py b/var/spack/repos/builtin/packages/lbann/package.py index 482d591a1c..e875832b92 100644 --- a/var/spack/repos/builtin/packages/lbann/package.py +++ b/var/spack/repos/builtin/packages/lbann/package.py @@ -402,12 +402,8 @@ def initconfig_package_entries(self): ) entries.append(cmake_cache_option("protobuf_MODULE_COMPATIBLE", True)) - if spec.satisfies("^python") and "+pfe" in spec: - entries.append( - cmake_cache_path( - "LBANN_PFE_PYTHON_EXECUTABLE", "{0}/python3".format(spec["python"].prefix.bin) - ) - ) + if spec.satisfies("+pfe ^python"): + entries.append(cmake_cache_path("LBANN_PFE_PYTHON_EXECUTABLE", python.path)) entries.append( cmake_cache_string("LBANN_PFE_PYTHONPATH", env["PYTHONPATH"]) ) # do NOT need to sub ; for : because diff --git a/var/spack/repos/builtin/packages/libcap-ng/package.py b/var/spack/repos/builtin/packages/libcap-ng/package.py index 373cf7c399..8256fc16a5 100644 --- a/var/spack/repos/builtin/packages/libcap-ng/package.py +++ b/var/spack/repos/builtin/packages/libcap-ng/package.py @@ -33,7 +33,7 @@ class LibcapNg(AutotoolsPackage): def setup_build_environment(self, env): if self.spec.satisfies("+python"): - env.set("PYTHON", self.spec["python"].command.path) + env.set("PYTHON", python.path) def configure_args(self): args = [] diff --git a/var/spack/repos/builtin/packages/mapserver/package.py b/var/spack/repos/builtin/packages/mapserver/package.py index e26951531e..c55cba0fea 100644 --- a/var/spack/repos/builtin/packages/mapserver/package.py +++ b/var/spack/repos/builtin/packages/mapserver/package.py @@ -3,8 +3,6 @@ # # SPDX-License-Identifier: (Apache-2.0 OR MIT) -import os - from spack.package import * @@ -60,9 +58,7 @@ def patch(self): # prefix. This hack patches the CMakeLists.txt for the Python # bindings and hard-wires in the right destination. A bit ugly, # sorry, but I don't speak cmake. - pyversiondir = "python{0}".format(self.spec["python"].version.up_to(2)) - sitepackages = os.path.join(self.spec.prefix.lib, pyversiondir, "site-packages") - filter_file(r"\${PYTHON_SITE_PACKAGES}", sitepackages, "mapscript/python/CMakeLists.txt") + filter_file(r"\${PYTHON_SITE_PACKAGES}", python_platlib, "mapscript/python/CMakeLists.txt") def cmake_args(self): args = [] diff --git a/var/spack/repos/builtin/packages/omnitrace/package.py b/var/spack/repos/builtin/packages/omnitrace/package.py index bc3ed31758..384f8ca2db 100644 --- a/var/spack/repos/builtin/packages/omnitrace/package.py +++ b/var/spack/repos/builtin/packages/omnitrace/package.py @@ -141,7 +141,3 @@ def setup_build_environment(self, env): files = glob.glob(pattern) if files: env.set("TAU_MAKEFILE", files[0]) - - def setup_run_environment(self, env): - if "+python" in self.spec: - env.prepend_path("PYTHONPATH", join_path(self.prefix.lib, "python", "site-packages")) diff --git a/var/spack/repos/builtin/packages/open3d/package.py b/var/spack/repos/builtin/packages/open3d/package.py index 2a63de25ca..a8f1beee6b 100644 --- a/var/spack/repos/builtin/packages/open3d/package.py +++ b/var/spack/repos/builtin/packages/open3d/package.py @@ -119,7 +119,7 @@ def install(self, spec, prefix): def test(self): if "+python" in self.spec: self.run_test( - self.spec["python"].command.path, + python.path, ["-c", "import open3d"], purpose="checking import of open3d", work_dir="spack-test", diff --git a/var/spack/repos/builtin/packages/opencv/package.py b/var/spack/repos/builtin/packages/opencv/package.py index 8e3f3236bb..64375458a3 100644 --- a/var/spack/repos/builtin/packages/opencv/package.py +++ b/var/spack/repos/builtin/packages/opencv/package.py @@ -1027,14 +1027,13 @@ def cmake_args(self): ) # Python - python_exe = spec["python"].command.path python_lib = spec["python"].libs[0] python_include_dir = spec["python"].headers.directories[0] if "+python3" in spec: args.extend( [ - self.define("PYTHON3_EXECUTABLE", python_exe), + self.define("PYTHON3_EXECUTABLE", python.path), self.define("PYTHON3_LIBRARY", python_lib), self.define("PYTHON3_INCLUDE_DIR", python_include_dir), self.define("PYTHON2_EXECUTABLE", ""), diff --git a/var/spack/repos/builtin/packages/openwsman/package.py b/var/spack/repos/builtin/packages/openwsman/package.py index 4be921bad9..9d9870706b 100644 --- a/var/spack/repos/builtin/packages/openwsman/package.py +++ b/var/spack/repos/builtin/packages/openwsman/package.py @@ -31,13 +31,9 @@ class Openwsman(CMakePackage): def patch(self): """Change python install directory.""" if self.spec.satisfies("+python"): - python_spec = self.spec["python"] - python_libdir = join_path( - self.spec.prefix.lib, "python" + str(python_spec.version.up_to(2)), "site-packages" - ) filter_file( "DESTINATION .*", - "DESTINATION {0} )".format(python_libdir), + "DESTINATION {0} )".format(python_platlib), join_path("bindings", "python", "CMakeLists.txt"), ) diff --git a/var/spack/repos/builtin/packages/precice/package.py b/var/spack/repos/builtin/packages/precice/package.py index 48484114fd..6426fa51f3 100644 --- a/var/spack/repos/builtin/packages/precice/package.py +++ b/var/spack/repos/builtin/packages/precice/package.py @@ -191,7 +191,7 @@ def cmake_args(self): python_library = spec["python"].libs[0] python_include = spec["python"].headers.directories[0] numpy_include = join_path( - spec["py-numpy"].prefix, spec["python"].package.platlib, "numpy", "core", "include" + spec["py-numpy"].package.module.python_platlib, "numpy", "core", "include" ) cmake_args.extend( [ diff --git a/var/spack/repos/builtin/packages/py-alphafold/package.py b/var/spack/repos/builtin/packages/py-alphafold/package.py index 674f3ebf10..0269a2abdf 100644 --- a/var/spack/repos/builtin/packages/py-alphafold/package.py +++ b/var/spack/repos/builtin/packages/py-alphafold/package.py @@ -77,7 +77,7 @@ class PyAlphafold(PythonPackage, CudaPackage): @run_after("install") def install_scripts(self): mkdirp(self.prefix.bin) - shebang = "#!{0}\n".format(self.spec["python"].command) + shebang = f"#!{python.path}\n" for fname in glob.glob("run_alphafold*.py"): destfile = join_path(self.prefix.bin, fname) with open(fname, "r") as src: diff --git a/var/spack/repos/builtin/packages/py-chainer/package.py b/var/spack/repos/builtin/packages/py-chainer/package.py index 1f78c76a30..a1ab6819a1 100644 --- a/var/spack/repos/builtin/packages/py-chainer/package.py +++ b/var/spack/repos/builtin/packages/py-chainer/package.py @@ -59,7 +59,7 @@ def test_chainermn(self): mnist_file = join_path(self.install_test_root.examples.chainermn.mnist, "train_mnist.py") mpirun = which(self.spec["mpi"].prefix.bin.mpirun) - opts = ["-n", "4", self.spec["python"].command.path, mnist_file, "-o", "."] + opts = ["-n", "4", python.path, mnist_file, "-o", "."] env["OMP_NUM_THREADS"] = "4" mpirun(*opts) diff --git a/var/spack/repos/builtin/packages/py-eccodes/package.py b/var/spack/repos/builtin/packages/py-eccodes/package.py index 107cddc03e..b53c545e4a 100644 --- a/var/spack/repos/builtin/packages/py-eccodes/package.py +++ b/var/spack/repos/builtin/packages/py-eccodes/package.py @@ -43,5 +43,4 @@ def setup_dependent_run_environment(self, env, dependent_spec): def test_selfcheck(self): """checking system setup""" - python = self.spec["python"].command python("-m", "eccodes", "selfcheck") diff --git a/var/spack/repos/builtin/packages/py-gmxapi/package.py b/var/spack/repos/builtin/packages/py-gmxapi/package.py index 77a03d6fd4..5869d1ad4b 100644 --- a/var/spack/repos/builtin/packages/py-gmxapi/package.py +++ b/var/spack/repos/builtin/packages/py-gmxapi/package.py @@ -49,5 +49,4 @@ def setup_build_environment(self, env): def install_test(self): with working_dir("spack-test", create=True): # test include helper points to right location - python = self.spec["python"].command python("-m", "pytest", "-x", os.path.join(self.build_directory, "test")) diff --git a/var/spack/repos/builtin/packages/py-gpaw/package.py b/var/spack/repos/builtin/packages/py-gpaw/package.py index 8a94a37b1d..6e4181afbd 100644 --- a/var/spack/repos/builtin/packages/py-gpaw/package.py +++ b/var/spack/repos/builtin/packages/py-gpaw/package.py @@ -59,7 +59,7 @@ def patch(self): python_include = spec["python"].headers.directories[0] numpy_include = join_path( - spec["py-numpy"].prefix, spec["python"].package.platlib, "numpy", "core", "include" + spec["py-numpy"].package.module.python_platlib, "numpy", "core", "include" ) libs = blas.libs + lapack.libs + libxc.libs diff --git a/var/spack/repos/builtin/packages/py-installer/package.py b/var/spack/repos/builtin/packages/py-installer/package.py index 01132b63bb..6ecf2c5b4e 100644 --- a/var/spack/repos/builtin/packages/py-installer/package.py +++ b/var/spack/repos/builtin/packages/py-installer/package.py @@ -36,6 +36,4 @@ def install(self, spec, prefix): python(*args) def setup_dependent_package(self, module, dependent_spec): - installer = dependent_spec["python"].command - installer.add_default_arg("-m", "installer") - setattr(module, "installer", installer) + setattr(module, "installer", python.with_default_args("-m", "installer")) diff --git a/var/spack/repos/builtin/packages/py-ipykernel/package.py b/var/spack/repos/builtin/packages/py-ipykernel/package.py index 2357fbb56d..80490e66f0 100644 --- a/var/spack/repos/builtin/packages/py-ipykernel/package.py +++ b/var/spack/repos/builtin/packages/py-ipykernel/package.py @@ -89,4 +89,4 @@ class PyIpykernel(PythonPackage): @run_after("install") def install_data(self): """install the Jupyter kernel spec""" - self.spec["python"].command("-m", "ipykernel", "install", "--prefix=" + self.prefix) + python("-m", "ipykernel", "install", "--prefix=" + self.prefix) diff --git a/var/spack/repos/builtin/packages/py-libensemble/package.py b/var/spack/repos/builtin/packages/py-libensemble/package.py index cf5cbd6b8d..5b553912ae 100644 --- a/var/spack/repos/builtin/packages/py-libensemble/package.py +++ b/var/spack/repos/builtin/packages/py-libensemble/package.py @@ -100,7 +100,6 @@ def run_tutorial_script(self, script): if not os.path.isfile(exe): raise SkipTest(f"{script} is missing") - python = self.spec["python"].command python(exe, "--comms", "local", "--nworkers", "2") def test_uniform_sampling(self): diff --git a/var/spack/repos/builtin/packages/py-nanobind/package.py b/var/spack/repos/builtin/packages/py-nanobind/package.py index 909175140b..6fb598acdc 100644 --- a/var/spack/repos/builtin/packages/py-nanobind/package.py +++ b/var/spack/repos/builtin/packages/py-nanobind/package.py @@ -60,5 +60,5 @@ class PyNanobind(PythonPackage): @property def cmake_prefix_paths(self): - paths = [join_path(self.prefix, self.spec["python"].package.platlib, "nanobind", "cmake")] + paths = [join_path(python_platlib, "nanobind", "cmake")] return paths diff --git a/var/spack/repos/builtin/packages/py-pip/package.py b/var/spack/repos/builtin/packages/py-pip/package.py index 023eab21ec..111e50911b 100644 --- a/var/spack/repos/builtin/packages/py-pip/package.py +++ b/var/spack/repos/builtin/packages/py-pip/package.py @@ -77,7 +77,5 @@ def install(self, spec, prefix): args.insert(0, os.path.join(whl, "pip")) python(*args) - def setup_dependent_package(self, module, dependent_spec): - pip = dependent_spec["python"].command - pip.add_default_arg("-m", "pip") - setattr(module, "pip", pip) + def setup_dependent_package(self, module, dependent_spec: Spec): + setattr(module, "pip", python.with_default_args("-m", "pip")) diff --git a/var/spack/repos/builtin/packages/py-pybind11/package.py b/var/spack/repos/builtin/packages/py-pybind11/package.py index 1fa9f7beee..f7c298309b 100644 --- a/var/spack/repos/builtin/packages/py-pybind11/package.py +++ b/var/spack/repos/builtin/packages/py-pybind11/package.py @@ -102,7 +102,6 @@ def install_test(self): with working_dir("spack-test", create=True): # test include helper points to right location - python = self.spec["python"].command py_inc = python( "-c", "import pybind11 as py; print(py.get_include())", output=str ).strip() diff --git a/var/spack/repos/builtin/packages/py-pymol/package.py b/var/spack/repos/builtin/packages/py-pymol/package.py index fe9bcce3a8..a4ab0b522d 100644 --- a/var/spack/repos/builtin/packages/py-pymol/package.py +++ b/var/spack/repos/builtin/packages/py-pymol/package.py @@ -58,7 +58,7 @@ def install_launcher(self): script = join_path(python_platlib, "pymol", "__init__.py") shebang = "#!/bin/sh\n" - fdata = 'exec {0} {1} "$@"'.format(self.spec["python"].command, script) + fdata = f'exec {python.path} {script} "$@"' with open(fname, "w") as new: new.write(shebang + fdata) set_executable(fname) diff --git a/var/spack/repos/builtin/packages/py-pyqt4/package.py b/var/spack/repos/builtin/packages/py-pyqt4/package.py index 31e481d305..00e27c994e 100644 --- a/var/spack/repos/builtin/packages/py-pyqt4/package.py +++ b/var/spack/repos/builtin/packages/py-pyqt4/package.py @@ -48,7 +48,7 @@ def configure_args(self): "--destdir", python_platlib, "--pyuic4-interpreter", - self.spec["python"].command.path, + python.path, "--sipdir", self.prefix.share.sip.PyQt4, "--stubsdir", diff --git a/var/spack/repos/builtin/packages/py-pyspark/package.py b/var/spack/repos/builtin/packages/py-pyspark/package.py index 087378d753..058ac47bf7 100644 --- a/var/spack/repos/builtin/packages/py-pyspark/package.py +++ b/var/spack/repos/builtin/packages/py-pyspark/package.py @@ -25,5 +25,5 @@ class PyPyspark(PythonPackage): depends_on("py-py4j@0.10.9", when="@3.0.1:3.1.3", type=("build", "run")) def setup_run_environment(self, env): - env.set("PYSPARK_PYTHON", self.spec["python"].command.path) - env.set("PYSPARK_DRIVER_PYTHON", self.spec["python"].command.path) + env.set("PYSPARK_PYTHON", python.path) + env.set("PYSPARK_DRIVER_PYTHON", python.path) diff --git a/var/spack/repos/builtin/packages/py-pythonsollya/package.py b/var/spack/repos/builtin/packages/py-pythonsollya/package.py index 84a715c8d1..c8f427f873 100644 --- a/var/spack/repos/builtin/packages/py-pythonsollya/package.py +++ b/var/spack/repos/builtin/packages/py-pythonsollya/package.py @@ -34,9 +34,4 @@ class PyPythonsollya(PythonPackage): @run_before("install") def patch(self): - filter_file( - "PYTHON ?= python2", - "PYTHON ?= " + self.spec["python"].command.path, - "GNUmakefile", - string=True, - ) + filter_file("PYTHON ?= python2", f"PYTHON ?= {python.path}", "GNUmakefile", string=True) diff --git a/var/spack/repos/builtin/packages/py-tensorflow/package.py b/var/spack/repos/builtin/packages/py-tensorflow/package.py index 9a25ea09d2..56d12ff231 100644 --- a/var/spack/repos/builtin/packages/py-tensorflow/package.py +++ b/var/spack/repos/builtin/packages/py-tensorflow/package.py @@ -410,7 +410,7 @@ def setup_build_environment(self, env): spec = self.spec # Please specify the location of python - env.set("PYTHON_BIN_PATH", spec["python"].command.path) + env.set("PYTHON_BIN_PATH", python.path) # Please input the desired Python library path to use env.set("PYTHON_LIB_PATH", python_platlib) diff --git a/var/spack/repos/builtin/packages/py-torch/package.py b/var/spack/repos/builtin/packages/py-torch/package.py index 5de618494d..e7824283fd 100644 --- a/var/spack/repos/builtin/packages/py-torch/package.py +++ b/var/spack/repos/builtin/packages/py-torch/package.py @@ -688,7 +688,5 @@ def install_test(self): @property def cmake_prefix_paths(self): - cmake_prefix_paths = [ - join_path(self.prefix, self.spec["python"].package.platlib, "torch", "share", "cmake") - ] + cmake_prefix_paths = [join_path(python_platlib, "torch", "share", "cmake")] return cmake_prefix_paths diff --git a/var/spack/repos/builtin/packages/py-xdot/package.py b/var/spack/repos/builtin/packages/py-xdot/package.py index bf898a117e..73d703a287 100644 --- a/var/spack/repos/builtin/packages/py-xdot/package.py +++ b/var/spack/repos/builtin/packages/py-xdot/package.py @@ -50,8 +50,7 @@ def post_install(self): dst, ) # regenerate the byte-compiled __init__.py - python3 = spec["python"].command - python3("-m", "compileall", dst) + python("-m", "compileall", dst) def setup_run_environment(self, env): spec = self.spec diff --git a/var/spack/repos/builtin/packages/python-venv/package.py b/var/spack/repos/builtin/packages/python-venv/package.py new file mode 100644 index 0000000000..cd0fd84371 --- /dev/null +++ b/var/spack/repos/builtin/packages/python-venv/package.py @@ -0,0 +1,100 @@ +# Copyright 2013-2023 Lawrence Livermore National Security, LLC and other +# Spack Project Developers. See the top-level COPYRIGHT file for details. +# +# SPDX-License-Identifier: (Apache-2.0 OR MIT) + +import os +import shutil + +import llnl.util.filesystem as fs + +from spack.package import * + + +class PythonVenv(Package): + """A Spack managed Python virtual environment""" + + homepage = "https://docs.python.org/3/library/venv.html" + has_code = False + + maintainers("haampie") + + version("1.0") + + extends("python") + + def install(self, spec, prefix): + # Create a virtual environment + python("-m", "venv", "--without-pip", prefix) + + def add_files_to_view(self, view, merge_map: Dict[str, str], skip_if_exists=True): + for src, dst in merge_map.items(): + if skip_if_exists and os.path.lexists(dst): + continue + + name = os.path.basename(dst) + + # Replace the VIRTUAL_ENV variable in the activate scripts after copying + if name.lower().startswith("activate"): + shutil.copy(src, dst) + fs.filter_file( + self.spec.prefix, + os.path.abspath(view.get_projection_for_spec(self.spec)), + dst, + string=True, + ) + else: + view.link(src, dst) + + @property + def bindir(self): + windows = self.spec.satisfies("platform=windows") + return join_path(self.prefix, "Scripts" if windows else "bin") + + @property + def command(self): + """Returns a python Executable instance""" + return which("python3", path=self.bindir) + + def _get_path(self, name) -> str: + return self.command( + "-Ec", f"import sysconfig; print(sysconfig.get_path('{name}'))", output=str + ).strip() + + @property + def platlib(self) -> str: + """Directory for site-specific, platform-specific files.""" + relative_platlib = os.path.relpath(self._get_path("platlib"), self.prefix) + assert not relative_platlib.startswith("..") + return relative_platlib + + @property + def purelib(self) -> str: + """Directory for site-specific, non-platform-specific files.""" + relative_purelib = os.path.relpath(self._get_path("purelib"), self.prefix) + assert not relative_purelib.startswith("..") + return relative_purelib + + @property + def headers(self): + return HeaderList([]) + + @property + def libs(self): + return LibraryList([]) + + def setup_dependent_run_environment(self, env, dependent_spec): + """Set PYTHONPATH to include the site-packages directory for the + extension and any other python extensions it depends on.""" + # Packages may be installed in platform-specific or platform-independent site-packages + # directories + for directory in {self.platlib, self.purelib}: + path = os.path.join(dependent_spec.prefix, directory) + if os.path.isdir(path): + env.prepend_path("PYTHONPATH", path) + + def setup_dependent_package(self, module, dependent_spec): + """Called before python modules' install() methods.""" + module.python = self.command + module.python_platlib = join_path(dependent_spec.prefix, self.platlib) + module.python_purelib = join_path(dependent_spec.prefix, self.purelib) diff --git a/var/spack/repos/builtin/packages/python/package.py b/var/spack/repos/builtin/packages/python/package.py index ad494226ee..b87ee81305 100644 --- a/var/spack/repos/builtin/packages/python/package.py +++ b/var/spack/repos/builtin/packages/python/package.py @@ -8,19 +8,16 @@ import os import platform import re -import stat import subprocess import sys from shutil import copy -from typing import Dict, List, Tuple +from typing import Dict, List import llnl.util.tty as tty -from llnl.util.filesystem import is_nonsymlink_exe_with_shebang, path_contains_subdirectory from llnl.util.lang import dedupe from spack.build_environment import dso_suffix, stat_suffix from spack.package import * -from spack.util.environment import is_system_path from spack.util.prefix import Prefix @@ -1115,7 +1112,7 @@ def platlib(self): path = self.config_vars["platlib"] if path.startswith(prefix): return path.replace(prefix, "") - return os.path.join("lib64", "python{}".format(self.version.up_to(2)), "site-packages") + return os.path.join("lib64", f"python{self.version.up_to(2)}", "site-packages") @property def purelib(self): @@ -1135,7 +1132,7 @@ def purelib(self): path = self.config_vars["purelib"] if path.startswith(prefix): return path.replace(prefix, "") - return os.path.join("lib", "python{}".format(self.version.up_to(2)), "site-packages") + return os.path.join("lib", f"python{self.version.up_to(2)}", "site-packages") @property def include(self): @@ -1163,39 +1160,6 @@ def setup_dependent_build_environment(self, env, dependent_spec): """Set PYTHONPATH to include the site-packages directory for the extension and any other python extensions it depends on. """ - # Ensure the current Python is first in the PATH - path = os.path.dirname(self.command.path) - if not is_system_path(path): - env.prepend_path("PATH", path) - - # Add installation prefix to PYTHONPATH, needed to run import tests - prefixes = set() - if dependent_spec.package.extends(self.spec): - prefixes.add(dependent_spec.prefix) - - # Add direct build/run/test dependencies to PYTHONPATH, - # needed to build the package and to run import tests - for direct_dep in dependent_spec.dependencies(deptype=("build", "run", "test")): - if direct_dep.package.extends(self.spec): - prefixes.add(direct_dep.prefix) - - # Add recursive run dependencies of all direct dependencies, - # needed by direct dependencies at run-time - for indirect_dep in direct_dep.traverse(deptype="run"): - if indirect_dep.package.extends(self.spec): - prefixes.add(indirect_dep.prefix) - - for prefix in prefixes: - # Packages may be installed in platform-specific or platform-independent - # site-packages directories - for directory in {self.platlib, self.purelib}: - env.prepend_path("PYTHONPATH", os.path.join(prefix, directory)) - - if self.spec.satisfies("platform=windows"): - prefix_scripts_dir = prefix.Scripts - if os.path.exists(prefix_scripts_dir): - env.prepend_path("PATH", prefix_scripts_dir) - # We need to make sure that the extensions are compiled and linked with # the Spack wrapper. Paths to the executables that are used for these # operations are normally taken from the sysconfigdata file, which we @@ -1241,9 +1205,7 @@ def setup_dependent_build_environment(self, env, dependent_spec): # invoked directly (no change would be required in that case # because Spack arranges for the Spack ld wrapper to be the # first instance of "ld" in PATH). - new_link = config_link.replace( - " {0} ".format(config_compile), " {0} ".format(new_compile) - ) + new_link = config_link.replace(f" {config_compile} ", f" {new_compile} ") # There is logic in the sysconfig module that is sensitive to the # fact that LDSHARED is set in the environment, therefore we export @@ -1256,66 +1218,23 @@ def setup_dependent_run_environment(self, env, dependent_spec): """Set PYTHONPATH to include the site-packages directory for the extension and any other python extensions it depends on. """ - if dependent_spec.package.extends(self.spec): - # Packages may be installed in platform-specific or platform-independent - # site-packages directories - for directory in {self.platlib, self.purelib}: - env.prepend_path("PYTHONPATH", os.path.join(dependent_spec.prefix, directory)) + if not dependent_spec.package.extends(self.spec) or dependent_spec.dependencies( + "python-venv" + ): + return + + # Packages may be installed in platform-specific or platform-independent site-packages + # directories + for directory in {self.platlib, self.purelib}: + env.prepend_path("PYTHONPATH", os.path.join(dependent_spec.prefix, directory)) def setup_dependent_package(self, module, dependent_spec): """Called before python modules' install() methods.""" - module.python = self.command - module.python_include = join_path(dependent_spec.prefix, self.include) module.python_platlib = join_path(dependent_spec.prefix, self.platlib) module.python_purelib = join_path(dependent_spec.prefix, self.purelib) - def add_files_to_view(self, view, merge_map, skip_if_exists=True): - # The goal is to copy the `python` executable, so that its search paths are relative to the - # view instead of the install prefix. This is an obsolete way of creating something that - # resembles a virtual environnent. Also we copy scripts with shebang lines. Finally we need - # to re-target symlinks pointing to copied files. - bin_dir = self.spec.prefix.bin if sys.platform != "win32" else self.spec.prefix - copied_files: Dict[Tuple[int, int], str] = {} # File identifier -> source - delayed_links: List[Tuple[str, str]] = [] # List of symlinks from merge map - for src, dst in merge_map.items(): - if skip_if_exists and os.path.lexists(dst): - continue - - # Files not in the bin dir are linked the default way. - if not path_contains_subdirectory(src, bin_dir): - view.link(src, dst, spec=self.spec) - continue - - s = os.lstat(src) - - # Symlink is delayed because we may need to re-target if its target is copied in view - if stat.S_ISLNK(s.st_mode): - delayed_links.append((src, dst)) - continue - - # Anything that's not a symlink gets copied. Scripts with shebangs are immediately - # updated when necessary. - copied_files[(s.st_dev, s.st_ino)] = dst - copy(src, dst) - if is_nonsymlink_exe_with_shebang(src): - filter_file( - self.spec.prefix, os.path.abspath(view.get_projection_for_spec(self.spec)), dst - ) - - # Finally re-target the symlinks that point to copied files. - for src, dst in delayed_links: - try: - s = os.stat(src) - target = copied_files[(s.st_dev, s.st_ino)] - except (OSError, KeyError): - target = None - if target: - os.symlink(os.path.relpath(target, os.path.dirname(dst)), dst) - else: - view.link(src, dst, spec=self.spec) - def test_hello_world(self): """run simple hello world program""" # do not use self.command because we are also testing the run env diff --git a/var/spack/repos/builtin/packages/qgis/package.py b/var/spack/repos/builtin/packages/qgis/package.py index 051a78660d..196a52622f 100644 --- a/var/spack/repos/builtin/packages/qgis/package.py +++ b/var/spack/repos/builtin/packages/qgis/package.py @@ -188,9 +188,7 @@ class Qgis(CMakePackage): @run_before("cmake", when="^py-pyqt5") def fix_pyqt5_cmake(self): cmfile = FileFilter(join_path("cmake", "FindPyQt5.cmake")) - pyqtpath = join_path( - self.spec["py-pyqt5"].prefix, self.spec["python"].package.platlib, "PyQt5" - ) + pyqtpath = join_path(self.spec["py-pyqt5"].package.module.python_platlib, "PyQt5") cmfile.filter( 'SET(PYQT5_MOD_DIR "${Python_SITEARCH}/PyQt5")', 'SET(PYQT5_MOD_DIR "' + pyqtpath + '")', @@ -210,7 +208,7 @@ def fix_qsci_sip(self): pyqtx = "PyQt6" sip_inc_dir = join_path( - self.spec["qscintilla"].prefix, self.spec["python"].package.platlib, pyqtx, "bindings" + self.spec["qscintilla"].package.module.python_platlib, pyqtx, "bindings" ) with open(join_path("python", "gui", "pyproject.toml.in"), "a") as tomlfile: tomlfile.write(f'\n[tool.sip.project]\nsip-include-dirs = ["{sip_inc_dir}"]\n') diff --git a/var/spack/repos/builtin/packages/qscintilla/package.py b/var/spack/repos/builtin/packages/qscintilla/package.py index bbf0b5a309..ce39d02d45 100644 --- a/var/spack/repos/builtin/packages/qscintilla/package.py +++ b/var/spack/repos/builtin/packages/qscintilla/package.py @@ -101,7 +101,7 @@ def make_qsci_python(self): with working_dir(join_path(self.stage.source_path, "Python")): copy(ftoml, "pyproject.toml") sip_inc_dir = join_path( - self.spec[py_pyqtx].prefix, self.spec["python"].package.platlib, pyqtx, "bindings" + self.spec[py_pyqtx].package.module.python_platlib, pyqtx, "bindings" ) with open("pyproject.toml", "a") as tomlfile: diff --git a/var/spack/repos/builtin/packages/redland-bindings/package.py b/var/spack/repos/builtin/packages/redland-bindings/package.py index 964f62108f..3c6bb42abc 100644 --- a/var/spack/repos/builtin/packages/redland-bindings/package.py +++ b/var/spack/repos/builtin/packages/redland-bindings/package.py @@ -3,8 +3,6 @@ # # SPDX-License-Identifier: (Apache-2.0 OR MIT) -import os - from spack.package import * @@ -27,11 +25,4 @@ class RedlandBindings(AutotoolsPackage): extends("python") def configure_args(self): - plib = self.spec["python"].prefix.lib - plib64 = self.spec["python"].prefix.lib64 - mybase = self.prefix.lib - if os.path.isdir(plib64) and not os.path.isdir(plib): - mybase = self.prefix.lib64 - pver = "python{0}".format(self.spec["python"].version.up_to(2)) - myplib = join_path(mybase, pver, "site-packages") - return ["--with-python", "PYTHON_LIB={0}".format(myplib)] + return ["--with-python", f"PYTHON_LIB={python_platlib}"] diff --git a/var/spack/repos/builtin/packages/z3/package.py b/var/spack/repos/builtin/packages/z3/package.py index e386e050c8..b42d46655a 100644 --- a/var/spack/repos/builtin/packages/z3/package.py +++ b/var/spack/repos/builtin/packages/z3/package.py @@ -53,13 +53,6 @@ def cmake_args(self): ] if spec.satisfies("+python"): - args.append( - self.define( - "CMAKE_INSTALL_PYTHON_PKG_DIR", - join_path( - prefix.lib, "python%s" % spec["python"].version.up_to(2), "site-packages" - ), - ) - ) + args.append(self.define("CMAKE_INSTALL_PYTHON_PKG_DIR", python_platlib)) return args diff --git a/var/spack/repos/duplicates.test/packages/python-venv b/var/spack/repos/duplicates.test/packages/python-venv new file mode 120000 index 0000000000..a9a1ce867a --- /dev/null +++ b/var/spack/repos/duplicates.test/packages/python-venv @@ -0,0 +1 @@ +../../builtin.mock/packages/python-venv \ No newline at end of file