hlrs-software-stack/spack
7
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity

Avoid hidden circular dependencies in spack.architecture (#25873)

Browse source 
* Refactor platform etc. to avoid circular dependencies

All the base classes in spack.architecture have been
moved to the corresponding specialized subpackages,
e.g. Platform is now defined within spack.platforms.

This resolves a circular dependency where spack.architecture
was both:
- Defining the base classes for spack.platforms, etc.
- Collecting derived classes from spack.platforms, etc.
Now it dopes only the latter.

* Move a few platform related functions to "spack.platforms"

* Removed spack.architecture.sys_type()

* Fixup for docs

* Rename Python modules according to review
This commit is contained in:
Massimiliano Culpo 2021-09-13 20:04:42 +02:00 committed by GitHub
parent 060582a21d
commit e9f1cfdaaf
No known key found for this signature in database
GPG key ID: 4AEE18F83AFDEB23
23 changed files with 492 additions and 478 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

2
lib/spack/docs/developer_guide.rst
View file

@ -211,7 +211,7 @@ Spec-related modules
yet.
:mod:`spack.architecture`
:func:`architecture.sys_type <spack.architecture.sys_type>` is used
:func:`architecture.default_arch <spack.architecture.default_arch>` is used
to determine the host architecture while building.
.. warning::

457
lib/spack/spack/architecture.py
View file

@ -2,28 +2,24 @@
# Spack Project Developers. See the top-level COPYRIGHT file for details.
#
# SPDX-License-Identifier: (Apache-2.0 OR MIT)
"""
This module contains all the elements that are required to create an
architecture object. These include, the target processor, the operating system,
and the architecture platform (i.e. cray, darwin, linux, etc) classes.
"""Aggregate the target processor, the operating system and the target
platform into an architecture object.
On a multiple architecture machine, the architecture spec field can be set to
build a package against any target and operating system that is present on the
platform. On Cray platforms or any other architecture that has different front
and back end environments, the operating system will determine the method of
compiler
detection.
compiler detection.
There are two different types of compiler detection:
1. Through the $PATH env variable (front-end detection)
2. Through the tcl module system. (back-end detection)
2. Through the module system. (back-end detection)
Depending on which operating system is specified, the compiler will be detected
using one of those methods.
For platforms such as linux and darwin, the operating system is autodetected
and the target is set to be x86_64.
For platforms such as linux and darwin, the operating system is autodetected.
The command line syntax for specifying an architecture is as follows:
@ -33,10 +29,8 @@
the command line and Spack will concretize using the default. These defaults
are set in the 'platforms/' directory which contains the different subclasses
for platforms. If the machine has multiple architectures, the user can
also enter front-end, or fe or back-end or be. These settings will concretize
to their respective front-end and back-end targets and operating systems.
Additional platforms can be added by creating a subclass of Platform
and adding it inside the platform directory.
also enter frontend, or fe or backend or be. These settings will concretize
to their respective frontend and backend targets and operating systems.
Platforms are an abstract class that are extended by subclasses. If the user
wants to add a new type of platform (such as cray_xe), they can create a
@ -47,334 +41,30 @@
new platform is added and the user wants that to be detected first.
Targets are created inside the platform subclasses. Most architecture
(like linux, and darwin) will have only one target (x86_64) but in the case of
(like linux, and darwin) will have only one target family (x86_64) but in the case of
Cray machines, there is both a frontend and backend processor. The user can
specify which targets are present on front-end and back-end architecture
Depending on the platform, operating systems are either auto-detected or are
set. The user can set the front-end and back-end operating setting by the class
Depending on the platform, operating systems are either autodetected or are
set. The user can set the frontend and backend operating setting by the class
attributes front_os and back_os. The operating system as described earlier,
will be responsible for compiler detection.
"""
import contextlib
import functools
import warnings
import six
import archspec.cpu
import llnl.util.lang as lang
import llnl.util.tty as tty
import spack.compiler
import spack.compilers
import spack.config
import spack.error as serr
import spack.paths
import spack.operating_systems
import spack.platforms
import spack.spec
import spack.util.classes
import spack.util.executable
import spack.target
import spack.util.spack_yaml as syaml
import spack.version
from spack.util.spack_yaml import syaml_dict
class NoPlatformError(serr.SpackError):
def __init__(self):
super(NoPlatformError, self).__init__(
"Could not determine a platform for this machine.")
def _ensure_other_is_target(method):
"""Decorator to be used in dunder methods taking a single argument to
ensure that the argument is an instance of ``Target`` too.
"""
@functools.wraps(method)
def _impl(self, other):
if isinstance(other, six.string_types):
other = Target(other)
if not isinstance(other, Target):
return NotImplemented
return method(self, other)
return _impl
class Target(object):
def __init__(self, name, module_name=None):
"""Target models microarchitectures and their compatibility.
Args:
name (str or Microarchitecture):micro-architecture of the
target
module_name (str): optional module name to get access to the
current target. This is typically used on machines
like Cray (e.g. craype-compiler)
"""
if not isinstance(name, archspec.cpu.Microarchitecture):
name = archspec.cpu.TARGETS.get(
name, archspec.cpu.generic_microarchitecture(name)
)
self.microarchitecture = name
self.module_name = module_name
@property
def name(self):
return self.microarchitecture.name
@_ensure_other_is_target
def __eq__(self, other):
return self.microarchitecture == other.microarchitecture and \
self.module_name == other.module_name
def __ne__(self, other):
# This method is necessary as long as we support Python 2. In Python 3
# __ne__ defaults to the implementation below
return not self == other
@_ensure_other_is_target
def __lt__(self, other):
# TODO: In the future it would be convenient to say
# TODO: `spec.architecture.target < other.architecture.target`
# TODO: and change the semantic of the comparison operators
# This is needed to sort deterministically specs in a list.
# It doesn't implement a total ordering semantic.
return self.microarchitecture.name < other.microarchitecture.name
def __hash__(self):
return hash((self.name, self.module_name))
@staticmethod
def from_dict_or_value(dict_or_value):
# A string here represents a generic target (like x86_64 or ppc64) or
# a custom micro-architecture
if isinstance(dict_or_value, six.string_types):
return Target(dict_or_value)
# TODO: From a dict we actually retrieve much more information than
# TODO: just the name. We can use that information to reconstruct an
# TODO: "old" micro-architecture or check the current definition.
target_info = dict_or_value
return Target(target_info['name'])
def to_dict_or_value(self):
"""Returns a dict or a value representing the current target.
String values are used to keep backward compatibility with generic
targets, like e.g. x86_64 or ppc64. More specific micro-architectures
will return a dictionary which contains information on the name,
features, vendor, generation and parents of the current target.
"""
# Generic targets represent either an architecture
# family (like x86_64) or a custom micro-architecture
if self.microarchitecture.vendor == 'generic':
return str(self)
return syaml_dict(
self.microarchitecture.to_dict(return_list_of_items=True)
)
def __repr__(self):
cls_name = self.__class__.__name__
fmt = cls_name + '({0}, {1})'
return fmt.format(repr(self.microarchitecture),
repr(self.module_name))
def __str__(self):
return str(self.microarchitecture)
def __contains__(self, cpu_flag):
return cpu_flag in self.microarchitecture
def optimization_flags(self, compiler):
"""Returns the flags needed to optimize for this target using
the compiler passed as argument.
Args:
compiler (spack.spec.CompilerSpec or spack.compiler.Compiler): object that
contains both the name and the version of the compiler we want to use
"""
# Mixed toolchains are not supported yet
if isinstance(compiler, spack.compiler.Compiler):
if spack.compilers.is_mixed_toolchain(compiler):
msg = ('microarchitecture specific optimizations are not '
'supported yet on mixed compiler toolchains [check'
' {0.name}@{0.version} for further details]')
warnings.warn(msg.format(compiler))
return ''
# Try to check if the current compiler comes with a version number or
# has an unexpected suffix. If so, treat it as a compiler with a
# custom spec.
compiler_version = compiler.version
version_number, suffix = archspec.cpu.version_components(
compiler.version
)
if not version_number or suffix not in ('', 'apple'):
# Try to deduce the underlying version of the compiler, regardless
# of its name in compilers.yaml. Depending on where this function
# is called we might get either a CompilerSpec or a fully fledged
# compiler object.
if isinstance(compiler, spack.spec.CompilerSpec):
compiler = spack.compilers.compilers_for_spec(compiler).pop()
try:
compiler_version = compiler.real_version
except spack.util.executable.ProcessError as e:
# log this and just return compiler.version instead
tty.debug(str(e))
return self.microarchitecture.optimization_flags(
compiler.name, str(compiler_version)
)
@lang.lazy_lexicographic_ordering
class Platform(object):
""" Abstract class that each type of Platform will subclass.
Will return a instance of it once it is returned.
"""
# Subclass sets number. Controls detection order
priority = None # type: int
#: binary formats used on this platform; used by relocation logic
binary_formats = ['elf']
front_end = None # type: str
back_end = None # type: str
default = None # type: str # The default back end target.
front_os = None # type: str
back_os = None # type: str
default_os = None # type: str
reserved_targets = ['default_target', 'frontend', 'fe', 'backend', 'be']
reserved_oss = ['default_os', 'frontend', 'fe', 'backend', 'be']
def __init__(self, name):
self.targets = {}
self.operating_sys = {}
self.name = name
def add_target(self, name, target):
"""Used by the platform specific subclass to list available targets.
Raises an error if the platform specifies a name
that is reserved by spack as an alias.
"""
if name in Platform.reserved_targets:
raise ValueError(
"%s is a spack reserved alias "
"and cannot be the name of a target" % name)
self.targets[name] = target
def target(self, name):
"""This is a getter method for the target dictionary
that handles defaulting based on the values provided by default,
front-end, and back-end. This can be overwritten
by a subclass for which we want to provide further aliasing options.
"""
# TODO: Check if we can avoid using strings here
name = str(name)
if name == 'default_target':
name = self.default
elif name == 'frontend' or name == 'fe':
name = self.front_end
elif name == 'backend' or name == 'be':
name = self.back_end
return self.targets.get(name, None)
def add_operating_system(self, name, os_class):
""" Add the operating_system class object into the
platform.operating_sys dictionary
"""
if name in Platform.reserved_oss:
raise ValueError(
"%s is a spack reserved alias "
"and cannot be the name of an OS" % name)
self.operating_sys[name] = os_class
def operating_system(self, name):
if name == 'default_os':
name = self.default_os
if name == 'frontend' or name == "fe":
name = self.front_os
if name == 'backend' or name == 'be':
name = self.back_os
return self.operating_sys.get(name, None)
@classmethod
def setup_platform_environment(cls, pkg, env):
""" Subclass can override this method if it requires any
platform-specific build environment modifications.
"""
@classmethod
def detect(cls):
""" Subclass is responsible for implementing this method.
Returns True if the Platform class detects that
it is the current platform
and False if it's not.
"""
raise NotImplementedError()
def __repr__(self):
return self.__str__()
def __str__(self):
return self.name
def _cmp_iter(self):
yield self.name
yield self.default
yield self.front_end
yield self.back_end
yield self.default_os
yield self.front_os
yield self.back_os
def targets():
for t in sorted(self.targets.values()):
yield t._cmp_iter
yield targets
def oses():
for o in sorted(self.operating_sys.values()):
yield o._cmp_iter
yield oses
@lang.lazy_lexicographic_ordering
class OperatingSystem(object):
""" Operating System will be like a class similar to platform extended
by subclasses for the specifics. Operating System will contain the
compiler finding logic. Instead of calling two separate methods to
find compilers we call find_compilers method for each operating system
"""
def __init__(self, name, version):
self.name = name.replace('-', '_')
self.version = str(version).replace('-', '_')
def __str__(self):
return "%s%s" % (self.name, self.version)
def __repr__(self):
return self.__str__()
def _cmp_iter(self):
yield self.name
yield self.version
def to_dict(self):
return syaml_dict([
('name', self.name),
('version', self.version)
])
@lang.lazy_lexicographic_ordering
@ -399,11 +89,13 @@ def __init__(self, plat=None, os=None, target=None):
@property
def concrete(self):
return all((self.platform is not None,
isinstance(self.platform, Platform),
self.os is not None,
isinstance(self.os, OperatingSystem),
self.target is not None, isinstance(self.target, Target)))
return all(
(self.platform is not None,
isinstance(self.platform, spack.platforms.Platform),
self.os is not None,
isinstance(self.os, spack.operating_systems.OperatingSystem),
self.target is not None, isinstance(self.target, spack.target.Target))
)
def __str__(self):
if self.platform or self.os or self.target:
@ -428,28 +120,28 @@ def __nonzero__(self):
__bool__ = __nonzero__
def _cmp_iter(self):
if isinstance(self.platform, Platform):
if isinstance(self.platform, spack.platforms.Platform):
yield self.platform.name
else:
yield self.platform
if isinstance(self.os, OperatingSystem):
if isinstance(self.os, spack.operating_systems.OperatingSystem):
yield self.os.name
else:
yield self.os
if isinstance(self.target, Target):
if isinstance(self.target, spack.target.Target):
yield self.target.microarchitecture
else:
yield self.target
def to_dict(self):
str_or_none = lambda v: str(v) if v else None
d = syaml_dict([
d = syaml.syaml_dict([
('platform', str_or_none(self.platform)),
('platform_os', str_or_none(self.os)),
('target', self.target.to_dict_or_value())])
return syaml_dict([('arch', d)])
return syaml.syaml_dict([('arch', d)])
def to_spec(self):
"""Convert this Arch to an anonymous Spec with architecture defined."""
@ -463,64 +155,25 @@ def from_dict(d):
return arch_for_spec(spec)
@lang.memoized
def get_platform(platform_name):
"""Returns a platform object that corresponds to the given name."""
platform_list = all_platforms()
for p in platform_list:
if platform_name.replace("_", "").lower() == p.__name__.lower():
return p()
def verify_platform(platform_name):
""" Determines whether or not the platform with the given name is supported
in Spack. For more information, see the 'spack.platforms' submodule.
"""
platform_name = platform_name.replace("_", "").lower()
platform_names = [p.__name__.lower() for p in all_platforms()]
if platform_name not in platform_names:
tty.die("%s is not a supported platform; supported platforms are %s" %
(platform_name, platform_names))
def arch_for_spec(arch_spec):
"""Transforms the given architecture spec into an architecture object."""
arch_spec = spack.spec.ArchSpec(arch_spec)
assert arch_spec.concrete
arch_plat = get_platform(arch_spec.platform)
arch_plat = spack.platforms.by_name(arch_spec.platform)
if not (arch_plat.operating_system(arch_spec.os) and
arch_plat.target(arch_spec.target)):
raise ValueError(
"Can't recreate arch for spec %s on current arch %s; "
"spec architecture is too different" % (arch_spec, sys_type()))
sys_type = str(default_arch())
msg = ("Can't recreate arch for spec {0} on current arch {1}; "
"spec architecture is too different")
raise ValueError(msg.format(arch_spec, sys_type))
return Arch(arch_plat, arch_spec.os, arch_spec.target)
@lang.memoized
def _all_platforms():
mod_path = spack.paths.platform_path
return spack.util.classes.list_classes("spack.platforms", mod_path)
@lang.memoized
def _platform():
"""Detects the platform for this machine.
Gather a list of all available subclasses of platforms.
Sorts the list according to their priority looking. Priority is
an arbitrarily set number. Detects platform either using uname or
a file path (/opt/cray...)
"""
# Try to create a Platform object using the config file FIRST
platform_list = _all_platforms()
platform_list.sort(key=lambda a: a.priority)
for platform_cls in platform_list:
if platform_cls.detect():
return platform_cls()
return spack.platforms.host()
#: The "real" platform of the host running Spack. This should not be changed
@ -531,44 +184,23 @@ def _platform():
#: context manager.
platform = _platform
#: The list of all platform classes. May be swapped by the use_platform
#: context manager.
all_platforms = _all_platforms
@lang.memoized
def default_arch():
"""Default ``Arch`` object for this machine.
See ``sys_type()``.
"""
"""Default ``Arch`` object for this machine"""
return Arch(platform(), 'default_os', 'default_target')
def sys_type():
"""Print out the "default" platform-os-target tuple for this machine.
On machines with only one target OS/target, prints out the
platform-os-target for the frontend. For machines with a frontend
and a backend, prints the default backend.
TODO: replace with use of more explicit methods to get *all* the
backends, as client code should really be aware of cross-compiled
architectures.
"""
return str(default_arch())
@lang.memoized
def compatible_sys_types():
"""Returns a list of all the systypes compatible with the current host."""
compatible_archs = []
"""Return a list of all the platform-os-target tuples compatible
with the current host.
"""
current_host = archspec.cpu.host()
compatible_targets = [current_host] + current_host.ancestors
for target in compatible_targets:
arch = Arch(platform(), 'default_os', target)
compatible_archs.append(str(arch))
compatible_archs = [
str(Arch(platform(), 'default_os', target)) for target in compatible_targets
]
return compatible_archs
@ -586,16 +218,15 @@ def __call__(self):
@contextlib.contextmanager
def use_platform(new_platform):
global platform, all_platforms
global platform
msg = '"{0}" must be an instance of Platform'
assert isinstance(new_platform, Platform), msg.format(new_platform)
assert isinstance(new_platform, spack.platforms.Platform), msg.format(new_platform)
original_platform_fn, original_all_platforms_fn = platform, all_platforms
original_platform_fn = platform
try:
platform = _PickleableCallable(new_platform)
all_platforms = _PickleableCallable([type(new_platform)])
# Clear configuration and compiler caches
spack.config.config.clear_caches()
@ -604,7 +235,7 @@ def use_platform(new_platform):
yield new_platform
finally:
platform, all_platforms = original_platform_fn, original_all_platforms_fn
platform = original_platform_fn
# Clear configuration and compiler caches
spack.config.config.clear_caches()

5
lib/spack/spack/binary_distribution.py
View file

@ -31,6 +31,7 @@
import spack.hash_types as ht
import spack.hooks.sbang
import spack.mirror
import spack.platforms
import spack.relocate as relocate
import spack.util.file_cache as file_cache
import spack.util.gpg
@ -1156,7 +1157,7 @@ def make_package_relative(workdir, spec, allow_root):
orig_path_names.append(os.path.join(prefix, filename))
cur_path_names.append(os.path.join(workdir, filename))
platform = spack.architecture.get_platform(spec.platform)
platform = spack.platforms.by_name(spec.platform)
if 'macho' in platform.binary_formats:
relocate.make_macho_binaries_relative(
cur_path_names, orig_path_names, old_layout_root)
@ -1267,7 +1268,7 @@ def is_backup_file(file):
]
# If the buildcache was not created with relativized rpaths
# do the relocation of path in binaries
platform = spack.architecture.get_platform(spec.platform)
platform = spack.platforms.by_name(spec.platform)
if 'macho' in platform.binary_formats:
relocate.relocate_macho_binaries(files_to_relocate,
old_layout_root,

2
lib/spack/spack/compilers/__init__.py
View file

@ -544,7 +544,7 @@ def arguments_to_detect_version_fn(operating_system, paths):
function by providing a method called with the same name.
Args:
operating_system (spack.architecture.OperatingSystem): the operating system
operating_system (spack.operating_systems.OperatingSystem): the operating system
on which we are looking for compilers
paths: paths to search for compilers

13
lib/spack/spack/concretize.py
View file

@ -37,6 +37,7 @@
import spack.error
import spack.repo
import spack.spec
import spack.target
import spack.tengine
import spack.variant as vt
from spack.config import config
@ -256,8 +257,7 @@ def concretize_architecture(self, spec):
# Get platform of nearest spec with a platform, including spec
# If spec has a platform, easy
if spec.architecture.platform:
new_plat = spack.architecture.get_platform(
spec.architecture.platform)
new_plat = spack.platforms.by_name(spec.architecture.platform)
else:
# Else if anyone else has a platform, take the closest one
# Search up, then down, along build/link deps first
@ -266,8 +266,7 @@ def concretize_architecture(self, spec):
spec, lambda x: x.architecture and x.architecture.platform
)
if platform_spec:
new_plat = spack.architecture.get_platform(
platform_spec.architecture.platform)
new_plat = spack.platforms.by_name(platform_spec.architecture.platform)
else:
# If no platform anywhere in this spec, grab the default
new_plat = spack.architecture.platform()
@ -612,9 +611,7 @@ def _adjust_target(self, spec):
# Try to adjust the target only if it is the default
# target for this platform
current_target = spec.architecture.target
current_platform = spack.architecture.get_platform(
spec.architecture.platform
)
current_platform = spack.platforms.by_name(spec.architecture.platform)
default_target = current_platform.target('default_target')
if PackagePrefs.has_preferred_targets(spec.name):
@ -633,7 +630,7 @@ def _adjust_target(self, spec):
for ancestor in microarchitecture.ancestors:
candidate = None
try:
candidate = spack.architecture.Target(ancestor)
candidate = spack.target.Target(ancestor)
candidate.optimization_flags(spec.compiler)
except archspec.cpu.UnsupportedMicroarchitecture:
continue

2
lib/spack/spack/main.py
View file

@ -666,7 +666,7 @@ def shell_set(var, value):
tty.die('shell must be sh or csh')
# print sys type
shell_set('_sp_sys_type', spack.architecture.sys_type())
shell_set('_sp_sys_type', str(spack.architecture.default_arch()))
shell_set('_sp_compatible_sys_types',
':'.join(spack.architecture.compatible_sys_types()))
# print roots for all module systems

16
lib/spack/spack/operating_systems/__init__.py
View file

@ -2,3 +2,19 @@
# Spack Project Developers. See the top-level COPYRIGHT file for details.
#
# SPDX-License-Identifier: (Apache-2.0 OR MIT)
from ._operating_system import OperatingSystem
from .cray_backend import CrayBackend
from .cray_frontend import CrayFrontend
from .linux_distro import LinuxDistro
from .mac_os import MacOs
__all__ = [
'OperatingSystem',
'LinuxDistro',
'MacOs',
'CrayFrontend',
'CrayBackend'
]
#: List of all the Operating Systems known to Spack
operating_systems = [LinuxDistro, MacOs, CrayFrontend, CrayBackend]

36
lib/spack/spack/operating_systems/_operating_system.py Normal file
View file

@ -0,0 +1,36 @@
# Copyright 2013-2021 Lawrence Livermore National Security, LLC and other
# Spack Project Developers. See the top-level COPYRIGHT file for details.
#
# SPDX-License-Identifier: (Apache-2.0 OR MIT)
import llnl.util.lang
import spack.util.spack_yaml as syaml
@llnl.util.lang.lazy_lexicographic_ordering
class OperatingSystem(object):
"""Base class for all the Operating Systems.
Each Operating System contain its own compiler finding logic, that is used
to detect compilers.
"""
def __init__(self, name, version):
self.name = name.replace('-', '_')
self.version = str(version).replace('-', '_')
def __str__(self):
return "%s%s" % (self.name, self.version)
def __repr__(self):
return self.__str__()
def _cmp_iter(self):
yield self.name
yield self.version
def to_dict(self):
return syaml.syaml_dict([
('name', self.name),
('version', self.version)
])

3
lib/spack/spack/operating_systems/cray_backend.py
View file

@ -10,9 +10,10 @@
import spack.error
import spack.version
from spack.operating_systems.linux_distro import LinuxDistro
from spack.util.module_cmd import module
from .linux_distro import LinuxDistro
#: Possible locations of the Cray CLE release file,
#: which we look at to get the CNL OS version.
_cle_release_file = '/etc/opt/cray/release/cle-release'

3
lib/spack/spack/operating_systems/cray_frontend.py
View file

@ -11,10 +11,11 @@
import llnl.util.lang
import llnl.util.tty as tty
from spack.operating_systems.linux_distro import LinuxDistro
from spack.util.environment import get_path
from spack.util.module_cmd import module
from .linux_distro import LinuxDistro
@contextlib.contextmanager
def unload_programming_environment():

3
lib/spack/spack/operating_systems/linux_distro.py
View file

@ -2,10 +2,9 @@
# Spack Project Developers. See the top-level COPYRIGHT file for details.
#
# SPDX-License-Identifier: (Apache-2.0 OR MIT)
import re
from spack.architecture import OperatingSystem
from ._operating_system import OperatingSystem
class LinuxDistro(OperatingSystem):

3
lib/spack/spack/operating_systems/mac_os.py
View file

@ -5,10 +5,11 @@
import platform as py_platform
from spack.architecture import OperatingSystem
from spack.util.executable import Executable
from spack.version import Version
from ._operating_system import OperatingSystem
# FIXME: store versions inside OperatingSystem as a Version instead of string
def macos_version():

40
lib/spack/spack/platforms/__init__.py
View file

@ -2,14 +2,54 @@
# Spack Project Developers. See the top-level COPYRIGHT file for details.
#
# SPDX-License-Identifier: (Apache-2.0 OR MIT)
import llnl.util.lang
from ._platform import Platform
from .cray import Cray
from .darwin import Darwin
from .linux import Linux
from .test import Test
__all__ = [
'Platform',
'Cray',
'Darwin',
'Linux',
'Test'
]
#: List of all the platform classes known to Spack
platforms = [Cray, Darwin, Linux, Test]
def host():
"""Detect and return the platform for this machine or None if detection fails."""
for platform_cls in sorted(platforms, key=lambda plt: plt.priority):
if platform_cls.detect():
return platform_cls()
return None
@llnl.util.lang.memoized
def cls_by_name(name):
"""Return a platform class that corresponds to the given name or None
if there is no match.
Args:
name (str): name of the platform
"""
for platform_cls in sorted(platforms, key=lambda plt: plt.priority):
if name.replace("_", "").lower() == platform_cls.__name__.lower():
return platform_cls
return None
def by_name(name):
"""Return a platform object that corresponds to the given name or None
if there is no match.
Args:
name (str): name of the platform
"""
platform_cls = cls_by_name(name)
return platform_cls() if platform_cls else None

126
lib/spack/spack/platforms/_platform.py Normal file
View file

@ -0,0 +1,126 @@
# Copyright 2013-2021 Lawrence Livermore National Security, LLC and other
# Spack Project Developers. See the top-level COPYRIGHT file for details.
#
# SPDX-License-Identifier: (Apache-2.0 OR MIT)
import llnl.util.lang
import spack.error
class NoPlatformError(spack.error.SpackError):
def __init__(self):
msg = "Could not determine a platform for this machine"
super(NoPlatformError, self).__init__(msg)
@llnl.util.lang.lazy_lexicographic_ordering
class Platform(object):
"""Base class for each type of Platform"""
# Subclass sets number. Controls detection order
priority = None # type: int
#: binary formats used on this platform; used by relocation logic
binary_formats = ['elf']
front_end = None # type: str
back_end = None # type: str
default = None # type: str # The default back end target.
front_os = None # type: str
back_os = None # type: str
default_os = None # type: str
reserved_targets = ['default_target', 'frontend', 'fe', 'backend', 'be']
reserved_oss = ['default_os', 'frontend', 'fe', 'backend', 'be']
def __init__(self, name):
self.targets = {}
self.operating_sys = {}
self.name = name
def add_target(self, name, target):
"""Used by the platform specific subclass to list available targets.
Raises an error if the platform specifies a name
that is reserved by spack as an alias.
"""
if name in Platform.reserved_targets:
msg = "{0} is a spack reserved alias and cannot be the name of a target"
raise ValueError(msg.format(name))
self.targets[name] = target
def target(self, name):
"""This is a getter method for the target dictionary
that handles defaulting based on the values provided by default,
front-end, and back-end. This can be overwritten
by a subclass for which we want to provide further aliasing options.
"""
# TODO: Check if we can avoid using strings here
name = str(name)
if name == 'default_target':
name = self.default
elif name == 'frontend' or name == 'fe':
name = self.front_end
elif name == 'backend' or name == 'be':
name = self.back_end
return self.targets.get(name, None)
def add_operating_system(self, name, os_class):
"""Add the operating_system class object into the
platform.operating_sys dictionary.
"""
if name in Platform.reserved_oss:
msg = "{0} is a spack reserved alias and cannot be the name of an OS"
raise ValueError(msg.format(name))
self.operating_sys[name] = os_class
def operating_system(self, name):
if name == 'default_os':
name = self.default_os
if name == 'frontend' or name == "fe":
name = self.front_os
if name == 'backend' or name == 'be':
name = self.back_os
return self.operating_sys.get(name, None)
@classmethod
def setup_platform_environment(cls, pkg, env):
"""Subclass can override this method if it requires any
platform-specific build environment modifications.
"""
@classmethod
def detect(cls):
"""Return True if the the host platform is detected to be the current
Platform class, False otherwise.
Derived classes are responsible for implementing this method.
"""
raise NotImplementedError()
def __repr__(self):
return self.__str__()
def __str__(self):
return self.name
def _cmp_iter(self):
yield self.name
yield self.default
yield self.front_end
yield self.back_end
yield self.default_os
yield self.front_os
yield self.back_os
def targets():
for t in sorted(self.targets.values()):
yield t._cmp_iter
yield targets
def oses():
for o in sorted(self.operating_sys.values()):
yield o._cmp_iter
yield oses

10
lib/spack/spack/platforms/cray.py
View file

@ -11,13 +11,15 @@
import llnl.util.tty as tty
from spack.architecture import NoPlatformError, Platform, Target
import spack.target
from spack.operating_systems.cray_backend import CrayBackend
from spack.operating_systems.cray_frontend import CrayFrontend
from spack.paths import build_env_path
from spack.util.executable import Executable
from spack.util.module_cmd import module
from ._platform import NoPlatformError, Platform
_craype_name_to_target_name = {
'x86-cascadelake': 'cascadelake',
'x86-naples': 'zen',
@ -51,7 +53,7 @@ def __init__(self):
# Make all craype targets available.
for target in self._avail_targets():
name = _target_name_from_craype_target_name(target)
self.add_target(name, Target(name, 'craype-%s' % target))
self.add_target(name, spack.target.Target(name, 'craype-%s' % target))
self.back_end = os.environ.get('SPACK_BACK_END',
self._default_target_from_env())
@ -63,12 +65,12 @@ def __init__(self):
# Setup frontend targets
for name in archspec.cpu.TARGETS:
if name not in self.targets:
self.add_target(name, Target(name))
self.add_target(name, spack.target.Target(name))
self.front_end = os.environ.get(
'SPACK_FRONT_END', archspec.cpu.host().name
)
if self.front_end not in self.targets:
self.add_target(self.front_end, Target(self.front_end))
self.add_target(self.front_end, spack.target.Target(self.front_end))
front_distro = CrayFrontend()
back_distro = CrayBackend()

6
lib/spack/spack/platforms/darwin.py
View file

@ -7,9 +7,11 @@
import archspec.cpu
from spack.architecture import Platform, Target
import spack.target
from spack.operating_systems.mac_os import MacOs
from ._platform import Platform
class Darwin(Platform):
priority = 89
@ -20,7 +22,7 @@ def __init__(self):
super(Darwin, self).__init__('darwin')
for name in archspec.cpu.TARGETS:
self.add_target(name, Target(name))
self.add_target(name, spack.target.Target(name))
self.default = archspec.cpu.host().name
self.front_end = self.default

6
lib/spack/spack/platforms/linux.py
View file

@ -6,9 +6,11 @@
import archspec.cpu
from spack.architecture import Platform, Target
import spack.target
from spack.operating_systems.linux_distro import LinuxDistro
from ._platform import Platform
class Linux(Platform):
priority = 90
@ -17,7 +19,7 @@ def __init__(self):
super(Linux, self).__init__('linux')
for name in archspec.cpu.TARGETS:
self.add_target(name, Target(name))
self.add_target(name, spack.target.Target(name))
# Get specific default
self.default = archspec.cpu.host().name

14
lib/spack/spack/platforms/test.py
View file

@ -2,10 +2,12 @@
# Spack Project Developers. See the top-level COPYRIGHT file for details.
#
# SPDX-License-Identifier: (Apache-2.0 OR MIT)
import platform
from spack.architecture import OperatingSystem, Platform, Target
import spack.operating_systems
import spack.target
from ._platform import Platform
class Test(Platform):
@ -24,13 +26,13 @@ class Test(Platform):
def __init__(self):
super(Test, self).__init__('test')
self.add_target(self.default, Target(self.default))
self.add_target(self.front_end, Target(self.front_end))
self.add_target(self.default, spack.target.Target(self.default))
self.add_target(self.front_end, spack.target.Target(self.front_end))
self.add_operating_system(
self.default_os, OperatingSystem('debian', 6))
self.default_os, spack.operating_systems.OperatingSystem('debian', 6))
self.add_operating_system(
self.front_os, OperatingSystem('redhat', 6))
self.front_os, spack.operating_systems.OperatingSystem('redhat', 6))
@classmethod
def detect(cls):

32
lib/spack/spack/spec.py
View file

@ -100,10 +100,12 @@
import spack.hash_types as ht
import spack.parse
import spack.paths
import spack.platforms
import spack.provider_index
import spack.repo
import spack.solver
import spack.store
import spack.target
import spack.util.crypto
import spack.util.executable
import spack.util.hash
@ -291,7 +293,7 @@ def os(self, value):
# will assumed to be the host machine's platform.
value = str(value) if value is not None else None
if value in spack.architecture.Platform.reserved_oss:
if value in spack.platforms.Platform.reserved_oss:
curr_platform = str(spack.architecture.platform())
self.platform = self.platform or curr_platform
@ -301,7 +303,7 @@ def os(self, value):
"arch platform (%s) isn't the current platform (%s)" %
(value, self.platform, curr_platform))
spec_platform = spack.architecture.get_platform(self.platform)
spec_platform = spack.platforms.by_name(self.platform)
value = str(spec_platform.operating_system(value))
self._os = value
@ -320,15 +322,15 @@ def target(self, value):
# will assumed to be the host machine's platform.
def target_or_none(t):
if isinstance(t, spack.architecture.Target):
if isinstance(t, spack.target.Target):
return t
if t and t != 'None':
return spack.architecture.Target(t)
return spack.target.Target(t)
return None
value = target_or_none(value)
if str(value) in spack.architecture.Platform.reserved_targets:
if str(value) in spack.platforms.Platform.reserved_targets:
curr_platform = str(spack.architecture.platform())
self.platform = self.platform or curr_platform
@ -338,7 +340,7 @@ def target_or_none(t):
"the arch platform (%s) isn't the current platform (%s)" %
(value, self.platform, curr_platform))
spec_platform = spack.architecture.get_platform(self.platform)
spec_platform = spack.platforms.by_name(self.platform)
value = spec_platform.target(value)
self._target = value
@ -416,7 +418,7 @@ def target_intersection(self, other):
# s_target_range is a concrete target
# get a microarchitecture reference for at least one side
# of each comparison so we can use archspec comparators
s_comp = spack.architecture.Target(s_min).microarchitecture
s_comp = spack.target.Target(s_min).microarchitecture
if not o_sep:
if s_min == o_min:
results.append(s_min)
@ -425,22 +427,22 @@ def target_intersection(self, other):
results.append(s_min)
elif not o_sep:
# "cast" to microarchitecture
o_comp = spack.architecture.Target(o_min).microarchitecture
o_comp = spack.target.Target(o_min).microarchitecture
if (not s_min or o_comp >= s_min) and (
not s_max or o_comp <= s_max):
results.append(o_min)
else:
# Take intersection of two ranges
# Lots of comparisons needed
_s_min = spack.architecture.Target(s_min).microarchitecture
_s_max = spack.architecture.Target(s_max).microarchitecture
_o_min = spack.architecture.Target(o_min).microarchitecture
_o_max = spack.architecture.Target(o_max).microarchitecture
_s_min = spack.target.Target(s_min).microarchitecture
_s_max = spack.target.Target(s_max).microarchitecture
_o_min = spack.target.Target(o_min).microarchitecture
_o_max = spack.target.Target(o_max).microarchitecture
n_min = s_min if _s_min >= _o_min else o_min
n_max = s_max if _s_max <= _o_max else o_max
_n_min = spack.architecture.Target(n_min).microarchitecture
_n_max = spack.architecture.Target(n_max).microarchitecture
_n_min = spack.target.Target(n_min).microarchitecture
_n_max = spack.target.Target(n_max).microarchitecture
if _n_min == _n_max:
results.append(n_min)
elif not n_min or not n_max or _n_min < _n_max:
@ -521,7 +523,7 @@ def from_dict(d):
d = d['arch']
operating_system = d.get('platform_os', None) or d['os']
target = spack.architecture.Target.from_dict_or_value(d['target'])
target = spack.target.Target.from_dict_or_value(d['target'])
return ArchSpec((d['platform'], operating_system, target))

162
lib/spack/spack/target.py Normal file
View file

@ -0,0 +1,162 @@
# Copyright 2013-2021 Lawrence Livermore National Security, LLC and other
# Spack Project Developers. See the top-level COPYRIGHT file for details.
#
# SPDX-License-Identifier: (Apache-2.0 OR MIT)
import functools
import warnings
import six
import archspec.cpu
import llnl.util.tty as tty
import spack.compiler
import spack.compilers
import spack.spec
import spack.util.spack_yaml as syaml
def _ensure_other_is_target(method):
"""In a single argument method, ensure that the argument is an
instance of ``Target``.
"""
@functools.wraps(method)
def _impl(self, other):
if isinstance(other, six.string_types):
other = Target(other)
if not isinstance(other, Target):
return NotImplemented
return method(self, other)
return _impl
class Target(object):
def __init__(self, name, module_name=None):
"""Target models microarchitectures and their compatibility.
Args:
name (str or Microarchitecture): microarchitecture of the target
module_name (str): optional module name to get access to the
current target. This is typically used on machines
like Cray (e.g. craype-compiler)
"""
if not isinstance(name, archspec.cpu.Microarchitecture):
name = archspec.cpu.TARGETS.get(
name, archspec.cpu.generic_microarchitecture(name)
)
self.microarchitecture = name
self.module_name = module_name
@property
def name(self):
return self.microarchitecture.name
@_ensure_other_is_target
def __eq__(self, other):
return (self.microarchitecture == other.microarchitecture and
self.module_name == other.module_name)
def __ne__(self, other):
# This method is necessary as long as we support Python 2. In Python 3
# __ne__ defaults to the implementation below
return not self == other
@_ensure_other_is_target
def __lt__(self, other):
# TODO: In the future it would be convenient to say
# TODO: `spec.architecture.target < other.architecture.target`
# TODO: and change the semantic of the comparison operators
# This is needed to sort deterministically specs in a list.
# It doesn't implement a total ordering semantic.
return self.microarchitecture.name < other.microarchitecture.name
def __hash__(self):
return hash((self.name, self.module_name))
@staticmethod
def from_dict_or_value(dict_or_value):
# A string here represents a generic target (like x86_64 or ppc64) or
# a custom micro-architecture
if isinstance(dict_or_value, six.string_types):
return Target(dict_or_value)
# TODO: From a dict we actually retrieve much more information than
# TODO: just the name. We can use that information to reconstruct an
# TODO: "old" micro-architecture or check the current definition.
target_info = dict_or_value
return Target(target_info['name'])
def to_dict_or_value(self):
"""Returns a dict or a value representing the current target.
String values are used to keep backward compatibility with generic
targets, like e.g. x86_64 or ppc64. More specific micro-architectures
will return a dictionary which contains information on the name,
features, vendor, generation and parents of the current target.
"""
# Generic targets represent either an architecture
# family (like x86_64) or a custom micro-architecture
if self.microarchitecture.vendor == 'generic':
return str(self)
return syaml.syaml_dict(
self.microarchitecture.to_dict(return_list_of_items=True)
)
def __repr__(self):
cls_name = self.__class__.__name__
fmt = cls_name + '({0}, {1})'
return fmt.format(repr(self.microarchitecture),
repr(self.module_name))
def __str__(self):
return str(self.microarchitecture)
def __contains__(self, cpu_flag):
return cpu_flag in self.microarchitecture
def optimization_flags(self, compiler):
"""Returns the flags needed to optimize for this target using
the compiler passed as argument.
Args:
compiler (spack.spec.CompilerSpec or spack.compiler.Compiler): object that
contains both the name and the version of the compiler we want to use
"""
# Mixed toolchains are not supported yet
if isinstance(compiler, spack.compiler.Compiler):
if spack.compilers.is_mixed_toolchain(compiler):
msg = ('microarchitecture specific optimizations are not '
'supported yet on mixed compiler toolchains [check'
' {0.name}@{0.version} for further details]')
warnings.warn(msg.format(compiler))
return ''
# Try to check if the current compiler comes with a version number or
# has an unexpected suffix. If so, treat it as a compiler with a
# custom spec.
compiler_version = compiler.version
version_number, suffix = archspec.cpu.version_components(
compiler.version
)
if not version_number or suffix not in ('', 'apple'):
# Try to deduce the underlying version of the compiler, regardless
# of its name in compilers.yaml. Depending on where this function
# is called we might get either a CompilerSpec or a fully fledged
# compiler object.
if isinstance(compiler, spack.spec.CompilerSpec):
compiler = spack.compilers.compilers_for_spec(compiler).pop()
try:
compiler_version = compiler.real_version
except spack.util.executable.ProcessError as e:
# log this and just return compiler.version instead
tty.debug(str(e))
return self.microarchitecture.optimization_flags(
compiler.name, str(compiler_version)
)

16
lib/spack/spack/test/architecture.py
View file

@ -9,8 +9,10 @@
import spack.architecture
import spack.concretize
import spack.operating_systems
import spack.platforms
import spack.spec
import spack.target
@pytest.fixture
@ -66,9 +68,9 @@ def test_dict_round_trip(sample_arch):
assert sample_arch == sample_arch_copy
for current_arch in (sample_arch, sample_arch_copy):
assert isinstance(current_arch, spack.architecture.Arch)
assert isinstance(current_arch.platform, spack.architecture.Platform)
assert isinstance(current_arch.os, spack.architecture.OperatingSystem)
assert isinstance(current_arch.target, spack.architecture.Target)
assert isinstance(current_arch.platform, spack.platforms.Platform)
assert isinstance(current_arch.os, spack.operating_systems.OperatingSystem)
assert isinstance(current_arch.target, spack.target.Target)
def test_platform(current_host_platform):
@ -111,7 +113,7 @@ def test_user_input_combination(config, target_str, os_str):
def test_operating_system_conversion_to_dict():
operating_system = spack.architecture.OperatingSystem('os', '1.0')
operating_system = spack.operating_systems.OperatingSystem('os', '1.0')
assert operating_system.to_dict() == {
'name': 'os', 'version': '1.0'
}
@ -129,7 +131,7 @@ def test_operating_system_conversion_to_dict():
('avx512', 'icelake'),
])
def test_target_container_semantic(cpu_flag, target_name):
target = spack.architecture.Target(target_name)
target = spack.target.Target(target_name)
assert cpu_flag in target
@ -159,7 +161,7 @@ def test_arch_spec_container_semantic(item, architecture_str):
def test_optimization_flags(
compiler_spec, target_name, expected_flags, config
):
target = spack.architecture.Target(target_name)
target = spack.target.Target(target_name)
compiler = spack.compilers.compilers_for_spec(compiler_spec).pop()
opt_flags = target.optimization_flags(compiler)
assert opt_flags == expected_flags
@ -182,7 +184,7 @@ def test_optimization_flags(
def test_optimization_flags_with_custom_versions(
compiler, real_version, target_str, expected_flags, monkeypatch, config
):
target = spack.architecture.Target(target_str)
target = spack.target.Target(target_str)
if real_version:
monkeypatch.setattr(
spack.compiler.Compiler, 'get_real_version',

6
lib/spack/spack/test/concretize.py
View file

@ -98,9 +98,6 @@ def current_host(request, monkeypatch):
cpu, _, is_preference = request.param.partition('-')
target = archspec.cpu.TARGETS[cpu]
# this function is memoized, so clear its state for testing
spack.architecture.get_platform.cache.clear()
monkeypatch.setattr(spack.platforms.Test, 'default', cpu)
monkeypatch.setattr(spack.platforms.Test, 'front_end', cpu)
if not is_preference:
@ -110,9 +107,6 @@ def current_host(request, monkeypatch):
with spack.config.override('packages:all', {'target': [cpu]}):
yield target
# clear any test values fetched
spack.architecture.get_platform.cache.clear()
@pytest.fixture()
def repo_with_changing_recipe(tmpdir_factory, mutable_mock_repo):

7
var/spack/repos/builtin/packages/pmix/package.py
View file

@ -2,11 +2,8 @@
# Spack Project Developers. See the top-level COPYRIGHT file for details.
#
# SPDX-License-Identifier: (Apache-2.0 OR MIT)
import os
import spack.architecture
from spack import *
@ -111,8 +108,8 @@ def configure_args(self):
# Versions < 2.1.1 have a bug in the test code that *sometimes*
# causes problems on strict alignment architectures such as
# aarch64. Work-around is to just not build the test code.
if 'aarch64' in spack.architecture.sys_type() and \
self.spec.version < Version('2.1.1'):
if (self.spec.satisfies('target=aarch64:') and
self.spec.version < Version('2.1.1')):
config_args.append('--without-tests-examples')
# Versions >= 3.0 also use hwloc