Commit graph

18787 commits

Author SHA1 Message Date
Massimiliano Culpo
1cdee03c4b concretizer: add conflict rules from packages
Conflict rules from packages are added as integrity
constraints in the ASP formulation. Most of the code
to generate them has been reused from PyclingoDriver.rules
2020-11-17 10:04:13 -08:00
Massimiliano Culpo
2595b58503 test_noversion_pkg: generalized the error to be caught
The new concretizer and the old concretizer solve constraints
in a different way. Here we ensure that a SpackError is raised,
instead of a specific error that made sense in the old concretizer
but probably not in the new.
2020-11-17 10:04:13 -08:00
Massimiliano Culpo
aaa75b831f compiler constraints: deduplicate the list of compilers before encoding one_of_iff rules
This fixes 8 bugs in test/concretize.py
2020-11-17 10:04:13 -08:00
Todd Gamblin
e56f90c3ef concretizer: add compiler version constraints
Add rules to account for compiler version
constraints in concretize.lp.
2020-11-17 10:04:13 -08:00
Todd Gamblin
115384afbd concretizer: use cardinality constraints for versions
Instead of python callbacks, use cardinality constraints for package
versions. This is slightly faster and has the advantage that it can be
written to an ASP program to be executed *outside* of Spack. We can use
this in the future to unify the pyclingo driver and the clingo text
driver.

This makes use of add_weight_rule() to implement cardinality constraints.
add_weight_rule() only has a lower bound parameter, but you can implement
a strict "exactly one of" constraint using it. In particular, wee want to
define:

    1 {v1; v2; v3; ...} 1 :- version_satisfies(pkg, constraint).
    version_satisfies(pkg, constraint) :- 1 {v1; v2; v3; ...} 1.

And we do that like this, for every version constraint:

    atleast1(pkg, constr) :- 1 {version(pkg, v1); version(pkg, v2); ...}.
    morethan1(pkg, constr) :- 2 {version(pkg, v1); version(pkg, v2); ...}.
    version_satisfies(pkg, constr) :- atleast1, not morethan1(pkg, constr).
    :- version_satisfies(pkg, constr), morethan1.
    :- version_satisfies(pkg, constr), not atleast1.

v1, v2, v3, etc. are computed on the Python side by comparing every
possible package version with the constraint.

Computing things like this has the added advantage that if v1, v2, v3,
etc. comprise *all* possible versions of a package, we can just omit the
rules for the constraint under consideration. This happens pretty
frequently in the Spack mainline.
2020-11-17 10:04:13 -08:00
Todd Gamblin
0ed019d4ef concretizer: first working version with pyclingo interface
- [x] Solver now uses the Python interface to clingo
- [x] can extract unsatisfiable cores from problems when things go wrong
- [x] use Python callbacks for versions instead of choice rules (this may
      ultimately hurt performance)
2020-11-17 10:04:13 -08:00
Todd Gamblin
14ab63f97c concretizer: add a configuration option to use new or old concretizer
- [x] spec.py can call out to the new concretizer
- [x] config.yaml now has an option to choose a concretizer (original, clingo)
2020-11-17 10:04:13 -08:00
Todd Gamblin
2dd06f14f9 concretizer: use repository names, not specs with is_virtual 2020-11-17 10:04:13 -08:00
Todd Gamblin
ac9405a80e concretizer: refactor to support multiple solver backends
There are now three parts:

- `SpackSolverSetup`
  - Spack-specific logic for generating constraints. Calls methods on
    `AspTextGenerator` to set up the solver with a Spack problem. This
    shouln't change much from solver backend to solver backend.

- ClingoDriver
  - The solver driver provides methods for SolverSetup to generates an ASP
    program, send it to `clingo` (run as an external tool), and parse the
    output into function tuples suitable for `SpecBuilder`.
  - The interface is generic and should not have to change much for a
    driver for, say, the Clingo Python interface.

- SpecBuilder
  - Builds Spack specs from function tuples parsed by the solver driver.
2020-11-17 10:04:13 -08:00
Todd Gamblin
5bb83be827 concretizer: set spec constraints correctly for body and head 2020-11-17 10:04:13 -08:00
Todd Gamblin
cd55fd4bd3 concretizer: allow non-default OS, inherit OS along dependencies 2020-11-17 10:04:13 -08:00
Todd Gamblin
51cb49743e tests: add framework to mock targets 2020-11-17 10:04:13 -08:00
Todd Gamblin
43e7255e19 concretizer: split platforms, OS, and targets apart in Python and ASP 2020-11-17 10:04:13 -08:00
Todd Gamblin
cb919c2e39 concretizer: targets are inherited like compilers 2020-11-17 10:04:13 -08:00
Todd Gamblin
afa74ea155 concretizer: change single-letter variables to descriptive names
The original implementation was difficult to read, as it only had
single-letter variable names.  This converts all of them to descriptive
names, e.g., P -> Package, V -> Virtual/Version/Variant, etc.
2020-11-17 10:04:13 -08:00
Todd Gamblin
35ae4c0ddd concretizer: handle compiler existence check settings
To handle unknown compilers propely in tests (and elsewhere), we need to
add unknown compilers from the spec to the list of possible compilers.

Rework how the compiler list is generated and includes compilers from
specs if the existence check is disabled.
2020-11-17 10:04:13 -08:00
Todd Gamblin
3b648c294e concretizer: add initial package existence check 2020-11-17 10:04:13 -08:00
Todd Gamblin
520b71e89b concretizer: handle virtual spec constraints better
Specs like hdf5 ^mpi were unsatisfiable because we added a requierment
for `node("mpi").`.  This can't be resolved because "mpi" is not a
package.

- [x] Introduce `virtual_node()`, which says *some* provider must be in
      the DAG.
2020-11-17 10:04:13 -08:00
Todd Gamblin
3ef7c06a48 concretizer: solve with compiler flags but preserve order
This adds compiler flags to the ASP solve so that we can have conditions
based on them in the solve.  But, it keeps order out of the solve to
avoid unneeded complexity and combinatorial explosions.

The solver determines which flags are on a spec, but the order is
determined by DAG precedence (childrens' flags take precedence over
parents' and are added on the right) and order (order flags were
specified on the command line is respected).

The solver is responsible for determining when to propagate flags, when
to inheit them from other nodes, when to take them from compiler
preferences, etc.
2020-11-17 10:04:13 -08:00
Todd Gamblin
7a1b5ca65e concretizer: add timers around phases 2020-11-17 10:04:13 -08:00
Todd Gamblin
5185ed1d28 concretizer: optimize microarchitectures, constrained by compiler support
Weight microarchitectures and prefers more rercent ones.  Also disallow
nodes where the compiler does not support the selected target.

We should revisit this at some point as it seems like if I play around
with the compiler support for different architectures, the solver runs
very slowly.  See notes in comments -- the bad case was gcc supporting
broadwell and skylake with clang maxing out at haswell.
2020-11-17 10:04:13 -08:00
Todd Gamblin
71726a9b33 concretizer bugfix: require at least one value for multi-value variants
We didn't have a cardinality constraint for multi-valued variants, so the
solver wasn't filling them in.

- [x] add a requirement for at least one value for multi-valued variants
2020-11-17 10:04:13 -08:00
Todd Gamblin
309ae856ab commands: add --json argument to spack solve 2020-11-17 10:04:13 -08:00
Todd Gamblin
cb8ca505ef concretizer: make some rules into facts 2020-11-17 10:04:13 -08:00
Todd Gamblin
810542d4fe concretizer bugfix: all variants need possible values
Variants like `cpu_target` on `openblas` don't have defineed values, but
they have a default.  Ensure that the default is always a possible value
for the solver.
2020-11-17 10:04:13 -08:00
Todd Gamblin
9b1f05df00 concretizer bugfix: fix generations of conditionals for dependencies
Spack was generating the same dependency connstraints twice in the output ASP:

```
declared_dependency("abinit", "hdf5", "link")
    :- node("abinit"),
       variant_value("abinit", "mpi", "True"),
       variant_value("abinit", "mpi", "True").
```

This was because `AspFunction` was modifying itself when called.

- [x] fix `AspFunction` so that every call returns a new object
2020-11-17 10:04:13 -08:00
Todd Gamblin
e31be3da56 concretizer bugfix: *at most* one provider for any virtual 2020-11-17 10:04:13 -08:00
Todd Gamblin
04295f6531 concretizer: optimized for preferred virtuals before recent versions 2020-11-17 10:04:13 -08:00
Todd Gamblin
f365373a3d concretizer: handle compiler preferences with optimization
- [x] Add support for packages.yaml and command-line compiler preferences.
- [x] Rework compiler version propagation to use optimization rather than
  hard logic constraints
2020-11-17 10:04:13 -08:00
Todd Gamblin
1859ff31c9 concretizer: deterministic order for asp output for better diffs
Technically the ASP output order does not matter, but it's hard to diff
two different solve fomulations unless we order it.

- [x] make sure ASP output is emitted in a deterministic order (by
      sorting all hash keys)
2020-11-17 10:04:13 -08:00
Todd Gamblin
36dae9ee05 concretizer: rename --dump to --show 2020-11-17 10:04:13 -08:00
Todd Gamblin
da215b50a3 concretizer: handle package namespaces 2020-11-17 10:04:13 -08:00
Todd Gamblin
4d34363c1d concretizer: handle constraints on dependencies, adjust optimization
This needs more thought, as I am pretty sure the weights are not correct.
Or, at least, I'm not convinced that they do what we want in all cases.
See note in concretize.lp.
2020-11-17 10:04:13 -08:00
Todd Gamblin
db62b00d58 concretizer: handle dependency types 2020-11-17 10:04:13 -08:00
Todd Gamblin
cde10692b0 concretizer: prioritize versions by package pref, newest, preferred, actual
Solver now prefers newer versions like the old concretizer.  Prefer
package preferences from packages.yaml, preferred=True, package
definition, and finally each version itself.
2020-11-17 10:04:13 -08:00
Todd Gamblin
18fba433f6 concretizer: Use "competition" output format to avoid extra parsing
Competition output only prints out one model, so we do not have to
unnecessarily parse all the non-optimal models.  We'll just look at the
best model and bring that in.

In practice, this saves a lot of JSON parsing and spec construction time.
2020-11-17 10:04:13 -08:00
Todd Gamblin
b4e6d9d28e concretizer: handle virtual provider preferences from packages.yaml 2020-11-17 10:04:13 -08:00
Todd Gamblin
36ec66d997 concretizer: use clingo json output instead of text
Clingo actually has an option to output JSON -- use that instead of
parsing the raw otuput ourselves.

This also allows us to pick the best answer -- modify the parser to
*only* construct a spec for that one rather than building all of them
like we did before.
2020-11-17 10:04:13 -08:00
Todd Gamblin
a332981f2f concretizer: require only one provider for any virtual in the DAG 2020-11-17 10:04:13 -08:00
Todd Gamblin
501cb371c9 concretizer: handle variant defaults with optimization
- Instead of using default logic, handle variant defaults by minimizing
  the number of non-default variants in the solution.

- This actually seems to be pretty fast, and it fixes the long-standing
  issue that writing this:

      spack install hdf5 ^mpich

  will fail if you don't specify hdf5+mpi.  With optimization and
  allowing enums to be enumerated, the solver seems to be able to quickly
  discover that +mpi is the only way hdf5 can depend on mpich, and it
  forces the switch to be thrown.
2020-11-17 10:04:13 -08:00
Todd Gamblin
1cab1b1994 concretizer: support conditional dependencies 2020-11-17 10:04:13 -08:00
Todd Gamblin
51af590e64 variants: allow MultiValuedVariants to be constructed incrementally 2020-11-17 10:04:13 -08:00
Todd Gamblin
be10568a6a concretizer: initial support for virtual dependencies
Add initial support for virtual dependencies.  Solver now knows about all
virtuals and can choose one to resolve a dependency.
2020-11-17 10:04:13 -08:00
Todd Gamblin
3f93553a08 concretizer: print out virtuals 2020-11-17 10:04:13 -08:00
Todd Gamblin
8a6207aa70 concretizer: handle versions with choice construct rather than conflicts
Use '1 { version(x); version(y); version(z) } 1.' instead of declaring
conflicts for non-matching versions.  This keeps the sense of version
clauses positive, which will allow them to be used more easily in
conditionals later.

Also refactor `spec_clauses()` method to return clauses that can be used
in conditions, etc. instead of just printing out facts.
2020-11-17 10:04:13 -08:00
Todd Gamblin
6e31430bec concretizer: add another definition pragma.
- single_value_variant may not be defined by the generated program.  Mark
  it to avoid warnings.
2020-11-17 10:04:13 -08:00
Todd Gamblin
a81258663c concretizer: cleanup 2020-11-17 10:04:13 -08:00
Todd Gamblin
6bbc64555b concretizer: use conditional literals for versions. 2020-11-17 10:04:13 -08:00
Todd Gamblin
4288639770 concretizer: mark depends_on/2 defined for solves without dependencies. 2020-11-17 10:04:13 -08:00
Todd Gamblin
60cf3fdb34 concretizer: add basic semantics for compilers
- This handles setting the compiler and falling back to a default
  compiler, as well as providing default values for compilers/compiler
  versions.

- Versions still aren't quite right -- you can't properly override
  versions on compiler specs.
2020-11-17 10:04:13 -08:00