spack/lib
Todd Gamblin 4beb9fc5d3 tests: improved spack test command line options
Previously, `spack test` automatically passed all of its arguments to
`pytest -k` if no options were provided, and to `pytest` if they were.
`spack test -l` also provided a list of test filenames, but they didn't
really let you completely narrow down which tests you wanted to run.

Instead of trying to do our own weird thing, this passes `spack test`
args directly to `pytest`, and omits the implicit `-k`.  This means we
can now run, e.g.:

```console
$ spack test spec_syntax.py::TestSpecSyntax::test_ambiguous
```

This wasn't possible before, because we'd pass the fully qualified name
to `pytest -k` and get an error.

Because `pytest` doesn't have the greatest ability to list tests, I've
tweaked the `-l`/`--list`, `-L`/`--list-long`, and `-N`/`--list-names`
options to `spack test` so that they help you understand the names
better.  you can combine these options with `-k` or other arguments to do
pretty powerful searches.

This one makes it easy to get a list of names so you can run tests in
different orders (something I find useful for debugging `pytest` issues):

```console
$ spack test --list-names -k "spec and concretize"
cmd/env.py::test_concretize_user_specs_together
concretize.py::TestConcretize::test_conflicts_in_spec
concretize.py::TestConcretize::test_find_spec_children
concretize.py::TestConcretize::test_find_spec_none
concretize.py::TestConcretize::test_find_spec_parents
concretize.py::TestConcretize::test_find_spec_self
concretize.py::TestConcretize::test_find_spec_sibling
concretize.py::TestConcretize::test_no_matching_compiler_specs
concretize.py::TestConcretize::test_simultaneous_concretization_of_specs
spec_dag.py::TestSpecDag::test_concretize_deptypes
spec_dag.py::TestSpecDag::test_copy_concretized
```

You can combine any list option with keywords:

```console
$ spack test --list -k microarchitecture
llnl/util/cpu.py  modules/lmod.py
```

```console
$ spack test --list-long -k microarchitecture
llnl/util/cpu.py::
    test_generic_microarchitecture

modules/lmod.py::TestLmod::
    test_only_generic_microarchitectures_in_root
```

Or just list specific files:

```console
$ spack test --list-long cmd/test.py
cmd/test.py::
    test_list                       test_list_names_with_pytest_arg
    test_list_long                  test_list_with_keywords
    test_list_long_with_pytest_arg  test_list_with_pytest_arg
    test_list_names
```

Hopefully this stuff will help with debugging test issues.

- [x] make `spack test` send args directly to `pytest` instead of trying
  to do fancy things.
- [x] rework `--list`, `--list-long`, and add `--list-names` to make
  searching for tests easier.
- [x] make it possible to mix Spack's list args with `pytest` args
  (they're just fancy parsing around `pytest --collect-only`)
- [x] add docs
- [x] add tests
- [x] update spack completion
2020-01-01 21:37:02 -08:00
..
spack tests: improved spack test command line options 2020-01-01 21:37:02 -08:00