Merge branch '2022-09-12-update-pytests-for-more-parellel-support' into next

To quote the author:
This series makes a further attempt to get closer to having all tests
run in parallel. It introduces a new 'make pcheck' option which runs
tests in parallel, skipping those that are not compatible.

A number of fixes are included for existing tests. The vboot test is
updated to only run a single scenario in 'quick' mode.

This makes use of pytest's parallel-testing features. The resulting
times (including incremental building with LTO) on a 16-core machine are
as follows:

   make pcheck        - 1 minute 6 seconds
   make qcheck        - 3 minutes
   make check         - 5 minutes 15 seconds

Note that this is not a fair comparison, since 'make pcheck' omits a
number of tests, even more than 'make qcheck'.

2 files changed, 26 insertions, 17 deletions

diff --git a/doc/develop/py_testing.rst b/doc/develop/py_testing.rst
index 06f919609b5..92fbd227219 100644
--- a/doc/develop/py_testing.rst
+++ b/doc/develop/py_testing.rst
@@ -121,31 +121,36 @@ more options.
 Running tests in parallel
 ~~~~~~~~~~~~~~~~~~~~~~~~~
 
-Note: This does not fully work yet and is documented only so you can try to
-fix the problems.
+Note: Not all tests can run in parallel at present, so the usual approach is
+to just run those that can.
 
 First install support for parallel tests::
 
+    sudo apt install python3-pytest-xdist
+
+or:::
+
     pip3 install pytest-xdist
 
-Then build sandbox in a suitable build directory. It is not possible to use
-the --build flag with xdist.
+Then run the tests in parallel using the -n flag::
 
-Finally, run the tests in parallel using the -n flag::
+    test/py/test.py -B sandbox --build --build-dir /tmp/b/sandbox -q -k \
+        'not slow and not bootstd and not spi_flash' -n16
 
-    # build sandbox first, in a suitable build directory. It is not possible
-    # to use the --build flag with -n
-    test/py/test.py -B sandbox --build-dir /tmp/b/sandbox -q -k 'not slow' -n32
+You can also use `make pcheck` to run all tests in parallel. This uses a maximum
+of 16 threads, since the setup time is significant and there are under 1000
+tests.
 
-At least the following non-slow tests are known to fail:
+Note that the `test-log.html` output does not work correctly at present with
+parallel testing. All the threads write to it at once, so it is garbled.
 
-- test_fit_ecdsa
-- test_bind_unbind_with_uclass
-- ut_dm_spi_flash
-- test_gpt_rename_partition
-- test_gpt_swap_partitions
-- test_pinmux_status
-- test_sqfs_load
+Note that the `tools/` tests still run each tool's tests once after the other,
+although within that, they do run in parallel. So for example, the buildman
+tests run in parallel, then the binman tests run in parallel. There would be a
+significant advantage to running them all in parallel together, but that would
+require a large amount of refactoring, e.g. with more use of pytest fixtures.
+The code-coverage tests are omitted since they cannot run in parallel due to a
+Python limitation.
 
 
 Testing under a debugger
diff --git a/doc/develop/testing.rst b/doc/develop/testing.rst
index 1abe4d7f0f0..5afeb42f696 100644
--- a/doc/develop/testing.rst
+++ b/doc/develop/testing.rst
@@ -28,8 +28,12 @@ run. Type this::
 
     make tcheck
 
+You can also run a selection tests in parallel with::
+
+    make pcheck
+
 All of the above use the test/run script with a paremeter to select which tests
-are run.
+are run. See :doc:`py_testing` for more information.
 
 
 Sandbox