Execution time varies widely with the existing tests. Provides a way to
produce a summary of the time taken for each test, along with a
histogram.

This is enabled with the --timing flag.

Enable it for sandbox in CI.

Example:

   Duration : Number of tests
   ======== : ========================================
       <1ms :  1
       <8ms :  1
      <20ms : # 20
      <30ms : ######## 127
      <50ms : ######################################## 582
      <75ms : ####### 102
     <100ms : ## 39
     <200ms : ##### 86
     <300ms : # 29
     <500ms : ## 42
     <750ms : # 16
      <1.0s : # 15
      <2.0s : # 23
      <3.0s :  13
      <5.0s :  9
      <7.5s :  1
     <10.0s :  6
     <20.0s :  12

Signed-off-by: Simon Glass <sjg@chromium.org>
Reviewed-by: Tom Rini <trini@konsulko.com>

This script can fail if there is no toolchain available for the board.
At present this is not handled very nicely, in that only the error
output is reported. It is much more useful to see everything, so
combine stdout and stderr and report them both.

Signed-off-by: Simon Glass <sjg@chromium.org>

The u-boot-test-getrole script runs before the normal environment
variables have been set up. This is unavoidable since the script is
providing necessary information to test.py

This means that U_BOOT_SOURCE_DIR is not set in the environment.

As a result, Labgrid uses its default source path, configured in its
environment variable. While this may happen to work, it is not correct.
Also, it causes problems when running from Gitlab, where the runner may
not have access to that source path.

Provide the required source path in U_BOOT_SOURCE_DIR so that Labgrid
does the right thing.

Signed-off-by: Simon Glass <sjg@chromium.org>
Fixes: bf89a8f1fc2 ("test: Introduce the concept of a role")

[1] https://patchwork.ozlabs.org/project/uboot/patch/20241211131858.520639-1-sjg@chromium.org/

Most test suites have a _test suffix. This is not necessary as there is
also a ut_ prefix.

Drop the suffix so that (with future work) the suite name can be used as
the linker-list name.

Remove the suffix from the pytest regex as well, moving it to the top of
the file, as it is a constant.

Signed-off-by: Simon Glass <sjg@chromium.org>

The logic in get_details() retrieves the default source directory from
the Labgrid settings. This is convenient for interactive use, since it
allows pytests to be run from any directory and still find the source
tree.

However, it is not actually correct.

Gitlab sets the current directory to the source tree and expects that to
be used. At present it is ignored. The result is that Gitlab builds
whatever happens to be in the default source directory, ignoring the
tree it is supposed to be building.

Fix this by using the directory of the source tree, always. This is
obtained by looking at the grandparent of the conftest.py file.

Signed-off-by: Simon Glass <sjg@chromium.org>
Reported-by: Tom Rini <trini@konsulko.com>
Fixes: bf89a8f1fc2 ("test: Introduce the concept of a role")
Tested-by: Tom Rini <trini@konsulko.com>

The Beagleplay board uses an SoC from the TI K3 family. This has both a
Cortex-R core and a Cortex-A core and the R core needs to come up before
the A core. In both cases we have U-Boot SPL then U-Boot proper being
used.

In practice this means we need two entirely separate builds to produce
an image.

Handle this in test.py by adding more parameters.

Signed-off-by: Simon Glass <sjg@chromium.org>

In Labgrid there is the concept of a 'role', which is similar to the
U-Boot board ID in U-Boot's pytest subsystem.

The role indicates both the target and information about the U-Boot
build to use. It can also provide any amount of other configuration.
The information is obtained using the 'labgrid-client query' operation.

Using this role, all required configuration for the board is stored
within the Labgrid environment, with pytest simply querying it. This
allows connecting to boards using an interactive console, something that
isn't possible without some kind of mapping. It also means that we don't
need to replicate the pytest functionality in tbot, since Labgrid can
handle the console and kick off builds as needed.

Make use of this in tests, so that only the role is required in gitlab
and other situations. The board type and other things can be queried
as needed.

Use a new 'u-boot-test-getrole' script to obtain the requested
information.

With this it is possible to run lab tests in gitlab with just a single
'ROLE' variable for each board.

Note that, without this feature:
- interactive use of boards with Labgrid-sjg would require repeating the
  id/board in a separate configuration file
- Gitlab yaml file would need to specify both the id and board

This feature is entirely optional, however, with the code gracefully
falling back to using a separate ID and board.

Link: https://tbot.tools

Signed-off-by: Simon Glass <sjg@chromium.org>

Sometimes we know that the board is already running the right software,
so provide an option to allow running of tests directly, without first
resetting the board.

This saves time when re-running a test where only the Python code is
changing.

Note that this feature is open to errors, since the user must know that
the board is in a fit state to execute tests. It is useful for repeated
iteration on a particular test, where it can save quite a bit of time.

Signed-off-by: Simon Glass <sjg@chromium.org>

When the connection to a board dies, assume it is dead forever until
some user action is taken. Skip all remaining tests. This avoids CI
runs taking an hour, with hundreds of 30-second timeouts all to no
avail.

Signed-off-by: Simon Glass <sjg@chromium.org>

The settings are decoded in two places. Combine them into a new
function, before (in a future patch) expanding the number of items.

Signed-off-by: Simon Glass <sjg@chromium.org>