Merge patch series "test: Improvements to ut command and test-suite running"

Simon Glass <[email protected]> says:

The current method of running unit tests relies on subcommands of the
ut command. Only the code in each subcommand knows how to find the tests
related to that subcomand.

This is not ideal and we now have quite a few subcommands which do
nothing but locate the relevant tests in a linker list, then call a
common function to run them.

This series adds a list of test suites, so that these subcommands can be
removed.

An issue with 'ut all' is that it doesn't record how many tests failed
overall, so it is necessary to examine copious amounts of output to look
for failures. This series adds a new 'total' feature allow recording the
total number of failed tests.

To help with 'ut all' a new pytest is created which runs it (as well as
'ut info') and makes sure that all is well. Due to the 'ut all' failures
this does not pass, so the test is disabled for now. It is here because
it provides security against misnaming a test suite and causing it not
to run.

Future work may:
- get 'ut all' passing
- enable test_suite() in CL, to ensure that 'ut all' keeps passing
- record duration of each suite
- allow running the tests in random order to tease out dependencies
- tweak the output to remove common prefixes
- getting rid of bootstd, optee and seame 'ut' subcommands

Link: https://lore.kernel.org/r/[email protected]

3 files changed, 1 insertions, 33 deletions

diff --git a/test/common/Makefile b/test/common/Makefile
index 95bd00741a3..1ad6c24b7e2 100644
--- a/test/common/Makefile
+++ b/test/common/Makefile
@@ -1,5 +1,4 @@
 # SPDX-License-Identifier: GPL-2.0+
-obj-y += cmd_ut_common.o
 obj-$(CONFIG_AUTOBOOT) += test_autoboot.o
 
 ifneq ($(CONFIG_$(XPL_)BLOBLIST),)
diff --git a/test/common/bloblist.c b/test/common/bloblist.c
index 4bca62110a5..9467abfa8e1 100644
--- a/test/common/bloblist.c
+++ b/test/common/bloblist.c
@@ -12,7 +12,7 @@
 
 /* Declare a new bloblist test */
 #define BLOBLIST_TEST(_name, _flags) \
-		UNIT_TEST(_name, _flags, bloblist_test)
+		UNIT_TEST(_name, _flags, bloblist)
 
 enum {
 	TEST_TAG		= BLOBLISTT_U_BOOT_SPL_HANDOFF,
@@ -602,13 +602,3 @@ static int bloblist_test_blob_maxsize(struct unit_test_state *uts)
 	return 0;
 }
 BLOBLIST_TEST(bloblist_test_blob_maxsize, UFT_BLOBLIST);
-
-int do_ut_bloblist(struct cmd_tbl *cmdtp, int flag, int argc,
-		   char *const argv[])
-{
-	struct unit_test *tests = UNIT_TEST_SUITE_START(bloblist_test);
-	const int n_ents = UNIT_TEST_SUITE_COUNT(bloblist_test);
-
-	return cmd_ut_category("bloblist", "bloblist_test_",
-			       tests, n_ents, argc, argv);
-}
diff --git a/test/common/cmd_ut_common.c b/test/common/cmd_ut_common.c
deleted file mode 100644
index 2f03a58af47..00000000000
--- a/test/common/cmd_ut_common.c
+++ /dev/null
@@ -1,21 +0,0 @@
-// SPDX-License-Identifier: GPL-2.0
-/*
- * Copyright (c) 2019 Heinrich Schuchardt <[email protected]>
- * Copyright (c) 2021 Steffen Jaeckel <[email protected]>
- *
- * Unit tests for common functions
- */
-
-#include <command.h>
-#include <test/common.h>
-#include <test/suites.h>
-#include <test/ut.h>
-
-int do_ut_common(struct cmd_tbl *cmdtp, int flag, int argc, char *const argv[])
-{
-	struct unit_test *tests = UNIT_TEST_SUITE_START(common_test);
-	const int n_ents = UNIT_TEST_SUITE_COUNT(common_test);
-
-	return cmd_ut_category("common", "common_test_", tests, n_ents, argc,
-			       argv);
-}