From cbd71fad6d468018727ab04b2bb912989aec0785 Mon Sep 17 00:00:00 2001 From: Simon Glass Date: Thu, 20 Oct 2022 18:22:50 -0600 Subject: test: Support tests which can only be run manually At present we normally write tests either in Python or in C. But most Python tests end up doing a lot of checks which would be better done in C. Checks done in C are orders of magnitude faster and it is possible to get full access to U-Boot's internal workings, rather than just relying on the command line. The model is to have a Python test set up some things and then use C code (in a unit test) to check that they were done correctly. But we don't want those checks to happen as part of normal test running, since each C unit tests is dependent on the associate Python tests, so cannot run without it. To acheive this, add a new UT_TESTF_MANUAL flag to use with the C 'check' tests, so that they can be skipped by default when the 'ut' command is used. Require that tests have a name ending with '_norun', so that pytest knows to skip them. Signed-off-by: Simon Glass --- doc/develop/tests_writing.rst | 27 +++++++++++++++++++++++++++ 1 file changed, 27 insertions(+) (limited to 'doc/develop') diff --git a/doc/develop/tests_writing.rst b/doc/develop/tests_writing.rst index 1ddf7a353a7..bb1145da268 100644 --- a/doc/develop/tests_writing.rst +++ b/doc/develop/tests_writing.rst @@ -74,6 +74,33 @@ NOT rely on running with sandbox, but instead should function correctly on any board supported by U-Boot. +Mixing Python and C +------------------- + +The best of both worlds is sometimes to have a Python test set things up and +perform some operations, with a 'checker' C unit test doing the checks +afterwards. This can be achieved with these steps: + +- Add the `UT_TESTF_MANUAL` flag to the checker test so that the `ut` command + does not run it by default +- Add a `_norun` suffix to the name so that pytest knows to skip it too + +In your Python test use the `-f` flag to the `ut` command to force the checker +test to run it, e.g.:: + + # Do the Python part + host load ... + bootm ... + + # Run the checker to make sure that everything worked + ut -f bootstd vbe_test_fixup_norun + +Note that apart from the `UT_TESTF_MANUAL` flag, the code in a 'manual' C test +is just like any other C test. It still uses ut_assert...() and other such +constructs, in this case to check that the expected things happened in the +Python test. + + How slow are Python tests? -------------------------- -- cgit v1.3.1