u-boot.git/tools/patman/test_util.py, branch v2023.01

patman: Don't buffer test output with a single test

2022-08-21T00:07:32Z

When a single test is run we don't need to buffer the test output. This has the unfortunate side effect of suppressing test output, in particular the binman output directory normally printed with the -X option. This is a huge problem since it blocks debugging of tests. We don't actually know how many tests will be run when we set up the suite, so as a work-around, assume that test_name being specified indicates that there is likely only one. Signed-off-by: Simon Glass

patman: Put the coverage command-line last

2022-08-21T00:07:32Z

Put this at the end so it is easier to copy it from the terminal. Signed-off-by: Simon Glass

patman: test_util: Print test stdout/stderr within test summaries

2022-06-28T02:09:51Z

While running tests for a python tool, the tests' outputs get printed in whatever order they happen to run, without any indication as to which output belongs to which test. Unittest supports capturing these outputs and printing them as part of the test summaries, but when a failure or error occurs it switches back to printing as the tests run. Testtools and subunit tests can do the same as their parts inherit from unittest, but they don't outright expose this functionality. On the unittest side, enable output buffering for the custom test result class. Try to avoid ugly outputs by not printing stdout/stderr before the test summary for low verbosity levels and for successful tests. On the subunit side, implement a custom TestProtocolClient that enables the same underlying functionality and injects the captured streams as additional test details. This causes them to be merged into their test's error traceback message, which is later rebuilt into an exception and passed to our unittest report class. Signed-off-by: Alper Nebi Yasak Reviewed-by: Simon Glass

patman: test_util: Customize unittest test results for more info

2022-06-28T02:09:51Z

By default, unittest test summaries only print extended info about tests that failed or couldn't run due to an error. Use a custom text result class to print info about more cases: skipped tests, expected failures and unexpected successes. Signed-off-by: Alper Nebi Yasak

patman: test_util: Use unittest text runner to print test results

2022-06-28T02:09:51Z

The python tools' test utilities handle printing test results, but the output is quite bare compared to an ordinary unittest run. Delegate printing the results to a unittest text runner, which gives us niceties like clear separation between each test's result and how long it took to run the test suite. Unfortunately it does not print info for skipped tests by default, but this can be handled later by a custom test result subclass. It also does not print the tool name; manually print a heading that includes the toolname so that the outputs of each tool's tests are distinguishable in the CI output. Signed-off-by: Alper Nebi Yasak Reviewed-by: Simon Glass

patman: test_util: Handle nonexistent tests while loading tests

2022-06-28T02:09:51Z

It's possible to request a specific test to run when trying to run a python tool's tests. If we request a nonexistent test, the unittest loaders generate a fake test that reports this as an error. However, we get these fake tests even when the test exists, because test_util can load tests from multiple places one by one and the test we want only exists in one. The test_util helpers currently remove these fake tests when printing test results, but that's more of a workaround than a proper solution. Instead, don't even try to load the missing tests. Signed-off-by: Alper Nebi Yasak Reviewed-by: Simon Glass

patman: test_util: Fix printing results for failed tests

2022-06-28T02:09:51Z

When printing a python tool's test results, the entire list of failed tests and their tracebacks are reprinted for every failed test. This makes the test output quite unreadable. Fix the loop to print failures and tracebacks one at a time. Signed-off-by: Alper Nebi Yasak Reviewed-by: Simon Glass

patman: Convert camel case in test_util.py

2022-02-09T19:30:12Z

Convert this file to snake case and update all files which use it. Signed-off-by: Simon Glass

patman: Convert camel case in command.py

2022-02-09T19:26:12Z

Convert this file to snake case and update all files which use it. Signed-off-by: Simon Glass

patman: Update test_util to run doc tests

2022-02-09T04:07:59Z

At present this function does not run the doctests. Allow the caller to pass these modules in as strings. Update patman to use this. Signed-off-by: Simon Glass