From ccf69386b79f6987cfebaeb823b5234a93cb118c Mon Sep 17 00:00:00 2001 From: Simon Glass Date: Sun, 7 Mar 2021 17:34:38 -0700 Subject: doc: Tidy up testing section Tweak this so the output looks a little better. Signed-off-by: Simon Glass Reviewed-by: Heinrich Schuchardt --- doc/develop/testing.rst | 20 ++++++++++++-------- 1 file changed, 12 insertions(+), 8 deletions(-) (limited to 'doc/develop') diff --git a/doc/develop/testing.rst b/doc/develop/testing.rst index 4bc9ca3a6ae..bc74eb53e35 100644 --- a/doc/develop/testing.rst +++ b/doc/develop/testing.rst @@ -8,14 +8,14 @@ tested and what tests you should write when adding a new feature. Running tests ------------- -To run most tests on sandbox, type this: +To run most tests on sandbox, type this:: make check in the U-Boot directory. Note that only the pytest suite is run using this command. -Some tests take ages to run. To run just the quick ones, type this: +Some tests take ages to run. To run just the quick ones, type this:: make qcheck @@ -35,9 +35,9 @@ either on sandbox or on real hardware. It relies on the U-Boot console to inject test commands and check the result. It is slower to run than C code, but provides the ability to unify lots of tests and summarise their results. -You can run the tests on sandbox with: +You can run the tests on sandbox with:: - ./test/py/test.py --bd sandbox --build + ./test/py/test.py --bd sandbox --build This will produce HTML output in build-sandbox/test-log.html @@ -58,10 +58,14 @@ Ad-hoc tests There are several ad-hoc tests which run outside the pytest environment: - test/fs - File system test (shell script) - test/image - FIT and legacy image tests (shell script and Python) - test/stdint - A test that stdint.h can be used in U-Boot (shell script) - trace - Test for the tracing feature (shell script) +test/fs + File system test (shell script) +test/image + FIT and legacy image tests (shell script and Python) +test/stdint + A test that stdint.h can be used in U-Boot (shell script) +trace + Test for the tracing feature (shell script) TODO: Move these into pytest. -- cgit v1.3.1 From 8d16ebdf81cda3e07eb76cd7f2efc52cbaf0475b Mon Sep 17 00:00:00 2001 From: Simon Glass Date: Sun, 7 Mar 2021 17:34:39 -0700 Subject: doc: Document make tcheck Add a comment about this option in the documentation. Also mention the script that runs these combinations. Signed-off-by: Simon Glass Reviewed-by: Heinrich Schuchardt --- doc/develop/testing.rst | 12 +++++++++++- 1 file changed, 11 insertions(+), 1 deletion(-) (limited to 'doc/develop') diff --git a/doc/develop/testing.rst b/doc/develop/testing.rst index bc74eb53e35..f01ca4dc408 100644 --- a/doc/develop/testing.rst +++ b/doc/develop/testing.rst @@ -15,10 +15,20 @@ To run most tests on sandbox, type this:: in the U-Boot directory. Note that only the pytest suite is run using this command. -Some tests take ages to run. To run just the quick ones, type this:: +Some tests take ages to run and are marked with @pytest.mark.slow. To run just +the quick ones, type this:: make qcheck +It is also possible to run just the tests for tools (patman, binman, etc.). +Such tests are included with those tools, i.e. no actual U-Boot unit tests are +run. Type this:: + + make tcheck + +All of the above use the test/run script with a paremeter to select which tests +are run. + Sandbox ------- -- cgit v1.3.1 From 4c8850aafc10d368cef575b31d1b931d3d2ca597 Mon Sep 17 00:00:00 2001 From: Simon Glass Date: Sun, 7 Mar 2021 17:34:42 -0700 Subject: doc: Explain how to run tests without pytest Add details about how to run a sandbox test directly, without using pytest. This is more convenient for rapid development, since it is faster and allows easier use of a debugger. Also mention sandbox_flattree as an example of the different sandbox builds available. Signed-off-by: Simon Glass --- doc/develop/index.rst | 1 + doc/develop/testing.rst | 9 +++++ doc/develop/tests_sandbox.rst | 80 +++++++++++++++++++++++++++++++++++++++++++ 3 files changed, 90 insertions(+) create mode 100644 doc/develop/tests_sandbox.rst (limited to 'doc/develop') diff --git a/doc/develop/index.rst b/doc/develop/index.rst index ac57fdb8f30..50b1de3bdff 100644 --- a/doc/develop/index.rst +++ b/doc/develop/index.rst @@ -33,3 +33,4 @@ Testing coccinelle testing py_testing + tests_sandbox diff --git a/doc/develop/testing.rst b/doc/develop/testing.rst index f01ca4dc408..87c90eee271 100644 --- a/doc/develop/testing.rst +++ b/doc/develop/testing.rst @@ -36,6 +36,7 @@ U-Boot can be built as a user-space application (e.g. for Linux). This allows test to be executed without needing target hardware. The 'sandbox' target provides this feature and it is widely used in tests. +See :doc:`tests_sandbox` for more information. Pytest Suite ------------ @@ -51,8 +52,16 @@ You can run the tests on sandbox with:: This will produce HTML output in build-sandbox/test-log.html +Some tests run with other versions of sandbox. For example sandbox_flattree +runs the tests with livetree (the hierachical devicetree) disabled. You can +also select particular tests with -k:: + + ./test/py/test.py --bd sandbox_flattree --build -k hello + See test/py/README.md for more information about the pytest suite. +See :doc:`tests_sandbox` for how to run tests directly (not through pytest). + tbot ---- diff --git a/doc/develop/tests_sandbox.rst b/doc/develop/tests_sandbox.rst new file mode 100644 index 00000000000..42b64882cd9 --- /dev/null +++ b/doc/develop/tests_sandbox.rst @@ -0,0 +1,80 @@ +.. SPDX-License-Identifier: GPL-2.0+ + +Tests Under the Hood +==================== + +Running sandbox tests directly +------------------------------ + +Typically tests are run using the pytest suite. Running pytests on sandbox is +easy and always gets things right. For example some tests require files to be +set up before they can work. + +But it is also possible to run some sandbox tests directly. For example, this +runs the dm_test_gpio() test which you can find in test/dm/gpio.c:: + + $ ./u-boot -T -c "ut dm gpio" + + + U-Boot 2021.01 + + Model: sandbox + DRAM: 128 MiB + WDT: Started with servicing (60s timeout) + MMC: mmc2: 2 (SD), mmc1: 1 (SD), mmc0: 0 (SD) + In: serial + Out: vidconsole + Err: vidconsole + Model: sandbox + SCSI: + Net: eth0: eth@10002000, eth5: eth@10003000, eth3: sbe5, eth6: eth@10004000 + Test: dm_test_gpio: gpio.c + Test: dm_test_gpio: gpio.c (flat tree) + Failures: 0 + +The -T option tells the U-Boot sandbox to run with the 'test' devicetree +(test.dts) instead of -D which selects the normal sandbox.dts - this is +necessary because many tests rely on nodes or properties in the test devicetree. +If you try running tests without -T then you may see failures, like:: + + $ ./u-boot -c "ut dm gpio" + + + U-Boot 2021.01 + + DRAM: 128 MiB + WDT: Not found! + MMC: + In: serial + Out: serial + Err: serial + SCSI: + Net: No ethernet found. + Please run with test device tree: + ./u-boot -d arch/sandbox/dts/test.dtb + Test: dm_test_gpio: gpio.c + test/dm/gpio.c:37, dm_test_gpio(): 0 == gpio_lookup_name("b4", &dev, &offset, &gpio): Expected 0x0 (0), got 0xffffffea (-22) + Test: dm_test_gpio: gpio.c (flat tree) + test/dm/gpio.c:37, dm_test_gpio(): 0 == gpio_lookup_name("b4", &dev, &offset, &gpio): Expected 0x0 (0), got 0xffffffea (-22) + Failures: 2 + +The message above should provide a hint if you forget to use the -T flag. Even +running with -D will produce different results. + +You can easily use gdb on these tests, without needing --gdbserver:: + + $ gdb u-boot --args -T -c "ut dm gpio" + ... + (gdb) break dm_test_gpio + Breakpoint 1 at 0x1415bd: file test/dm/gpio.c, line 37. + (gdb) run -T -c "ut dm gpio" + Starting program: u-boot -T -c "ut dm gpio" + Test: dm_test_gpio: gpio.c + + Breakpoint 1, dm_test_gpio (uts=0x5555558029a0 ) + at files/test/dm/gpio.c:37 + 37 ut_assertok(gpio_lookup_name("b4", &dev, &offset, &gpio)); + (gdb) + +You can then single-step and look at variables as needed. + -- cgit v1.3.1 From e56c09457e07dc32cffeac5b7fdbb06f5a773d16 Mon Sep 17 00:00:00 2001 From: Simon Glass Date: Sun, 7 Mar 2021 17:34:43 -0700 Subject: doc: Document how sandbox_spl_tests are run Add a few notes about the sandbox_spl tests, since they are special. Signed-off-by: Simon Glass Acked-by: Pratyush Yadav --- doc/develop/testing.rst | 5 +++ doc/develop/tests_sandbox.rst | 82 +++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 87 insertions(+) (limited to 'doc/develop') diff --git a/doc/develop/testing.rst b/doc/develop/testing.rst index 87c90eee271..b181c2e2e41 100644 --- a/doc/develop/testing.rst +++ b/doc/develop/testing.rst @@ -58,6 +58,11 @@ also select particular tests with -k:: ./test/py/test.py --bd sandbox_flattree --build -k hello +There are some special tests that run in SPL. For this you need the sandbox_spl +build:: + + ./test/py/test.py --bd sandbox_spl --build -k test_spl + See test/py/README.md for more information about the pytest suite. See :doc:`tests_sandbox` for how to run tests directly (not through pytest). diff --git a/doc/develop/tests_sandbox.rst b/doc/develop/tests_sandbox.rst index 42b64882cd9..2b2c4be2dd1 100644 --- a/doc/develop/tests_sandbox.rst +++ b/doc/develop/tests_sandbox.rst @@ -78,3 +78,85 @@ You can easily use gdb on these tests, without needing --gdbserver:: You can then single-step and look at variables as needed. + +Running sandbox_spl tests directly +---------------------------------- + +SPL is the phase before U-Boot proper. It is present in the sandbox_spl build, +so you can run SPL like this:: + + ./spl/u-boot-spl + +SPL tests are special in that they run (only in the SPL phase, of course) if the +-u flag is given:: + + ./spl/u-boot-spl -u + + U-Boot SPL 2021.01-00723-g43c77b51be5-dirty (Jan 24 2021 - 16:38:24 -0700) + Running 5 driver model tests + Test: dm_test_of_plat_base: of_platdata.c (flat tree) + Test: dm_test_of_plat_dev: of_platdata.c (flat tree) + Test: dm_test_of_plat_parent: of_platdata.c (flat tree) + Test: dm_test_of_plat_phandle: of_platdata.c (flat tree) + Test: dm_test_of_plat_props: of_platdata.c (flat tree) + Failures: 0 + + + U-Boot 2021.01-00723-g43c77b51be5-dirty (Jan 24 2021 - 16:38:24 -0700) + + DRAM: 128 MiB + ... + +It is not possible to run SPL tests in U-Boot proper, firstly because they are +not built into U-Boot proper and secondly because the environment is very +different, e.g. some SPL tests rely on of-platdata which is only available in +SPL. + +Note that after running, SPL continues to boot into U-Boot proper. You can add +'-c exit' to make U-Boot quit without doing anything further. It is not +currently possible to run SPL tests and then stop, since the pytests require +that U-Boot produces the expected banner. + +You can use the -k flag to select which tests run:: + + ./spl/u-boot-spl -u -k dm_test_of_plat_parent + +Of course you can use gdb with sandbox_spl, just as with sandbox. + + +Running all tests directly +-------------------------- + +A fast way to run all sandbox tests is:: + + ./u-boot -T -c "ut all" + +It typically runs single-thread in 6 seconds on 2021 hardware, with 2s of that +to the delays in the time test. + +This should not be considered a substitute for 'make check', but can be helpful +for git bisect, etc. + + +What tests are built in? +------------------------ + +Whatever sandbox build is used, which tests are present is determined by which +source files are built. For sandbox_spl, the of_platdata tests are built +because of the build rule in test/dm/Makefile:: + + ifeq ($(CONFIG_SPL_BUILD),y) + obj-$(CONFIG_SPL_OF_PLATDATA) += of_platdata.o + else + ...other tests for non-spl + endif + +You can get a list of tests in a U-Boot ELF file by looking for the +linker_list:: + + $ nm /tmp/b/sandbox_spl/spl/u-boot-spl |grep 2_dm_test + 000000000001f200 D _u_boot_list_2_dm_test_2_dm_test_of_plat_base + 000000000001f220 D _u_boot_list_2_dm_test_2_dm_test_of_plat_dev + 000000000001f240 D _u_boot_list_2_dm_test_2_dm_test_of_plat_parent + 000000000001f260 D _u_boot_list_2_dm_test_2_dm_test_of_plat_phandle + 000000000001f280 D _u_boot_list_2_dm_test_2_dm_test_of_plat_props -- cgit v1.3.1 From e1b12e39452fba018e39c7d9005870ab80450b8c Mon Sep 17 00:00:00 2001 From: Simon Glass Date: Sun, 7 Mar 2021 17:35:16 -0700 Subject: test: sandbox: Move sandbox test docs into doc/develop At present some of the documentation about running sandbox tests is in the sandbox docs. It makes more sense to put it in with the other testing docs, with a link there from sandbox. Update the documentation accordingly. Also add a paragraph explaining why sandbox exists and the test philosophy that it uses. Signed-off-by: Simon Glass --- doc/arch/sandbox.rst | 44 +++++++++---------------------------------- doc/develop/tests_sandbox.rst | 44 +++++++++++++++++++++++++++++++++++++++++-- 2 files changed, 51 insertions(+), 37 deletions(-) (limited to 'doc/develop') diff --git a/doc/arch/sandbox.rst b/doc/arch/sandbox.rst index 1ccb5344ac6..1638b050002 100644 --- a/doc/arch/sandbox.rst +++ b/doc/arch/sandbox.rst @@ -17,6 +17,12 @@ of the sandbox U-Boot. The purpose of running U-Boot under Linux is to test all the generic code, not specific to any one architecture. The idea is to create unit tests which we can run to test this upper level code. +Sandbox allows development of many types of new features in a traditional way, +rather than needing to test each iteration on real hardware. Many U-Boot +features were developed on sandbox, including the core driver model, most +uclasses, verified boot, bloblist, logging and dozens of others. Sandbox has +enabled many large-scale code refactors as well. + CONFIG_SANDBOX is defined when building a native board. The board name is 'sandbox' but the vendor name is unset, so there is a @@ -486,42 +492,10 @@ Testing ------- U-Boot sandbox can be used to run various tests, mostly in the test/ -directory. These include: - -command_ut: - Unit tests for command parsing and handling -compression: - Unit tests for U-Boot's compression algorithms, useful for - security checking. It supports gzip, bzip2, lzma and lzo. -driver model: - Run this pytest:: - - ./test/py/test.py --bd sandbox --build -k ut_dm -v - -image: - Unit tests for images: - test/image/test-imagetools.sh - multi-file images - test/image/test-fit.py - FIT images -tracing: - test/trace/test-trace.sh tests the tracing system (see README.trace) -verified boot: - See test/vboot/vboot_test.sh for this - -If you change or enhance any of the above subsystems, you shold write or -expand a test and include it with your patch series submission. Test -coverage in U-Boot is limited, as we need to work to improve it. - -Note that many of these tests are implemented as commands which you can -run natively on your board if desired (and enabled). - -To run all tests use "make check". - -To run a single test in an existing sandbox build, you can use -T to use the -test device tree, and -c to select the test: - - /tmp/b/sandbox/u-boot -T -c "ut dm pci_busdev" +directory. -This runs dm_test_pci_busdev() which is in test/dm/pci.c +See :doc:`../develop/tests_sandbox` for more information and +:doc:`../develop/testing` for information about testing generally. Memory Map diff --git a/doc/develop/tests_sandbox.rst b/doc/develop/tests_sandbox.rst index 2b2c4be2dd1..dd15692f656 100644 --- a/doc/develop/tests_sandbox.rst +++ b/doc/develop/tests_sandbox.rst @@ -1,7 +1,47 @@ .. SPDX-License-Identifier: GPL-2.0+ -Tests Under the Hood -==================== +Sandbox tests +============= + +Test Design +----------- + +Most uclasses and many functions of U-Boot have sandbox tests. This allows much +of the code to be checked in an developer-friendly environment. + +Sandbox provides a way to write and run unit tests. The traditional approach to +unit tests is to build lots of little executables, one for each test or +category of tests. With sandbox, so far as possible, all the tests share a +small number of executables (e.g. 'u-boot' for sandbox, 'u-boot-spl' and +'u-boot' for sandbox_spl) and can be run very quickly. The vast majority of +tests can run on the 'sandbox' build, + +Available tests +--------------- + +Some of the available tests are: + + - command_ut: Unit tests for command parsing and handling + - compression: Unit tests for U-Boot's compression algorithms, useful for + security checking. It supports gzip, bzip2, lzma and lzo. + - image: Unit tests for images: + + - test/image/test-imagetools.sh - multi-file images + - test/py/tests/test-fit.py - FIT images + - tracing: test/trace/test-trace.sh tests the tracing system (see + README.trace) + - verified boot: test/py/tests/test_vboot.py + +If you change or enhance any U-Boot subsystem, you should write or expand a +test and include it with your patch series submission. Test coverage in some +older areas of U-Boot is still somewhat limited and we need to work to improve +it. + +Note that many of these tests are implemented as commands which you can +run natively on your board if desired (and enabled). + +To run all tests, use 'make check'. + Running sandbox tests directly ------------------------------ -- cgit v1.3.1 From fc3283314539d6c3fb577359f6cb364c19c13726 Mon Sep 17 00:00:00 2001 From: Simon Glass Date: Sun, 7 Mar 2021 17:35:17 -0700 Subject: doc: Explain briefly how to write new tests Add a second on writing tests, covering when to use Python and C, where to put the tests, etc. Add a link to the existing Python test documentation. Signed-off-by: Simon Glass --- doc/develop/index.rst | 1 + doc/develop/py_testing.rst | 3 +- doc/develop/testing.rst | 2 + doc/develop/tests_sandbox.rst | 7 + doc/develop/tests_writing.rst | 346 ++++++++++++++++++++++++++++++++++++++++++ 5 files changed, 358 insertions(+), 1 deletion(-) create mode 100644 doc/develop/tests_writing.rst (limited to 'doc/develop') diff --git a/doc/develop/index.rst b/doc/develop/index.rst index 50b1de3bdff..41c0ba1ebd9 100644 --- a/doc/develop/index.rst +++ b/doc/develop/index.rst @@ -33,4 +33,5 @@ Testing coccinelle testing py_testing + tests_writing tests_sandbox diff --git a/doc/develop/py_testing.rst b/doc/develop/py_testing.rst index 7f01858cfda..c4cecc0a01b 100644 --- a/doc/develop/py_testing.rst +++ b/doc/develop/py_testing.rst @@ -13,7 +13,8 @@ results. Advantages of this approach are: U-Boot; there can be no disconnect. - There is no need to write or embed test-related code into U-Boot itself. It is asserted that writing test-related code in Python is simpler and more - flexible than writing it all in C. + flexible than writing it all in C. But see :doc:`tests_writing` for caveats + and more discussion / analysis. - It is reasonably simple to interact with U-Boot in this way. Requirements diff --git a/doc/develop/testing.rst b/doc/develop/testing.rst index b181c2e2e41..ced13ac8bb4 100644 --- a/doc/develop/testing.rst +++ b/doc/develop/testing.rst @@ -117,6 +117,8 @@ or is covered sparingly. So here are some suggestions: is much easier to add onto a test - writing a new large test can seem daunting to most contributors. +See doc:`tests_writing` for how to write tests. + Future work ----------- diff --git a/doc/develop/tests_sandbox.rst b/doc/develop/tests_sandbox.rst index dd15692f656..84608dcb840 100644 --- a/doc/develop/tests_sandbox.rst +++ b/doc/develop/tests_sandbox.rst @@ -200,3 +200,10 @@ linker_list:: 000000000001f240 D _u_boot_list_2_dm_test_2_dm_test_of_plat_parent 000000000001f260 D _u_boot_list_2_dm_test_2_dm_test_of_plat_phandle 000000000001f280 D _u_boot_list_2_dm_test_2_dm_test_of_plat_props + + +Writing tests +------------- + +See :doc:`tests_writing` for how to write new tests. + diff --git a/doc/develop/tests_writing.rst b/doc/develop/tests_writing.rst new file mode 100644 index 00000000000..1ddf7a353a7 --- /dev/null +++ b/doc/develop/tests_writing.rst @@ -0,0 +1,346 @@ +.. SPDX-License-Identifier: GPL-2.0+ +.. Copyright 2021 Google LLC +.. sectionauthor:: Simon Glass + +Writing Tests +============= + +This describes how to write tests in U-Boot and describes the possible options. + +Test types +---------- + +There are two basic types of test in U-Boot: + + - Python tests, in test/py/tests + - C tests, in test/ and its subdirectories + +(there are also UEFI tests in lib/efi_selftest/ not considered here.) + +Python tests talk to U-Boot via the command line. They support both sandbox and +real hardware. They typically do not require building test code into U-Boot +itself. They are fairly slow to run, due to the command-line interface and there +being two separate processes. Python tests are fairly easy to write. They can +be a little tricky to debug sometimes due to the voluminous output of pytest. + +C tests are written directly in U-Boot. While they can be used on boards, they +are more commonly used with sandbox, as they obviously add to U-Boot code size. +C tests are easy to write so long as the required facilities exist. Where they +do not it can involve refactoring or adding new features to sandbox. They are +fast to run and easy to debug. + +Regardless of which test type is used, all tests are collected and run by the +pytest framework, so there is typically no need to run them separately. This +means that C tests can be used when it makes sense, and Python tests when it +doesn't. + + +This table shows how to decide whether to write a C or Python test: + +===================== =========================== ============================= +Attribute C test Python test +===================== =========================== ============================= +Fast to run? Yes No (two separate processes) +Easy to write? Yes, if required test Yes + features exist in sandbox + or the target system +Needs code in U-Boot? Yes No, provided the test can be + executed and the result + determined using the command + line +Easy to debug? Yes No, since access to the U-Boot + state is not available and the + amount of output can + sometimes require a bit of + digging +Can use gdb? Yes, directly Yes, with --gdbserver +Can run on boards? Some can, but only if Some + compiled in and not + dependent on sandboxau +===================== =========================== ============================= + + +Python or C +----------- + +Typically in U-Boot we encourage C test using sandbox for all features. This +allows fast testing, easy development and allows contributors to make changes +without needing dozens of boards to test with. + +When a test requires setup or interaction with the running host (such as to +generate images and then running U-Boot to check that they can be loaded), or +cannot be run on sandbox, Python tests should be used. These should typically +NOT rely on running with sandbox, but instead should function correctly on any +board supported by U-Boot. + + +How slow are Python tests? +-------------------------- + +Under the hood, when running on sandbox, Python tests work by starting a sandbox +test and connecting to it via a pipe. Each interaction with the U-Boot process +requires at least a context switch to handle the pipe interaction. The test +sends a command to U-Boot, which then reacts and shows some output, then the +test sees that and continues. Of course on real hardware, communications delays +(e.g. with a serial console) make this slower. + +For comparison, consider a test that checks the 'md' (memory dump). All times +below are approximate, as measured on an AMD 2950X system. Here is is the test +in Python:: + + @pytest.mark.buildconfigspec('cmd_memory') + def test_md(u_boot_console): + """Test that md reads memory as expected, and that memory can be modified + using the mw command.""" + + ram_base = u_boot_utils.find_ram_base(u_boot_console) + addr = '%08x' % ram_base + val = 'a5f09876' + expected_response = addr + ': ' + val + u_boot_console.run_command('mw ' + addr + ' 0 10') + response = u_boot_console.run_command('md ' + addr + ' 10') + assert(not (expected_response in response)) + u_boot_console.run_command('mw ' + addr + ' ' + val) + response = u_boot_console.run_command('md ' + addr + ' 10') + assert(expected_response in response) + +This runs a few commands and checks the output. Note that it runs a command, +waits for the response and then checks it agains what is expected. If run by +itself it takes around 800ms, including test collection. For 1000 runs it takes +19 seconds, or 19ms per run. Of course 1000 runs it not that useful since we +only want to run it once. + +There is no exactly equivalent C test, but here is a similar one that tests 'ms' +(memory search):: + + /* Test 'ms' command with bytes */ + static int mem_test_ms_b(struct unit_test_state *uts) + { + u8 *buf; + + buf = map_sysmem(0, BUF_SIZE + 1); + memset(buf, '\0', BUF_SIZE); + buf[0x0] = 0x12; + buf[0x31] = 0x12; + buf[0xff] = 0x12; + buf[0x100] = 0x12; + ut_assertok(console_record_reset_enable()); + run_command("ms.b 1 ff 12", 0); + ut_assert_nextline("00000030: 00 12 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................"); + ut_assert_nextline("--"); + ut_assert_nextline("000000f0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 12 ................"); + ut_assert_nextline("2 matches"); + ut_assert_console_end(); + + ut_asserteq(2, env_get_hex("memmatches", 0)); + ut_asserteq(0xff, env_get_hex("memaddr", 0)); + ut_asserteq(0xfe, env_get_hex("mempos", 0)); + + unmap_sysmem(buf); + + return 0; + } + MEM_TEST(mem_test_ms_b, UT_TESTF_CONSOLE_REC); + +This runs the command directly in U-Boot, then checks the console output, also +directly in U-Boot. If run by itself this takes 100ms. For 1000 runs it takes +660ms, or 0.66ms per run. + +So overall running a C test is perhaps 8 times faster individually and the +interactions are perhaps 25 times faster. + +It should also be noted that the C test is fairly easy to debug. You can set a +breakpoint on do_mem_search(), which is what implements the 'ms' command, +single step to see what might be wrong, etc. That is also possible with the +pytest, but requires two terminals and --gdbserver. + + +Why does speed matter? +---------------------- + +Many development activities rely on running tests: + + - 'git bisect run make qcheck' can be used to find a failing commit + - test-driven development relies on quick iteration of build/test + - U-Boot's continuous integration (CI) systems make use of tests. Running + all sandbox tests typically takes 90 seconds and running each qemu test + takes about 30 seconds. This is currently dwarfed by the time taken to + build all boards + +As U-Boot continues to grow its feature set, fast and reliable tests are a +critical factor factor in developer productivity and happiness. + + +Writing C tests +--------------- + +C tests are arranged into suites which are typically executed by the 'ut' +command. Each suite is in its own file. This section describes how to accomplish +some common test tasks. + +(there are also UEFI C tests in lib/efi_selftest/ not considered here.) + +Add a new driver model test +~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +Use this when adding a test for a new or existing uclass, adding new operations +or features to a uclass, adding new ofnode or dev_read_() functions, or anything +else related to driver model. + +Find a suitable place for your test, perhaps near other test functions in +existing code, or in a new file. Each uclass should have its own test file. + +Declare the test with:: + + /* Test that ... */ + static int dm_test_uclassname_what(struct unit_test_state *uts) + { + /* test code here */ + + return 0; + } + DM_TEST(dm_test_uclassname_what, UT_TESTF_SCAN_FDT); + +Replace 'uclassname' with the name of your uclass, if applicable. Replace 'what' +with what you are testing. + +The flags for DM_TEST() are defined in test/test.h and you typically want +UT_TESTF_SCAN_FDT so that the devicetree is scanned and all devices are bound +and ready for use. The DM_TEST macro adds UT_TESTF_DM automatically so that +the test runner knows it is a driver model test. + +Driver model tests are special in that the entire driver model state is +recreated anew for each test. This ensures that if a previous test deletes a +device, for example, it does not affect subsequent tests. Driver model tests +also run both with livetree and flattree, to ensure that both devicetree +implementations work as expected. + +Example commit: c48cb7ebfb4 ("sandbox: add ADC unit tests") [1] + +[1] https://gitlab.denx.de/u-boot/u-boot/-/commit/c48cb7ebfb4 + + +Add a C test to an existing suite +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +Use this when you are adding to or modifying an existing feature outside driver +model. An example is bloblist. + +Add a new function in the same file as the rest of the suite and register it +with the suite. For example, to add a new mem_search test:: + + /* Test 'ms' command with 32-bit values */ + static int mem_test_ms_new_thing(struct unit_test_state *uts) + { + /* test code here*/ + + return 0; + } + MEM_TEST(mem_test_ms_new_thing, UT_TESTF_CONSOLE_REC); + +Note that the MEM_TEST() macros is defined at the top of the file. + +Example commit: 9fe064646d2 ("bloblist: Support relocating to a larger space") [1] + +[1] https://gitlab.denx.de/u-boot/u-boot/-/commit/9fe064646d2 + + +Add a new test suite +~~~~~~~~~~~~~~~~~~~~ + +Each suite should focus on one feature or subsystem, so if you are writing a +new one of those, you should add a new suite. + +Create a new file in test/ or a subdirectory and define a macro to register the +suite. For example:: + + #include + #include + #include + #include + #include + + /* Declare a new wibble test */ + #define WIBBLE_TEST(_name, _flags) UNIT_TEST(_name, _flags, wibble_test) + + /* Tetss go here */ + + /* At the bottom of the file: */ + + int do_ut_wibble(struct cmd_tbl *cmdtp, int flag, int argc, char *const argv[]) + { + struct unit_test *tests = UNIT_TEST_SUITE_START(wibble_test); + const int n_ents = UNIT_TEST_SUITE_COUNT(wibble_test); + + return cmd_ut_category("cmd_wibble", "wibble_test_", tests, n_ents, argc, argv); + } + +Then add new tests to it as above. + +Register this new suite in test/cmd_ut.c by adding to cmd_ut_sub[]:: + + /* Within cmd_ut_sub[]... */ + + U_BOOT_CMD_MKENT(wibble, CONFIG_SYS_MAXARGS, 1, do_ut_wibble, "", ""), + +and adding new help to ut_help_text[]:: + + "ut wibble - Test the wibble feature\n" + +If your feature is conditional on a particular Kconfig, then you can use #ifdef +to control that. + +Finally, add the test to the build by adding to the Makefile in the same +directory:: + + obj-$(CONFIG_$(SPL_)CMDLINE) += wibble.o + +Note that CMDLINE is never enabled in SPL, so this test will only be present in +U-Boot proper. See below for how to do SPL tests. + +As before, you can add an extra Kconfig check if needed:: + + ifneq ($(CONFIG_$(SPL_)WIBBLE),) + obj-$(CONFIG_$(SPL_)CMDLINE) += wibble.o + endif + + +Example commit: 919e7a8fb64 ("test: Add a simple test for bloblist") [1] + +[1] https://gitlab.denx.de/u-boot/u-boot/-/commit/919e7a8fb64 + + +Making the test run from pytest +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +All C tests must run from pytest. Typically this is automatic, since pytest +scans the U-Boot executable for available tests to run. So long as you have a +'ut' subcommand for your test suite, it will run. The same applies for driver +model tests since they use the 'ut dm' subcommand. + +See test/py/tests/test_ut.py for how unit tests are run. + + +Add a C test for SPL +~~~~~~~~~~~~~~~~~~~~ + +Note: C tests are only available for sandbox_spl at present. There is currently +no mechanism in other boards to existing SPL tests even if they are built into +the image. + +SPL tests cannot be run from the 'ut' command since there are no commands +available in SPL. Instead, sandbox (only) calls ut_run_list() on start-up, when +the -u flag is given. This runs the available unit tests, no matter what suite +they are in. + +To create a new SPL test, follow the same rules as above, either adding to an +existing suite or creating a new one. + +An example SPL test is spl_test_load(). + + +Writing Python tests +-------------------- + +See :doc:`py_testing` for brief notes how to write Python tests. You +should be able to use the existing tests in test/py/tests as examples. -- cgit v1.3.1