The test role is very similar to the application role, it also builds an executable binary.

The difference is that is also builds for each target a test script that executes the test target either on the host (MODE=host) or a target platform (currently only qemu).

The role accepts all make variables that are accepted by the application role. The only difference is that the TARGET variable is not required. If it is missing, the source directory will be scanned for source files that fit the pattern test_*.c[c] and create one target for each of them.

Note: It is possible to still use SRC_C[C] when targets are determined automatically. In that case the specified sources will be used in addition* to the main test_*.c[c] source.

In addition to the variables above, there are a number of variables that control how the test is executed. All these variables may be used as a global variable that applies to all test or, if the target name is added as a suffix, set for a specific target only.

TEST_TARGET: Name of binary containing the test (default: same as TARGET).

TARGET_$(ARCH): When TARGET is undefined, these targets are added to the list of targets for the specified architecture. For all targets SRC_C[C] files must be defined separately.

TEST_KERNEL_ARGS: Arguments to append to the kernel command line. These are also appended when specifying custom ones via a .t-file's -f parameter or when using -d.

TEST_EXPECTED: File containing expected output. By default the variable is empty, which means the test binary is expected to produce TAP test output, that can be directly processed. When the TEST_TAP_PLUGINS variable is given, TEST_EXPECTED is ignored.

TEST_EXPECTED_REPEAT: Number of times the expected output should be repeated, by default 1. When set to 0 then output is expected to repeat forever. This is particularly useful to make sure that stress tests that are meant to run in an endless loop are still alive. Note that such endless tests can only be run by directly executing the test script. They will be skipped when run in a test harness like prove.

TEST_TAP_PLUGINS: Specify the plugins that are used to process the output of the test run. The syntax is of the values is:

plugin1:arg1=a,arg2=b plugin2:arg=foo

Multiple plugins separated by a space are loaded in order. Spaces are not allowed inside a plugin specification. One or more arguments are optionally passed to the plugin separated by commas and delimited by a colon.

If the variable is not specified the plugins for TAPOutput and OutputMatching (depending on the TEST_EXPECTED variable) are automatically loaded.

For the supported plugins and their options please refer to their in-line documentation in tool/lib/L4/TapWrapper/Plugin/. The plugin name corresponds to the file stem name in that directory.

TEST_TIMEOUT: Non-standard timeout after which the test run is aborted (useful for tests involving sleep).

NED_CFG: LUA configuration file for startup to give to Ned

REQUIRED_MODULES: Additional modules needed to run the test. By adding [opts] to the name of a module you can add module options that are reflected in the generated modules.list.

BOOTSTRAP_ARGS: Additional parameters to supply to bootstrap.

QEMU_ARGS: Additional parameters to supply to QEMU.

MOE_ARGS: Additional parameters to supply to moe.

TEST_ARGS: Additional arguments for the TEST_STARTER (tapper-wrapper per default).

TEST_ROOT_TASK: Alternative root task to be used during a test instead of moe.

TEST_ROOT_TASK_ARGS: Arguments passed to TEST_ROOT_TASK if TEST_ROOT_TASK is different from moe.

KERNEL_CONF: Features the L4Re Microkernel must have been compiled with. A space-separated list of config options as used by Kconfig. run_test looks for a globalconfig.out file in the same directory as the kernel and checks that all options are enabled. If not, the test is skipped. Has only an effect if the globalconfig.out file is present.

L4RE_CONF: Features the L4Re userland must have been compiled with. A space-separated list of config options as used by Kconfig. run_test will look for these in the .kconfig file in the L4Re build directory.

L4LINUX_CONF: Features the L4Linux kernel must have been compiled with. Similar to KERNEL_CONF but checks for a .config file in the directory of the L4Linux kernel.

TEST_SETUP: Command to execute before the test is run. The test will only be executed if the command returns 0. If the exit code is 69, the test is marked as skipped with the reason provided in the final line of stdout.

TEST_LOGFILE: Append output of test execution to the given file unless TEST_WORKDIR is given.

TEST_WORKDIR: Create logs, temp and other files below the given directory. That directory is taken as base dir for more automatically created subdir levels using the current test path, in order to guarantee conflict-free usage when running many different tests with a common workdir. When TEST_WORKDIR is provided then TEST_LOGFILE is ignored as it is organized below workdir.

TEST_TAGS

List of conditions for tags provided during execution of a test. A tag can be set to 1, set to 0 or be unspecified via TEST_RUN_TAGS during execution. Therefore there are 4 possible conditions for a tag that can be specified in TEST_TAGS: tag, !tag, +tag and -tag. The following table shows the conditions they represent.

TEST_RUN_TAGS \ TEST_TAGS	tag	!tag	+tag	-tag
tag or tag=1	y		y
unspecified		y	y
tag=0		y		y

Example usage:

The tag long-running is used by tests which take a long time and should be skipped by default. These tests are marked with the tag long-running unprefixed.

The tag hardware is set to 1 at runtime when the tests will run on real hardware. Tests that must not run on real hardware are marked with !hardware.

The tag +impl-def is used by tests that test implementation details. Due to the nature of this flag we require the "+" prefix to be used, so they are run by default but can be excluded from execution by setting TEST_RUN_TAGS to impl-def=0 at runtime.

If you want to specify multiple tag conditions they need to be separated with a comma.

TEST_PLATFORM_ALLOW and TEST_PLATFORM_DENY

Deny and allow lists of platforms a test is banned from or limited to. If you list platforms in the TEST_PLATFORM_ALLOW variable the test will only be run on these listed platforms and will be skipped on any other platform. If you list platforms in the TEST_PLATFORM_DENY variable the test will be skipped on the listed platforms and will be run on any other platform. You can only use one of these variables per test, not both. See mk/platforms/ for the various identifiers.

Example usage:

# Do not run this test on the Raspberry Pi platform
TEST_PLATFORM_DENY_test_xyz := rpi

# Only run this test on this test on the RCar3 platform.
TEST_PLATFORM_ALLOW_test_abc := rcar3

TAPARCHIVE: Filename for an archive file to store the resulting TAP output.

In addition to compiled tests, it is also possible to create tests where the test binary or script comes from a different source. These tests must be listed in EXTRA_TARGET and for each target a custom TEST_TARGET must be provided.

Running Tests

The make role creates a test script which can be found in <builddir>/test/t/<arch>/<api>. It is possible to organise the tests further in subdirectories below by specifying a TEST_GROUP.

To be able to execute the test, a minimal test environment needs to be set up by exporting the following environment variables:

KERNEL_<arch>, KERNEL: L4Re Microkernel binary to use. The test runner is able to check if the kernel has all features necessary for the test and skip tests accordingly. In order for this to work, the globalconfig.out config file from the build directory needs to be available in the same directory as the kernel.

L4LX_KERNEL_<arch>, L4LX_KERNEL: L4Linux binary to use. This is only required to run tests in mode=l4linux. If no L4Linux kernel is set then these tests will simply be skipped. The test runner is also able to check if the kernel has all features compiled in that are required to run the test successfully (see make variable L4LINUX_CONF above). For this to work, the .config configuration file from the build directory needs to be available in the same directory as the kernel.

LINUX_RAMDISK_<arch>, LINUX_RAMDISK: Ramdisk to mount as root in L4Linux. This is only required to run tests in mode=l4linux. If not supplied, L4Linux tests will be skipped. The ramdisk must be set up to start the test directly after the initial startup is finished. The name of the test binary is supplied via the kernel command line option l4re_testprog. The tool/test directory contains an example script launch-l4linux-test. which can be copied onto the ramdisk and started by the init script.

TEST_HWCONFIG and TEST_FIASCOCONFIG

Some userland tests rely on external information about the underlying platform and the configuration of the L4Re Microkernel to decide whether or not to test specific features or to determine which and how much resources are available. Some examples for this are whether or not virtualization is supported by the platform, how many cores the platform has, how many cores the kernel supports or how much memory the platform provides. To convey this information to these tests you can set the two environment variables TEST_HWCONFIG and TEST_FIASCOCONFIG.

Using TEST_HWCONFIG requires a plain text document containing key-value pairs separated by a = symbol. On top of that comment lines starting with # are supported. Simply create a plain text file such as the following and set TEST_HWCONFIG to its absolute path.

VIRTUALIZATION=y
MP=y
NUM_CORES=4
MEMORY=2048

Using TEST_FIASCOCONFIG is easier since it only needs to contain the absolute path of the globalconfig.out file in the L4Re Microkernel's build directory. The build system will then extract the information when a test is started.

When starting a test the build system will read both files and provide their content as a lua table to the test. A ned script can then make decisions based on them. To simplify some decisions the build system merges some information by itself, e.g. virtualization is only available if both the platform and the L4Re Microkernel support this feature. More details can be obtain from the perl module in tool/lib/L4/TestEnvLua.pm.

In addition to these variables, the following BID variables can be overwritten at runtime: PT (for the plaform type) and TEST_TIMEOUT. You may also supply QEMU_ARGS and MOE_ARGS which will be appended to the parameters specified in the BID test make file.

Once the environment is set up, the tests can be run either by simply executing all of them from the build directory with

make test

or executing them directly, like

test/t/amd64_amdfam10/l4f/l4re-core/moe/test_namespace.t

or running one or more tests through the test harness prove, like

prove test/t/amd64_amdfam10/l4f/l4re-core/moe/test_namespace.t
prove -r test/t/amd64_amdfam10/l4f/l4re-core/
prove -rv test/t/amd64_amdfam10/l4f/l4re-core/

TEST_TAGS allow for a way to include or exclude whole groups of tests during execution, primarily with prove. You can specify which tests to run at runtime using one of the following ways:

$ test/t/amd64_amdfam10/l4f/l4re-core/test_one.t --run-tags slow,gtest-shuffle=0
$ test/t/amd64_amdfam10/l4f/l4re-core/test_one.t -T slow,gtest-shuffle=0
$ prove -r test/t/amd64_amdfam10/l4f/l4re-core/ :: -T slow,gtest-shuffle=0
$ TEST_RUN_TAGS=slow,gtest-shuffle=0 prove -r test/t/amd64_amdfam10/l4f/l4re-core/

For each test tag requirements defined in the corresponding TEST_TAGS Makefile variable are tested. If the requirements for tags do not match the test is skipped. The SKIP message will provide insight why the test was skipped:

$ make test
...
test/t/amd64_amdfam10/test_one.t .... ok
test/t/amd64_amdfam10/test_two.t .... skipped: Running this test requires tag slow to be set to 1.
test/t/amd64_amdfam10/test_three.t .. ok

When tags are provided, the tests requiring those tags are now also executed while the tests that forbid them are skipped:

$ TEST_RUN_TAGS=slow,gtest-shuffle
$ make test
...
test/t/amd64_amdfam10/test_one.t .... ok
test/t/amd64_amdfam10/test_two.t .... ok
test/t/amd64_amdfam10/test_three.t .. skipped: Running this test requires tag gtest-shuffle to be set to 0 or not specified.

For further details on how values in TEST_TAGS and TEST_RUN_TAGS interact, see the help text for TEST_TAGS.

Running Tests in External Programs

You can hand-over test execution to an external program by setting the environment variable EXTERNAL_TEST_STARTER to the full path of that program:

export EXTERNAL_TEST_STARTER=/path/to/external/test-starter
make test

EXTERNAL_TEST_STARTER

This variable is evaluated by tool/bin/run_test (the backend behind make test) and contains the full path to the tool which actually starts the test instead of the test itself.

The EXTERNAL_TEST_STARTER can be any program instead of the default execution via make qemu E=maketest. Its output is taken by run_test as the test output.

Usually it is just a bridge to prepare the test execution, e.g., it could create the test as image and start that image via a simulator.

Running Tests in a Simulator

Based on above mechanism there is a dedicated external test starter tool/bin/teststarter-image-telnet.pl shipped in BID which assumes an image to be started with another program which provides test execution output on a network port.

This can be used to execute tests in a simulator, like this:

export EXTERNAL_TEST_STARTER=$L4RE_SRC/tool/bin/teststarter-image-telnet.pl
export SIMULATOR_START=/path/to/configured/simulator-exe
make test

After building the image and starting the simulator it contacts the simulator via a network port (sometimes called "telnet" port) to pass-through its execution output as its own output so it gets captured by run_test as usual.

The following variables control teststarter-image-telnet.pl:

SIMULATOR_START: This points to the full path of the program that actually starts the prepared test image. Most often this is the frontend script of your simulator environment which is pre-configured so that it actually works in the way that teststarter-image-telnet.pl expects from the following settings.

SIMULATOR_IMAGETYPE: The image type to be generated via make $SIMULATOR_IMAGETYPE E=maketest. Default is elfimage.

SIMULATOR_HOST: The simulator will be contacted via socket on that host to read its output. Default is localhost.

SIMULATOR_PORT: The simulator will be contacted via socket on that port to read its output. Default is 11111.

SIMULATOR_START_SLEEPTIME: After starting the simulator it waits that many seconds before reading from the port. Default is 1 (second).

Running tests without tapper-wrapper

In case you want to replace the tapper-wrapper test starter, you can replace the default one by setting the environment variable TEST_STARTER to the path of your test starter. Then your test starter can use the same environment which is normally set up for the default starter, which includes environment variables provided by the build system as well as the test itself. Among these are SEARCHPATH, MODE, ARCH, MOE_CFG, MOE_ARGS, TEST_TIMEOUT, TEST_TARGET, TEST_EXPECTED, QEMU_ARGS and many more.

Debugging Tests

The test script is only a thin wrapper that sets up the test environment as it was defined in the make file and then executes two scripts: tapper-wrapper and run_test.

The main work horse of the two is tool/bin/run_test. It collects the necessary files and starts qemu to execute the test. This script is always required.

There is then a second script wrapped around the test runner: tool/bin/tapper-wrapper. This tool inspects the output of the test runner and reformats it, so that it can be read by tools like prove. If the test produces tap output, then the script scans for this output and filters away all the debug output. If TEST_EXPECTED was defined, then the script scans the output for the expected lines and prints a suitable TAP message with success or failure. It also makes sure that qemu is killed as soon as the test is finished.

There are a number of command-line parameters that allow to quickly change test parameters for debugging purposes. Run the test with '–help' for more information about available parameters.