.. SPDX-License-Identifier: Apache-2.0 Copyright (C) 2020-2022 OKTET Labs Ltd. All rights reserved. te-parent: te_engine .. index:: pair: group; Tester .. _doxid-group__te__engine__tester: Tester ====== .. toctree:: :hidden: .. _doxid-group__te__engine__tester_1te_engine_tester_introduction: Introduction ~~~~~~~~~~~~ :ref:`Tester ` is an application of :ref:`Test Engine ` that controls running tests according to configuration files. .. image:: /static/image/ten_tester_context.png :alt: Tester context in TE In the context of Test Environment tests are organized in groups called Test Packages. From the Tester point of view, the Test Package is a set of executables (binary or script files can be run in shell with set of parameters passed via command line) accompanied by a file package.xml describing how to run these scripts (execution order, test parameter values, etc.). See :ref:`Tester Concepts ` for more details. It is possible to run the same test with different parameters. Parameters for a test are specified in package.xml file. Tests may be iterated with different combinations of parameter values. Such iterations are performed by :ref:`Tester `. :ref:`Tester ` has complex parameters iteration and multiplication logic, see :ref:`Iteration and usecases `. Some tests may be marked as (associated with) checking the particular requirement(s) of the product. :ref:`Tester ` allows to run tests checking a particular set of requirements. See :ref:`Test Requirements `. :ref:`Tester ` may be asked to build the tests from sources. In this case tests should be built using meson or GNU tools (make/autoconf/automake). All TE libraries used by the tests should be specified in the :ref:`Builder ` configuration file and be built/installed before :ref:`Tester ` starting. :ref:`Tester ` is responsible for the error recovery. It utilizes a service provided by :ref:`Configurator ` to make a backup configuration for each package and test and to restore backups if necessary. :ref:`Tester ` utilizes logging facilities provided by :ref:`Logger ` to log test/package starting and finishing events. This information is then used by Report Generator to split the log to sections corresponding to particular tests/packages. .. _doxid-group__te__engine__tester_1te_engine_tester_concepts: Tester Concepts ~~~~~~~~~~~~~~~ **Test** is a complete sequence of actions required to achieve a specific purpose (check that tested system provides a required functionality, complies a standard, etc.) and producing a verdict pass/fail (possibly accompanied by additional data). **Test** **Package** is a group of tightly related tests or test packages, which may share internal libraries and usually run together (one-by-one or simultaneously). Test Package may consist of one test. It may have a prologue (performing some initialization) and epilogue (releasing resources and restoring TE configuration). **Test** **Script** is a test which is a minimal structural unit of a test harness. **Test** **Suite** is a Test Package which may be considered as standalone entity from organisational point of view and build issues. The hierarchy of these objects is: * Test Suite consists of several Test Packages; * Test Package includes several Tests; * each Test has a corresponding Test Script. .. _doxid-group__te__engine__tester_1te_engine_tester_req: Test Requirements ~~~~~~~~~~~~~~~~~ Requirements are string labels which describe: * product features; * configuration features; * scenario description. Test can be associated with a set of requirements. I.e. * **LINUX** the test should be executed only on Linux; * **LINUX-3.2** specific kernel version is required; * **PRODUCTION_BUILD** test should be executed only on production build (non-debug); * **SELECT** test checks select() function behaviour; * **IGMP** test checks IGMP-support of the product; * **LONG** test scenario is really long; * **BROKEN** test scenario is broken - avoid!. Requirements can be normal and sticky. Sticky requirements are inherited by all descendants (see the hierarchy description in :ref:`Tester Concepts `). Tester can be asked to run test satisfying certain requirements via tester-req option which can be set to logical expression where logical variables are test requirements and logical operators are ! (not; should be escaped in bash), \| (or) and & (and). For example, .. code-block:: shell $ dispatcher.sh --tester-req=\!BROKEN : skip broken tests $ dispatcher.sh --tester-req="MSG_MORE|TCP_CORK" : run all the tests involving MSG_MORE flag or TCP_CORK socket option If you specify multiple tester-req options, it works as if their values are concatenated into a single logical expression by means of & (and) logical operator and passed with a single tester-req option. .. code-block:: shell $ dispatcher.sh --tester-req=IGMP --tester-req=\!LONG : run fast IGMP tests .. _doxid-group__te__engine__tester_1te_engine_tester_conf: Configuration File ~~~~~~~~~~~~~~~~~~ .. _doxid-group__te__engine__tester_1te_engine_tester_conf_root: Tester Root Configuration File ------------------------------ Tester configuration file is a root of testing scenario. This file contains some auxiliary information like description and list of maintainers as well as information about test suites. XSD schema of :ref:`Tester ` configuration file can be found at ${TE_BASE}/doc/xsd/tester_config.xsd file. :ref:`Tester ` configuration file can look like: .. ref-code-block:: xml IPv6 Protocol Testing Here is some description of used TAGs of configuration file: * maintainer - specify information about the maintainer of this configuration; * description - optional description of this configuration. Please note that this is rather configuration file description and it has nothing in common to the description of test suite(s) - you can refer to the same test suite from different :ref:`Tester ` configuration files; * suite - specify information about the location of top level test package (test suite). This directive tells :ref:`Tester ` where to find sources or binaries of a test suite. The only suites mentioned in suite directives are built (if necessary) by the :ref:`Tester `. suite tag can keep the following attributes: * name - specifies test suite name that is used as an identifier associated with this test suite; * src - tells :ref:`Tester ` where to find sources of mentioned test suite; * bin - tells :ref:`Tester ` where to find binarires of mentioned test suite (please note that src and bin attributes are mutualy exclusive, only one of them can present at the same time in suite directive). It is possible not to have both src and bin attributes in suite tag, which means you ask :ref:`Tester ` to build a test suite located under ${TE_BASE}/suites/[suite name] directory. * run - a block of tests, packages to run. In the above example we tell :ref:`Tester ` : * to build a test suite located under ${TE_TS_IPV6_HOST} directory. :ref:`Tester ` will ask :ref:`Builder ` to build and install this test suite; * to associate a directory ${TE_TS_IPV6_ROUTER_BIN} with installed version of test suite reffered as ipv6_router; * to run a test suite named ipv6_host (it will be run from the place where it is installed by :ref:`Builder `); * to run a test suite named ipv6_router (it will be run from ${TE_TS_IPV6_ROUTER_BIN}directory that is previously associated with bin attribute of suite directive. If you only need to build a set of test suites you can have :ref:`Tester ` configuration file as following: .. ref-code-block:: xml Build IPv6 related test suites This configuration file asks :ref:`Tester ` to: * build a test suite whose sources located at ${TE_TS_IPV6_HOST} directory; * build a test suite named ipv6_router. As no src attribute is given, :ref:`Tester ` looks for it under ${TE_INSTALL_SUITE}, i.e. among the already installed suites. .. _doxid-group__te__engine__tester_1te_engine_tester_conf_pkg: Tester Package Description File ------------------------------- Test package description files contain nodes of testing scenario tree. Testing scenario tree is built of test script nodes to be executed with particular set of parameters and particular execution order. In order to group some test scripts together there are entities called sessions and packages. Test script is an external applications to be called by Tester. Test session is a complex structure that can include different components to run. Test sessions may have prologue that is run at the start of a session, epilogue that is executed after the session and keep-alive validation to run before and after each test script execution. XSD schema of Package description files can be found at ${TE_BASE}/doc/xsd/test_package.xsd. Here is an example of the simplest package description file: .. ref-code-block:: xml Device LED operation Test Suite Power USB Ethernet Error **track_conf** can have the following values: * **no**. Tester does not track configuration changes at all, so it does not try to restore configuration state if it was changed. * **yes** or **barf**. This is the default behaviour described above. * **yes_nohistory** or **barf_nohistory**. This differs from the default behaviour only in that Tester does not try to roll configuration changes back by executing configuration commands stored in history in reverse order. Instead it jumps directly to restoring configuration by changing only those configuration objects whose state differs from configuration backup (normally this is done only if restoring configuration from history fails). * **silent**. Tester tracks and restores configuration state as usual, but it does not set to **DIRTY** results of the tests which changed configuration. This is convenient if you do not want to undo all changes manually in tests cleanup but instead rely on Tester to do it for you. * **nohistory** or **silent_nohistory**. Works like **silent** but also does not try to restore configuration from history like **yes_nohistory**. There is also **sync** flag which can be added to these values (for **no** it makes no sense though), like **nohistory|sync**. When the flag is set, Tester synchronizes configuration state with Test Agents before checking whether it is changed. This way it can catch fully unexpected changes (like side effects of commands such as ioctl(SIOCETHTOOL/ETHTOOL_RESET)). However synchronizing configuration with all Test Agents takes extra time. That's why it is not done by default and extra flag is required to enable it. **track_conf** attribute can be specified not only for ** 10G 25G is logged as *Test that 10G link speed is correctly advertised* for the first iteration and *Test that 25G link speed is correctly advertised* for the second one. The names that can be referenced are the arguments and the handed-down session variables of the run item. A compound value provides one name per field, spelled ``argument_field``, exactly as the fields appear among the logged parameters. For instance, if a run item is given a compound argument ``dev`` with fields ``pci`` and ``name``, its objective may refer to ``${dev_pci}`` and ``${dev_name}``. A compound value provides no name for the argument as a whole, so a bare ``${dev}`` is reported as unresolvable instead of quietly expanding to nothing. The syntax is the one te_string_expand_parameters() implements, see ``lib/tools/te_expand.h``: in particular ``${name:-default}``, ``${name:+alternate}`` and filters such as ``${name|upper}``. A reference to a name that no argument or variable of the run item can provide is a fatal error: the Tester refuses to start any test. Every such reference of the configuration is reported, so that a single run tells everything that has to be fixed. The limits of the check are: * An argument declared as ```` takes its value from the parent run item. The name is known, so ``${x}`` is confirmed to be declared. The value is not: should it be a compound, the field names it carries are settled only when the parent supplies it, and two iterations may supply different ones. A reference beginning ``x_``, such as ``${x_field}``, is therefore accepted without checking the field, and the check does not follow the argument up to the parent to enumerate the possibilities. Every reference outside that prefix is still verified. * A reference nested inside another one, such as ``typo`` in ``${name:-${typo}}``, is not checked, because what the enclosing reference expands to is not known in advance. * A missing value is not an error when the reference says so: write ``${name:-default}`` where absence is intentional. The diagnostic is logged with ``ERROR()``, not printed to the console. When a run refuses to start, the log (or a run with ``--log-txt``) has the message naming the package file, the run item and the reference that could not be resolved. Test parameters themselves are never expanded, so expansion cannot affect iteration identity, the iteration hash or test history. The execution plan carries objectives as written; expansion happens when the run item starts. .. note:: ``trc_update`` stores the objective it reads from the log as the objective of the test in the TRC database. For an objective with references that is the text of the last processed iteration, not the generic wording from the package file. .. _doxid-group__te__engine__tester_1te_engine_tester_iteration: Iteration and usecases ~~~~~~~~~~~~~~~~~~~~~~ .. _doxid-group__te__engine__tester_1te_engine_tester_iteration_parameters: Parameters ---------- Every test has a set of parameters and every parameter has certain values. In the package.xml file it's represented as: .. ref-code-block:: xml