NeXtMidas Coverage Testing

Introduction

The NeXtMidas baseline (SYS option tree) uses a testing strategy based on unit tests and macro tests. These tests provide the basis upon which the coverage tests are built upon.

Definitions

All Paths
A type of White-Box Testing that tests all possible execution paths in a module. This type of test is impossible to write for methods that include loops with a variable number of iterations or share resources with external actors or threads.
All Lines
A type of White-Box Testing that tests to see that all lines are executed at least once during the course of testing. This type of test is impossible to write for methods that have exception-handling statements to account for "impossible" conditions.
Black-Box Testing
A set of tests designed to verify that the functionality of an application matches its design. Black-box tests ignore the actual implementation of an application and test only against design requirements.
Coverage Tests
A set of tests designed to exercise all possible paths of execution of an application.
Interactive Tests
Tests that require user interaction, usually to test graphical functions.
JUnit
The de facto standard for the Unit Testing of Java programs. (See http://www.junit.org/)
Non-Interactive Tests
Tests that run without any user interaction, usually to test computational functions.
Test Macro
A NeXtMidas macro that runs a set of functionality tests (collectively called "Macro Tests") to test NeXtMidas primitives and intrinsics. Test Macros include both Non-Interactive Tests (that run all the time) and Interactive Tests (that run when the /INTERACTIVE switch is set).
Unit Testing
An industry-standard technique to test the functionality of component parts (typically functions or methods) of a computer program. Unit Testing involves writing test cases for various units (collectively called "Unit Tests") that, when run, validate the output of those units.
White-Box Testing
A set of tests designed to verify that all possible functions of an application match its design. White-box tests are generated based on a careful analysis of the code to ensure that all possible branches of code are tested.

Where Coverage Tests are Used in NeXtMidas

Due to the extremely high cost of writing coverage tests their direct use within the NeXtMidas baseline is limited. Significantly more emphasis is put on writing the black-box tests that make up the backbone of the NeXtMidas regression tests (please see "NeXtMidas Regression Testing" for more information). Only when writing unit tests is there an explicit attempt to do white-box testing of a method.

This approach has been found to be quite cost-effective in that the black-box tests cover a high percentage of the lines of code in the libraries. The following table shows the percentage of lines that are tested by the standard confidence tests only (CONFIDENCE and CONFIDENCE/BASELINE/WEB=NO).

Area Line Coverage
intr 67%
lib 61%
libg 54%(1)
libm 28%(2)
net 45%
prim 67%
(1) Most libg functions need to be tested using interactive tests, the increased cost of adding such tests tends to keeps this number down.
(2) libm contains auto-generated code for math functions, a significant portion of which is not used.

The above chart does not account for the code that can not be tested under normal, stand-alone, conditions on a single Linux system (i.e. the above chart counts such code as "untested" although it may be tested through other means). Such code normally supports:

When one accounts for such factors, the estimated coverage is closer to 85% than 65%. While this leaves around 15% as untested it shows that the confidence tests are doing a good job of line-coverage though they were primarily written with a black-box perspective.

Adding Coverage Tests to NeXtMidas

Unit tests are slowly being added to the NeXtMidas baseline with the future goal of providing line coverage testing within the unit tests. The high cost of writing unit tests and even higher cost of writing coverage tests has forced the NeXtMidas Development team to choose an incremental approach where a few new tests are included in each release. The testing of "all paths" is seen as cost-prohibitive and there are no plans to add this level of testing to the NeXtMidas baseline in the near future.