The NeXtMidas baseline (SYS option tree) uses a testing strategy based on unit tests and macro tests. These tests provide the basis upon which the coverage tests are built upon.
/INTERACTIVE
switch is set).Due to the extremely high cost of writing coverage tests their direct use within the NeXtMidas baseline is limited. Significantly more emphasis is put on writing the black-box tests that make up the backbone of the NeXtMidas regression tests (please see "NeXtMidas Regression Testing" for more information). Only when writing unit tests is there an explicit attempt to do white-box testing of a method.
This approach has been found to be quite cost-effective in that the
black-box tests cover a high percentage of the lines of code in the
libraries. The following table shows the percentage of lines that are
tested by the standard confidence tests only
(CONFIDENCE
and CONFIDENCE/BASELINE/WEB=NO
).
Area | Line Coverage |
---|---|
intr | 67% |
lib | 61% |
libg | 54%(1) |
libm | 28%(2) |
net | 45% |
prim | 67% |
(1) Most libg functions need to be tested using
interactive tests, the increased cost of adding such tests
tends to keeps this number down. (2) libm contains auto-generated code for math functions, a significant portion of which is not used. |
The above chart does not account for the code that can not be tested under normal, stand-alone, conditions on a single Linux system (i.e. the above chart counts such code as "untested" although it may be tested through other means). Such code normally supports:
When one accounts for such factors, the estimated coverage is closer to 85% than 65%. While this leaves around 15% as untested it shows that the confidence tests are doing a good job of line-coverage though they were primarily written with a black-box perspective.
Unit tests are slowly being added to the NeXtMidas baseline with the future goal of providing line coverage testing within the unit tests. The high cost of writing unit tests and even higher cost of writing coverage tests has forced the NeXtMidas Development team to choose an incremental approach where a few new tests are included in each release. The testing of "all paths" is seen as cost-prohibitive and there are no plans to add this level of testing to the NeXtMidas baseline in the near future.