NeXtMidas Regression Testing

Introduction

The NeXtMidas baseline (SYS option tree) uses a testing strategy based on unit tests and macro tests. The core tests are integrated into a set of Confidence Tests which are be easily launched by a NeXtMidas user.

Definitions

Confidence Tests
A set of tests run by an end user to verify an application is functioning normally. Confidence tests are easy to run and report an over-all rating of "Pass" (system is operating as expected) or "Fail" (a malfunction was detected).
Interactive Tests
Tests that require user interaction, usually to test graphical functions.
JUnit
The de facto standard for the Unit Testing of Java programs. (See http://www.junit.org/)
Non-Interactive Tests
Tests that run without any user interaction, usually to test computational functions.
Test Macro
A NeXtMidas macro that runs a set of functionality tests (collectively called "Macro Tests") to test NeXtMidas primitives and intrinsics. Test Macros include both Non-Interactive Tests (that run all the time) and Interactive Tests (that run when the /INTERACTIVE switch is set).
Unit Testing
An industry-standard technique to test the functionality of component parts (typically functions or methods) of a computer program. Unit Testing involves writing test cases for various units (collectively called "Unit Tests") that, when run, validate the output of those units.

Use of Test Macros

In NeXtMidas, macro tests are the closest to real-world testing. They are the best approach to mimic the conditions of actual NeXtMidas-based applications.

For more details on writing macro tests, goto the "Macro Editing" section of the NeXtMidas User's Guide which explains how to use the macro-based testing facilities of NeXtMidas. Also check out the CONFIDENCE;SYS explain file.

Use of Unit Tests

NeXtMidas developers use unit tests for rigorous testing. Unit tests rely on calling individual Java methods from the Java level. They are expensive to write. Often the amount of code written to test a method exceeds the original method. Therefore, unit tests are primarially created to test core library functions where the benefits of the test far outweigh the cost.

For more details, look at the UNITTEST;SYS explain file and the documentation for the nxm.sys.test.junit.NMTestCase class.

Testing Strategy

NeXtMidas developers use the following chart to prioritize testing and devote the most attention to the most-frequently-used functions.

Priority Function/Description Tests Used
1 Ubiquitous Library Functionality
(Tables, String utilities, ...)
Unit Tests,
Macro Tests
2 Essential Macro Language Functionality
(IF..THEN..ELSE, RESULTS, CALL, ...)
Macro Tests
3 File and Pipe I/O
 
Macro Tests
4 Core Primitives & Intrinsics
(FFT, NOOP, ...)
Macro Tests
5 Basic Functionality of Graphical Primitives
(PLOT, PANEL, GCONTROL)
Interactive Macro Tests
6 Basic Networking Functionality
(RMIF, ...)
Macro Tests,
Manual Tests
7 Other Customer-Identified Key Functions
 
As Appropriate
8 Advanced Functionality of Graphical Primitives
 
Interactive Macro Tests,
Manual Tests
9 Advanced Networking Functionality
 
Manual Tests
10 Utility Primitives & Intrinsics
(QUERY, HELP, ...)
Manual Tests

Current Status & Plans

(Revised June 2005)

Currently the NeXtMidas confidence tests consist of over 78 test macros and 4 complete unit test classes. In all they produce over a thousand Pass/Fail tests and many interactive tests.

The tests are successful. Throughout the NeXtMidas 1.9.X development series no bugs were got past the unit tests and only a handful got past the macro tests. Bugs passing a macro test did so for one of the following reasons:

The remaining bugs are categorized as follows:

The NeXtMidas developers assembled a larger number of test to lower the number of platform-dependent bugs that go undetected. They are also experimenting with a system that runs non-interactive confidence tests every night and emails the results to developers.

The majority of the bugs reported in the 2.0.X release of NeXtMidas are graphical ones which require manual testing. The NeXtMidas developers hope to add a dedicated tester to the team to perfrorm extensive manual testing.

Throughout the 2.1.X development series, the NeXtMidas developers will extend the existing set of macro tests to include tests for additional functionality and add unit tests for other core library functions. This will reduce the number of non-tested functions and will add an extra set of tests to many existing functions.