The NeXtMidas baseline (SYS option tree) uses a testing strategy based on unit tests and macro tests. The core tests are integrated into a set of Confidence Tests which are be easily launched by a NeXtMidas user.
/INTERACTIVE
switch is set).In NeXtMidas, macro tests are the closest to real-world testing. They are the best approach to mimic the conditions of actual NeXtMidas-based applications.
For more details on writing macro tests, goto the "Macro Editing"
section of the NeXtMidas User's Guide which explains how to use
the macro-based testing facilities of NeXtMidas. Also check out the
CONFIDENCE;SYS
explain file.
NeXtMidas developers use unit tests for rigorous testing. Unit tests rely on calling individual Java methods from the Java level. They are expensive to write. Often the amount of code written to test a method exceeds the original method. Therefore, unit tests are primarially created to test core library functions where the benefits of the test far outweigh the cost.
For more details, look at the UNITTEST;SYS
explain file
and the documentation for the nxm.sys.test.junit.NMTestCase
class.
NeXtMidas developers use the following chart to prioritize testing and devote the most attention to the most-frequently-used functions.
Priority | Function/Description | Tests Used |
---|---|---|
1 | Ubiquitous Library Functionality (Tables, String utilities, ...) |
Unit Tests, Macro Tests |
2 | Essential Macro Language Functionality ( IF..THEN..ELSE , RESULTS ,
CALL , ...) |
Macro Tests |
3 | File and Pipe I/O |
Macro Tests |
4 | Core Primitives & Intrinsics ( FFT , NOOP , ...) |
Macro Tests |
5 | Basic Functionality of Graphical Primitives ( PLOT , PANEL , GCONTROL ) |
Interactive Macro Tests |
6 | Basic Networking Functionality ( RMIF , ...) |
Macro Tests, Manual Tests |
7 | Other Customer-Identified Key Functions |
As Appropriate |
8 | Advanced Functionality of Graphical Primitives |
Interactive Macro Tests, Manual Tests |
9 | Advanced Networking Functionality |
Manual Tests |
10 | Utility Primitives & Intrinsics ( QUERY , HELP , ...) |
Manual Tests |
Currently the NeXtMidas confidence tests consist of over 78 test macros and 4 complete unit test classes. In all they produce over a thousand Pass/Fail tests and many interactive tests.
The tests are successful. Throughout the NeXtMidas 1.9.X development series no bugs were got past the unit tests and only a handful got past the macro tests. Bugs passing a macro test did so for one of the following reasons:
The remaining bugs are categorized as follows:
The NeXtMidas developers assembled a larger number of test to lower the number of platform-dependent bugs that go undetected. They are also experimenting with a system that runs non-interactive confidence tests every night and emails the results to developers.
The majority of the bugs reported in the 2.0.X release of NeXtMidas are graphical ones which require manual testing. The NeXtMidas developers hope to add a dedicated tester to the team to perfrorm extensive manual testing.
Throughout the 2.1.X development series, the NeXtMidas developers will extend the existing set of macro tests to include tests for additional functionality and add unit tests for other core library functions. This will reduce the number of non-tested functions and will add an extra set of tests to many existing functions.