Using NeXtMidas as an X-Midas Option Tree (NeXtOpt)

NeXtMidas may be used as an X-Midas option tree. This enables an X-Midas macro to call NeXtMidas primitives.

Requirements
Current Capabilities
Building
Running
Using X-Midas Results from NeXtMidas
Using X-Midas Pipes from NeXtMidas
Using X-Midas Messages from NeXtMidas
Using X-Midas AUX List from NeXtMidas
Concept and Design
Debugging
Top Ten Things To Remember

Requirements

Install libXt-devel (RHEL/CentOS) or appropriate package that provides required development headers for X11/Intrinsic.h.

In order to build NeXtMidas as an X-Midas option tree you will need to use three compilers:

A Fortran compiler (for the host primitives)
A C compiler (for the Java Native Interface, JNI)
A Java compiler (for NeXtMidas)

The following table shows the supported compilers and X-Midas versions. Other combinations of compliers for X-Midas 3.8.X and 4.2.X are likely work out-of-the-box, they are red here because (as of the time of this writing) no one has tried them yet. It is possible that a few minor changes will be necessary before X-Midas 3.6.5 and 3.7.1 are supported. Any help is welcome in getting this matrix filled in.

	Fortran Compilers						C Compilers
	Promula	Intel (ifort)		PGF			GCC			Intel (icc)
	6.5	9.1	10.1	5.2	6.0	8.0	3.2.X	3.4.X	4.1.X	10.1
X-Midas 4.8.X	Yes⁽¹⁾	Yes	Yes	Yes⁽¹⁾	Yes⁽¹⁾	Yes	Yes	Yes	Yes	Yes
X-Midas 4.6.X	Yes⁽¹⁾	Yes	Yes	Yes⁽¹⁾	Yes⁽¹⁾	Yes	Yes	Yes	Yes	Yes
The below X-Midas version are no longer supported.
X-Midas 4.4.X	Yes⁽¹⁾	Yes	Yes	Yes	Yes	Yes	Yes	Yes	Yes	Yes
X-Midas 4.2.X	Yes	No	No	Yes⁽¹⁾	Yes⁽¹⁾	No	Yes	Yes⁽¹⁾	No	No
X-Midas 3.8.3	Yes	No	No	No	No	No	Yes	No	No	No
X-Midas 3.7.1	No	No	No	No	No	No	No	No	No	No
X-Midas 3.6.5	No	No	No	No	No	No	No	No	No	No
All versions require Java 7 or later
⁽¹⁾ User tested.

Current Capabilities

As of this writing, NeXtMidas primitives run from the X-Midas shell can:

Read from X-Midas pipes.

Limitations:	NeXtMidas supports dynamic allocation and connection of pipes. This is not supported when using X-Midas pipes. The NeXtMidas primitive must open an X-Midas pipe inside its' `open()` method, otherwise the primitive will not see the start of the pipe. On the NeXtMidas side the pipe name must be prefixed with `xmpipe:`.

Create and write to X-Midas pipes.

Limitations:

Limitations:	NeXtMidas supports dynamic allocation and connection of pipes. This is not supported when using X-Midas pipes. NeXtMidas has no control over the size of the pipes (pipe size is set by X-Midas). When a NeXtMidas primitive is creating an X-Midas pipe, it must set the file size (note that this is not the same as pipe size) before it opens the pipe. The NeXtMidas primitive must open an X-Midas pipe inside its' `open()` method. The X-Midas `PIPE` or `XPIPE` must have the `/NP=` set to a value large enough to account for the pipes created by NeXtMidas primitives. On the NeXtMidas side the pipe name must be prefixed with `xmpipe:`.

NeXtMidas supports dynamic allocation and connection of pipes. This is not supported when using X-Midas pipes.
NeXtMidas has no control over the size of the pipes (pipe size is set by X-Midas).
When a NeXtMidas primitive is creating an X-Midas pipe, it must set the file size (note that this is not the same as pipe size) before it opens the pipe.
The NeXtMidas primitive must open an X-Midas pipe inside its' open() method.
The X-Midas PIPE or XPIPE must have the /NP= set to a value large enough to account for the pipes created by NeXtMidas primitives.
On the NeXtMidas side the pipe name must be prefixed with xmpipe:.

Read an X-Midas pipe created by another NeXtMidas primitive. (This will, however, be slower than just using NeXtMidas pipes.)

Read/write results from/to the X-Midas results table.

Limitations:	NeXtMidas supports result groups or collections called TABLES and can have result objects. These features are not supported in the X-Midas results table.

Read and set the AUX list to find and create files. NeXtMidas has all capabilites X-Midas has to dynamically read and set the AUX list.

Limitations:	NeXtMidas supports AUX names as strings, but X-Midas only allows numbers. NeXtMidas supports URLs but X-Midas only allows local directories.

Send messages from X-Midas to NeXtMidas and send messages from NeXtMidas to X-Midas.

Limitations:	The NeXtMidas macro/primitive to receive the X-Midas message must have been started from an X-Midas macro with "`NM <command>/ID=<id>`".

Import the X-Midas option tree at the start of the JVM (nm/on/bg/id=999).
Learn the position of graphical NeXtMidas primitives relative to X-Midas graphic windows.

All NeXtMidas commands default to using NeXtMidas pipes. This is because the NeXtMidas pipes provide better performance and more flexibility than X-Midas pipes. If you want to use an X-Midas pipe from NeXtMidas you need to prefix the X-Midas pipe name with xmpipe: (e.g. use xmpipe:_mypipe to read the X-Midas pipe named _mypipe).

Building

Building NeXtMidas as an X-Midas option tree is simple:

Define NMROOT, the root install directory of NeXtMidas. On Unix for example,
```
$ setenv NMROOT /home/smith/nxm210 
```
Add the NeXtMidas SYS directory as an X-Midas option tree named NXM
```
X-Midas> xmopt nxm $NMROOT/nxm/sys 
```
Add NXM to your path
```
X-Midas> xmp +nxm 
```
Build the UCL option
```
X-Midas> xmbopt ucl
```
Next, build the NXM option
```
X-Midas> xmbopt nxm
```

Run the confidence tests.

X-Midas> home $NMROOT/nxm/sys/test
X-Midas> mtrans %confidence
X-Midas> %confidence

Note: Always use XMBOPT NXM command to build the NXM option tree. Other build commands will not properly build and link the JNI libraries required.

Running NeXtMidas Commands from X-Midas

An example macro is:

X-Midas> sd360/nmplot 1e9

Look for additional examples in $NMROOT/nxm/sys/mcr/ or on the Using NeXtMidas as an X-Midas Option Tree Demos page.

The NM command is used to start a Java Virtual Machine (JVM) in which all subsequent NeXtMidas commands are launched. The steps for running a NeXtMidas command are:

Turn on the JVM
```
X-Midas> nm/on/bg/id=999 
```
Note: The /ID=[n] switch is optional but is required for subsequent commands to send and receive messages. The value of the ID is not important.

Run the NeXtMidas command(s)

X-Midas> nm plot nxm.sys.dat.world|nxm.sys.dat.coneflags5000

X-Midas> nm plot nxm.sys.dat.Sunset.jpg

X-Midas> nm plot nxm.sys.dat.alaska view=xyz

Turn off the JVM
```
X-Midas> nm/off
```

Inside an X-Midas macro this would look like:

startmacro
  ...
  nm/on/bg/id=999  ! Start the JVM, permitting messages with /ID switch
  xpipe/setup on
    ...
    ! Run some NeXtMidas commands
    nm plot nxm.sys.dat.world
    ...
  xpipe off
  nm/off    ! Stop the JVM
  ...
endmacro

Using X-Midas Results from NeXtMidas

The global table XM in NeXtMidas maps directly to the X-Midas results table.

X-Midas
X-Midas> res temp 123.0
NeXtMidas
nM> res/global xm.temp
D: XM.TEMP      = 123.0

Using X-Midas Pipes from NeXtMidas

NeXtMidas can directly use X-Midas pipes. To prevent confusion all X-Midas pipes are given an xmpipe: prefix in NeXtMidas. For example:

        startmacro
          nm/on/bg/id=999  ! Start the JVM, permitting messages with /ID switch
          xpipe/setup on
            waveform _mypipe sf 1e9
            nm plot xmpipe:_mypipe
          xpipe off
          nm/off    ! Stop the JVM
        endmacro

Using X-Midas Messages from NeXtMidas

X-Midas messages are automatically mapped to NeXtMidas messages using the XBC message mapping tools (see XBC: Sending Messages to a Host Primitive for details). Many examples are included in the following files:

$NMROOT/nxm/sys/test/test_xmmessage.txt - X-Midas Macro
$NMROOT/nxm/sys/test/test_xmmessage.mm - NeXtMidas Macro
$NMROOT/nxm/sys/prim/testxmmsg.java - NeXtMidas Primitive

Using X-Midas AUX List from NeXtMidas

The NeXtMidas AUX list is directly linked to the X-Midas AUX list. Any changes made in NeXtMidas show up in X-Midas and any changes made in X-Midas show up in NeXtMidas.

In addition NeXtMidas maps the familiar AUX names to their respective X-Midas counterparts:

NeXtMidas X-Midas

HOME Mc.home_path

CWD 0

HOMEPATH 98

DAT 99

RAM *

XMPIPE *

* X-Midas has no parallel to RAM or XMPIPE. These are special AUX entries that only NeXtMidas can see. Since they are required for the correct functioning of the pipe system, they will always be included on the READ AUX in NeXtMidas.

NeXtMidas	X-Midas
HOME	Mc.home_path
CWD	0
HOMEPATH	98
DAT	99
RAM	*
XMPIPE	*
* X-Midas has no parallel to RAM or XMPIPE. These are special AUX entries that only NeXtMidas can see. Since they are required for the correct functioning of the pipe system, they will always be included on the READ AUX in NeXtMidas.

Note that AUX entries 0, 98 and 99 have special handling in X-Midas. In particular, changes to AUX 0 made after NeXtMidas has been started (e.g. via a cd in the shell) will not be visible to NeXtMidas.

Concept and Design

The design is loosely modelled after the X-Midas RSM concept. That is to turn on the JVM, run the Java commands, and then turn off the JVM. The NM command uses Inter Process Communication (IPC) to signal events between X-Midas and the shared JVM.

Turning on the JVM - This starts a Java Virtual Machine (JVM) and connects to the X-Midas global sections for access to the X-Midas results table, message queue and pipes.
Running a Command - The JVM is capable of interpreting NeXtMidas commands and running them in the same (single) JVM via IPC. The NeXtMidas command in run in its own Java thread inside the JVM. It has access to the X-Midas resources.
Turning off the JVM - This signals (via IPC) the JVM to clean up and shut down

The NM;NXM command has the following switches:

        Global Switches
        /BG     - Runs the command in the background with respect
                  to the X-Midas shell.
        /PN=str - Sets the process name.
        /XS=n   - Allows a graphical NeXtMidas primitive to LEARN
                  window positions (see XPIPE).

        Standard Switches
        /ON     - Turns the shared JVM on (must be used with /BG).
        /OFF    - Turns the shared JVM off.
        /STATUS - Reports the status of the shared JVM.

        Debugging Switches
        /DEBUG  - Turns on general debugging messages.
        /JDEBUG - Turns on debugging messages in the JNI code.

        Special-Purpose Switches
        /CP=str       - Adds elements on the front of the classpath.
        /JARS=str     - Adds JAR files to the end of the classpath.
        /VERBOSEJNI   - Sets the "-verbose:jni" flag for the Java VM.
        /VERBOSEGC    - Sets the "-verbose:gc"  flag for the Java VM.
        /VERIFYALL    - Sets the "-Xverify:all" flag for the Java VM.
        /XCHECKJNI    - Sets the "-Xcheck:jni"  flag for the Java VM.
        /JVMOPTIONS=s - Sets special Java VM flags.

Debugging / Troubleshooting

Usually debugging Java applications it much easier than debugging those written in C, C++ or Fortran. This is because Java has a well-developed exception handling model that allows the application to detect and (frequently) recover from most errors.

When running as an X-Midas option tree NeXtMidas needs to go outside of the standard Java safety-net in order to interact with X-Midas. This makes is much easier for errors to occur that will result in core dumps or the application hanging. While every effort has been made to aviod this situation, developers are likely to see it from time to time. Below is the list of the things most-likely to cause problems:

NeXtMidas primitive causes an exception in the open() method.
-- Causes application to hang while waiting at the X-Midas M$SYNC.
NeXtMidas primitive uses an X-Midas pipe incorrectly (see list of limitations above).
-- Causes a Signal 11 (core dump) inside X-Midas.
IPC is not responding or out of memory (either due to previous core dumps or other processes using IPC incorrectly).
-- Causes application to die following a WARNING: JNI.C: Got back a strange answer via IPC message or an IPC error message saying it could not create a message queue because it is out of memory.

To help identify where things are going wrong, add the /JDEBUG/DEBUG switches to the NM command. This will turn on normal debugging (/DEBUG) plus debugging for the JNI code (/JDEBUG). Most of the time this will help determine the point where the problem first occurs.

What to do to recover from a problem depends on the type of problem. Here are some possible solutions to common problems:

Problem: Can't Run NeXtMidas Commands
Always try running the following command first:
```
X-Midas> NM/OFF
```
Problem: Application Hangs
If running in an XPIPE try using END MACRO first. If you are able to use END MACRO, run NM/OFF at the X-Midas prompt (this will usually clean up everything). If not in an XPIPE or END MACRO does not work, use CONTROL+C to kill the current process. After a CONTROL+C kill all processes named nm.exe and run:
```
$ $NMROOT/os/unix/ipcclean
```
Problem: Application Core Dumps
Exit X-Midas with XMEND, kill all processes named nm.exe and run:
```
$ $NMROOT/os/unix/ipcclean
```
Problem: IPC is Not Responding or IPC is Out of Memory
Try this first: Exit X-Midas with XMEND, kill all processes named nm.exe and run:
```
$ $NMROOT/os/unix/ipcclean
```
If that doesn't work: Stop all X-Midas sessions with XMEND, kill all stale X-Midas processes, close other applications you are using (i.e. any started with your USER ID) using IPC and run:
```
$ $NMROOT/os/unix/ipcclean
```
Problem: Getting "WARNING: jni$startjsm: No space left on device" message when running NM/ON/BG/ID=999
You have hit the max number of message queue from Linux (default of 16). Run ipcs -l to see your current OS limit for number of message queues. e.g.
```
$ ipcs -l
...
------ Messages: Limits --------
max queues system wide = 16                 // kernel.msgmni
max size of message (bytes) = 65536         // kernel.msgmnb
default max size of queue (bytes) = 65536   // kernel.msgmax
            
```
Edit /etc/sysctl.conf (requires root/sudo access) and change/add the kernel.msgmni property, e.g.
```
# increase the number of message queues (from default of 16)
kernel.msgmni = 1024
```
Run the following to re-load the updated /etc/sysctl.conf (requires root/sudo access):
```
$ sysctl -p
```
Problem: Getting a "too large for message queue" ERROR and X-Midas macro goes into suspend mode.
You have hit the max default message queue size allocated for your X-Midas macro (normally 512 slots of 8-bytes = 4096 bytes).
You can increase the default message queue size using the /NMQ= switch on your startmacro line or your message up line in your macro. Since NeXtMidas 2.7.1, messages sent from NeXtMidas Primitives are automatically translated and normally takes up 540 slots (4320 bytes) and forward to the interested X-Midas macro/command.
It is very common for X-Midas macros that needs to send and receive messages from NeXtMidss to use a larger X-Midas message queue size. For example the $NMROOT/nxm/sys/mcr/sd360.txt macro sets message queue to 8K slots (or 65536 bytes)
```
startmacro/nmq=8k/ps=128k
```
-- or on the message up line in your .txt macro:
```
message/nmq=8k up
```

Top Ten Things To Remember

These are the top ten tips to remember when running NeXtMidas commands from X-Midas.

X-Midas will not let you use AUX paths or NMROOT with upper-case letters, URLs, or length greater than 80 characters.
NeXtMidas pipes are the default in NeXtMidas, use the xmpipe: prefix to use an X-Midas pipe in NeXtMidas. Also make sure that /NP= is large enough to account for the pipes created by NeXtMidas.
NeXtMidas messages are the default in NeXtMidas, when sending a message to X-Midas use ID=XM.<XmId> and remember to include ARGS= and TYPE= in the table for unconfigured messages.
X-Midas only allows numbered AUXs.
X-Midas does not allow NeXtMidas commands run from an X-Midas macro to have a named ID.
X-Midas messages are painful to use. NeXtMidas trys to make them earier to use but it can't do magic (at least not in this release).
When run from X-Midas the normal NeXtMidas startup files (including nmstartup.mm) are bypassed.
X-Midas option trees and xmpath are automatically imported into NeXtMidas
Use X-Midas /pname=<desired_process_name> switch to name your the NM;NXM process, e.g. NM/ON/BG/ID=999/PNAME=NXM_JVM to name the shared JVM for NeXtOpt "NXM_JVM".
Don't forgot to call NM/OFF to shutdown NeXtMidas and the shared JVM.