Timelines - Keywords, Updating, Known Issues

A data file may have a TimeLine object that supports time adjustments ("time hacks") from the regular time delta specified in the header. This TimeLine can be populated from various forms of timeline keywords (extended header keywords), via PipeResource for incoming pipe data, or directly from a user's primitive.

Topics covered in this page

Enabling CoreIO.IOOptions designed for fixing issues with timeline propagation
The new TimeLine warnings
What they mean and how to address the underlying issues
How to turn the warnings off
Supported timeline keywords and how to convert between them
Modification or deletion of a DataFile's timeline Keywords
Setting or editing a TimeLine via setTimeAt(...) methods
TimeLine tolerance
Normal operation in a pipe


Enabling CoreIO.IOOptions designed for fixing issues with timeline propagation


There are two major known issues with timeline propagation in a pipe.
  1. Pipe restarts (due to things such as fft frame size changes) can cause timeline information to be lost and time to go "back in time"
  2. All keywords are naturally forwarded throughout the pipe. This is not appropriate for timeline keywords.
    • PipeResource already takes care of propagating time via setTimeAt() at each TL
    • Timeline keywords that are based on elements (rather than bytes) need to be adjusted when BPE changes in a primitive
Enabling the CoreIO.IOOptions via Java
getMidas().io.setOptions("+DoNotPropagateTimelineKeywordsInPipe");
getMidas().io.setOptions("+FixPipeRestartTimeLines");
    
Enabling the CoreIO.IOOptions via the macro language
env set IOOptions +FixPipeRestartTimeLines
env set IOOptions +DoNotPropagateTimelineKeywordsInPipe
    

Note: As of NeXtMidas 3.9.0, these fixes are not enabled by default. They do constitute a change in behavior. We are hoping our major users will vet these changes and they can become default in the near future.



The new TimeLine warnings - added in 3.7.2


What they mean and how to address the underlying issues

Previously bad TimeLine entries could be written (or read in from a file) and there was no warning to the users. A major user requested that warnings be issues, for problems like back-in-time entries.

ERROR: Invalid TimeLine entry that occurs later in the timeline, but is back in time ...
This means that a timeline entry with a later offset has a "back-in-time" time
If this is a file in a pipe and restarts are occuring, this may be caused by the problem in our PipeResource code listed above. Please try enabling the CoreIO.IOOptions.FixPipeRestartTimeLines
Especially if you get an Exception(rather than a WARN: ERROR), this may also be caused by back-in-time issues caused by user code.
ERROR: Adding in a missed TimeLine entry with an earlier offset is currently disabled
This means that there a timeline already exists and the offset specified for this new entry is not or greater than the last offset added
If this is a file in a pipe and there are timeline keywords in the original source file, this is probably caused by the problem in our code where ALL keywords are propagated (including timeline keywords) and then our pipe code tries to create the TimeLine by setting offset/time at each transfer length. Please try enabling the CoreIO.IOOptions.DoNotPropagateTimelineKeywordsInPipe.
If you are indeed trying to manual edit older entries in an existing TimeLine, please enable TimeLine editing (i.e. setEnableEditing(true)).
If you are getting this warning outside of these circumstances, the problem is likely in user code.
Globally turning warnings messages off via Java
getMidas().io.setOptions("+SuppressInvalidTimeLineEntryErrors");
    
Globally turning warnings messages off via the macro language
env set IOOptions +SuppressInvalidTimeLineEntryErrors
    
Controlling warnings actions via user setTimeAt(...) API

New API allows the user to specify a notification level on the TimeLine warnings

TimeLine.setTimeAt (double index, double dbpe, Time time, NotifyInvalidEntry notifyLevel)
    


Supported timeline keywords and how to convert between them


Supported Timeline Keywords
Original BLUE microsecond TIMELINE keyword
High Precision BLUE TC Keywords
TCSAMPLE_#, TC_WHOLE_#, TC_FRAC_#, where # is the element (1 based)
High Precision Platinum Keywords
TIMETAG.OFFSET, TIMETAG.TIME, TIMETAG.TIME.UNITS, TIME_EPOCH", TIME_DELTA, TIME_TICKS
Conversion via CONVERT command
! Read in BLUE TIMELINE file and write out matching file with BLUE TC Keywords
  convert timelinepenny KWCONV tcpenny{USEBLUETCTIMELINE=true} OPTIONS={}

! Read in BLUE TC keywords file and write out matching file with BLUE TIMELINE Keyword
  convert tcpenny KWCONV timelinefromtcpenny{USEBLUETCTIMELINE=false} OPTIONS={}

! Convert PLATINUM to BLUE: convert keywords and do not keep original Platinum keywords
  convert ^plat_file PLAT2BLUE ^blue_file options={TOA_SIGMA={VALUE=0.0},FOA_SIGMA={VALUE=0.0}}
    
Via Java using file qualifier
DataFile df = new DataFile(getMidasContext(), "mydatafile.tmp{USEBLUETCTIMELINE=true}");
    


Modification or deletion of a DataFile's timeline Keywords


DataFile API available
deleteTimeLineKeywords() - since NeXtMidas 3.8.0
deletes any timeline keywords, but preserves TimeLine format and tolerance
updateTimeLineFromKeywords() - since NeXtMidas 3.9.1
Should be used directly or indirectly after updating TimeLine Keywords
If a TimeLine exists for a DataFile, Keywords.putAll updates of TimeLine keywords will not update the timeLine and the updated keywords will not be saved in the file, unless this method is called to recreate the TimeLine with the latest keywords
updateKeywordsAndTimeLine(Map map) - since NeXtMidas 3.9.1
effectively the combination of Keywords.putAll(Map) and updateTimeLineFromKeywords ()
Timeline keywords can also be deleted via the KEYWORD TCDEL command
Example update of a DataFile's BLUE TC TimeLine Keywords
    Table kwdsForUpdate = new Table("{D:TCSAMPLE_2=2.0,D:TC_WHOLE_2=2.146032E9,D:TC_FRAC_2=0.9}");
    // note original in file is D:TCSAMPLE_2=3.0,D:TC_WHOLE_2=2.146032001E9,D:TC_FRAC_2=0.228808
    try (DataFile dfTCKeywords = new DataFile(M,tcOutFile)) {

      // Open the file with TC Keywords
      dfTCKeywords.setFlags(BaseFile.INPUT|BaseFile.OUTPUT);
      dfTCKeywords.open();

      // get this fie's current Keywords object and update some keywords
      Keywords kwTCFile = dfTCKeywords.getKeywordsObject();
      kwTCFile.putAll(kwdsForUpdate);

      // Since this file had a timeline, we need to update it based on the updated keywords
      // (otherwise, the actual timeline will not get updated and the updates will NOT appear in
      // the file when it is closed)
      dfTCKeywords.updateTimeLineFromKeywords();
      ...


Setting or editing a TimeLine via setTimeAt(...) methods


Example creating a TimeLine from a Table

Note: An existing TimeLine can be converted to a table via a method like TimeLine.toTable(int decimalPlaces, double bpe), modified, and then used to create a new TimeLine. DataFile.setTimeLineObject(TimeLine tl) can then be used to swap the DataFile's TimeLine.

public static TimeLine timelineFromTable (double delta, int bpe, Table elemTimeTable) {
  TimeLine timeline = new TimeLine(TimeLine.GROW_AS_NEEDED, delta, 0);
  for (String offset : elemTimeTable) {
    int off = Integer.parseInt(offset);
    Time timelineEntry = new Time(elemTimeTable.getString(offset));
    timeline.setTimeAt(off, bpe, timelineEntry);
  }
  return timeline;
}
  
Correcting an earlier TimeLine entry
timeline.setEditingEnabled(true);
// Note: This entry must still be within range of the current TimeLine entries arrays.
// This should not be an issue, unless the timeline has overflowed.
boolean wasInserted = timeline.setTimeAt(offset, 1,time, NotifyInvalidEntry.WARNING);
    


TimeLine tolerance


Normal operation in a pipe

PipeResource handles propagation of the the current time
based on a combination of the most recent adjustment due to a timeline Extended Header Keyword (if applicable) and the current element and time delta
done at each transfer length (if the incoming data has intermediate timeline keyword entries, they will not be propagated)
handles the restart case (but probably needs CoreIO.IOOptions.FixPipeRestartTimeLines set to handle it properly)
Enabling the CoreIO.IOOptions fixes mentioned earlier is recommended


Other Timeline related CoreIO.IOptions

CoreIO.IOOptions.PreferBlueTCTimeLine
Prefer BLUE TC (Time Code) keywords as outlined in the JICD 4.2 ICD
CoreIO.IOOptions.PreferPlatinumTimeLine
Prefer Platinum timeline keywords (high precision) over the NeXtMidas TIMELINE keyword (microsecond precision)


Your feedback is appreciated