Timelines - Keywords, Updating, Known Issues
A data file may have a TimeLine object that supports time adjustments ("time hacks") from the regular
time delta specified in the header. This TimeLine can be populated from various forms of timeline
keywords (extended header keywords), via PipeResource for incoming pipe data, or directly from a user's
primitive.
Topics covered in this page
- Enabling
CoreIO.IOOptions
designed for fixing issues with timeline propagation
- The new TimeLine warnings
- What they mean and how to address the underlying issues
- How to turn the warnings off
- Supported timeline keywords and how to convert between them
- Modification or deletion of a DataFile's timeline Keywords
- Setting or editing a TimeLine via
setTimeAt(...)
methods
- TimeLine tolerance
- Normal operation in a pipe
Enabling CoreIO.IOOptions designed for fixing issues with timeline propagation
There are two major known issues with timeline propagation in a pipe.
- Pipe restarts (due to things such as fft frame size changes) can cause timeline information to be lost and time to go "back in time"
- All keywords are naturally forwarded throughout the pipe. This is not appropriate for timeline keywords.
- PipeResource already takes care of propagating time via setTimeAt() at each TL
- Timeline keywords that are based on elements (rather than bytes) need to be adjusted when BPE changes in a primitive
Enabling the CoreIO.IOOptions via Java
getMidas().io.setOptions("+DoNotPropagateTimelineKeywordsInPipe");
getMidas().io.setOptions("+FixPipeRestartTimeLines");
Enabling the CoreIO.IOOptions via the macro language
env set IOOptions +FixPipeRestartTimeLines
env set IOOptions +DoNotPropagateTimelineKeywordsInPipe
Note: As of NeXtMidas 3.9.0, these fixes are not enabled by default. They do constitute a change in behavior.
We are hoping our major users will vet these changes and they can become default in the near future.
The new TimeLine warnings - added in 3.7.2
What they mean and how to address the underlying issues
Previously bad TimeLine entries could be written (or read in from a file) and there was no warning to the users.
A major user requested that warnings be issues, for problems like back-in-time entries.
ERROR: Invalid TimeLine entry that occurs later in the timeline, but is back in time ...
- This means that a timeline entry with a later offset has a "back-in-time" time
- If this is a file in a pipe and restarts are occuring, this may be caused by the problem in our
PipeResource code listed above. Please try enabling the
CoreIO.IOOptions.FixPipeRestartTimeLines
- Especially if you get an Exception(rather than a WARN: ERROR), this may also be caused by
back-in-time issues caused by user code.
ERROR: Adding in a missed TimeLine entry with an earlier offset is currently disabled
- This means that there a timeline already exists and the offset specified for this new entry is not
or greater than the last offset added
- If this is a file in a pipe and there are timeline keywords in the original source file, this is
probably caused by the problem in our code where ALL keywords are propagated (including timeline
keywords) and then our pipe code tries to create the TimeLine by setting offset/time at each transfer length.
Please try enabling the
CoreIO.IOOptions.DoNotPropagateTimelineKeywordsInPipe
.
- If you are indeed trying to manual edit older entries in an existing TimeLine, please enable
TimeLine editing (i.e. setEnableEditing(true)).
- If you are getting this warning outside of these circumstances, the problem is likely in
user code.
Globally turning warnings messages off via Java
getMidas().io.setOptions("+SuppressInvalidTimeLineEntryErrors");
Globally turning warnings messages off via the macro language
env set IOOptions +SuppressInvalidTimeLineEntryErrors
Controlling warnings actions via user setTimeAt(...) API
New API allows the user to specify a notification level on the TimeLine
warnings
TimeLine.setTimeAt (double index, double dbpe, Time time, NotifyInvalidEntry notifyLevel)
Supported timeline keywords and how to convert between them
Supported Timeline Keywords
- Original BLUE microsecond TIMELINE keyword
- High Precision BLUE TC Keywords
- TCSAMPLE_#, TC_WHOLE_#, TC_FRAC_#, where # is the element (1 based)
- High Precision Platinum Keywords
- TIMETAG.OFFSET, TIMETAG.TIME, TIMETAG.TIME.UNITS, TIME_EPOCH", TIME_DELTA, TIME_TICKS
Conversion via CONVERT command
! Read in BLUE TIMELINE file and write out matching file with BLUE TC Keywords
convert timelinepenny KWCONV tcpenny{USEBLUETCTIMELINE=true} OPTIONS={}
! Read in BLUE TC keywords file and write out matching file with BLUE TIMELINE Keyword
convert tcpenny KWCONV timelinefromtcpenny{USEBLUETCTIMELINE=false} OPTIONS={}
! Convert PLATINUM to BLUE: convert keywords and do not keep original Platinum keywords
convert ^plat_file PLAT2BLUE ^blue_file options={TOA_SIGMA={VALUE=0.0},FOA_SIGMA={VALUE=0.0}}
Via Java using file qualifier
DataFile df = new DataFile(getMidasContext(), "mydatafile.tmp{USEBLUETCTIMELINE=true}");
Modification or deletion of a DataFile's timeline Keywords
DataFile API available
deleteTimeLineKeywords() - since NeXtMidas 3.8.0
- deletes any timeline keywords, but preserves TimeLine format and tolerance
updateTimeLineFromKeywords() - since NeXtMidas 3.9.1
- Should be used directly or indirectly after updating TimeLine Keywords
- If a TimeLine exists for a DataFile, Keywords.putAll updates of TimeLine keywords will
not update the timeLine and the updated keywords will not be saved in the file, unless
this method is called to recreate the TimeLine with the latest keywords
updateKeywordsAndTimeLine(Map map) - since NeXtMidas 3.9.1
- effectively the combination of Keywords.putAll(Map) and updateTimeLineFromKeywords ()
Timeline keywords can also be deleted via the KEYWORD TCDEL command
Example update of a DataFile's BLUE TC TimeLine Keywords
Table kwdsForUpdate = new Table("{D:TCSAMPLE_2=2.0,D:TC_WHOLE_2=2.146032E9,D:TC_FRAC_2=0.9}");
// note original in file is D:TCSAMPLE_2=3.0,D:TC_WHOLE_2=2.146032001E9,D:TC_FRAC_2=0.228808
try (DataFile dfTCKeywords = new DataFile(M,tcOutFile)) {
// Open the file with TC Keywords
dfTCKeywords.setFlags(BaseFile.INPUT|BaseFile.OUTPUT);
dfTCKeywords.open();
// get this fie's current Keywords object and update some keywords
Keywords kwTCFile = dfTCKeywords.getKeywordsObject();
kwTCFile.putAll(kwdsForUpdate);
// Since this file had a timeline, we need to update it based on the updated keywords
// (otherwise, the actual timeline will not get updated and the updates will NOT appear in
// the file when it is closed)
dfTCKeywords.updateTimeLineFromKeywords();
...
Setting or editing a TimeLine via setTimeAt(...) methods
Example creating a TimeLine from a Table
Note: An existing TimeLine can be converted to a table via a method like
TimeLine.toTable(int decimalPlaces, double bpe)
, modified, and then used to
create a new TimeLine. DataFile.setTimeLineObject(TimeLine tl)
can then be used
to swap the DataFile's TimeLine.
public static TimeLine timelineFromTable (double delta, int bpe, Table elemTimeTable) {
TimeLine timeline = new TimeLine(TimeLine.GROW_AS_NEEDED, delta, 0);
for (String offset : elemTimeTable) {
int off = Integer.parseInt(offset);
Time timelineEntry = new Time(elemTimeTable.getString(offset));
timeline.setTimeAt(off, bpe, timelineEntry);
}
return timeline;
}
Correcting an earlier TimeLine entry
timeline.setEditingEnabled(true);
// Note: This entry must still be within range of the current TimeLine entries arrays.
// This should not be an issue, unless the timeline has overflowed.
boolean wasInserted = timeline.setTimeAt(offset, 1,time, NotifyInvalidEntry.WARNING);
TimeLine tolerance
- Determines whether the current time/offset pair arriving via setTimeAt has enough
variance from the expected to be set as a TimeLine entry
- Default is 0.1 seconds
- Can be set directly using the method
TimeLine.setTolerance(double)
- Will automatically be set appropriately when high precision timeline keywords are present
- User code that is handling the setting TimeLine for each frame (or batch of data) and not using
a pipe (and indirectly PipeResouces) to handle the setting via setTimeAt methods ...
- should make sure an appropriate tolerance is set for the TimeLine
- should call setTimeAt each frame/batch and let TimeLine handle whether a timeline entry is added
- should NOT do selective calling of setTimeAt based on a separate tolerance in effect in the users
code, from what is in affect in TimeLine. This can cause missing or erroneous entries
Normal operation in a pipe
- PipeResource handles propagation of the the current time
- based on a combination of the most recent adjustment due to a timeline Extended Header Keyword
(if applicable) and the current element and time delta
- done at each transfer length (if the incoming data has intermediate timeline keyword entries,
they will not be propagated)
- handles the restart case (but probably needs
CoreIO.IOOptions.FixPipeRestartTimeLines
set to handle it properly)
- Enabling the CoreIO.IOOptions fixes mentioned earlier is recommended
Other Timeline related CoreIO.IOptions
CoreIO.IOOptions.PreferBlueTCTimeLine
- Prefer BLUE TC (Time Code) keywords as outlined in the JICD 4.2 ICD
CoreIO.IOOptions.PreferPlatinumTimeLine
- Prefer Platinum timeline keywords (high precision) over the NeXtMidas TIMELINE keyword (microsecond precision)
Your feedback is appreciated
- Experiencing issues: If you are experiencing any problems are unexpected warnings from TimeLine and timeline
Keywords use, the NeXtMidas team would greatly appreciate your feedback. Please include details about your use case,
details on the problem you experiences, and, if possible, instructions on how to replicate the problem.
- CoreIO.IOOptions help (or at least don't cause any problems): If you have used the CoreIO.IOOptions recommended
and they work well, we would like to know this as well. We are hoping to move these fixes to default behavior.
- Documentation inadequate: Please let us know type of information is missing or is unclear.
- Unusual use case or just some information you would like to relay: Feedback is always welcome. Understanding our
different users' use cases can be invaluable as we try to have NextMidas meet the needs of the community. Thanks!