### Timelines - Keywords, Updating, Known Issues

A data file may have a TimeLine object that supports time adjustments ("time hacks") from the regular time delta specified in the header. This TimeLine can be populated from various forms of timeline keywords (extended header keywords), via PipeResource for incoming pipe data, or directly from a user's primitive.

#### Topics covered in this page

Enabling `CoreIO.IOOptions` designed for fixing issues with timeline propagation
The new TimeLine warnings
  What they mean and how to address the underlying issues
  How to turn the warnings off
Supported timeline keywords and how to convert between them
Modification or deletion of a DataFile's timeline Keywords
Setting or editing a TimeLine via `setTimeAt(...)` methods
TimeLine tolerance
Normal operation in a pipe

#### Enabling CoreIO.IOOptions designed for fixing issues with timeline propagation

##### There are two major known issues with timeline propagation in a pipe.

1. Pipe restarts (due to things such as fft frame size changes) can cause timeline information to be lost and time to go "back in time"
2. All keywords are naturally forwarded throughout the pipe. This is not appropriate for timeline keywords.
   - PipeResource already takes care of propagating time via setTimeAt() at each TL
   - Timeline keywords that are based on elements (rather than bytes) need to be adjusted when BPE changes in a primitive

##### Enabling the CoreIO.IOOptions via Java

    getMidas().io.setOptions("+DoNotPropagateTimelineKeywordsInPipe");
    getMidas().io.setOptions("+FixPipeRestartTimeLines");

##### Enabling the CoreIO.IOOptions via the macro language

    env set IOOptions +FixPipeRestartTimeLines
    env set IOOptions +DoNotPropagateTimelineKeywordsInPipe

Note: As of NeXtMidas 3.9.0, these fixes are not enabled by default. They do constitute a change in behavior. We are hoping our major users will vet these changes and they can become default in the near future.

#### The new TimeLine warnings - added in 3.7.2

##### What they mean and how to address the underlying issues

Previously bad TimeLine entries could be written (or read in from a file) and there was no warning to the users. A major user requested that warnings be issues, for problems like back-in-time entries.

`ERROR: Invalid TimeLine entry that occurs later in the timeline, but is back in time ...`
  This means that a timeline entry with a later offset has a "back-in-time" time
  If this is a file in a pipe and restarts are occuring, this may be caused by the problem in our PipeResource code listed above. Please try enabling the `CoreIO.IOOptions.FixPipeRestartTimeLines`
  Especially if you get an Exception(rather than a WARN: ERROR), this may also be caused by back-in-time issues caused by user code.
`ERROR: Adding in a missed TimeLine entry with an earlier offset is currently disabled`
  This means that there a timeline already exists and the offset specified for this new entry is not or greater than the last offset added
  If this is a file in a pipe and there are timeline keywords in the original source file, this is probably caused by the problem in our code where ALL keywords are propagated (including timeline keywords) and then our pipe code tries to create the TimeLine by setting offset/time at each transfer length. Please try enabling the `CoreIO.IOOptions.DoNotPropagateTimelineKeywordsInPipe`.
  If you are indeed trying to manual edit older entries in an existing TimeLine, please enable TimeLine editing (i.e. setEnableEditing(true)).
  If you are getting this warning outside of these circumstances, the problem is likely in user code.

##### Globally turning warnings messages off via Java

    getMidas().io.setOptions("+SuppressInvalidTimeLineEntryErrors");

##### Globally turning warnings messages off via the macro language

    env set IOOptions +SuppressInvalidTimeLineEntryErrors

##### Controlling warnings actions via user setTimeAt(...) API

New API allows the user to specify a notification level on the `TimeLine` warnings

    TimeLine.setTimeAt (double index, double dbpe, Time time, NotifyInvalidEntry notifyLevel)

#### Supported timeline keywords and how to convert between them

##### Supported Timeline Keywords

Original BLUE microsecond TIMELINE keyword
High Precision BLUE TC Keywords
  TCSAMPLE_#, TC_WHOLE_#, TC_FRAC_#, where # is the element (1 based)
High Precision Platinum Keywords
  TIMETAG.OFFSET, TIMETAG.TIME, TIMETAG.TIME.UNITS, TIME_EPOCH", TIME_DELTA, TIME_TICKS

##### Conversion via CONVERT command

    ! Read in BLUE TIMELINE file and write out matching file with BLUE TC Keywords
      convert timelinepenny KWCONV tcpenny{USEBLUETCTIMELINE=true} OPTIONS={}

    ! Read in BLUE TC keywords file and write out matching file with BLUE TIMELINE Keyword
      convert tcpenny KWCONV timelinefromtcpenny{USEBLUETCTIMELINE=false} OPTIONS={}

    ! Convert PLATINUM to BLUE: convert keywords and do not keep original Platinum keywords
      convert ^plat_file PLAT2BLUE ^blue_file options={TOA_SIGMA={VALUE=0.0},FOA_SIGMA={VALUE=0.0}}

##### Via Java using file qualifier

    DataFile df = new DataFile(getMidasContext(), "mydatafile.tmp{USEBLUETCTIMELINE=true}");

#### Modification or deletion of a DataFile's timeline Keywords

##### DataFile API available

`deleteTimeLineKeywords() - since NeXtMidas 3.8.0`
  deletes any timeline keywords, but preserves TimeLine format and tolerance
`updateTimeLineFromKeywords() - since NeXtMidas 3.9.1`
  Should be used directly or indirectly after updating TimeLine Keywords
  If a TimeLine exists for a DataFile, Keywords.putAll updates of TimeLine keywords will not update the timeLine and the updated keywords will not be saved in the file, unless this method is called to recreate the TimeLine with the latest keywords
`updateKeywordsAndTimeLine(Map map) - since NeXtMidas 3.9.1`
  effectively the combination of Keywords.putAll(Map) and updateTimeLineFromKeywords ()

##### Timeline keywords can also be deleted via the KEYWORD TCDEL command

##### Example update of a DataFile's BLUE TC TimeLine Keywords

        Table kwdsForUpdate = new Table("{D:TCSAMPLE_2=2.0,D:TC_WHOLE_2=2.146032E9,D:TC_FRAC_2=0.9}");
        // note original in file is D:TCSAMPLE_2=3.0,D:TC_WHOLE_2=2.146032001E9,D:TC_FRAC_2=0.228808
        try (DataFile dfTCKeywords = new DataFile(M,tcOutFile)) {

          // Open the file with TC Keywords
          dfTCKeywords.setFlags(BaseFile.INPUT|BaseFile.OUTPUT);
          dfTCKeywords.open();

          // get this fie's current Keywords object and update some keywords
          Keywords kwTCFile = dfTCKeywords.getKeywordsObject();
          kwTCFile.putAll(kwdsForUpdate);

          // Since this file had a timeline, we need to update it based on the updated keywords
          // (otherwise, the actual timeline will not get updated and the updates will NOT appear in
          // the file when it is closed)
          dfTCKeywords.updateTimeLineFromKeywords();
          ...

#### Setting or editing a TimeLine via setTimeAt(...) methods

##### Example creating a TimeLine from a Table

Note: An existing TimeLine can be converted to a table via a method like `TimeLine.toTable(int decimalPlaces, double bpe)`, modified, and then used to create a new TimeLine. `DataFile.setTimeLineObject(TimeLine tl)` can then be used to swap the DataFile's TimeLine.

    public static TimeLine timelineFromTable (double delta, int bpe, Table elemTimeTable) {
      TimeLine timeline = new TimeLine(TimeLine.GROW_AS_NEEDED, delta, 0);
      for (String offset : elemTimeTable) {
        int off = Integer.parseInt(offset);
        Time timelineEntry = new Time(elemTimeTable.getString(offset));
        timeline.setTimeAt(off, bpe, timelineEntry);
      }
      return timeline;
    }

##### Correcting an earlier TimeLine entry

    timeline.setEditingEnabled(true);
    // Note: This entry must still be within range of the current TimeLine entries arrays.
    // This should not be an issue, unless the timeline has overflowed.
    boolean wasInserted = timeline.setTimeAt(offset, 1,time, NotifyInvalidEntry.WARNING);

#### TimeLine tolerance

- Determines whether the current time/offset pair arriving via setTimeAt has enough variance from the expected to be set as a TimeLine entry
- Default is 0.1 seconds
- Can be set directly using the method `TimeLine.setTolerance(double)`
- Will automatically be set appropriately when high precision timeline keywords are present
- User code that is handling the setting TimeLine for each frame (or batch of data) and not using a pipe (and indirectly PipeResouces) to handle the setting via setTimeAt methods ...
  - should make sure an appropriate tolerance is set for the TimeLine
  - should call setTimeAt each frame/batch and let TimeLine handle whether a timeline entry is added
  - should NOT do selective calling of setTimeAt based on a separate tolerance in effect in the users code, from what is in affect in TimeLine. This can cause missing or erroneous entries

#### Normal operation in a pipe

PipeResource handles propagation of the the current time
  based on a combination of the most recent adjustment due to a timeline Extended Header Keyword (if applicable) and the current element and time delta
  done at each transfer length (if the incoming data has intermediate timeline keyword entries, they will not be propagated)
  handles the restart case (but probably needs `CoreIO.IOOptions.FixPipeRestartTimeLines` set to handle it properly)
Enabling the CoreIO.IOOptions fixes mentioned earlier is recommended

#### Other Timeline related CoreIO.IOptions

`CoreIO.IOOptions.PreferBlueTCTimeLine`
  Prefer BLUE TC (Time Code) keywords as outlined in the JICD 4.2 ICD
`CoreIO.IOOptions.PreferPlatinumTimeLine`
  Prefer Platinum timeline keywords (high precision) over the NeXtMidas TIMELINE keyword (microsecond precision)

#### Your feedback is appreciated

- Experiencing issues: If you are experiencing any problems are unexpected warnings from TimeLine and timeline Keywords use, the NeXtMidas team would greatly appreciate your feedback. Please include details about your use case, details on the problem you experiences, and, if possible, instructions on how to replicate the problem.
- CoreIO.IOOptions help (or at least don't cause any problems): If you have used the CoreIO.IOOptions recommended and they work well, we would like to know this as well. We are hoping to move these fixes to default behavior.
- Documentation inadequate: Please let us know type of information is missing or is unclear.
- Unusual use case or just some information you would like to relay: Feedback is always welcome. Understanding our different users' use cases can be invaluable as we try to have NextMidas meet the needs of the community. Thanks!