As an overview of the time-related keywords found in FITS files from the optical instruments at Keck Observatory please see this computer-generated documentary list.
In the subsequent discussion it is important to distinguish between KTL keywords and FITS keywords.
KTL keywords are obtained by interprocess (and often intermachine) communication when a client process indicates interest. The values of the KTL keywords communicate information about the state of the various subsystems of the Keck telescopes and instruments. Depending on the nature of the quantity described by the keyword it may be continuously varying during an observation, or it may never change. Each KTL keyword is associated with a "service" that corresponds to some subsystem of the operations at Keck. Some KTL services are the responsibility of the Keck staff and describe observatory-wide systems (e.g., DCS describes the telescope pointing and dome, ACS describes the primary mirror). Other KTL services are associated with the particular instrument (e.g., CCD subsystems, motor control subsystems, etc.).
FITS keywords are recorded in the headers of the files which are produced by the instrument systems when they obtain an image. In may cases there is a one-to-one correspondence between KTL keywords and FITS keywords. There are, however, a number of FITS keywords inserted into every image which do not correspond to any KTL keyword. There can also be FITS keywords which correspond to KTL keywords, but where the FITS keyword name does not match the KTL keyword.
Finally, it is relevant to note that the FITS DATE keyword indicates a time stamp related to the construction of the FITS file itself. There should be no expectation that the value of the FITS DATE keyword is related to the time at which the data in the FITS file were acquired.
During the development of the keyword handling systems at Keck Observatory there was no requirement for a method of obtaining a precise time stamp. None of the specifications for the initial optical instruments at Keck (HIRES, LRIS, ESI, DEIMOS) included any requirements for precise timing. As a result, the ability of these instruments at Keck to indicate the time for any exposure-related event is haphazard.
The Keck telescope pointing control system (DCS) uses commercial GPS receivers as the time base. The time provided by these receivers has experienced "interesting" events during the GPS W1K rollover in 1999 and during theater-level GPS jamming experiments conducted by DoD. It is not clear what the DCS system reports during a leap second. Other than in these exceptional conditions the time base used by DCS is relatively reliable.
Any of the Keck instruments is able to ask DCS for its current values of its timing keywords. Most notable among these KTL keywords are DATE-OBS, UTC, and MJD. Reading the KTL DATE-OBS keyword from DCS provides a FITS Y2K-agreement-compliant value of the calendar date according to Coordinated Universal Time. Reading the KTL UTC keyword from DCS provides a sexagesimal character string representation of the value of Coordinated Universal Time. It is important to note that these are two separate keywords. It is not possible to request both keywords simultaneously, nor to receive values which are guaranteed to refer to the same instant. As a result it is possible that around 0 hours UTC the values of the KTL DATE-OBS and UTC keywords from DCS can refer to different calendar days -- thus resulting in a one day discrepancy in the available notion of the time. (Given that 0 hours UTC occurs around mid-day in Hawaii this will probably never be an issue for observational data.)
The manner by which DCS can be queried for date and time is a KTL keyword request. In the current scheme the KTL keyword read generates a EPICS request that is sent over the network to the DCS systems. After that request is received and processed in DCS the resulting value is sent back over the network to the KTL requestor. The design of the system provides no guarantees about how long this round trip takes. Under normal circumstances the response to the KTL read is received within a small fraction of a second, but under adverse circumstances the DCS system has been observed to take as long as 30 seconds. In these adverse circumstances it is not clear how to ascertain what point in the transaction is represented by the time value.
The initial optical instruments at Keck (HIRES, LRIS, ESI, DEIMOS) monitor the sequence of events of an exposure using a process named watch_ccd. The watch_ccd process is responsible for gathering most of the KTL keywords which will be written into the FITS file along with the image data of an exposure. The watch_ccd process has a list of KTL keywords which indicates when each keyword should be gathered during the exposure sequence. The three points during the exposure are erase, shutter open, and shutter close.
In these instruments the computer which has command of the CCD is called the CCD crate. The CCD crate issues signals to the CCD controller to initiate all operations of the CCD electronics and the shutter. When the CCD crate issues exposure-related signals to the CCD controller it also transmits a MUSIC broadcast message to the traffic process. The traffic process then re-transmits that MUSIC broadcast message to every other process, one of which is watch_ccd. Each of these hops between processes on the network takes time.
When watch_ccd receives an exposure-related event it proceeds to issue KTL read requests for each of the keywords whose value is desired at that event. These KTL read requests are issued sequentially. Each request must complete before the next KTL read is done. A successful KTL read involves a round trip between watch_ccd and the system(s) which know(s) the value(s) of each keyword. Usually each KTL read succeeds in a small fraction of a second, but under adverse circumstances the servers for the keywords may respond slowly, or not respond at all.
Each KTL keyword read has a timeout. If the server does not respond within that timeout a KTL read will fail, and watch_ccd will proceed to read the next keyword in its list. There are often dozens, even hundreds of keywords in the lists that watch_ccd has to acquire. The total time to gather the keywords, even under ideal circumstances can be several seconds, and under adverse circumstances it has been monitored to be as many as 30 seconds or more. If the delay is long then the values of the KTL DATE-OBS and (more importantly) UTC keywords from DCS may not bear much relation to the instant at which the shutter events happened.
These issues regarding the validity of the time-related keywords came to the attention of the UCO/Lick Scientific Programming Group during the retrofit which added the exposure meter to HIRES. Starting with that deployment the CCD readout software has been tailored to provide the best possible indication of event times given the constraints of systems which were not specified with that as a design requirement. These principles are now in use with HIRES and DEIMOS. They will be applied to LRIS during the Red Mosaic upgrade.
The first remedy was to carefully arrange the order of the keywords in the lists provided to watch_ccd. In the more recent deployments the list has the KTL UTC keyword as the first in the list. There can be no quicker way of getting that keyword from DCS.
Another remedy was to enhance watch_ccd to distribute the keyword collection over three events rather than two. Initially the keywords were only gathered in response to shutter open and shutter close. Those lists were rather large, and not optimally arranged. In the more recent versions of watch_ccd it is possible to gather keywords in response to the intiation of the erase of the CCD prior to opening the shutter. The objective is to gather keywords whose value tends not to change during erase, and thus reduce the length of the lists of keywords desired at shutter open and shutter close.
Another remedy was to enhance watch_ccd to be able to rename a KTL keyword when writing it to the FITS file. This renaming is often done when it is desired to sample the value of a KTL keyword more than once during an observation. In the case of HIRES the KTL UTC and DATE-OBS keywords are read in response to both shutter open and shutter close. For shutter open they are written to the FITS file with the same name, and for shutter close they are written to the FITS file with the names UTC-END and DATE-END. This has been in use for all instruments deployed or upgraded since the HIRES exposure meter. (Note that in the unusual case of an exposure very near to 0 h UTC the combination of all four of these keywords can probably be used to ascertain whether the pair of keywords for one of the shutter events may have straddled the day boundary.)
The most definitive remedy was to create an independent method for watch_ccd to give some indication of the times of the shutter events. In recent versions when watch_ccd receives the shutter events it immediately queries the system to get the Unix system time. The values of these queries are inserted into the KTL keywords DATE_BEG (for shutter open) and DATE_END (for shutter close). These keywords are only as precise as one second. Furthermore, these keywords rely on the Unix system clock being set correctly.
In normal operation the Keck systems should be using NTP to keep their clocks on time, but this is not guaranteed. In particular, since the late 1990s the timekeeping on the Keck summit relied on two Datum (later Symmetricom, later Microsemi) TymServe 2100 GPS-based NTP servers. These reached end of support during 2010, but they were still in operation in 2015. When the GPS satellite broadcasts started to include the announcement of the 2015-06-30 leap second, some TymServe 2100 units began to give timestamps off by one second. It is unknown whether the Keck summit NTP service was affected by that one second offset. It is known that as of 2015-05-03 all TymServe 2100 units slipped back by a full 1024 weeks. From that date until Keck receives replacement devices it is known that the Keck summit NTP service is slow by about 150 milliseconds.
In any instrument which supplies the FITS DATE_BEG and DATE_END keywords along with the UTC and UTC-END keywords it is prudent to inspect their values. If the various subsystems were operating nominally then the keywords for shutter open will be in reasonable agreement, as will the keywords for shutter close. Furthermore, the difference between the times of shutter close and shutter open should be in reasonable agreement with the various keywords that indicate the duration of the exposure. If all of these agree, then their values are probably reliable. If they do not agree, then their discrepancies give some indication of how unreliable the time stamps may be.