7.5.3: Specifications for DEIMOS Output Data Products

The tangible products with scientific value from an astronomical instrument are the data. These data return home with the investigator for further analysis. In keeping with astronomical convention, all data products from DEIMOS shall be stored within FITS files.

7.5.3.1: Required Data Products

The primary data product from DEIMOS is a CCD image containing either a direct image or spectra from the slitlets. This image is stored in FITS format. DEIMOS uses the newer Keck Data Acquisition System (DAS) rather than the older one long in use at Lick Observatory. The Keck DAS permits such FITS files to be stored on disk and/or tape.

The identities of the objects in each spectrum on the CCD image cannot be documented without including auxiliary information. This information is based upon the observer's original input catalog and upon the slitmask configuration which was used for the observation. It provides a complete description of the mapping between spectra, slitlets, objects and their celestial coordinate locations. This auxiliary information shall be stored in a FITS ASCII table extension.

The archivally correct scheme to keep the FITS table extension properly associated with the FITS image would be to append the table extension to the FITS header and data unit (HDU) for the primary image. Unfortunately most current FITS image readers ignore anything beyond the primary HDU (PHDU); when reprocessing such an image in place on disk they will typically destroy any extension HDUs. Also, the Keck DAS is not currently designed to append arbitrary FITS extensions to the image HDUs. For these reasons we choose to create the table extension for DEIMOS in a separate FITS file.

In order to document the relationships between these separate files the FITS Grouping Convention shall be used. The grouping convention requires that an additional FITS table be created. This 'GROUPING' table contains a list of all the related HDUs. In the case of the current DEIMOS data the list of related HDUs would be

The primary HDU(s) containing the components of the CCD mosaic image
The overall FITS GROUPING table (lists all others)
The image FITS GROUPING table (lists image HDUs)
The ASCII table containing input star/galaxy catalog
The ASCII table containing the slitmask description
The ASCII table containing the CCD defects
The ASCII table(s) containing status & event logs for the observation
ASCII tables pointing to calibration frames for pipeline reduction

Other tables could also be included or referenced. For example:

pointers to calibration frames (dark, flat, bias, comparison lamps) as part of a scheme of pipeline data reduction
pointers to other frames taken of the same field (as designated during the mask design process)

7.5.3.2: Existing Practice

During an observing session the Keck DAS produces FITS files on disk with names such as /full/path/prefixNNNN.fits. The full directory path (/full/path/) can be modified and is stored in the FITS header as the string value of the keyword OUTDIR. The file name (prefix) can also be modified and is stored in the FITS header as the string value of the keyword OUTFILE. The sequence number (NNNN) is always zero-padded to 4 digits and is stored in the FITS header as the integer value of the keyword FRAMENO (and for compatibility with the Lick DAS also as OBSNUM).

The sequence number typically increments by one after each saved CCD readout, but the observer is allowed to modify this sequence number such that more than one FITS file has the same value of FRAMENO. There is thus no guarantee of a single unique identifier for each image, but in practice it is extremely uncommon for an observer so to modify the sequence number.

7.5.3.2.1: A point of difficulty

The FITS Grouping Convention works well when all HDUs are stored within the same FITS file. It also works well when HDUs are stored in separate named files on a directory-structured file system with Uniform Resource Identifier (URI)-like characteristics. Because FITS tapes lack a universal scheme for associating files with file names the grouping convention can only work well if the tape reader uses the FRAMENO, OUTFILE, and OUTDIR keywords to reconstruct a file name. One possible solution would be to adopt the NOAO convention of storing the file system name in the PHDU using the FITS keyword IRAFNAME. Other solutions are in use by other projects.

For DEIMOS the FITS file containing the grouping table and other table extensions should be prepended to the image data by the FITS Grouper tool. In the long run the lack of a single FITS keyword which can serve as a unique, file system-oriented identifier in the Keck DAS needs to be addressed.

7.5.3.3: Other Requirements

While these images are stored on disk their file names can also be used to indicate their association. The tables can be stored in a file whose name is clearly associated with the image. As an example, for image d0001 the files containing the n image sections shall be d0001_n.fits grouping table and the other tables shall be d0001.dei.

7.5.3.3.1: Grouping keywords in the CCD image HDU

The guidelines of the FITS grouping convention do not require the FITS file containing the CCD image to contain indications that it is a member of a group. However, the grouping convention strongly recommends that two additional FITS keywords be added to the image header. These keywords provide the indication that the primary HDU is a member of a group by pointing to the grouping table that contains the PHDU.

GRPID1 = -1: This FITS grouping keyword requires an integer value. This indicates that the HDU is a member of a group. The negative sign indicates that the grouping table resides in a different FITS file; therefore there must also be a GRPLC1 keyword. Taking ABS(-1) indicates that the grouping table to which this HDU belongs has EXTVER = 1.
GRPLC1 = 'prefixNNNN.dei': This FITS grouping keyword gives a partial URI (in this case the name of another file in the same directory) in which the grouping table resides.

Using GRPIDn and GRPLCn a FITS HDU may indicate its membership in as many as 999 groups. In this current specification the DEIMOS PHDU belongs to only one. As we develop the ideas of the automated data reduction pipeline we me see further relationships between the image frames and, e.g., calibration frames. These could be documented by membership in and pointing to other groups.

7.5.3.3.2: Structure of the FITS file with the tables

The FITS grouping file must consist of several different HDUs. The detailed structure of each HDU is indicated within the links.

A primary HDU with zero-size image: FITS requires that a PHDU be at the beginning of all files.
An overall grouping table: Documents the relationship of the CCD image and the other FITS HDU(s).
The input catalog table: This reiterates the total content of all input catalogs provided by the user. If the assignment procedure creates apertures at positions not contained in the input catalogs (e.g., ``sky slitlets'') then the celestial coordinates of those positions are also included.
The slitmask configuration table: This contains a description of the apertures (to be) cut into the slitmask which was in use during a spectral observation. Model information which was used to plan the observation and cut the slitlets. It should contain slit identifiers, object identifiers and much more. In particular, the WCS information used by the quick look tool to report wavelength, object ID and position should be stored here. [It has not yet been defined]
The DEIMOS hardware table(s): Documents the DEIMOS model used in the software. Specifically, such things as the refraction coefficients used in the design of the mask, the distortion coefficients describing the WCS transformations, etc. [These have not yet been defined]

Section 7.5.3.5 contains a data dictionary that describes the DEIMOS-specific keywords in the above HDUs. It is essential reading for understanding of the DEIMOS and its data products.

Back to the DEIMOS home page.

Steve Allen <sla@ucolick.org>

$Date: 1996/03/18 20:01:26 $