7.4: Functional Requirements for DEIMOS Data Storage Formats

7.4.1: FITS conformance

FITS is the IAU-approved standard for interchange and archiving of astronomical images; DEIMOS shall use FITS for these purposes. Data from DEIMOS will be more complex than can be stored in the keyword/value pairs of simple FITS images. The DEIMOS data should be stored in a combination of FITS images and FITS tables. FITS and Quick Look analysis

During readout the CCD controllers will assemble the data streams from the various amplifiers in a segment of shared memory. The quick look analysis will be done using the shared memory. The FITS header and data need not be contiguous in memory at this stage. Upon completion of the readout the image and headers should be written to disk as a suitable collection of FITS files. Optimizing FITS Grouping for processing

From an archival standpoint the various data products from a DEIMOS readout should be stored in a single FITS file composed of many image and table HDUs. In recognition of this STScI has recently contributed a FITS Image Extension Kernel for IRAF. With this kernel it becomes possible to use IRAF on FITS files that contain multiple image and table extensions. This facility was created in anticipation of the NICMOS and STIS instruments for HST. However, these HST instruments have detectors much smaller than DEIMOS.

A single FITS file from DEIMOS of size 140 (or 280) Mbytes consisting of multiple image HDUs is unwieldy for data processing. If a FITS header near the beginning of the file should need to be increased in size it would be necessary to copy all of the following bits. Such copying would imply a need for a file locking scheme if multiple processes were to be modifying the file. DEIMOS should make use of FITS grouping to facilitate automated recognition of the component FITS files.

The FITS grouping convention permits the related HDUs to be stored in several manners. All HDUs may be stored within a single FITS file. Alternatively, the various related HDUs can be stored separately in different FITS files. The separate FITS files may be in different directories or even scattered over different machines on the Internet. DEIMOS shall store the HDUs in separate files as is best suited for mountain top processing of the particular kind of data. A design for such storage is given in section 7.5. Grouping for transport

For interchange and archival purposes all the HDUs should be combined into a single FITS file with many different HDUs one after another. (This corresponds exactly to the datasets proposed for NICMOS and STIS.) DEIMOS should provide a FITS Grouper tool which can recognize the component FITS files of typical DEIMOS groups and then combine them into single archival files. The Grouper should recognize and suitably rewrite all of the mosaic-specific image section and WCS keywords. Note that if the FITS images have been processed into floating point these archival FITS files will be as large as 280 Mbytes. Ungrouping after transport

DEIMOS should provide a UnGrouper which can be used to split archivally-packed DEIMOS data into components suited for data processing. By default the UnGrouper should unpack into the same form as would be used on the mountain top. Upon request of the user the UnGrouper should split the HDUs into other combinations as may best suit the available data processing programs. The UnGrouper should recognize and suitably rewrite all of the mosaic-specific image section and WCS keywords.

7.4.2: Mosaic geometry

There should be FITS keywords whose values give a complete description of the geometry and origin of the pixel data. A design for such keywords follows in section 7.5.

7.4.3: World Coordinate Systems

Each image produced by DEIMOS should be accompanied by a set of WCS keywords which describe the conversion from pixel coordinates to sky coordinates. There should be separate WCS information for the section of image from each CCD. A design for such keywords follows in section 7.5.

For spectral images the multislit WCS information is too complex to store in keyword/value pairs. The WCS information for the slitlets should be stored in a FITS table that is appended to the data from the CCD readout.

7.4.4: Documentary metadata

In general, DEIMOS CCD images should have appended metadata which completely document all measured quantities. The metadata should permit reconstruction of all elements of an observation aside from those of the environment.

Because of requirements for data acquisition and quick look the designed characteristics of the slitmask for each DEIMOS spectral image will already be in the database (see Chapter 9). These data will include the input catalog and the designed slitlet layout. Tables containing these data should be appended to the data produced by each spectral CCD readout. It should be possible to take a DEIMOS image from an archive and use the included tables re-fabricate an identical slitmask, or to redesign with modifications.

The DEIMOS Guider requirements specify that it should be possible to obtain a co-added frame of the guide image during a CCD exposure. They also require that the Guider should maintain a history of guiding offsets, sky background values, guide star brightness values, etc. All of these data should be appended to the data from each CCD readout either as image extensions or event-log tables.

The information management component of the DEIMOS design is described in Chapter 9. This database will be recording events throughout each CCD exposure. Most of these events are environmental. Other data represent long-term status or parameterizations of physical models used during the observation. At the end of each exposure the database should be queried and FITS tables containing logs of relevant events and status values should be appended to the CCD image data.

Steve Allen <sla@ucolick.org>
$Date: 1996/03/20 08:38:46 $