7.4: Functional Requirements for DEIMOS Data Storage Formats
7.4.1: FITS conformance
FITS is the IAU-approved standard for interchange and archiving
of astronomical images; DEIMOS shall use FITS for these purposes.
Data from DEIMOS will be more complex than can be stored in the
keyword/value pairs of simple FITS images. The DEIMOS data should be
stored in a combination of FITS images and FITS tables.
7.4.1.1: FITS and Quick Look analysis
During readout the CCD controllers will assemble the data streams from
the various amplifiers in a segment of shared memory. The
quick look analysis will be done using the shared memory. The FITS
header and data need not be contiguous in memory at this stage.
Upon completion of the readout the image and headers should be written
to disk as a suitable collection of FITS files.
7.4.1.2: Optimizing FITS Grouping for processing
From an archival standpoint the various data products from a DEIMOS
readout should be stored in a single FITS file composed of many
image and table HDUs. In recognition of this
STScI has recently contributed a FITS Image Extension Kernel for IRAF.
With this kernel it becomes possible to use IRAF on FITS files that
contain multiple image and table extensions.
This facility was
created in anticipation of the NICMOS and STIS instruments for HST.
However, these HST instruments have detectors much smaller than
DEIMOS.
A single FITS file from DEIMOS of size 140 (or 280) Mbytes consisting
of multiple image HDUs is unwieldy for data processing. If a FITS
header near the beginning of the file should need to be increased in
size it would be necessary to copy all of the following bits.
Such copying would imply a need for a file locking scheme if multiple
processes were to be modifying the file.
DEIMOS should make use of FITS grouping to facilitate automated
recognition of the component FITS files.
The FITS grouping convention permits the related HDUs to be stored in
several manners. All HDUs may be stored within a single FITS file.
Alternatively, the various related HDUs can be stored separately in
different FITS files. The separate FITS files may be in different
directories or even scattered over different machines on the Internet.
DEIMOS shall store the HDUs in separate files as is best suited for
mountain top processing of the particular kind of data.
A design for such storage is given in section 7.5.
7.4.1.3: Grouping for transport
For interchange and archival purposes all the HDUs should be combined
into a single FITS file with many different HDUs one after another.
(This corresponds exactly to the datasets proposed for NICMOS and STIS.)
DEIMOS should provide a FITS Grouper tool which can recognize the
component FITS files of typical DEIMOS groups and then combine them
into single archival files.
The Grouper should recognize and suitably
rewrite all of the mosaic-specific image section and WCS keywords.
Note that if the FITS images have been
processed into floating point these archival FITS files will be as large as
280 Mbytes.
7.4.1.4: Ungrouping after transport
DEIMOS should provide a UnGrouper which can be used to split
archivally-packed DEIMOS data into components suited for
data processing. By default the UnGrouper should unpack into
the same form as would be used on the mountain top. Upon
request of the user the UnGrouper should split the HDUs
into other combinations as may best suit the available data
processing programs.
The UnGrouper should recognize and suitably
rewrite all of the mosaic-specific image section and WCS keywords.
7.4.2: Mosaic geometry
There should be FITS keywords whose values give a complete description of
the geometry and origin of the pixel data.
A design for such keywords follows in section 7.5.
7.4.3: World Coordinate Systems
Each image produced by DEIMOS should be accompanied by a set of
WCS keywords which describe the conversion from pixel coordinates to
sky coordinates. There should be separate WCS information for the section
of image from each CCD. A design for such keywords follows in section 7.5.
For spectral images the multislit WCS information is too complex to store
in keyword/value pairs. The WCS information for the slitlets should
be stored in a FITS table that is appended to the data from the CCD readout.
7.4.4: Documentary metadata
In general, DEIMOS CCD images should have
appended metadata which completely document all measured
quantities. The metadata should permit reconstruction of all
elements of an observation aside from those of the environment.
Because of requirements for data acquisition and quick look the
designed characteristics of the slitmask for each DEIMOS spectral
image will already be in the database (see Chapter 9).
These data will include
the input catalog and the designed slitlet layout. Tables containing
these data should be appended to the data produced by each spectral
CCD readout. It should be possible to take a DEIMOS image from an
archive and use the included tables re-fabricate an identical slitmask,
or to redesign with modifications.
The DEIMOS Guider requirements specify that it should be possible to
obtain a co-added frame of the guide image during a CCD exposure.
They also require that the Guider should maintain a history of guiding
offsets, sky background values, guide star brightness values, etc.
All of these data should be appended to the data from each CCD
readout either as image extensions or event-log tables.
The information management component of the DEIMOS design is described
in Chapter 9. This database will be recording events throughout each
CCD exposure. Most of these events are environmental.
Other data represent long-term status or parameterizations of
physical models used during the observation.
At the end of each exposure the database should be queried and
FITS tables containing logs of relevant events and status values
should be appended to the CCD image data.
Steve Allen <sla@ucolick.org>
$Date: 1996/03/20 08:38:46 $