7.7: Data Storage Format, Additional Resources Needed
7.8: Data Storage Format, Interdependencies
The data storage formats represent the scientific end product of all other
components of DEIMOS.
The telescope, spectrograph, CCD controllers, and environmental monitors
provide inputs which must be classified and named.
Many components of the relational database described in Chapter
9 must be isomorphic with FITS keywords and tables.
The Image Display and Quick Look tools in Chapter 8 use these formats
as their data inputs. The Pipeline Data Reduction of Chapter 11
depends on the storage formats to document everything that is
relevant for automated operation.
7.9: Data Storage Format, Concerns
The storage schemes for DEIMOS images described above are a compromise
between archival correctness and existing technology. The images from
different CCDs are combined in a manner which keeps the related data
together at the expense of WCS considerations because software for
handling the FITS grouping conventions is not yet commonplace.
7.9.1: FITS World Coordinate System Issues
The proposed FITS WCS conventions (see
URL=ftp://fits.cv.nrao.edu/fits/documents/wcs) do not consider the
case of storing data from a multiple CCD detector array into a single
FITS image. If the current FITS WCS and grouping proposals were
well-established and tested then the right way to store the images
would be as 8 separate FITS header and data units (HDUs). Each HDU
would have WCS keywords relating it to a coordinate system in the
focal plane of the dewar. Each FITS file would also contain FITS
grouping tables. The ensemble of FITS HDUs, whether stored in
separate files or concatenated together, would be a completely
self-documenting data structure with standard-conforming WCS keywords.
The scheme proposed for DEIMOS blurs the distinction
between different CCDs when it stores their data into single
FITS HDUs. This may confuse some existing FITS viewers, but at the
same time it will permit existing FITS viewers to display
entire DEIMOS images. A better solution might be to propose and
implement a standard for the viewing of mosaicked CCD images, but
this seems to be beyond the scope of the DEIMOS project.
7.9.2: Multi-HDU FITS files vs. separate FITS files
the grouping of images is hierarchical it is possible for
an application to use the FITS grouping tables to gather the separated
FITS HDUs into a single FITS file for easier transport.
Naive image processing systems (almost all existing
systems) do not handle FITS tables appended to FITS images. The
typical action of most existing FITS readers would be to ignore
appended tabular information when reading a file and thus to omit it
when writing out processed frames.
The scheme described for DEIMOS chooses to use separate FITS files
for images and tables.
There is a tradeoff here. Placing this tabular information in an
auxiliary file instead of as tables appended to each image file
less likely that the tables will be inadvertently destroyed by
a naive image processing system. However it becomes more likely that
the files might be separated from each other by a naive user.
Inadvertent separation is probably less troublesome than inadvertent
destruction. Recovery of the separated data files should be easier
than recovery of destroyed data in FITS tables appended to images.
Thus we choose separate FITS files rather than multi-HDU FITS files.
Steve Allen <firstname.lastname@example.org>
$Date: 1996/03/18 23:44:34 $