7.7: Data Storage Format, Additional Resources Needed


7.8: Data Storage Format, Interdependencies

The data storage formats represent the scientific end product of all other components of DEIMOS. The telescope, spectrograph, CCD controllers, and environmental monitors provide inputs which must be classified and named. Many components of the relational database described in Chapter 9 must be isomorphic with FITS keywords and tables. The Image Display and Quick Look tools in Chapter 8 use these formats as their data inputs. The Pipeline Data Reduction of Chapter 11 depends on the storage formats to document everything that is relevant for automated operation.

7.9: Data Storage Format, Concerns

The storage schemes for DEIMOS images described above are a compromise between archival correctness and existing technology. The images from different CCDs are combined in a manner which keeps the related data together at the expense of WCS considerations because software for handling the FITS grouping conventions is not yet commonplace.

7.9.1: FITS World Coordinate System Issues

The proposed FITS WCS conventions (see URL=ftp://fits.cv.nrao.edu/fits/documents/wcs) do not consider the case of storing data from a multiple CCD detector array into a single FITS image. If the current FITS WCS and grouping proposals were well-established and tested then the right way to store the images would be as 8 separate FITS header and data units (HDUs). Each HDU would have WCS keywords relating it to a coordinate system in the focal plane of the dewar. Each FITS file would also contain FITS grouping tables. The ensemble of FITS HDUs, whether stored in separate files or concatenated together, would be a completely self-documenting data structure with standard-conforming WCS keywords.

The scheme proposed for DEIMOS blurs the distinction between different CCDs when it stores their data into single FITS HDUs. This may confuse some existing FITS viewers, but at the same time it will permit existing FITS viewers to display entire DEIMOS images. A better solution might be to propose and implement a standard for the viewing of mosaicked CCD images, but this seems to be beyond the scope of the DEIMOS project.

7.9.2: Multi-HDU FITS files vs. separate FITS files

Whenever the grouping of images is hierarchical it is possible for an application to use the FITS grouping tables to gather the separated FITS HDUs into a single FITS file for easier transport.

Naive image processing systems (almost all existing systems) do not handle FITS tables appended to FITS images. The typical action of most existing FITS readers would be to ignore appended tabular information when reading a file and thus to omit it when writing out processed frames.

The scheme described for DEIMOS chooses to use separate FITS files for images and tables. There is a tradeoff here. Placing this tabular information in an auxiliary file instead of as tables appended to each image file makes it less likely that the tables will be inadvertently destroyed by a naive image processing system. However it becomes more likely that the files might be separated from each other by a naive user. Inadvertent separation is probably less troublesome than inadvertent destruction. Recovery of the separated data files should be easier than recovery of destroyed data in FITS tables appended to images. Thus we choose separate FITS files rather than multi-HDU FITS files.

Steve Allen <sla@ucolick.org>
$Date: 1996/03/18 23:44:34 $