next up previous
Next: Generating Source Code Up: History of Development (keywords/memes) Previous: Database Memes

Conversion of Data

Having reached the point where FITS headers, FITS table extensions, and Sybase tables all looked more or less alike (like bundles of memes), we were quite ready for the next step: converting between the different arbitrary realizations of the abstract concept a bundle of memes.

First, I attempted a tool which would generate a Sybase table definition (or more, as needed) for any FITS header. From the definition of the FITS header (a bundle) I would generate correct Sybase DDL to create a table or tables in which the contents of the target FITS header (or table extension) could be stored. This was an important step for DEIMOS operations, in which we hoped to establish several Sybase tables for logging operational data. The automatic generation of these tables from keyword definitions would be a major proof-of-concept. (See the main page for a demo).

Secondly, we began to consider the export of Sybase tables as FITS table extensions. It seemed a trivial mutation of existing code, so we made the attempt and quite quickly achieved a tool which would dump any meme bundle as a FITS table extension (in correct FITS card-image format). We now had a portable way of transporting and sharing Sybase table data. As long as the Sybase table had been documented in the Memes/Mbundles schema, it could be converted to FITS.

We also now had a way of rotating and flattening FITS headers. Example: we generate Sybase DDL which creates a table suitable for storing HIRES headers. We store 1000 HIRES headers in this table. We now run the utility which generates a FITS table extension from a Sybase table. The list of HIRES headers is now transformed into a single FITS table extension, serving (for example) as an index for the data from an observing run.

The ingestion of a FITS table extension back into a Sybase table is a thornier problem, since the data have now become FITS and FORTRAN I/O is required to read them reliably. However, Steve has succeeded in writing 500 lines of FORTRAN which in turn produce a bcp (bulk-copy program) flat ascii datafile and associated format file; the Sybase utility bcp can then be used to load the data into the target Sybase table in one operation.

The Sybase-to-FITS and FITS-to-Sybase data paths give us an alternate means of backing up and restoring database tables, as well as a means of sharing data which we store as Sybase tables with sites which do not use relational database engines, but are FITS-literate.


next up previous
Next: Generating Source Code Up: History of Development (keywords/memes) Previous: Database Memes

De Clarke
Mon Sep 9 16:46:16 PDT 1996