Son-of-Spectro (SOS) Documentation

This document desribes the real-time reductions of spectroscopic data taken with the SDSS 2.5-m. These reductions are robotically posted to the SOS web page and archived at the Princeton mailing list.

In this document:

Related documents:

How to Launch Son-of-Spectro

Son-of-Spectro is a cron job that runs from the "observer" account on "sos.apo.nmsu.edu". It should always be running, and should even recover (as any cron job) if that machine is power-cycled. It has been run previously on the machines "hoggpt.apo" and "plate-mapper.apo".

Log into "sos.apo" as user "observer". Check if the SOS is running by typing: 

  % sos_status
If it is not running, then you will see a response beginning with: 
  The Son-of-Spectro cron job is NOT running for USER=observer MACHINE=sos.
If it is running, then you will see a response beginning with: 
  The Son-of-Spectro cron job IS running for USER=observer MACHINE=sos.
This is then followed with some other information that may be useful for trouble-shooting.

To start up SOS, type: 

  % sos_start

To stop up SOS, type: 

  % sos_stop

All of these commands can also be run from a remote machine. A second argument should be the machine name with an optional user-name, i.e.: 

  % sos_status sos
  % sos_status observer@sos.apo.nmsu.edu

Note that the following directories should exist on sos.apo and be write-able by the observer account: 

  /data/spectro
  /data/spectro/astrolog
  /data/spectro/spectrologs
  /data/spectro/spectrologs/html

Note that the following link should exist if running on sos.apo: 

  ln -s /usr/local/www/data /data/spectro/spectrologs/html
Note that the following link should exist if running on hoggpt.apo: 
  ln -s /data/spectro/spectrologs/html /usrdevel/skent/inventory/www/apo2d
Note that the following link should exist if running on plate-mapper.apo: 
  ln -s /data/spectro/spectrologs/html /home/skent/inventory/www/apo2d

Backing out to an older version

  1. You can run a different version of Son-of-Spectro by setting up the relevant version of idlspec2d before you launch it with "sos_start". This should be done from the "observer" account on "sos.apo.nmsu.edu". Check which version of the code is being run with: 
      % sos_status
    Stop this version from running with: 
      % sos_stop
  2. Wait a minute or two for any running SOS reductions to complete.
  3. The output files of different versions of Son-of-Spectro can be incompatable with one another. We recommend deleting (or at least renaming) the current working directory: 
      mv /data/spectro/spectrologs/$MJD /data/spectro/spectrologs/$MJD.old
    The file whose existence will cause a new version of code to crash is: 
      /data/spectro/spectrologs/$MJD/logfile-$MJD.fits
  4. Now set up and start the new version: 
      % setup idlspec2d v4_9_1
  % sos_start
    If you type "sos_status", it should tell you which version is now running.

Note that Son-of-Spectro is actually general enough to be run from any account on any remote computer. The only requirements are as following:

Installing a new version

Follow the instructions for CVS install w/UPS and evilinstall. This is the short-cut to installing code using UPS. It will cvs-export code, build it, install it, and declare it current. If the code is already installed, then it simply declares it the specified version current.

At APO, install both on the SOS machine (currently sos.apo), and on sdsshost.apo. This is because the documentation on the sdsshost.apo procedures web page is linked to the installed version on that machine.

Now you are ready to launch this new version of the code.

How Son-of-Spectro Works

Son-of-Spectro is a cron job that runs from the "observer" account on "sos.apo.nmsu.edu". This cron job invokes rsync once every minute to copy raw spectro data from "sdsshost" to "sos.apo", then launches a package of shell scripts and IDL commands to reduce the data one frame at a time. It is part of the idlspec2d data product.

There are actually two rsync processes that run, one for the blue camera images and one for the red. We did this so that both processors on the computer can be utilized. All 4 cameras should be reduced for an exposure within a few minutes.

The raw data is copied into these directories on sos.apo: 

  /data/spectro/astrolog/$MJD  - Copy of the same from sdsshost
  /data/spectro/$MJD           - Copy of /astrolog from sdsshost

The data reduction is not the same as the full-up spectroscopic pipeline. A number of shortcuts are taken to speed up the reductions and make them very robust. For example, boxcar extraction of the spectra is used instead of optimal extraction.

The reduced data will appear in the directory: 

  /data/spectro/spectrologs/$MJD
Files like the following can be found in that directory: 
  splog-b1-00006541.log              - Text history of exp #6541 reduction
  tset-51795-0389-00006541-b1.fits   - Fiber traces (from flat exposure #6541)
  wset-51795-0389-00006542-r1.fits   - Wavelength solution (from arc exp #6542)
  fflat-51795-0389-00006542-b1.fits  - Flat-field vectors (w/ arc exp #6542)
  sci-0389-b2-00006546.fits          - Science spectra for plate 389 (exp #6546)
  logfile-51795.fits                 - Summary info for night in FITS format
  logfile-51795.html                 - Summary info for night in HTML format
  snplot-51795-0389.ps               - Signal-to-noise plot for nplate 389
If a particular frame does not get reduced and appear on the Son-of-Spectro web site, then you can look at its log file (splog*.log). If the file does not end with the line "Finished at", then that exposure must have crashed.

The HTML files and S/N plots for the web page are copied into the directory: 

  /data/spectro/spectrologs/html
The most recent HTML file has a second copy as the file "logfile-current.html". This is the file pointed to by the SOS web page.

FITS header cards (and how to change them)

Required FITS header cards

The following FITSheader cards are required to be correct for either Son-of-Spectro or Spectro-2D to reduce the data properly. The AIRMASS is not read from the header, but computed from RADEG,DECDEG and the TAI-BEG,TAI-END keywords. 

EXPOSURE= Exposure number.  We always override the value of this keyword
          with the exposure number in the file name.
CAMERAS = Camera name, e.g. "b1".  We always override the value of this keyword
          with the exposure number in the file name.
FLAVOR  = The flavor of the observation, which can be "bias", "dark",
          "arc", "flat", and "science" or "target".  Files with other
          flavors, such as "unknown", are not reduced.
MJD     = MJD of observation, which must agree with the directory in which
          the file resides, "/data/spectro/$MJD".
PLATEID = The plate ID, which must match that in the NAME header card.
NAME    = Plug name, e.g. "0328-52277-01" for plate 328, plugged on 52277
          with a plugging ID of 01.  We assume that there is a plug-map
          file at APO with the name "/astrolog/$MJD/plPlugMapM-$NAME.par".
EXPTIME = The exposure time, which is used along with TAI-BEG to compute
          the airmass.  It is also used to determine whether the sky
          brightness or scattered light is larger than tolerances.
TAI-BEG = The TAI time for the beginning of the exposure, which is used
          to compute the airmass and sky-brightness gradients for sky-
          subtraction.  If this keyword is missing, we guess it from
          EXPTIME and TAI.
TAI-END = Used with TAI-BEG.  If this keyword is missing, we guess it from
          EXPTIME and TAI.
TAI     = This keyword is used if TAI-BEG,TAI-END are missing.
FFS     = Flat-field screen positions, which should be
          "0 0 0 0 0 0 0 0" for science exposures (all petals open)
          "1 1 1 1 1 1 1 1" for flat or arc exposures (all petals closed)
FF      = Flat-field (quartz) lamp status, which should be
          "0 0 0 0" for science or arc exposures (all off)
          "1 1 1 1" for flat exposures (all on)
NE      = Neon lamp status, which should be
          "0 0 0 0" for science or flat exposures (all off)
          "1 1 1 1" for arc exposures (all on)
HGCD    = HgCd lamp status, which should be
          "0 0 0 0" for science or flat exposures (all off)
          "1 1 1 1" for arc exposures (all on)
OBSCOMM = This keyword is used to identify the dithered flat sequence
          taken in the Monthly Checkout with the "specFlats" script in SOP.
          The flats in this sequence must have OBSCOMM="{dithered flats-flat}".
          The arcs in this sequence must have OBSCOMM="{dithered flats-arc}".
          This keyword also identifies the Hartmann focus exposures, with
          the entries OBSCOMM="{focus, hartmann l}" or "{focus, hartmann r}".
          Science exposures all have OBSCOMM="science ".
QUALITY = The observer-input quality for an exposure: "excellent", "test",
          or "bad".  Exposures are declared "test" or "bad" using the
          APOFIX procedure.

Informational FITS header cards

The following FITS header cards are informational. We would very much appreciate the observers to correct errors in these keywords so that our book-keeping of survey progress is done properly: 

TILEID  = Tile ID for this plate from the plug-map file
CARTID  = Cartridge ID number (1 through 9)
RA      = Right ascension of telescope boresight (in degrees)
DEC     = Declination of telescope boresight (in degrees)
RADEG   = Right ascension of plate center for plug-map file (in degrees)
DECDEG  = Declination of plate center for plug-map file (in degrees)
AIRTEMP = Temperature (deg C)

How to change FITS header cards

Incorrect FITS header cards for the raw spectro sdR files can be corrected by adding entries in the sdHdrFix file "/astrolog/$MJD/sdHdrFix-$MJD.par". The interesting header cards from these files can be listed with the APOHEADER procedure run from the IDL prompt on sos.apo. The proc APOFIX can be used to add entries to the sdHdrFix file to denote edits.

This structure should have the following definition in the sdHdrFix file: 

   typedef struct {
     char fileroot;    # Root of file name, without any ".fit" suffix
     char keyword;     # Keyword name
     char value;       # Keyword value (as a string)
   } ophdrfix;

Following are some examples of what should appear in the sdHdrFix files in order to correct faulty headers. (The typedef struct above must also be in the file.)

If you have edited the sdHdrFix file and need to re-reduce an exposure, you can do so with the sos_redo command. When Son-of-Spectro reads in the raw FITS files, it then patches the headers with the information in the ophdrfix entries of the sdHdrFix file. But note that the one piece of information that cannot be patched this way is MJD, since we need the MJD in order to find the correct sdHdrFix file!!

Son-of-Spectro Outputs

Tabulated values

Son-of-Spectro reduces four flavors of observations: bias/dark, flat, arc, & science/smear. Select information is tabulated for each of these types of observation. These values are tabulated in yellow if they are going out of spec, and in red if they are very much out of spec.

The values reported are:

The exact values of these yellow/red limits and further explanation can be found in the "idlspec2d" product in the file "examples/opLimits.par".

WARNING and ABORT messages

There are a number of WARNING and ABORT messages that can appear if the pipeline runs into trouble when processing a frame. These messages appear at the bottom of each table. Each one-line message begins with the relevant file name, WARNING or ABORT, then a brief plain-text message.

Note that a single problem may cascade into a large number of warning messages. For example, an out-of-focus spectrograph will first produce the "Median spatial widths" message, probably followed by warnings about bad sky-residuals.

The following messages may appear:

The following warning messages are all based upon the quality of the sky-subtraction. Typically, we are able to model the sky spectrum (from the 16 sky fibers on each spectrograph) with a relative chi^2 of around unity. At very bright sky lines, like 5577 Ang, the relative chi^2 may be as large at 25. If the relative chi^2 is large at other wavelengths, this means that there is excess light down the fibers that varies across the plate. This could be due to a light source near the telescope, or possibly a bright, non-uniform sky. Strong auroral activity is something that can do this, since the O I lines at 6300 and 6366 Ang are resolved on the sky.

Log files

If the reduction of an exposure is catastrophically bad, it may not appear at all in the Son-of-Spectro table. However, there should still be a log file for this exposure on the sos.apo computer: 

    /data/spectro/spectrologs/$MJD/splog-$CAMERA-$EXPOSURE.log
Reading this file should tell you what failed. The first and last lines of these files should contain "Started at" and "Finished at" followed by timestamps.

S/N figures

A median signal-to-noise is computed for each object in the wavelength ranges [4000,5500] Angstroms (synthetic g-band) and [6910,8500] Angstroms (synthetic i-band). We plot these S/N values versus the PHOTO fiber magnitudes, which were measured in approximately a 3-arcsec diameter aperature (the same size as our fibers). If everything is working perfectly, then our S/N values should correlate very well with these PHOTO magnitudes.

We determine whether a plate is "done" based upon the signal-to-noise of the fainter objects on the plate. We do this by fitting a line to the (S/N)-vs.-magnitude plot in a specified wavelength range, then evaluating this fit at g=20.2 mag (blue CCDs), and i=19.9 mag (red CCDs). Approximately 10% of the main galaxy sample, 60% of the BRG's and 15% of the QSO's are fainter than g=20.2. Very few objects (1, 1, and 5%) are fainter than i=19.9.

When the sky level is higher, we gain S/N more slowly at the fainter magnitudes where we are sky-limited rather than photon-limited. Without moon, we have typically found: 

  log(S/N_g) = (zeropoint) - 0.31 * g
  log(S/N_i) = (zeropoint) - 0.31 * i
With partial moon, the slope steepens to -0.34 or worse.

The fitting regions are denoted on the plot with vertical dotted lines. Arrows point to the evaluation of the fit on each of the 4 cameras, with the top panels corresponding to the blue CCD's (synthetic g-band) and the bottom panels corresponding to the red (synthetic r-band). The thick blue line on the figure is only a meaningless, fixed reference line.

The right-hand figures plot the residuals of each object from the fit. Symbol sizes on those right-hand plots indicate the magnitude of the deviation from the fit line. Symbol color is the same on the left as on the right, so green objects have more flux and red ones less. If the scale of the telescope is wrong, then you will see a radial drop-off in flux (red points on the edge of the plate). If you are observing too far over in air mass, then typically you correct to first order with a scale change, but a quadropole is left in these residuals. If one spectrograph has problems, then this will show up as red points in half of one of these figures.

Note that the (S/N)^2 totals listed in the table and the figure might not exactly agree. This is because the fitting to (S/N)-vs.-magnitude is done on individual frames for the table, but on the summed S/N over all frames for the figure. The tabulated values are the ones we use to declare a plate done.

Trouble-shooting Son-of-Spectro

The Son-of-Spectro (SOS) reduction robot has proven to be quite robust. However, if it appears to not be working one could check the following:

  1. Has a "loadCartridge" command been issued?
    If this has not been done from SOP, then there is no PLATE entry in the images. This prevents SOS from reducing anything but bias and dark frames.
  2. Is the cron job running?
    Log into "observer@sos.apo.nmsu.edu", and issue a "sos_status" command. This should report that SOS is running. If not, then restart the cron job.
  3. Is the data being copied over?
    The raw data files should be copied from "sdsshost" to "sos.apo" in the /data/spectro/$MJD directory. If it is not, then the disk may be full (see SOS disk is full warning message). Or, the rsync command may not be working. This could be due to a rogue "aporsync" process sitting around. The "sos_status" command will report any such processes. If you suspect this as the problem, just kill that process. It's a harmless thing to do, since SOS will just start it up again within a minute.
  4. Are there old "lock" files sitting around?
    SOS generates "lock" files to prevent multiple processes from changing these files simultaneously. The "sos_status" command will report all "lock" files and their last modification times. If any have been sitting around for several minutes or more, then something is wrong. You should delete the file ending with ".lock". However, this could result in a corrupted "logfile*.fits".
  5. Is the IDL license manager running?
    You can issue the following command to check that the IDL license manager is indeed running: 
        % $IDL_DIR/bin/lmstat -a
    This should display the number of licenses available, and the current number in use. The server running on galileo.apo has 110 licenses, each IDL session uses 6, so there can be at most 18 IDL sessions running concurrently. Note that Son-of-Spectro, the fiber mapper, and HoggPT all use IDL licenses.
  6. Is anything amiss with running the scripts?
    Perhaps there is a communication problem between sdsshost.apo and sos.apo, or a problem running the code on sos.apo. There is logging information written once per minute into the file 
        /data/spectro/spectrologs/sos.log
    which may indicate such problems.
  7. Is the logfile corrupted?
    This is probably the worst thing that could happen. If the file 
        /data/spectro/spectrologs/$MJD/logfile-$MJD.fits
    gets corrupted, then SOS will repeatedly crash. One could just delete this file, which would basically wipe out all the previous reductions for that night from the web page. Alternatively, one could delete all the raw and reduced data for that night from "sos.apo" (but not from sdsshost.apo!). Best to stop the SOS robot before doing this: 
        % sos_stop
    % sos_status   <-- Wait for any running processes to complete.
    % rm -rf /data/spectro/spectrologs/$MJD
    % rm -rf /data/spectro/$MJD
    % sos_start
    That might sound extreme, but SOS will simply start re-copying everything from "sdsshost" and re-build the reductions for the night. Obviously, this could take a couple of hours if its already late into the night.

Re-Reducing (Failed) Exposures

There is a mechanism to re-reduce exposures with "sos_redo". You may need to do this if a reduction failed, or the pre-calibs failed and SOS could not reduce the science exposures, or if you needed to edit the sdR header information by putting entries in the sdHdrFix file (see APOFIX).

It is fastest to only re-reduce a specific exposure number. To force a re-reduction of exposure number 12345 on all 4 cameras (whether it has already been reduced or not) use the EXPNUM option: 

  % ssh observer@sos.apo.nmsu.edu
  % sos_redo expnum=12345
If you want to find and reduce only failed reductions for a plate use the PLATE option. This may take several more minutes because all of the FITS headers for the night are parsed. To re-reduce only missing exposures for plate 666: 
  % ssh observer@sos.apo.nmsu.edu
  % sos_redo plate=666
The last exposure taken is not re-reduced unless explicitly told to with the EXPNUM option. Currently, either EXPNUM or PLATE must be specified.

Note that science exposures for a plate cannot be reduced until both flat and arc exposures have been successfully reduced for that same plate.

This is from version $Name: $ of procedures.

Maintained by David Schlegel and Scott Burles