HIRES Data Reduction [v1.0; April 2005]

HIRES Suggested Calibrations

1. 10 trace flat images (per setup)
   Note - It is recommended to take a set of Arcs right before and/or after
   creating the trace flat images!!
2. Arc Calib:
   • One pair taken at the same time as the trace flat
   • One image taken before (or after) each set of science exposures
3. One Standard star     (each setup)

Pre-Reduction Setup

• Computer Resources
  • >600 MHz processor
  • 20G disk space / night
  • 1G RAM
  • Linux, Solaris, MacOSX
• Solaris WARNINGS (These do not apply to Linux and less to MacOSX)
  • Solaris+IDL does not release memory until IDL is exited
  • Consider exiting IDL occasionally and monitor memory usage
  • IDL+Solaris freezes up on the CPU intermittently
    • Possibly bug in licensing software
    • Forced to kill IDL process and restart
• Software
  • gcc or cc
  • IDL v6.1 or higher (most and maybe all of the code will work with v5.4)
  • IDL packages
    1. idlutils SDSS IDL package (Burles/Schlegel/Sloan)
    2. idlspec2d SDSS IDL package (Sloan public)
    3. xidl XIDL package (JXP)
    4. HIRES HIRES package (Burles)
• Log sheets [somewhat optional]
• This cookbook

     Setup

Initial Setup (Repeat for each night)

1. Create a new directory for the night (e.g. 08sep04) and enter it
2. Create a 'Raw/' directory and put all the raw data in it.
3. If the filenames do not read hires####.fits it is quite likely the code will be unhappy
4. gzip the data (gzip *.fits)
5. Launch idl in the directory above Raw/ :: (e.g. idl -32 or idlde -32)
6. hires_strct :: Create the HIRES structure.
   • This structure organizes the entire night of data and is the most important file created.
   • The routine creates a few things:
     1. An IDL structure in memory with whatever name you choose (e.g. hires).
        Here is a copy of the file which defines the structure and lists the tags.
     2. The file 'hiresstrct.fits' which is a fits version of the structure;
     3. The file 'hires.list' which is an ASCII version of the fits file which lists the values of some of the tags.
     To view the fits structure outside of IDL, I recommend the program 'fv' which I think stands for fitsview. It allows you to examine binary fits tables.
   • If you run with /EDIT the program will launch a gui which allows some editing of the hires structure (see next step). We recommend you use an IDL script to do the editing. It will save time if you have to start over from scratch and it gives a permanent record of what you changed.
         Example: IDL> hires_strct, hires, [/EDIT]
         Time : 2s per image
   • You know the code is working   properly when:
     1. It runs through the completion after checking each Raw file
     2. The majority of files are correctly identified by type. Note the code will not get them all right all of the time. We typically create a script like this one to set a number of the tags.

7. hires_editstrct :: Modify the HIRES structure. The previous step creates the structure and takes a guess at the initial values of many of the tags based on the header card info. It is difficult, however, to automate all of the values for the tags and therefore the user should carefully check the structure. Also, the user should set flg_anly = 0 for all of images which should be ignored during data reduction (bad flats, etc.). For most of the important tags, one can use hires_editstrct. The rest must be done from the command line by hand or through a simple IDL script (recommended). The obvious tags to modify are:
   • Obj :: Object name (this propagates into the final spectra and should have no spaces!)
   • flg_anly :: Include in analysis (defaulted to 1 for yes)
   • type :: ARC, TFLT, MFLT, TWI, OBJ, STD, ZRO, etc.
   • setup :: 1L, 2L, etc. (unique integer for each instrument setup including slit width)
       Example: IDL> hires_editstrct, hires
       Time : User interaction
       Example: IDL> set_crds, hires    (see doc/set_crds.pro for an example)
   • You know the code is working   right when the GUI appears.

8. hires_updstrct :: Append new files to the current structure. This is mainly useful at the telescope as you are taking new data. The code simply searches the Raw directory, notes any new files, and adds them to the structure.
    Example: IDL> hires_updstrct, hires
    Time : Fast
9. You know the code is working   right when it finds the files.

10. hires_wrstrct :: Write the HIRES structure to disk. In IDL you can modify the values of any of the tags. You can then save the structure in fits form and rewrite the ASCII file with the routine hires_wrstrct.
        Example: IDL> hires_wrstrct, hires, FITS='hires_name.fits'
        Time : fast

11. If you exit (or crash) IDL, you will need to read the structure back in.
    hires_ar :: Read the HIRES structure from disk. If no name is given, the file looks for the first fits file starting 'hires' that contains a '_'.
        Example: IDL> hires = hires_ar()
        Example: IDL> hires = hires_ar('hires_name.fits')
        Time : fast
12. You know the code is working   right when IDL> help, hires, /str
    lists the structure.

Setup

• A unique setup is defined by the slit width, the binning, the angles of the echelle and cross-disperser, the blocking filter, the cross-dispersor, and the collimator used. Obviously, each setup requires its own set of calibration files.
• It is important to make sure the objects are properly named at this point. At the least, multiple exposures of the same object should have identical object name (tag: hires.obj).
• hires_setup :: This routine examines the hires structure and looks for calibration files associated with the various setups. It groups together exposures with identical Obj name (hires.obj) and sets the obj_id tags accordingly. A list of the setups and the corresponding files is put in 'hires_summ.txt'. See 'setup_summ.txt' for a very brief listing.
• Note, this program is useful to run if you happen to crash IDL before saving the HIRES structure.
      Example: IDL> hires_setup, hires
      Time : Fast
• You know the code is working right when you   check the output file 'hires_summ.txt' and everything is properly setup. Modify the obj_id and other tags and rerun   hires_setup as necessary.
• Perhaps the main failure mode of this routine is when the cross-disperser angle floats between exposures. Right now the default tolerance is 0.0015 for the angle. So if your exposures differ by more than this, the code will assign unique setup values.

     Calibrations

Create Bias (Zero) frames [Optional and NOT recommended]

• The default method of bias subtraction is to use the overscan region and the bias row. If the user prefers, one can also subtract off a combined bias frame. This routine creates that combined frame from a series of Bias frames. This routine will create 1 bias frame per side per binning mode.
  Alternatively, one may perform this step as a test of the bias subtraction algorithm. The resulting images should have 0 counts with no gross structure.
      Example: IDL> hires_mkbias, hires
      Time : 3-5min
• You know the code is working right when you   check the Bias frame : IDL> xatv, 'Bias/name.fits'
  and it shows a generally blank image with no counts. Of course, there is the possibility that you have bad bias frames and the code is still working fine.

Set Gain

• The stability of the gain with the new HIRES ccd's is not well known at this time. At present, we choose to measure the value by comparing a series of flats.
• Note that the gain was changed from approximately 1 to 2 (Low mode) on September 16, 2004. You should definitely run this routine if your data is from before then.
• hires_findgain :: Loops on the Milky Flats and performs statistics (in hires_gain) to determine the gain and apply it to the gain TAG.
      Example: IDL> hires_setgain, hires, setup
      Time : 3-5min
• You know the code is working right when you   see it looping through all of the flat files and reporting gain values within the ballpark of the published values. You might also print out the gain tags after completion:
  IDL> print, hires.gain

Process Flats

• hires_mktflat :: Combines the series of trace flats (currently twilight flats) to create one high S/N image for order and slit tracing. Because of the thermal gradients, it is highly recommended that one use only a coeval set of trace flats.
      Example: IDL> hires_mktflat, hires, setup, [side]
      Time : 1min
  • You know the code is working right when you   see it process and stack the various trace flat files. You should then check the Trace Flat:
    IDL> xatv, 'Flats/Flat_B_01_T.fits'

• hires_trcflat :: Routine to trace the trace flat created above to determine the order curvature.
  • The main driver is using trace_crude on a 'sawtooth' image of the trace flat. At present this is one of the weaker spots in the automated pipeline. This is especially true given that there is no 'standard' prescription for creating trace flats with HIRES. The main failing is that the code may miss or misidentify the slit edges. As such, we highly recommend that you run this routine with /INTER and (or) /CHK to interactively check the order edges. It is particularly important that the code has the first few orders correct before making an automatic attempt.
  • We have recently (Mar 2005) turned the PCA fitting back on for the red side. I suspect we turned it off for a reason initially, so this may cause problems. You can turn off the PCA by using the /NOBASIS switch.
  • The code also makes a first guess (usually incorrect) to the physical order numbers for the orders.
  • Example: IDL> hires_trcflat, hires, setup, [side], /INTER, /CHK
        Time : 1min
  • You know the code is working right when   the reduced_chi^2 for the fits are small (<0.005). You can also check the QA plots in (e.g. 'QA/Flats01/qa_trcflt_01B.ps'). The X0 and PCA0 values should vary significantly but be well fit while the PCA1-4 values should have minor variations.
  • You can also check the tracing with hires_chktrcflat. Along with xatv, this routine is used to check the results from hires_trcflat.
        IDL> xatv, 'Flats/Flat_B_01_T.fits'
        IDL> hires_chktrcflat, hires, setup, 1, /NOSTOP

• hires_fittflat :: Performs a 2D fit to the traces created in the previous step. This gives the full 2D optical model for each side of the instrument. The relevant information is written in the order structure in the Flats/ directory (e.g. OStr_B_01.fits).
      Example: IDL> hires_fittflat, hires, setup, [side]
      Time : 1min
  • You know the code is working right when   the RMS for the edges are less than 0.1 (blue) and 0.01 (red). The code produces a QA file (e.g. 'QA/Flats01/qa_fittflt_01B.ps') which also lists the RMS values.
  • You can also check the fits as above but with the /FIT keyword enabled.
        IDL> xatv, 'Flats/Flat_B_01_T.fits'
        IDL> hires_chktrcflat, hires, setup, 1, /NOSTOP, /FIT

• hires_allflat :: The previous 4 routines have been bundled into one simple script. We recommend using this routine once you are comfortable that the code is working well with your setup, etc. We recommend you run hires_chktrcflat after the procedure is through to examine the solutions.
      Example: IDL> hires_allflat, hires, setup, [side, /INTER]
      Time : <20min per side
  • You know the code is working right when   you go through all of the checking steps listed above.

Arc Images

• General :: The arcs are processed through a series of steps which:
  1. Process the raw frame (hires_procarc)
  2. Determine the offset between the trace flat and the position of the arcs due to thermal expansion (hires_arcalign)
  3. Derive a 1D solution down the center of each order (hires_fitarc)
  4. Create a 2D solution (hires_fit2darc)
  5. Traces the individual arc lines (hires_tracearc)
  6. Fits the changing slope of the arc lines (hires_fittrcarc)
  7. Create a 2D wavelength image (hires_mkaimg)
  Because of shifts in the arc lines during the night (expected to be due to thermal expansion), the arcs require special care. In particular, one wants to associate each science frame with the temporally closest arc image. This bookkeeping is done in hires_setup and is repeated in the first step below.

• hires_allarc: There are two modes of processing the arcs. We strongly recommend against running the arc steps individually unless you are debugging. In fact, it is possible that some procedures will not run on their own at the present time.
  1. Process individually. This is generally only recommended for redoing specific arcs. It is also useful for testing the code on non-standard setups before running on all of the arcs. The user inputs the indice(s) of the arc(s) to process.
         Example: IDL> hires_allarc, hires, [indx], /INDX
         Time : 5min per arc
  2. Process all together. This is recommended If you are reducing a full night of data, I suggest the latter.
         Example: IDL> hires_allarc, hires, setup, [side]
         Time : 5min per arc

• Procedures
  • hires_setarcm: For the processing to work, the code must first measure the curvature of the arc lines. This involves processing one arc to near completion using the code hires_setarcm. This procedure is called automatically by hires_allarc and we recommend against calling hires_setarcm directly. In any case, here is the calling sequence (raw_fil is the name of an Arc file, e.g. 'Raw/mb0428.fits'):
        Example: IDL> hires_setarcm, raw_fil, setup, side
        Time : <10min
    • You know the code is working right when   it runs to completion. Actually, if this code runs successfully then it is very likely the rest of the arcs will process fine. This code calls hires_procarc, hires_fitarc, hires_fit2darc, hires_trcarc and you should review these codes (below) to see how to check for success.
    
    
  • hires_procarc :: Process the Arcs. This step bias subtracts (blue side only due to leakage in overscan region on the red side) and flat fields the arc images. It chooses the arc image closest in UT time to the science exposure. This routine will also calculate the x,y shifts between the arcs being processed and a 'template' arc. The `template' arc should be the arc taken most closely in time to the trace flats. By default, the first Arc of the night is taken as the template. Output are fits files in the 'Arcs/' directory.
        Example: IDL> hires_procarc, hires, setup, obj_id, [side]
        Example: IDL> rslt = hires_procarc_sngl('raw.fits', setup, side)
        Time : 1min per image
    • You know the code is working right when   it finishes. It would be quite unusual for this procedure to fail.
    
    
  • hires_arcalign :: This routine compares the Arc line profiles against the trace flat profiles to determine an offset between the two. The offset is calcualte for a series of orders and a 1st order polynomial fit is written to the arc_xyoff tag.
        IDL> hires_arcalign, hires, setup, side, /CHK
    • You know the code is working right when   the output to the screen shows a reasonable boxcar profile and the fractional shifts (indicated by the vertical blue line) are smaller than 0.2 or so. Also, the total shifts should by on the order of 2 unbinned pixels for the blue and less than 1 unbinned pixel for the red.
  • hires_fitarc :: This routine extracts a 1D spectrum down the center of each order and determines the wavelength solution.
    • There are two levels of interaction with this routine. The most interaction (not recommended) is to use /INTER which prompts the user to identify and fit the Arc lines. One can also use the option /PINTER which has the program attempt to identify a set of lines in each order. The user than interactively fits the lines using the routine x_identify which calls x1dfit. As long as your arc lines are within several pixels of my solution, things ought to run smoothly in the non-interactive mode.
    • At this point, we recommend using the full AUTO mode. If it fails, we would hope to fix it. The output is an IDL file containing the arc lines identified and their pixel centroids: 'Arcs/Fits/Arc_name_fit.idl'. The program can also create a ps file to examine the quality of fits.
          Example: IDL> hires_fitarc, hires, setup, obj_id, [side, /INTER, /PINTER, /PSOUT]
          Example: IDL> result = hires_fitarc_work('arc_file', setup, side)
          Time : 5min per side per arc image
    • You know the code is working right when   the RMS of the fits to each order are generally less than 0.1 pixel. There will be several exceptions per full exposure. Also, it is common for the code to fail for the very bluest orders of the blue side (at least with the original blue CCD).
      The code produces a QA file (e.g. 'QA/Arcs01/qa_arcfit_mb0038.ps.gz') which shows the residuals to the order by order fits including the RMS (in pixels).
  
  
  • hires_fit2darc :: This routine fits a 2D solution to the wavelengths as a function of their y pixel value and their order number. It is rather straightforward.
        Example: IDL> hires_fit2darc, hires, setup, obj_id, [side]
        Example: IDL> result = hires_fit2darc_work('arc_file', setup, side)
        Time : fast
    • You know the code is working right when   the RMS of the 2D fit is less than 0.1 pix or so. The code produces a QA file (e.g. 'QA/Arcs01/qa_arc2dfit_mb0038.ps.gz') which shows the 2D solution and lists the RMS.

  • hires_tracearc :: This routine traces arc lines in each order. It traces the lines in the curved order frame of the original image and then fits a straight line to each arc line. The slope and centroid of the arc line is recorded to a file for later use.
        Example: IDL> hires_tracearc, hires, setup, obj_id, [side]
        Example: IDL> result = hires_tracearc_work('arc_file', setup, side)
        Time : 2min per side per arc image
    • You know the code is working right   by reviewing the QA file it produces (e.g. 'QA/Arcs01/qa_tracearc_mb0034.ps.gz'). So long as the arc lines and the fits look reasonably straight, everything should be fine.
    • You can check the output by using   hires_chkarctrc
          Example: IDL> xatv, arc_img
          Example: IDL> hires_chkarctrc, hires, indx

  • hires_fittrcarc :: Using the slopes derived in the previous step, this routine fits for the slope throughout the 2D arc image. That is the slope as a function of order and vertical height in the image. Uses the usual least-squares algorithm.
        Example: IDL> hires_fittrcarc, hires, setup, obj_id, [side], /CLOBBER, /CHK
        Example: IDL> result = hires_fittrcarc_work('arc_file', setup, side)
        Time : Fast
    • You know the code is working right when   RMS3 is less than 0.01. The code produces a QA file (e.g. 'QA/Arcs01/qa_fittrcarc_mb0038.ps.gz').

  • hires_mkaimg :: Using the 2D slope solution and the 1D arc solution from step 2, this routine calculates a wavelength value for every science pixel in the image. The final wavelength image has air wavelengths and are saved as alog10 double values. Output is : 'Arcs/Arc_nameI.fits'
        Example: IDL> hires_mkaimg, hires, setup, obj_id, [side], /CHK
        Example: IDL> result = hires_mkaimg_work('arc_file', setup, side, XOFF=)
        Time : 1min
    • You know the code is working right provided it does not complain about an order not being single-valued in wavelength. Also one can examine the BAD_ANLY parameter (it should be 0 upon completion).

Slit Profile

• hires_slitflat :: This routine determines the slit profile for each order. This is crucial for optimal sky subtraction, particularly given the short slit length used in HIRES. The profile is used to correct the illumination pattern of the science frames. The procedure is also important for optimal extraction. It must be run after the arc calibrations as the slope of the arc lines is a necessary input.
      Example: IDL> hires_slitflat, hires, setup, [side], [/chk], [/detilt]
      Time : 10min per setup per side
  • You know the code is working right when you check the QA file (e.g. 'QA/Flats01/qa_slitflat_01B.ps.gz') and the slit profiles look reasonably well behaved and the chi^2 values are around 0.5.
  • If you want to see the fits in real-time, use /chk
  • If the slitflat has a large tilt, which you determine is due to illumination from the source, use /detilt to remove this from the applied profile in later steps.

     Extraction

• The following routines all apply to a single object, i.e. multiple exposures of that object will be reduced together.
• Most of the following routines take the hires structure, and the setup, side and obj_id tags.
• hires_allobj :: Run all extraction routines described below. Currently, it is preferable to run hires_allobj with these options:
  IDL> hires_allobj , hires, 1, /procall, /nocr

Process the Image

1. hires_proc :: Bias subtract and flat field the Raw image. This routine takes the index number of the hires structure as input or keywords setup and obj. The index number is the integer in the first column of the file 'hires.list'. The resulting image is output in 'Final/' and is a flattened flux and inverse variance fits file (one gzipped fits image with two extensions per file). Output is 'Final/f_name.fits'
       Example: IDL> hires_proc, hires, SETUP=setup, OBJ=obj_id
       Example: IDL> rslt = hires_proc_sngl('rawfil', [side])
       Time : 1min per image
       Check : xatv, 'Final/f_name.fits'
   • You know the code is working right when you check the image and it looks reasonable.

2. hires_objcr (Optional, recommended with caution) :: Compares two or more images to identify cosmic rays. For two images, it uses the difference of the two images scaled to the exposure times and is quite conservative. One should NOT use this on multiple exposures of the same object if the object has moved along the slit or if thermal expansion has resulted in a large shift in the data.
       Example: IDL> hires_objcr, hires, setup, obj_id, side, [exp], /CHK
       Time : 3min per pair
   • You know the code is working right when you run with the /CHK keyword and the code is not identifiying excessive CR's along the object profile. This will happen (in particular) if the object has moved significantly along the slit in between exposures.

Identify and Trace the Object

• hires_fntobj :: This code automatically identifies the object in each order (rectify+collapse) and then traces using trace_crude. A PCA analysis is then performed on the coefficients of each trace to determine a smoothed (quasi-2D) solution that is useful for interpolation and extrapolation. The code then creates and Object structure hiresobstrct that contains the trace and will contain the 1D extractions. Output: 'Extract/Obj_name.fits'
      Example: IDL> hires_fntobj, hires, setup, obj_id, side, /CHK
      Time : fast
• This code creates an object structure in the Extract/ directory (e.g. 'Extract/Obj_mb0033.fits.gz') which holds information relating to the trace and extraction of the spectra. Here is a link to the code which lists the tags used.
• You know the code is working right when you run with the /CHK keyword and the fit to the position in the slit fraction is nearly constant and well fit as a function of trace number. Also, the image with the trace overplotted should look sensible. Alternatively (or in addition), check the QA output (e.g. 'QA/Obj01/qa_fntobj_mr0022.ps.gz'). The X0 and PCA0 coefficients should be well fit and the remaining PCA coefficients should show small variation. Finally, the reduced_chi2 should be < 0.1 (blue) or <0.01 (red).

Sky Subtraction

• hires_skysub :: Performs sky subtraction on an order by order basis using a bspline fitting algorithm to all sky pixels in a given order (the object is masked). The sky spectrum is also cross-correlated with the UVES sky line list to investigate a shift. At the moment, while it calculates the shift, it does nothing with the value.
• Output: Sky subtracted 2D image appended to the unsubtracted flux and the variance 2D images (e.g. 'Final/f_name.fits'). Also the bspline coefficients are written to 'Sky/sky_name.fits'
      Example: IDL> hires_skysub, hires, setup, obj_id, side, [/CHK, /FCHK]
      Time : 10min per exposure
• Examine the final product (and get wavelength info) by:
      IDL> xatv, 'Final/f_name.fits', getsky=2, WVIMG='Arcs/Arc_nameI.fits'
• You know the code is working right when you run with /FCHK and the final image shows 'random' noise in places where sky lines used to be. You can also run with /CHK to examine the order by order fits. Finally, the QA file (e.g. 'QA/Obj01/qa_skysub_mr0022.ps.gz') shows a zoom in on key sky lines in the red or several small patches in the blue.

Extraction

• hires_box :: This procedure first performs a boxcar extraction of all orders and uses this result to gauge the SNR in each order. The routine than loops on each order in decreasing SNR and fits a unique profile within each order. This profile and a uniform sky is fit to the entire order and the object flux is optimally extracted.
• Output: The tags .box_fx, .box_var, .box_wv, .fx, .var, .wave are all filled
      Example: IDL> hires_box, hires, setup, obj_id, side, /RESCHK
      Time : 15min per exposure (longer for high SNR data)
• Examine the final product:
      IDL> hires_specplt, [/BOX]
• You know the code is working right when the reduced chi^2 values are 1.2 or lower (a bit higher for very bright objects).

Standard Star (Optional, and recommended)

• At this step it is recommended to process a standard star to 'test' the performance of extraction and to create a sensitivity function used to flux the data (currently set to only use the calibration files archived on the ESO web site). This is not required, however. The user can used an archived sensitivity function or forgo fluxing altogether. The next sequence of steps are roughly the same as what will be performed on individual science objects. Greater detail is given on these procedures below. Note that the variable index below refers to the index number of the standard in the hires structure. Also, you will need to use the /STD keyword for most of the routines.

• hires_proc :: Process the raw image (bias sub, flatten).
      Example: IDL> hires_proc, hires, index
      Time : 1min
• hires_fntobj :: Identify the object in the slit and trace it.
      Example: IDL> hires_fntobj, hires, setup, index, side, /STD
      Time : 2min
• hires_skysub :: Sky subtract the image.
      Example: IDL> hires_skysub, hires, setup, index, side, /STD, /CHK
      Time : 2min
• hires_box :: Extract the standard star (boxcar and optimal).
      Example: IDL> hires_box, hires, setup, index, side, /STD, /CHK
      Time : 15min

• hires_calibstd :: Calibrate the extracted 1D standard star spectrum. At present, this step requires a calibration file from the ESO web site:
  Grab the appropriate file (e.g. fhr4468.dat) and go. This will create a sensitivity function in the Extract/ directory which can be used to flux science objects.
      Example: IDL> hires_calibstd, hires, index, ESOFIL='fhr4468.dat'
      Time : 3min

Flux

• hires_flux :: This routine fluxes the data using a sensitivity function. Use either the one you have created for that given night (set using the FLUXFIL keyword) or an archived solution (if it exists for the binning and orders of interest).
• Output: The tags .flux and .sig are filled in the Object structure.
      Example: IDL> hires_flux, hires, setup, obj_id, side, FLUXFIL=, /STD
      Time : 1min
• Examine the final product:
      IDL> hires_specplt, /FLUX