
***   BAD  FIBER  LIST   ********

Plate  camera   fiber (0 indexed)      Problem
0410     r1       90                   Ill constrained flat field applied

* Look at plate 306, MJD 51637, fiber#295 (0-indexed 294).
  Appears that fluxing for b1 is bad redward of 6000 Ang.

----------------------------------------------------------------------------
* Do we need to try to sky-subtract the arcs to make the centering more
  robust?  Also, check the value of MAXDEV, which Scott set to 2.0e-5
  from SOS reductions (passed there from QUICKWAVE).

* Output Cartridge# on SOS web page. (But where is this information??)
  Also look for "fail" in sos.log, in case the disks fill.

* Exclude low-S/N frames from co-addition of a plate?

* Do halo correction before scattered-light correction?

* Confirm proper dispset in combined spectra
     Currently using variance weighted dispersion, but this has scatter

* Report non-gaussian errors in SDSSPROC from the overscan region.
  Search for other possible electronics problems.
  Do a QUICKBIAS to reduce the bias frames and look for these problems.

* Have APO2D report whopping fibers.

* SPPLAN1D needs a fully-qualified TOPINDIR, but should be able to work
  relative from current directory.

* Add 3% to the errors in each spectrum for when combining high S/N data.
   --- or add errors due to uncertainties in flat field/telluric corrections

* Let MYFLUXCORR work for different pluggings

* Look for cosmics on raw frame as ratio of neighboring points.

* Watch for the flux-calib factor going negative.
  Mask out any wavelength regions without a calibration.

* Looks like fibermask is no longer used in DIVIDEFLAT.

* Let TRACESET2XY return a wave map of type double.

* What the heck is CURANS in the routine XY2TRACESET?

* The routine traceset2pix should also support 'fchebyshev_split',
  as does func_fit().

* Rename heliocentric -> barycentric?

* Warning if telluric correction is out of range (not between 0 and 1.5?)

* Calling script for SPREDUCE2D, SPCOMBINE, SPREDUCE1D

* Is the 5460 sky line at the wrong wavelength?

* Dispersion width needs to be adjusted by measuring the width of bright
  sky lines (on sky fibers), and change units from pixels to log10-lambda.

   --- Here we need Robust fitting of skyline, arcline widths 
   --- we could just try to fit sky fibers to see how widths have changed

* Modify SDSSPROC to read in the binned images.

* For SPFLATTEN2, set any bad columns (from opBC file) to unity.

* When combining files, set an additional mask bit for when part of a
  spectrum has been constructed with only one input spectrum.  In those
  cases, we don't have information for CR rejection.

* Add /DEBUG keyword for writing out spArc and spFlat files.

* Add keywords: INSTRUME ?

* In SPALLREDUCE, don't send the name UNKNOWN to COMBINE2DOUT.

* Measure relative shifts between all the flat-field traces in SPCALIB,
  then for each object pick the flat-field with the most similar traces.
  We can do this cheaply by measuring the offset to just the first good flat,
  then use the known relative shifts to pick the best match.

   Can easily use match_trace to solve the item above.

* Add in error as a fraction of the whopping terms?
  Or at least set a pixelmask bit.

* Bug in fitans image, do not get smooth fits for centershift and
     sigmashift, what is going on?

     --- Shifts were likely too large
     --- Now fitansimage is cleaner, and hopefully more robust

* Don't create pointers for SPCALIB, or if I do, then be sure to discard them
  with arcstruct_destroy(), flatstruct_destroy().

* Need to have better (or any) standard star SED prediction based
    on u-g, g-r, r-i.

       Look at: Stone, R. P. 1996, ApJS 107, 423, 
                Gunn & Stryker 

       BD17 is now assumed with AB_79 = 9.405

* Implement PSF variable sky subtraction

* Change FIBERMASK not in FIBERFLAT, but in SUPERFLAT.

* BUG: For arc 51441/sdR-b2-00001347.fit (plate 214), bizarre wavelength
  solution!!  Were too many fibers rejected or something??
  This then shows up as horrible sky-subtraction!!
  Something must be broken in FITARCIMAGE or FINALARCFIT.
  Reverting to the old code, now the arc 51441/sdR-r2-00001347.fit
    looks somewhat poor in first 40 rows.  Damn it!

* BUG: The ends of the spectra have poor flux-calibration.  This is most
  obvious when one looks at the overlap of individual exposures.

  --> Could this be due to scattered light errors near the edge of frames?

* Looks like combining now fails if blue side is missing (21-Dec-99).

* Do something fancier for overscan-correction?
  JEG points out that the b1 CCD has a terrible transient at the beginning.
  also first and last columns in all cameras.

* Choose bkpts in combine2dout more carefully, bad behavior ensues
  with current choice (uniform bins).

* Special-case a combine routine for only 2 exposures that will reject CR's

* Sky-subtraction: Fix edge effects in inverse variance scaling

* Measure gain + read noise

* Somehow measure nonlinearity + saturation.
  Maybe look at doublets with known line ratios?

* Measure system throughput (Q.E.) for g,r,i !!

* The b1 CCD has too much overscan subtracted on rows 0 and 1.
  Check all overscan regions in SDSSPROC.

* Output in header PSF information (spatial + wavelength) to final files.

-------------------------------------------------------------------------------
;   These have been dealt with
-------------------------------------------------------------------------------
* Reject entire fibers for use in SKY if those fibers appear inconsistent.

* For calls to COMBINE1FIBER, first normalize to ergs/1e-4 log-lam,
  just as is done for flux-calibration.

* Use DJS_FILEPATH() where appropriate.

* Look at the bias frames

* Pass batch1d.par, batch2d.par file names as environ variables, or use
  defaults.

* Preserve header info in APOREDUCE.
  Don't bother (Jan 2001).

* Need to save WARNING/ABORT messages as a separate HDU from APOREDUCE.
  Fixed Jan 2001.

* The telluric regions #5 and #6 accidentally overlap in TELLURIC_CORR.
  Fix it!!  Fixed Jan 2001.

* Horrible bug in plate 343, but evident in other plates:
  If throughput varies for some fibers (due to atmospheric refraction
  in this case) relative to the standard star, then fluxing between
  frames is horrible.  When combining multiple spectra of high S/N objects,
  lots of random points get rejected, and the combined spectra looks horrible.
  Fixed v4_x with flux-corr vectors.

* Fit spline to each spectrum before full combine2dout spline to
   register each spectrum in scale (including red/blue merge)
  Fixed v4_x with flux-corr vectors.

* Re-instate S/N plot, and include global S/N values in spPlate files,
  evaluated at 20th mag or so just like in APO-2D.

* Apply illumination correction, as done for the PT. 

* Improve mask bits for bright sky features.

* Track total exposure time in combined spectra
  some attempt is made, if both red and blue spectra are used in combining,
    total exposure time is exposure time of all / 2

* Replace slatec_splinefit and slatec_bvalu with their IDL counterparts 

* Put back Scott's conservevariance terms in COMBINE2DOUT.
   --- taken care of by correct_dlam

* Calculate synthetic g',r',i' in fluxed spectra and begin QA
    for spectrophotometry

* Do higher order shifting then "bestlag" as real flexure changes
  scale as well.  in place: match_trace

* Don't allow too many whopping terms.

* Focal plane mapping

* Reject masked or deviant sky fibers from the supersky (as done for superflat)

* Plate 306, fibers 172-175 now look worse than in July.

  ---> Looks better now (6-Sep-00)

* Move iteration code from XY2TRACESET to FUNC_FIT(), letting it call itself
  recursively.

  ---> Now use DJS_REJECT().

* Looks like /halfintwo is no longer used in FUNC_FIT().
  Should probably just have a different function call than fchebyshev,
  like fchebyshev_split.

* trace_crude gives a wrong xstart for fiber 172 in 
    plate 192:  Tracing 320 fibers in sdR-r1-00001602.fit
    The fiber overlaps with fiber 171, but is not set as a
    bad fiber, so maybe this is why it is not fixed.

* Use a proc SDSSHEAD in place of HEADFITS, which calls SDSSPROC to get
  the header info correct (e.g., the CAMERA keyword).

* Try not to overwrite PostScript log file.

* Put the arc-fitting QA logs into the QAPLOT code.

* Rationalize 01, 1b, b1 --> b1.

* For the calibration op*.par files, make versions based upon MJD in /examples
    --- opECalib-51574.par  is up to date with latest gains.

* Make bad pixel masks for "opBC.par" file.
  Include bad pixels from pixflat.

* Have FIBERFLAT only do a very low-order correction now that the
  flat fields are so much better.
----> Done (17-Dec-99).  Now only fit a cubic to each.
      But I'm still not using FIBERMASK everywhere I should.

* Plot the superflat vector as a QA plot

* Decide on vacuum or air wavelengths - document this!

* QA plot for how well the telluric-removal is doing

* Compare throughput from dome flats vs. throughput on bright sky lines.
----> Gaussians are fit to 5577 in each spectrum, after flat scaling

* Need some logic to make certain we're looking at a flat, arc, etc.

* Add HELIOCENTRIC code into current cvs.

* Need to pass rejection information back from extract_row.c
    to add into pixelmask

   ---> done, put in another parameter to return to extract_row.pro
          also put in rejection thresholds as a two elements array

* BUG: trace320crude is failing for fiber 276 on plate 217: sdR-r2-00001459.fit
    The xcen is off by > 1 pixel thr whole length of the fiber, and
    flat field extraction is horrible, object extraction is bad as well.

   ---> Should be fixed 03-Mar-00.

* CR zapping ---> Current code seems to deal with this.

* Problems with CCD electronics?  Bleads with low counts?
    -- This was the shutter open

* Characterize flexure 

* In flat-fields, be smarter with break points
  Also, tune pix-to-pix flat with global flat to be high-pass / low-pass filter

* When doing red/blue merging, apply a window function to the **variance**
  of each to smoothly merge the two.  E.g., at the long-wavelengths of blue,
  increase the variance -> infinity.
---->   current window of 100 pixels at red/blue merge is in place.

* Discard flats that saturate in SPREDUCE or SPPLAN.
----> Done (17-Dec-99) in SPREDUCE, but maybe should implement in SPPLAN?

* BUG: telluric_corr crashed on call to SLATEC_SPLINEFIT, because
  TEMPIVAR was all zeros.  Should test for this.  Failure was
  on 51433/plate#198/sdR-r1-00001137.fit.

-------------------------------------------------------------------------------
;   These are not likely to be done in near future:
-------------------------------------------------------------------------------
* Implement Proper variance (and covariance) terms in combine2dout
    -Larger problem of returning general covariance terms with
     slatec_efc and slatec_bvalu.  Probably requires doing customized
     B-spline fitting.  Also a change to deal with bkpts gracefully.

-------------------------------------------------------------------------------
-------------------------------------------------------------------------------

2D Pre-Processing:
  * Overscan + bias subtraction
  * Pixel-to-pixel flat-fielding

2D Reductions:
  * Spatial tracing from flat-field image
      - Never, ever, ever mis-identify fiber numbers
      - Fit traces to smooth functions
      - Inspection of traces

  * Optimal extraction of flats, arcs, objects
      - Characterize PSF in spatial direction
      - Reject cosmics
      - Remove cross-talk

  * Wavelength calibration
      - Fit to arc lines + sky
      - Robust against bad centroids on some fibers
        (e.g., consistent fits between fibers)
      - Achieve < 0.4 pixel residuals (30 km/s)
      - Characterize PSF in wavelength direction

  * Flat-fielding
      - Remove fiber-to-fiber variations
      - Remove dichroic effects
      - Model relative wavelength response between fibers

  * Object extraction
      - Tweak spatial traces from flat -> object frame
      - Extraction: the "3 Step Plan"
        [1] Characterize exact spatial profile shape.
            Characterize scattered light.
        [2] Demand smoothly varying profiles (fit)
            Demand smoothly varying scattered light (fit)
        [3] Extract, fixing profile shape, center + scattered light terms
            e.g., ONLY fit flux level
      - Proper handling of bright objects.
        Use 1-param fit to exponential tails.

  * Sky-subtraction
      - Construct "super-sky"
      - Deal with the variable PSF

  * Telluric absorption removal
      - Template
      - Correction methods

  * Rebinning spectra + co-adding red+blue / multiple exposures
      - Deal with the variable PSF
      - More outlier rejection
      - Demonstrate no residuals from dichroic

  * Variance maps
      - Retain bad pixel info
      - Correct for flat-fielding
      - Include sky-sub variance term
      - Properly rebin along with the spectra

Code requirements (cf, "Requirements Document"):
   * Support on SDSS platforms
   * "Adequate development/debugging environment"
   * Code reliability
   * Code management (CVS, UPD)
   * Distribute code in "timely fashion"
   * Timing: Extract 1 night in < 1 day processing
   * Run tests on real data
   * Documentation

Data Model:
  * Which files are needed for reductions?
  * Which files are obsolete?
  * Fermi archiving issues.

-------------------------------------------------------------------------------
-------------------------------------------------------------------------------
Standalone routines:

readidlout

-------------------------------------------------------------------------------
-------------------------------------------------------------------------------
Unused routines:

extract2
finalarcfit
findobjects
fit_tset
fitwithmx (deprecated)
funccheb
gauss_smooth
read2dout
svdcheb
trace_fix
writeapid

-------------------------------------------------------------------------------
-------------------------------------------------------------------------------
Unused files in /etc:

BG14.txt
FG3.txt
altbg14.txt
fluxcorr.bkpts
intrinsic1.dat
intrinsicF2.dat
lamplist.dat

-------------------------------------------------------------------------------
-------------------------------------------------------------------------------
