6. CCD Control

6.1 Introduction

Since much of the software for the CCD system depends on the specifics of selected CCD controller hardware, we begin with a brief discussion of that hardware.

6.1.1. CCD controller hardware plans

Both first-light optical instruments on Keck 1 (i.e., HIRES and LRIS) use the first-generation CCD controller electronics designed by Bob Leach of the Astronomy Department of San Diego State University (SDSU). These same CCD controllers are also currently used at Lick on the MOS Spectrograph and the Lick Infrared Camera (LIRC2). In addition, controllers of this type are planned for use on the Lick Prime Focus Camera (PFCAM) currently being built for the Shane 3-m on Mt. Hamilton, and for the Echellette Spectrograph and Imager, currently under development at Lick for installation on Keck 2.

The default CCD electronics hardware design for DEIMOS is based on a second-generation SDSU system which was originally planned for delivery in September 1995. However, this system is still under development, with delivery now planned for later this year. Some of the parameters of this new system are described in Table 6.1, which compares the performance characteristics of the first- and second-generation SDSU systems. Should the second-generation system not become available in time for DEIMOS, the fall-back CCD electronics design for DEIMOS is a brute-force replication of the first-generation system used for HIRES, scaled up to handle up to 16 amplifiers per dewar rather than 2. We hope to avoid this fall-back plan, since it results in a CCD controller that is larger, heavier, more expensive, and generates more heat than the second-generation system. In either case, the overall system block diagram is similar, and is shown in Figure 6.1 , which conservatively assumes the fall-back design. If the second-generation system is used, as planned, the number of analog boards needed per controller is half that shown in figure 6.1.

Prior to the first DEIMOS instrument PDR in November 1994, several other CCD controller options for DEIMOS were investigated by Richard Stover, the director of the UCO/Lick CCD Detector Laboratory. These options included:

Stover's conclusion at that time was that only the ARCON Controller could be considered as a reasonable alternative to the SDSU system, since all of the others were either still in early stages of development or were not considered likely to be available as finished printed-circuit boards. However, the ARCON system did not appear to offer an overwhelming advantage over the SDSU system, and given the several man-years of experience at Lick in using the SDSU system, the latter was clearly the more practical choice. A more detailed discussion of this choice can be found in section 4.2.3.3 of the DEIMOS PDR document from the first PDR that was held in November 1994, and is reproduced here as appendix 6.10.1. 
		Table 6.1 SDSU Controller Characteristics

			    First-generation		Second-generation
			    -------------------		-----------------

Typical readout time	    (21+2 x N) us/pixel		18 us/pixel
for N amplifiers

Fastest readout time	    26 usec			3.2 usec
for N=8 amplifiers

Readout noise		    0.7 ADU			0.7 ADU

A/D converter		    16-bits,			16-bits, 
			    10 usec convert		2 usec convert
							(19-bit with
							auto-scaling)

Selectable gain		    2 choices			4 choices

Timing sequencer	    DSP56001			DSP56002
			    100nsec/instruction		40 nsec/instruction

CCD clock drivers	    12 drivers,			14 drivers,
			    in 0.1 V steps		in 0.01 V steps

Board size		    10 x 26 mm			10 x 23 mm

Power dissipation for	    23 watts			18 watts
two readouts

Boards needed for	    6				4
four readouts

Practical maximum	    8				16
number of readouts

Computer interfaces	    VME, S-bus			VME, S-bus

6.2 Figures

Figure 6.1 shows a block diagram of the various hardware components of the fall-back design for the DEIMOS CCD controller electronics. It also identifies the worst-case bandwidth requirement assuming a 10 microsecond/pixel read time and dual-amplifier readout from each CCD of the mosaic. It also shows the dewars for both beams of the instrument, even though only one beam will be built for first-light. The functions of the various CCD controller boards is defined in the glossary below.

6.3 Glossary 


ADU		- Analog/Digital Unit, sometime also referred to as DN, for
		  digital number.  These correspond to one resolution unit of
		  the analog to digital converter used to digitize the CCD
		  signal.

Analog Board	- In the first-generation SDSU system, the analog board (in
		  response to commands sent from the timing board) generates
		  the analog voltages used for CCD bias levels and CCD clock
		  waveforms.  It also performs the video processing of the
		  CCD signal and the analog to digital conversion of that
		  signal, returning the digitized value to the timing board.

ATM		- Asynchronous transfer mode: a high speed data transmission
		  protocol which can provide transfer rates in the tens to
		  hundreds of megabits per second.  This standard is still
		  somewhat in flux, but boards for interfacing ATM links to
		  various computers are now commercially available.

Clock Driver	- In the second-generation SDSU CCD controller systems,
Board		  the functions of the first-generation analog board have
		  been split between this board and a separate Video
		  Processing Board.  The Clock Driver board (in response
		  to commands sent from the timing board) generates the
		  analog voltages used for CCD bias levels and CCD Clock
		  waveforms.

DAC		- Digital to analog converter: a device used to generate
		  an analog voltage from a digital input.

DSP		- Digital signal processor: a high-speed microprocessor chip
		  optimized for signal processing application.  The SDSU CCD
		  controllers use Motorola 56000-series DSP chips.

fast-ethernet	- An updated ethernet protocol which operates at 100 megabits
		  per second rather than at the 10 megabits/second of
		  conventional ethernet.

FDDI		- Fiber distributed data interchange: a high speed data
		  transmission standard using fiber optic cables providing
		  transfer rates in excess of a hundred megabits per second.
		  This standard has been in place for some time, but ATM is
		  starting to make in-roads on FDDI.

ESI     	- Echellette Spectrograph and Imager, currently under
          	  development at Lick for installation on Keck 2

GUI		- Graphical User Interface

HIRES		- High Resolution Echelle Spectrometer built at Lick and
          	  now in use on Keck I

Instrument-	  This is the computer at which the astronomer sits, and
Computer	  which receives the CCD images from the CCD controller via
		  the high-speed industry-standard network interface from the
		  CCD [VME] crate.  The instrument computer displays the CCD
		  images in real-time as they are being read from the CCD
		  detectors, and records these images to disk when the readout
		  is complete.

Leach controller- The SDSU CCD controller is sometimes referred to as the
		  Leach CCD Controller.  Bob Leach prefers that the former
		  name be used.

LIRC2		- Lick Infrared Camera, built at Lick and used on both the
		  Shane 3-m and Nickel 1-m telescopes at Mt. Hamilton.

LRIS		- Low-Resolution Imaging Spectrograph built at CIT and
		  now in use on Keck I.  This spectrograph, like DEIMOS,
		  employs multi-aperture slit masks, and also shares
		  some similarity in optical design

MOS		- Fiber-fed prime-focus Multi-Object Spectrograph now
		  being commissioned on Shane 3-m telescope at Lick

overscan pixels - An arbitrary number (0 to N) of non-image pixels which can
		  be read out at the end of each CCD row by continuing to
		  shift the serial shift register after all of the image
		  pixels have been shifted out.  Overscan pixels are often
		  used for CCD baseline compensation or other low-level
		  engineering purposes.


overscan rows-	  An arbitrary number (0 to N) of non-image rows  which can
		  be read out after the image data has been completely
		  shifted out of the CCD chip.  The reading of over overscan
		  rows includes both parallel and serial transfer.  Overscan
		  row are sometimes used for CCD baseline compensation or
		  other low-level engineering purposes.

PFCAM   	- Prime Focus Camera, currently under development at
          	  Lick for installation on the Shane 3-m telescope

prescan pixels  - A fixed number of CCD pixels at the beginning of each CCD
		  row which have been masked on the CCD chip itself so that
		  they receive no light.  The number of prescan pixels is set
		  by the design of a given CCD.  These pixels are often used
		  for CCD baseline compensation or for other low-level
		  engineering tests.

prescan rows	- An arbitrary number (0 to N) of non-image rows which can
		  be read out immediately prior to the readout of an image.
		  The reading of prescan rows includes on serial transfer,
		  and thus can be used to independently measure serial
		  transfer noise.  Prescan rows are used primarily for
		  low-level engineering tests.

RAID		- Redundant Arrays of Inexpensive Disks.  The name for a
		  technology which combines many separate, relatively slow
		  and small disks into a device which appears to be a single,
		  large, fast disk.

Sbus		- A computer interface bus found inside of Sun Sparc
		  architecture computers into which relatively small and
		  inexpensive interface boards can be plugged.  A typical
		  Sun Sparc computer usually has at least one, and sometimes
		  several available Sbus slots.

SDSS		- Sloan Digital Sky Survey

SDSU CCD 	- The CCD controller system developed by Bob Leach of the
controller	  Astronomy Department at San Diego State University.  The
		  second-generation of this controller is now nearing
		  completion, with working prototype boards to demonstrated
		  in April 1996.

Timing Board	- In both the first- and second-generation SDSU CCD controller
		  systems, the timing board provides the overall control and
		  system synchronization and directs the operations of the
		  other boards in the system.  The timing board receives
		  external commands and transmits back status and digitized
		  CCD pixel data via a dual-fiber fiber-optic interface.

Unix 		- A Unix operating system, which varies in form depending on
		  the underlying hardware architecture and/or vendor.  In the
		  context of the VME crate for DEIMOS, the specific version
		  will most probably be Solaris 2.x.

Utility Board	- In both the first- and second-generation SDSU CCD controller
		  systems, the utility board (in response to commands sent
		  from the timing board) handles various utility functions
		  such as exposure timing and shutter control, plus control
		  and monitoring of CCD dewar temperature  and CCD controller
		  enclosure temperature.  In HIRES, the utility board was also
		  involved in the control loop which automatically refills the
		  dewar with liquid nitrogen as needed, but that function will
		  be done elsewhere in DEIMOS.

Vdd		- CCD Output Drain bias voltage, also often abbreviated as OD.
		  This bias voltage has a profound impact on the gain of the
		  output amplifier, and on some CCDs, can be used effectively
		  to adjust the output gain over a relatively wide range.
		  However, it is vital that this voltage not be adjusted
		  above the maximum rated value for the chip.

Video Processing- In the second-generation SDSU CCD controller systems,
Board		  the functions of the first-generation analog board have
		  been split between this board and a separate Clock Driver
		  Board.  The Video Processing Board (in response to commands
		  sent from the timing board) performs the video processing
		  of the CCD signal and the analog to digital conversion of
		  that signal, returning the digitized value to the timing
		  board.

VME Board	- A circuit board which conforms to the mechanical and
		  electrical standards of the VME Bus Specification, and which
		  plugs into a slot of a VME backplane inside of a VME chassis.

VME Crate	- The CCD VME crate serves as a buffer and protocol converter
		  between the SDSU CCD controller and the instrument computer.
		  It simplifies future upgrades and/or replacement of the
		  instrument computer, since the custom (i.e., non-industry-
		  standard) SDSU fiber-optic interface is installed in the
		  VME crate rather than the instrument computer.  (This design
		  goal is discussed in depth in
		  CCD Data Acquisition Systems at Lick and Keck Observatories
		  by Kibrick, Stover, and Conrad, pgs. 277-288 in ADASS II,
		  ASP Conference Series, Vol. 52, 1993, Hanisch, Brissenden,
		  Barnes, eds.)
		  The VME crate consists of a separate 6U-format VME chassis
		  containing one fiber-optic interface board for each timing
		  board in the corresponding SDSU CCD controller to which it
		  is connected via the dual-fiber-optic cable.  The VME crate
		  also contains an embedded CPU board and VME memory board(s).
		  The fiber-optic interface board which receives the dual-fiber
	 	  optic cable from the SDSU CCD controller timing board may
		  either be a separate VME board that plugs into a slot on
		  the VME backplane or an Sbus board which plugs into an Sbus
		  slot on the embedded CPU board.  An industry-standard high-
		  speed network interface (e.g., FDDI, ATM, or fast-ethernet)
		  connects the embedded CPU in the VME crate with a matching
		  network interface in the instrument computer.  The embedded
		  CPU board either runs the VxWorks real-time kernel (like
		  HIRES, LRIS, and MOS) or a Unix operating system (DEIMOS,
		  LIRC2).

VxWorks		- A real-time kernel which was used in the VME crates for
		  HIRES and LRIS, as mandated by a Keck I standard.  VxWorks
		  is not a standard for Keck II instruments.

6.4 Functional Requirements

The functional requirements for the DEIMOS CCD system apply to the overall subsystem, which includes 3 distinct layers of hardware: In general, most requirements will have some impact on the software running on all 3 levels of the hardware, so in most cases we will not attempt to map requirements to specific layers of the hardware. Where appropriate, functions localized to a given layer will be noted with comments delimited by square brackets.

While the detailed software functional requirements for the overall DEIMOS CCD system contain elements which are defined in Chapter 7: Data Storage Formats and Chapter 8: Image Display and Quick-Look , the primary requirements are summarized below, starting with an enumeration of the various readout modes to be supported. Those functional requirements which are already met (at least for readout of non-mosaicked CCDs) by the HIRES/LRIS CCD system are marked with an asterisk (*).

  1. Two independent beams: Each beam of the spectrograph can be set up to expose and read out separately, or the two sides can be ganged together with a single command. Each beam will be treated as its own Keck subsystem (i.e, each beam will have its own distinctly-named keyword library, although these two libraries will be nearly identical, so that identical keywords will perform the same functions on each beam).
  2. * Regular exposure mode: The shutter is opened for a specified period of time, it closes [exposure timing and shutter control are performed by the utility board in the SDSU CCD controller], and the CCD is read out. The minimum commandable non-dark exposure time is 0.1 s (the shutter will not be uniform for this short a time, but the data may be useful for bright objects) and the maximum time is 20,000 seconds. (Note: the minimum exposure time currently available on HIRES/LRIS is 1 second.)
  3. * Multiple exposure mode with single readout: This involves N exposures of M seconds each, with a single readout at the end. The spectrograph may give or receive commands between each exposure. For example, DEIMOS may command the telescope to move between exposures and change either spectrograph or telescope focus, producing a focus sequence on one image. This mode is also useful to determine shutter timing error: a series of 1 second exposures in multiple mode is divided by a single regular exposure of the same total duration. The resultant is a map of the shutter timing error at 1 second. NOTE: Multiple exposure modes have implications for FITS headers which have not been adequately addressed in the HIRES and LRIS systems, nor have they been fully addressed in the DEIMOS data format specifications in Chapter 7: Data Storage Formats . We propose that the FITS headers for each sub-exposure (containing the relevant parameters for that sub-exposure, such as exposure time, telescope position, spectrograph settings) of a multiple-exposure image be retained in a FITS extension table, and that the primary FITS header for the multiple-exposure image contain the relevant data (e.g., number of sub-exposures, total integration time and total dark time, etc.) for the combined image.
  4. * Multiple exposure mode with shift of image on the CCD by N rows up: This mode is similar to item 3, but the image is now parallel-shifted N rows towards the serial register before the start of each sub-exposure. The data from the N top rows of each of these N-row shift-sequences are discarded. It may be faster in some cases to produce multiple exposures this way than by moving the telescope. It is also useful for tests within DEIMOS itself. The same concerns regarding FITS headers also apply to this multiple exposure mode.
  5. Multiple exposure mode with shift of image on the CCD by N rows up and down: This mode is similar to item 4, but the image is now alternately parallel-shifted N rows towards or N rows away from the serial register before the start of each sub-exposure. For those shift-sequences which shift rows towards the serial register, the data from the top N rows are discarded. This readout mode is one of several which might be used to reduce the effects of fringing within the CCD. The same concerns regarding FITS headers that applied to the previous two multiple-exposure modes also apply to this one.
  6. * Dark exposure: Same as regular mode, but shutter remains closed.
  7. Bias exposure: A dark exposure of zero seconds.
  8. * Specific amplifiers: Reads out only a specified subset of the amplifiers in a given mosaic. A given chip may be read out through either one of the two readout amplifiers (single-amplifier readout mode) or via both amplifiers simultaneously (dual-amplifier readout mode). The appropriate de-scrambling algorithms are applied so that the displayed image is geometrically correct. NOTE: Since there may be as many as 16 and as few as 8 usable readout amplifiers per mosaic, the number of possible subsets is quite large, and not all possible combinations may be feasible or useful. In particular, when reading out the full mosaic (or any portion of the mosaic larger than a single CCD chip) the dual-amplifier readout mode does not save ANY observing time unless ALL CCD chips being read out have two working amplifiers. This is because you can't open the shutter and start the next exposure until you have finished reading out the slowest CCD chip(s). The severity of this limitation depends on whether the exposure being read out contains a direct image or spectra. The dual-amplifier readout mode will not save any observing time if: Accordingly, when the time comes to populate the DEIMOS mosaic, if all 8 chips do not have two usable readout amplifiers, then those chips that do should be placed in the top half of the mosaic that is used for direct imaging. In that way, at least the direct imaging readouts can benefit from reduced readout time by using dual-amplifier readout mode, even though the spectroscopic readouts can't. If there are less than 4 chips which have two usable readout amplifiers, then they should probably be placed near the center portion of the top half, although this is not certain. A further complication arises when trying to optimize the placement of chips within the mosaic for the second beam, if and when it is built. If doubled-barreled drift scanning is to be supported, then that might argue for placement of the dual-amplifier chips into the bottom half of that mosaic. Clearly, this area needs more detailed analysis.
  9. Matched bias levels between CCD amplifiers: Mechanisms must be provided to insure adequate matching of the bias levels between amplifiers on the same CCD, as well as between CCDs that are part of the same mosaic. The goal is to get the bias level of all amplifiers in a mosaic equal to within 5 ADU. This match may be accomplished by both analog and digital adjustment. If needed, the software will provide a separate digital offset for each CCD channel.
  10. Matched gains between CCD amplifiers: Mechanisms must also be provided to insure adequate matching of the gains between amplifiers on the same CCD, as well as between CCDs that are part of the same mosaic. The goal is that all gains should match to within +/- 1%. If possible, this match should be accomplished by analog adjustment, in order to avoid the need for floating point calculations in the pixel data transmission pipeline. If adequate balancing of the gains cannot be accomplished by adjustment of the CCD bias voltages to each amplifier or via adjustment of preamplifier gains, it may be necessary to do this gain adjustment in software somewhere in the pixel data transmission pipeline as the data are being read out, probably in the software which runs in the VME crate. During any such analog or digital adjustments of the relative gains between amplifiers, a re-calibration of the absolute gain of the mosaic should be performed using an Fe55 X-ray source.
  11. * Overscan regions: Provision will be made for reading a user-specified number of overscan pixels per row and overscan rows per image, and to optionally make these data available as part of the recorded image.
  12. Specific subregions: Reads out only specific subregions of the mosaic. This mode will be used primarily during the mask alignment procedure, which is described in more detail in Chapter 5: Guider System and Object Acquisition Software . Since these readouts of the subregions are being used for alignment rather than detailed scientific analysis, pre-scan and over-scan pixels will not be retained in this readout mode. Furthermore, all of the subregions to be read out in a given exposure are subject to the following constraints: In the case where two sub-regions overlap each other, the readout software data pipeline will replicate the overlapped pixel data as needed. More details regarding the format in which these multiple subregion images are written to disk can be found in section 7.5.1.5: Disk Storage of mask alignment images contained in 7: Data Storage Formats .
  13. * On chip-binning: Images can be binned on chip by 1 x 2, 2 x 1, and 2 x 2. Additional binning combinations will be supported, provided that these do not result in discontinuities in the image at the boundary which divides each row into the two halves that are read respectively by the amplifiers at each end. In general, values for column binning will be constrained to be evenly divisible into the sum of the number of prescan pixels per row plus the number of imaging pixels read per row per amplifier (i.e., evenly divisible into {# of prescan pixels + 1024} if dual-amplifier readout mode, or into {# of prescan pixels + 2048} if single-amplifier mode). Values for row binning will be constrained to be evenly divisible into the number of rows (i.e., 4096). In the case of a binned and windowed readout, the window size will be adjusted upwards to insure that the binning parameters divide evenly into the corresponding dimension of the readout window.
  14. Rapid readout: A fast mode that pushes the CCD controller electronics and intervening data transmission paths as fast as they can go, with a concomitant increase in readout noise. This mode would be useful for focusing and alignment tests. The exact parameters of this mode remain TBD, since they depend on the characteristics of the selected CCD chip and CCD controller.
  15. * Software selectable gains: Both a fixed high-gain (default) and low-gain readout mode will be provided and made directly selectable by the observer. If the second-generation SDSU CCD controller hardware is used, as many as 4 different fixed software-selectable gains may be provided. In either case, selecting one of these discrete gains is accomplished by switching between alternative gains provided by the SDSU CCD controller's video processing circuitry. Such observer-accessible gain adjustments will be applied globally to the entire mosaic, and not to individual amplifiers. (Adjustment of individual amplifier gains will be an engineering function reserved to the instrument support staff, since such adjustments will change the absolute gain calibration of the instrument, and will require re-calibration with an Fe55 Xray source. When such re-calibration is done, it will be performed for each one of the observer-accessible fixed gain settings.) Typically, the fixed high-gain mode might provide a gain of order 1 electron/ADU, and the fixed low-gain mode a gain of order 2 electrons/ADU. NOTE: Some CCDs have amplifiers with very high gain when they are operated to achieve the lowest possible noise. Some observers might want to operate with a lower gain (by lowering the output drain bias voltage) to obtain better dynamic range at the cost of increased noise. Depending on the CCDs used for DEIMOS, it might be possible to make relatively large gain changes by allowing the observer to adjust (within a limited, safe range) the output drain bias voltage. This adjustment mechanism could be implemented two different ways:
    1. Allow the observer to select between a discrete set of pre-calibrated output drain bias voltages whose resulting gains have already been accurately calibrated using an Fe55 Xray source. In this case, the observers would not be allowed to manipulate the output drain bias voltage directly. This implementation would require that we measure in the lab all combinations of gain settings that result both from this discrete set of output drain bias voltages and from the discrete set of gain settings provided by the video processing circuitry in the SDSU CCD controller.
    2. Allow the observer to adjust the output drain bias voltage directly (within a safe range of voltages), and leave it to the observer to calibrate the corresponding gain at that voltage. Provided that there is at least one standard output drain voltage that has been accurately calibrated against the Fe55 Xray source and can thus be used as a reference, the observer could then calibrate other OD voltages via observations of a standard star obtained at both the reference OD voltage and the adjusted OD voltage.
    Input from the DEIMOS science team and user community is requested regarding which of these two implementation options is preferred.
  16. Statistics / diagnostic pipeline: As each image is read out and its pixel data is transmitted via the software pipeline, statistics will be computed "on-the-fly" to provide a brief summary of the number of defects or artifacts in the data, to assist the observer in making a rapid determination of image quality. Such statistics will include the number of saturated pixels, "dead" pixels, bad columns, etc. These statistics are described in more detail in section 8.4.1.3: Gathering Image Statistics, in Chapter 8: Image Display and Quick-Look .
  17. Eventual drift-scan mode: If optical studies show that distortion is low enough for drift-scanning, we will study the requirements, and if not too costly, will build the necessary hooks into the CCD controller electronics. Drift-scanning introduces additional considerations regarding the placement of CCD chips within the mosaics for each beam (see functional requirement 8 in section 6.4 above) and the organization of analog and timing boards within the SDSU CCD controllers. We will also study the possibility of a read-out mode similar to drift-scanning that might be used to reduce the effects of CCD fringing. However, we do not intend to implement drift-scanning for first light as that would require considerable extra software and testing, and the demand for it (as well as its feasibility) has not been proven. Were drift scanning to be added, it might be done as part of either the first or second year post-commissioning upgrades. Neither do we currently plan to implement the fringe-flattening readout mode for first-light.
  18. Readout times: The total time to read out, display, and store on disk an un-binned, full-frame CCD-mosaic raw exposure is estimated to be as follows: 
							Goal	Specification
							----	-------------
Read out CCDS						42s		120s

Transfer data from timing board to VME crate memory	 0s*		  0s*

Transfer data from VME crate to instrument computer	 0s*		  0s*

De-interleave pixel data and stitch into mosaic		 0s*		  0s*

On-the-fly statistics					 0s*		  0s*

Display raw image(s) on monitor(s)			 0s*		  0s*

Write raw image(s) to disk				13s		 20s

Total time					 	55s		140s
The CCD readout time goal of 42 seconds corresponds to a readout rate of 100 kilopixels/second/amplifier (or 10-microseconds/pixel/amplifier) with dual-amplifier readout per CCD chip, while the specification of 120 seconds corresponds to a 33 kilopixel/second/amplifier rate, which is comparable to the existing HIRES and LRIS systems. (The "goal" value is what we hope to achieve, and the "specification" value is the minimum that we plan to deliver. However, under certain hardware scenarios, even the specification might not be met, as described in section 6.9.4 below). The intervening processing steps between CCD readout and writing to disk are assigned zero time (and marked with *) to indicate that these operations are overlapped with the CCD readout using a similar data transmission and processing pipeline to that used for HIRES and LRIS.

Writing of the image(s) to disk is not overlapped with the CCD readout because the image(s) must be completely de-interleaved and stitched together before disk writing operations can be initiated. The disk write time goal of 13 seconds assumes a sustained disk write rate of 20 megabytes/second, while the specification assumes a more conservative rate of 13 MB/sec. Either of these implies the use of RAID disk technology. The exact format in which the images are written to disk depends on both the selected readout mode and the disk format selected by the observer. The various possible disk formats are described in detail in Chapter 7: Data Storage Formats .

6.5 Preliminary or Prototype Design

As noted in Chapter 1, the DEIMOS CCDs, dewars, and CCD controller hardware and software will be subject to a separate design review in mid-1996. Until then, a number of significant details regarding the CCD controller hardware will remain uncertain. Since many aspects of the software for the CCD system deeply depend on the specifics of the controller hardware, the design described here should be considered tentative.

6.5.1 Design impacts of using first-generation vs. second generation SDSU controllers

The choice between using the first- or second-generation SDSU controller impacts the design in the following ways:

6.5.1.1 Impacts on downstream hardware

If a second-generation SDSU controller is used rather than a first-generation controller, this will probably reduce the total number analog boards, since each video processing board handles two amplifiers rather than one. However, whether or not it reduces the number of timing boards (and hence the fiber-optic cables and interface boards to which they are connected) depends on how many video processing and clock driver boards can be reliably controlled by a single second-generation timing board.

The maximum number of analog boards (for a first-generation system) or clock driver and video processing boards (for a second-generation system) that can be controlled reliably from a single timing board is unknown at this time. (For the first-generation system we know we can reliably control at least 4.) Figure 6.1 assumed that a first-generation timing board could drive either 8 or 16 analog boards, but recent experiments involving first-generation boards indicate that this is doubtful due to issues involving the drive capacity of the bus-driver chips on the analog boards coupled with issues of VME bus termination. It appears that controlling 4 analog boards may be the practical limit for first-generation timing boards, although it is conceivable this might be stretched to 8 given careful attention to bus terminations and insertion of wait states for certain critical data transfer transactions on the SDSU controller backplane bus. Further experiments in this area are needed. Since we have not yet received any second-generation boards to test, we do not know the corresponding practical limits for that system.

The total number of analog boards (or clock driver and video processing boards in the case of the second-generation system) and hence the total number of timing boards and fiber-optical cable pairs is also a function of the total number of amplifiers to be read out from each mosaic. We are assuming the worst-case demand that would result from dual-amplifier readout mode being used on all CCDs within the mosaic, namely, that there will be 16 amplifiers to be simultaneously read out from each mosaic.

However, dual-amplifier readout mode suffers from two significant limitations. First, as noted in the discussion of functional requirement 8 in section 6.4, dual-amplifier readout mode does not provide any savings in observing time unless all of the CCD chips being read out have two usable readout amplifiers. Second, it suffers from the problem of electronic crosstalk between readout amplifiers on the same CCD chip. While the exact mechanism for this crosstalk is not yet known, it is suspected to be related to supplying common bias voltages to multiple amplifiers on the same chip. In this case, a large CCD signal in one amplifier will create current demands which result in variations in the bias voltages seen by the other amplifiers, and this is suspected to produce the ghost shadows observed in the images from those other amplifiers. However, there is also some evidence that this crosstalk may result through some other coupling internal to the CCD, and this may vary significantly from one type of CCD to another, or even between CCDs of a given type. The particular type of CCD to be used for DEIMOS is not known at this time. However, recent tests on a Lincoln Lab device with readout amplifiers similar to those that might be used for DEIMOS indicated that while providing separate CCD bias voltages to each quadrant of the detector reduced the problem of amplifier crosstalk by about a factor of two, it did not eliminate it.

Even if we can't solve the electronic crosstalk problem, it is less serious for spectra than it is for direct imaging. Since we need to read out twice as many pixels for spectra, there is still considerable motivation for providing dual-amplifier readout to shorten the readout time for spectra. Unfortunately, as noted earlier, the dual-amplifier mode only saves observing time if all of the chips being read out have two usable readout amplifiers. In the case of reading spectra from the full-mosaic, this means that every readout amplifier on every chip must be usable. While direct images are somewhat less impacted by this limitation, they are more seriously impacted by the electronic crosstalk problem. Given these constraints, we need to weigh carefully whether the potential advantage (of reduced readout time) that the dual-amplifier readout mode might provide is worth the extra costs in additional hardware (and software complexity) required to support it. Input from the DEIMOS science team and user community is needed on this question. If it can be demonstrated that the read time specification of 120-seconds can always be met using the single-amplifier readout mode, should the dual-amplifier mode be dropped as a requirement?

Thus, there is considerable uncertainty at this time regarding the total number of amplifiers that will need to be read from each mosaic (we will assume 16 for now), as well as the total number of analog, timing, and fiber-optic interface boards. Given this uncertainty, we will attempt to develop the software for each of the hardware levels (SDSU controller, VME crate, and instrument computer) to be easily configurable to a variety of such configurations. How difficult this will prove to be in practice is not yet known. We expect that these issues will be resolved and addressed at the CCD Detectors/Controllers review that will be held later this year.

6.5.1.2 Bias Matching

Under item 9 of the functional requirements in section 6.4, the bias levels between the various amplifiers used to read out the DEIMOS CCD mosaic must be matched within 5 ADU. With the first-generation analog boards, the digital adjustment (via a DAC) of this bias level (i.e., DAC 0, the video offset level) is too coarse to allow the bias levels between amplifiers to be matched within the 5 ADU specification. Accordingly, the software will provide a separate digital offset value for each amplifier which will be subtracted from each pixel during its transmission to the instrument computer, either by the timing board software or the software running in the VME crate. This digital offset value for each amplifier will be adjustable by the observer, to allow for compensation of small drifts in the bias level which may occur over time. While the second-generation analog boards may provide sufficient resolution in the adjustment of this bias level to make this separate software-based digital adjustment unnecessary, we will plan on implementing it anyway.

6.5.1.3 Gain Matching

Under item 10 of the functional requirements in section 6.4, the gain levels between the various amplifiers used to read out the DEIMOS CCD mosaic must be matched to within +/- 1%. With the first-generation analog boards, the digital adjustment (via DACs) of the analog bias levels which affect the amplifier gain are too coarse to allow the gain levels between amplifiers to be matched the +/- 1% specification. Accordingly, the software will provide a separate floating point gain adjustment factor for each amplifier. As the raw digital data from each amplifier is transmitted down the pipeline, it will be multiplied by this gain adjustment factor, and rounded to an integer value. Since the timing board does not contain floating point hardware, this gain adjustment will probably be performed in the VME crate software. (It should be noted that the integer rounding portion of this gain adjustment mechanism will contribute a slight increase to the readout noise of the system, and careful thought needs to be given as to whether this adjustment should be performed in the raw data pipeline, or rather in the pipeline of data as it flows to the image display. The crucial question here is whether or not the "raw" data that are recorded to disk should be scaled by this gain adjustment factor. The implications of this should be carefully considered, and we need the DEIMOS users to reach a consensus regarding the proper answer to this question.)

With the second-generation clock driver boards, the resolution of the digital adjustment (via DACs) of the CCD bias voltages which affect the amplifier gain may be sufficiently precise to allow the gains to be matched to the +/- 1% specification. In that case, the software-based gain adjustment proposed above might be unnecessary, in which case the question of noise introduced by integer rounding could be avoided. At this time, however, we will assume that some sort of software-based amplifier gain adjustment factor will need to be applied somewhere in either the raw data pipeline or the image display pipeline, and that the appropriate keywords will be reserved for specifying and documenting these adjustment factors.

Regardless of whether the first- or second-generation SDSU controllers are used, and regardless of whether the matching of individual amplifier gains is accomplished purely via the adjustment of CCD bias voltages or by application of floating point gain multipliers, to the extent that these adjustments affect the raw data that is recorded to disk, they should be restricted to qualified instrument support personnel and not accessible to the observers. That is, any adjustments which effectively change the absolute gain calibration of the overall mosaic should not be made without re-calibration of the gain using an Fe55 Xray source, and this is not something that observers will be equipped to do. (Providing a means for adequately illuminating the DEIMOS mosaic in the lab with an Fe55 Xray source needs to be factored into the DEIMOS dewar mechanical design.)

If it is decided that the floating point gain multipliers are not to be applied to the raw data, but will only be used for matching the effective gains of the amplifiers as displayed on the image display, then such multipliers should be adjustable by the observer, and procedures should be provided to allow for automated computation of these multipliers using some sort of flat field calibration frame.

6.5.1.4 Changes to controller DSP firmware and support tools

If the first-generation SDSU system is used for DEIMOS, much of the timing board software developed for HIRES, LRIS, and MOS will be directly applicable. In addition, the "wavegen" CCD waveform-generation tool, developed by Richard Stover, will be directly usable as well. Wavegen provides a graphical user interface (GUI) for defining arbitrary CCD waveforms, and automatically generates the corresponding data tables that will generate those waveforms when those tables are down-loaded into SDSU first-generation timing boards.

Since the underlying SDSU "analog" board architecture and the CCD waveform encoding scheme of the second-generation SDSU timing board is considerably different and not upwardly-compatible from the first-generation timing board, if we use the second-generation SDSU system for DEIMOS, most of the timing board software developed for HIRES, LRIS, and MOS will need to be re-designed and re-written, as will the wavegen tool. Thus, while the second-generation SDSU hardware has the potential to be more compact, capable, and cost-effective, it does have an associated cost in terms of additional software development.

6.5.2 Design issues not impacted by choice of first-generation vs. second-generation SDSU hardware

We can now proceed with detailed design of a considerable amount of the CCD control software independent of which generation of SDSU CCD controller hardware is ultimately selected. In fact, a significant amount of effort has already been made to define the formats in which the pixel data that originates from the CCD controller will be arranged in memory, displayed on the image display, and written to disk. These topics are described in more detail in Chapter 7: Data Storage Formats and in Chapter 8: Image Display and Quick-Look , and will not be repeated here. Also, since the utility board hardware remains unchanged between the first and second-generation systems, we can proceed with detailed design of the software responsible for exposure timing, shutter control, and control and monitoring of the dewar temperature and CCD controller enclosure temperature. With regard to utility board functions, it should be noted that unlike the HIRES system, in which the automated-dewar filling mechanism was treated as a CCD control function and controlled from the utility board, in DEIMOS this mechanism is considered part of the spectrograph control software, and does not directly involve any of the SDSU CCD Controller hardware for its operation. Details regarding the automated-dewar filling mechanism for DEIMOS can be found in Chapter 4: Spectrograph Control .

6.6 Inherited/Existing Software, Tools, and Hardware

Much of the overall CCD control hardware and software model for DEIMOS will be directly inherited from the system developed for HIRES and LRIS (and which is also used on MOS). If the first-generation SDSU CCD controller hardware is used for DEIMOS, then considerably more software at the level of the timing board will be transferable, as will the wavegen tool. We note here the primary differences between the DEIMOS CCD controller hardware and software and that used by HIRES, LRIS, and MOS. Since the differences between the first and second-generation SDSU CCD controller systems have already been described, we will assume for the purposes of this comparison that DEIMOS will be using the first-generation system:
  1. The fiber-optic interface board(s) for DEIMOS will be either the SDSU VME interface board or the SDSU Sbus interface board, rather than the custom-made wire-wrapped "Harris/Ricketts" fiber-optic interface board used for HIRES and LRIS. (This also eliminates the need for the IKON DMA interface board.) The changes to the HIRES/LRIS software to use the SDSU VME interface board instead of the "Harris/Ricketts" board have already been made and tested by Cromer at CIT for the blue-side upgrade to LRIS, and Cromer has already made this code available to UCO/Lick.
  2. We do not plan to use the VxWorks real-time kernel in the DEIMOS VME crate, but rather to run a standard Unix-system there. This saves DEIMOS the licensing costs and numerous other difficulties associated with the VxWorks development environment. However, it does involve porting a significant amount of HIRES/LRIS VME crate software from the VxWorks to the Unix environment. On the other hand, much of that software will require some modification anyway to support the requirements of the DEIMOS CCD mosaic.
  3. We do not plan to use the UDP-based image transmission protocol currently employed by HIRES/LRIS to send images from the VME crate to the instrument computer. To provide the needed bandwidth, we plan to use more modern networking technologies than conventional ethernet, and will be considering FDDI, ATM, or fast ethernet, whichever proves more cost effective. Given our current understanding of the CCD mosaic readout bandwidth requirements and the actual performance achievable with the newer network technologies, a TCP-based protocol should provide the needed bandwidth. We also plan to explore mechanisms that would allow this transmission protocol to be directed to multiple destinations, and to optionally include some form of in-line image compression/decompression, in order to better support the needs of remote observing.
  4. While Keck figdisp is one of the possible image displays being considered for DEIMOS, if it is selected it will require extensive modifications to support the requirements of the DEIMOS CCD mosaic. If it is not selected, then the HIRES/LRIS software which runs on the instrument computer and which receives the CCD image data from the VME crate will need to be interfaced to a different image display server.
As a result of these differences, it should be clear that while DEIMOS will benefit significantly from the underlying hardware and software models used for HIRES and LRIS, it will not be able to use such software "off-the-shelf", and that significant porting and updating of this software will be needed.

6.7 Additional Resources Needed

As noted in Chapter 1, we have already assembled a Dec-Alpha-based platform to use as a test-bed for prototyping the DEIMOS instrument computer software. In the UCO/Lick CCD Detector lab the DEIMOS software group also has access to a reasonable complement of first-generation SDSU CCD controller boards for prototyping the DEIMOS CCD controller software, and a VME chassis, Unix-capable VME CPU and memory boards, and an SDSU VME fiber-optic interface board for prototyping VME crate software. With the exception of the expansion RAM for the Dec-Alpha-based platform, none of this equipment was acquired with DEIMOS funds. The hardware and software resources which we are currently lacking, and which will need to be acquired later this year for end-to-end testing of a prototype DEIMOS CCD readout pipeline are:
  1. An SDSU Sbus fiber-optic interface board, to compare its cost effectiveness and performance relative to the SDSU VME interface board
  2. An SDSU second-generation CCD controller board set, including one timing board, and several clock generation and video processing boards
  3. A pair of higher speed networking boards (e.g., FDDI, ATM, or fast-ethernet), one for the VME crate and the other for the instrument computer
  4. A more capable CPU board for use in the VME crate. Currently, we have several Force 3/CE boards capable of running Solaris 1.x (i.e., SunOS 4.1.3) which can be used for early tests, but we will ultimately need a faster board and licensing for Solaris 2.x.
  5. A test dewar capable of supporting multiple CCD chips, especially 2K by 4K sized chips of the types used for DEIMOS. The primary purpose of this test dewar would be to allow DEIMOS CCD mosaic noise and gain measurements to be made using various configurations of SDSU controllers, prior to the availability of thinned DEIMOS chips or the actual DEIMOS dewar. Ideally, this test dewar would provide space and wiring to support a full 4 X 2 DEIMOS mosaic, but if this were not possible, then it should at least support two 2K by 4K devices. However, this test dewar would not need to keep the test CCDs coplanar, nor would all of the CCDs need to be illuminated by light. The CCDs used in this test dewar could be thick, engineering grade devices, so long as they had readout noise levels comparable to the thinned, science grade chips to be used in the real DEIMOS CCD mosaic and dewar. Provision for illuminating the test CCDs with an Fe55 Xray source would be needed. This test dewar would also need to be pumped and cooled to normal CCD operating temperatures.

6.8 Dependencies upon other Components

As noted earlier, the DEIMOS CCD control system involves software running on three different levels of hardware (SDSU controller, VME crate, and instrument computer). That portion of the software which runs on the instrument computer has strong dependencies with both the quick-look/image display software that is described in Chapter 8: Image Display and Quick-Look and the software which formats the data and writes it to disk as described in Chapter 7: Data Storage Formats . In terms of accessing tables of CCD defects and various calibration and adjustment factors, it will have significant interactions with the database software that is described in Chapter 9: DEIMOS Information Management . The utility board software which monitors and controls the CCD dewar temperature, CCD Controller enclosure temperature, and CCD controller power- supplies will interact with various environment monitoring, logging, and alarm mechanisms running on both the VME crate and the instrument computer, as described in Chapter 10: Environmental Monitoring .

6.9 Outstanding Issues/Concerns

There are several outstanding issues which impact the design of the DEIMOS CCD control software: We expect that many of these issues will be resolved at the upcoming DEIMOS reviews on detectors, controllers, and flexure compensation. In the meantime, we are focusing much our design effort on those aspects of the system (e.g., data formats in memory and on disk) which are not directly dependent on these open issues. In addition, we have several areas of concerns which warrant further discussion here:

6.9.1 Delays in delivery of the second-generation SDSU CCD controller

This is a serious concern. We had originally expected the availability of the second-generation system in September 1995. The development effort at SDSU has taken longer than expected, an in late January 1996 a team from UCO/Lick (Stover, Wei, Wright, and Kibrick) visited Bob Leach in his lab at SDSU to review the progress of the second-generation system. We received schematics for all of the new boards, and were shown a working prototype of the second-generation timing board. A preliminary layout and partially tested clock driver board was also displayed, but layout of the video processing board had not yet begun. In general, the team was quite impressed with the design of the second-generation boards, but disappointed at the delay in their availability. During this visit, Leach indicated that his schedule called for working prototypes of all 3 boards (timing, clock driver, and video processing) to be ready for review in early April, and as of early March he indicated that the development of these boards was progressing on schedule. We will be checking with Leach in April to determine if he is still on schedule. However, until such time as DEIMOS has working second-generation boards in hand, the availability of these boards remains a serious concern.

6.9.2 Unknown limitations on the number of analog boards per timing board

Our fall-back plan assumes that we can use the first-generation system to read out a full DEIMOS mosaic of 8 CCD chips and up to 16 separate amplifiers per dewar. During the last few months, it has become apparent that the first-generation system might not be able to support more than 4 analog boards from a single timing board, due to problems involving the bus-driving capabilities of the bus transceiver chips used on these boards, coupled with inadequate termination of the SDSU controller backplane. Even if 8 analog boards could be made to function with a single timing board (which at the moment seems questionable), that would not be sufficient if dual-amplifier readout is to be used for each CCD chip. Thus, should we need to fall back to the first-generation system, there is the very real possibility that we will need to have more than one timing board per dewar. In fact, there are also other practical reasons why it would be advantageous to have more than one timing board per dewar.

6.9.3 Noise problems using unsynchronized timing boards to read CCD mosaic

As presently designed, the first-generation SDSU system does not provide any direct method for synchronizing the operation of two separate timing boards. This creates a serious electronic noise problem when attempting to read out the mosaic using more than one timing board, since without synchronization the two timing boards will be performing different functions at the same time. For example, one timing board might be performing parallel transfer on the portion of the mosaic that it is reading out, while the other timing board is doing serial pixel processing and trying to sample and digitize data. Since the two timing boards will be running off of separate 40 MHz clocks, their activities will be asynchronous, and the noise generated by one timing board will not be cancelled by the double-correlated sampling done by the analog boards controlled by the other timing board.

G. Luppino at IFA has experienced this very problem with a 4 x 2 CCD mosaic that he built which is in essentially the same format as the DEIMOS mosaic. Luppino used two separate first-generation SDSU controllers, each containing 1 timing board and 4 analog boards. Each controller reads one side of the mosaic. Unfortunately, because of the noise problem that results from the unsynchronized timing boards, Luppino is only able to read out one side of the mosaic at a time, hence doubling the readout time.

While the first-generation SDSU controller was not designed with any provision for synchronizing multiple timing boards, we have several ideas as to how this might be done, and propose that this be tested as soon as possible using existing first-generation hardware that we currently have available in the UCO/Lick CCD detector lab. In order to conduct such a test, we would like to have a test dewar capable of supporting at least 2 CCD chips in close proximity. However, we do not have such a dewar available at this time, and propose an alternate (and perhaps more severe) test where we try to simultaneously read out from separate amplifiers of a single CCD using two different first-generation SDSU controllers that we have attempted to synchronize. (This will require careful thought regarding common power supplies and avoidance of ground loops between the two controllers.) If we can demonstrate that at least one of our synchronization schemes allows separate first-generation controllers to be adequately synchronized to solve the readout noise problem, we could then request Luppino verify this method using his CCD mosaic. If he agrees and our synchronization scheme works for his mosaic, then we would have some confidence that the first-generation system represents a viable fall-back for DEIMOS. If, however, we are unable to develop a synchronization scheme that works, then using the first-generation system becomes a serious concern, especially if we decide the dual-amplifier readout is both feasible and essential.

6.9.4 Performance degradation under various CCD controller hardware scenarios

The following matrix summarizes the functionality that could be provided depending upon which SDSU controller configurations are possible. It assumes that the second-generation control system will in fact support readout of 16 amplifiers from a single timing board, as claimed by the documentation. 
----------------------------------------------------------+--------------------
   SDSU Controller Configuration Options		  |    Functionality 
--------------------------------------------------------  | -------------------
	         1st-generation can     1st-generation    | Read      Dual-
2nd-generation   be made to work with   controllers can   | Full      Amp	
available?	 N analog boards?	be synchronized?  | Mosaic?   Readout?
--------------	 --------------------   ----------------  | -------   --------
    YES		    doesn't matter	 doesn't matter   |  YES	YES
--------------------------------------------------------  | -------------------
    NO			N=8			YES	  |  YES	YES	
--------------------------------------------------------  | -------------------
    NO			N=4			YES       |  YES	YES
--------------------------------------------------------  | -----------------
    NO			N=8			NO	  |  YES	NO
--------------------------------------------------------  | -----------------
    NO			N=4			NO	  |  NO         NO
--------------------------------------------------------  | -----------------
In the worst case scenario shown in the bottom row of the matrix, we end up with a system like Luppino's, in which we can only readout half of the mosaic at a time, and only using single-amplifier readout per CCD. While this would allow us to obtain images, it would not meet any of the readout time specifications for DEIMOS (i.e, it would takes 240 seconds to read out the full mosaic rather than the specified value of 120 seconds) that were established under the functional requirements section 6.4, and would result in a very significant waste of Keck 2 telescope time.

6.10 Appendices

6.10.1 Material extracted from section 4.2.3.3 of DEIMOS Instrument PDR, held November 1994

Justification of Leach (SDSU) Controller The Leach Controller was selected for the Keck instruments because it was an available, working system long before any other controller. At Lick we now have years of experience in programming, developing, debugging, and actual on-telescope use of the current Leach system. There are working systems on the Keck telescope (HIRES and LRIS spectrographs) and on the Shane telescope (MOS dual-beam fiber-fed spectrograph). We are currently building another Leach system to run our IR detector arrays on Mt. Hamilton. We also have a close working relationship with Bob Leach, which has proven very valuable in debugging and testing both hardware and software. In addition, the existing Leach system is in use at several other sites in Hawaii (i.e., CFHT and IFA), as well as numerous other sites in the US and Europe. There is an organized users' group and associated e-mail mailing list which allows Leach users to communicate with each other and with Leach.

Although the existing Leach system has some deficiencies in terms of its packaging and power-up/power-down transients, neither of these has proved fatal, and both should be addressed by the new Leach Controller. For the HIRES Spectrograph at Keck and the MOS Spectrograph at Lick, we have found that the existing Leach system meets our current performance specifications with respect to both readout rate and readout noise. The existing system also meets the readout noise requirements of DEIMOS, although it falls short of our readout rate goal by more than a factor of three. The second-generation Leach system should satisfy the readout rate required by DEIMOS and should also result in a more compact and efficient package for a CCD mosaic system.

While the new Leach Controller will have some important differences, it will in many respects be similar to the current system. In the new system the controlling DSP processor will be faster, but it will be the same type of device, and our programming skills and tools are already in place for these devices. The details of the clocking waveforms will be different, but the general approach to creating waveforms from memory tables will remain the same. Equally important, the existing CCD Controller support code that runs in the VME Crate and the host computer will, in most cases, need little modification to work with the new Leach Controller. All of these points demonstrate that the adoption of the new Leach system will be an evolutionary change rather than a revolutionary one. The infrastructure already exists at Lick (and Keck) to work with these systems. Discarding that infrastructure in favor of a new system would be very costly in dollars and time and could be justified only if there was an overwhelming advantage.

That overwhelming advantage does not appear to exist. As indicated in the previous section, the ARCON controller is the only other system comparable to the Leach system. It is a proven system, working now on CTIO telescopes. However, it is a very different system from the Leach Controller. An entirely new infrastructure would have to be developed to support the ARCON system. But there is no obvious advantage in cost, complexity, data quality, or scientific productivity that could justify the switch. In fact, in most parameters of interest to DEIMOS (primarily readout speed and ability to handle many readout channels), the systems are comparable.

In the previous paragraph we said that discarding our Leach Controller infrastructure would be costly. That is an over-simplification since we could not actually discard it. We already have Leach Controllers running that must be maintained and supported, so we would have to add a new support infrastructure if we were to adopt a different controller. This would be even more expensive both in immediate costs to DEIMOS and in terms of long-term maintenance, support, and development. All of this expense and personnel expertise would have to be duplicated at Lick and at Keck. None of this can be justified.