Shane AO  

Welcome/Index | Architecture | Procedures | Problems | Features | Documents | Resources | Chronology  



Reverse Chronology

2023-01-10: NSF statement on NSF and SpaceX Astronomy Coordination Agreement:
https://beta.nsf.gov/news/statement-nsf-astronomy-coordination-agreement

Contains the following bullet point regarding Starlink satellites and the LCH:

SpaceX agreed to analyze the impact of astronomical facility lasers on its satellites. Following this analysis, the Laser Clearinghouse removed the coordination requirements for these lasers. Therefore, adaptive optics lasers at ground-based facilities will no longer undergo multiple closures every time the SpaceX satellites pass nearby.

2021-05-26: E. Gates writes:
Installed an external filter wheel on ShARCS with ND and the narrowband filters that do not fit in the ShARCS dewar (i.e. ND2, ND4, 2.2/0.04, Kcont, Hcont). The filters are par-focal. Sub-optimal because the filters are warm, but makes them available.

Problem: Occasionally, communication to the Sci Filter wheel become unresponsive. The recovery is to re-start the saomot5 service.

2021-05-10: M. Shetrone writes:
In 2021B we do not expect to have the field steering tip-tilt for the laser working. Thus, we expect you will have AO LGS on axis in the same way it operated in 2021A. Please confirm with me that this is acceptable. If not please let me know if you want to withdraw your proposal or if you would prefer AO NGS.
2021-03-19: E. Gates writes:
The WFS Dichroic stage has been fixed (this is the first time it has worked since the instrument arrived on Mount Hamilton in 2014). Modified and aligned so that we should have the full field steering range expected with the WFS stage upgrade. The saomot GUI has changed slightly to reflect the new working state of the motor and the two WFS Dichroic options (NIR Splitter and Empty). For now we shall always want the NIR Splitter in until such time as we get a visible dichroic beam splitter so we can do I-band (and maybe R-band) imaging with ShARCS. The following URL has been fully updated: http://mtham.ucolick.org/techdocs/instruments/ShaneAO/saomot/ Other web pages that refer to this stage and shall be updated. There is one pending repair to the WFS Dichroic stage, which is to fix the cable loopback signal, which is currently not working. This is not critical for operations, so will likely be fixed in April 2021 or as soon as Gabe can find time to get up to the mountain to look into the SAO wiring.
2020-10-26--2020-10-30:
Re-commissioning of ShaneAO system follwing intervention to replace Wave Front Sensor mechanism.

The content of some GUIs has been modified as a result.

Some motors/keywords, formerly used by the pre-2020-10-26 WFS mechanism have been removed (e.g. STEER2RY, STEER2RX, STEER1RX, STEER1RY, WFSFOCUS).

2019-10-28:
P. Zachary writes (AO TT Y-stage update):

I removed the brakes from both the X and Y axis and tried moving the stages by hand, both seemed just fine. Next, I put a load on the stage and in the horizontal orientation things seemed fine but when I raised the stage to vertical I saw 8 inch-Lbs of force needed to hold the stage and about 11 needed to move it. This continued for almost half the stage travel at which point the 8 inch-lbs became zero and the force needed to move it rose very slightly. From this it was obvious that there was a significant amount of drag introduced to the system. Disconnecting the lead screw made it obvious the the slide bearings were just fine and the problem was in the motor drive. Loosening the motor drive from the stage showed the ball-nut was free to move and the binding was absolutely in the motor itself. Disassembling the motor seems to indicate the front bearings are fine and the problem was the rotor interfering with the windings. There is some chance this contamination originated in the manufacture or handling of the motor or stage since it is not completely sealed but I feel there is a high probability this contamination this material may have originated from within the brake located on the back (and unfortunately above) of the motor. I'm cleaning the motor out now and then we will wire it up for Will to test. I expect to be done with the mechanical part later today but not sure when D. Black might have it wired for W. Deich to test. Overall, it feels like neither an endorsement nor condemnation of the Griffin stage but is something to remember should we have problems in future. It seems to me that it might be worth making sure the design wiring allows removal of a motor without too much pain and then we could buy a motor to have in case of trouble. D. Cowley and C. Ratcliffe both feel it is not worth reinstalling this stage in any case since the upgrade is just around the corner.

2019-09-07:
Standard set of LGS Alignment Stars near zenith introduced.

2019-08-20:
E. Gates writes:

I moved the telescope to see how the TTCamY stage moves at different declinations. Movement is improved with the higher torque limit, but we still cannot move the stage properly everywhere on the sky. I believe we can do dithering as normal in the Dec range of +15 degrees to +70 degrees, with the lower declination limit being a little uncertain. Certainly at a Dec of +7 deg (30 degrees south of zenith), the stage could not do moves against gravity successfully. At Dec=+7 degrees, the torque was pegged at its maximum (-6.25V) just holding its position. Our observer (Jon Zink) on Aug 21st only has targets below Dec +7 deg, so his program will be unable to use LGS dithering for any of their targets, so we will have to work around that doing any offsets with the field steer motors with the telescope near zenith, changing the mode to Pos OFF, then acquiring their off-axis guide stars, and doing separate sky background frames.

I believe we need to further investigate the counterweight option for TTCAMY so we can have proper motion over the entire 45 degree zenith distance range of LGS operations.

We also have temperature monitoring of the motor housing working, so I will keep track of that through the night to make sure we do not cook the motor. Temperature of the motor housing right now is about 42C with no extra gravity load, which I am going to assume is normal.

W. Deich writes:

Yesterday afternoon I did some detailed tests to characterize the motor further. Dale had posited that the Hall sensors might not be reliably showing correct voltage, so I did a check of that, too.

One of the tests was to power-cycle the Galil -- to ensure consistent starting conditions -- and then do a simple brushless init, carefully monitoring the Galil-reported brushless phase. The init consists of:

 
   BIF=-1		# Tell the Galil to use standard Hall inputs 
   BCF		# Tell Galil to learn the precise phase angle. 
   SHA		# Motor on 
   SPA=500		# Slow speed... 
   PRA=2500	# ...for 2500 counts of motion 
   BGA		# Begin motion. 
   

Two noteworthy points:

a) Sometimes the motor would draw excessive current as soon as 'SHA' was given, which suggests the Galil did not get the correct phase from the Halls. One thing I did *not* do, which in hindsight would be useful, is to record the Hall sensor values immediately before and after power-cycling the Galil. That would provide a check on whether it is remaining correct.

b) I monitored the Hall sensor outputs while the galil moved through circa 120 changes in Hall value. Each time, the transition came at about the expected number of encoder counts (1111.1111), and the Halls showed the expected value.

2019-06-17:
W. Deich writes:

We recently encountered an odd error with the TTCAMX and TTCAMY motors: the Galil claimed it was putting out maximum continuous torque (indicated by +-4V on TTCAMX, +-3V on TTCAMY) to try and hold position at angles where much less should be fine.

This was the case on Sunday night (2019-06-16), when TTCAMX was unable to move correctly, and it was the case Sunday morning, when TTCAMY was misbehaving.

The Galil has to know the brushless phase (with respect to the motor windings). At initial power-up, it doesn't know the phase precisely, so it uses the Hall sensors to get a good enough value - it will typically be in error by 30 degrees, but this still is enough to move the motor. We tell it to learn the very accurate phase by watching for the moment at which the on Hall sensors switch state. If the Galil has the wrong phase, it will be range between more difficult and impossible to move the motor.

On the off chance that the Galil somehow lost brushless phase information, I told the Galil to re-learn the phase, and that resolved the problem. Here's how to do this.

(I do not know why TTCAMX and TTCAMY were at incorrect phase. The Galil claimed that the axes were brushless-initialized, yet a re-init corrected their behavior. It would require a combination of the motor encoder not reading out, while the stage drifted by a hundred counts or more.)

  1. First, this will not help if the Galil requires the usual amount of 'torque' (voltage to the amplifier), and is simply holding position poorly.

    This might help if the Galil is putting out full power (+- 4V on TTCAMX; +- 3V on TTCAMY) to hold, when you are pretty sure that is not normal.

  2. Put the telescope at zenith (more or less).
  3. Put the stage into the Halt state, using the GUI.
  4. From shred (it has the saomot.io command installed), issue low-level commands to turn on the motor, start a "brushless calibration", and start jogging slowly:

    For TTCAMX:

     
       saomot.io ctrl1aux='SHF' 
       saomot.io ctrl1aux='BCF;JGF=500;BGF' 
       
    Verify that it is moving by observing that the TTCAMX raw position is increasing by about 500 cts/sec, then stop it:

    saomot.io ctrl1aux='STF'

    For TTCAMY:

     
       saomot.io ctrl1aux='SHE' 
       saomot.io ctrl1aux='BCE;JGE=500;BGE' 
       
    Verify that it's moving by observing that the TTCAMY raw position is increasing by about 500 cts/sec, then stop it:

     
       saomot.io ctrl1aux='STE' 
       
    You can now resume normal controls. The torque should no longer be at max output.

Trouble issue(s): 11866; 11819; 11788.

2019-05-30:
TTCamX stage won't calibrate:

It appears to move fine until it gets to the Find Index stage, where it turns the motor off and gives:

 
   error -5413 "Decel or stop due to Off-on-Error - can be positioning error, amplifier error, or an 'abort' input." 
   

It moves down fine, but has a lot of trouble going up. I can sometimes get it to go up if I give it a helping hand. I'm suspecting we need better tuning (happening tomorrow), but maybe a mechanical help of a stronger spring might make a difference?

Trouble issue(s): 11866.

2019-04-18:
TTCam stage workaround for the rest of the run:

TTCamY stage won't move properly at all at Dec < 7 degrees, Hence LGS mode dithering or field steering is not possible at Dec < 7 degrees.

If observing at a 7 < Dec < 20 degrees, set OFE=-2.5: user@shred: saomot.io ctrl1aux='OFE=-2.5'

If observing at a Dec > 20 deg, set OFE=0: user@shred: saomot.io ctrl1aux='OFE=0'

If you have difficulty at Decs between 30 and 37, try using OFE=-2.5. user@shred: saomot.io ctrl1aux='OFE=-2.5'

If failures continue, employ on-axis LGS observing and nod to sky to get backgrounds instead of dithering.

Trouble issue(s): 11819; 11788.

2019-04-18:
Intervention to address recent TT Cam X, TT Cam Y motor failures. Limited success.

On receiving instrument for night operation, beam path was found to be obstructed by Cal Y stage. Galil motor controller 2 (which controls Cal Y stage via channel F) was found to be in error state. Inspection of cabling to Galil motor controller 2 revealed terminal strip fan-out connector/card that attaches externally to the Galil controller was not properly seated (despite the caputure screws being securely fastened). Trouble: 11788; 11819; 11820.

2019-02-01:
Replacement Galil hardware and accompanying upgraded firmware and code installed. This should reslove a number of troubles with TTCam brushless motors (e.g. Lenslet, TTCamX, TTCamY). Note:
  1. Whenever the Galils are power-cycled, the saomot dispatchers must be restarted.
  2. To calibrate the brushless motor stages (lenslet, TTCamX, and TTCamY) after a Galil power-cycle, it is necessary to issue the calibrate commnad twice.
Trouble: 11717.
2018-12-07:
Update of procedure to accommodate USAF Space Command additional/extra shutdowns.
2018-10-26:
  • Introduction of 8x NGS mode scripts:

    /local/u/user/observers/lgs/scripts/saoBxytmp8xNGS <nexpo> (center)

    and,

    /local/u/user/observers/lgs/scripts/on-axis8

2018-08-29:
2018-05-30--2018-07-02:
  • ShARCS on UCSC campus for overhaul.

    Trouble: 11269; Trouble: .

    ShARCS returned to the mountain after filter wheel repairs. After the first round of ShARCS servicing it was discovered that filter wheel #2 was now repaired, but filter wheel #1 no longer moved. When warmed up, ShARCS #1 moved fine, so there was clearly something wrong when the dewar was cold. Over the (2018-05-19, 2018-05-20) weekend Dale Sandford took things apart, rebuilt wheel #1 with the correct bearing, and cooled and tested. ShARCS is now (2018-05-22) cooling and just got cold enough to test motor motion.

    Right now everything looks like it is working, need to do the following:

    1. Check pupil alignment
    2. Check repeatability of aperture wheel positioning
    3. Check general alignment
    Right now general alignment looks OK, with the white light source coming in close to the position it was in before on ShARCS when loops are closed.
2018-04-05--2018-05-15:
  • ShARCS on UCSC campus for investigation/replacement of filter wheel bearings and all other necessary measurements (o-rings, etc.) preparatory to the servicing/overhaul foreseen for June 2018.

    Trouble: 11269.

2018-03-09:
ShARCS Filter Wheel Bearings:
  • ShARCS filter wheel #2 bearing is failing so its motion has been disbled with the filter wheel locked in the open position. Any filters in filter wheel #2 (e.g. Fe II; CH-1.2) will not be available for the AO run between 2018-03-26--2018-04-03.

    For Darks, use alternative script:

    user@shimmy > /u/user/observers/lgs/scripts/darkpos

    The ShARCS dewar is scheduled to be shipped to UCSC campus for investigation during April 2018 and repair during June 2018.

2018-02-XX:
2018-01-30:
  • Shane primary mirror support readjustment and migration of LGS parameters.
2017-12-28:
  • For LGS operations slinelamp_fe Spare2 power supply outlet has been repaired and laser TT mirror has been re-connected to the Spare2 power supply. Spare2 should again be used routinely from now on (following 2017-11-29 fall-back to Spare1).

    Laser startup checklist in SAO_ops.html and the shutdown checklist in SAO_shutdown (as root on mtham) have been updated to reflect this change.

2017-12-03:
    Low ambient temperature (circa 37 °F / 2.7 °C):
  • WFS lenslet array fails to move from 16x to 8x mode.
    Trouble: 10159.
  • Woofer heater rendered insufficient.
    Trouble: 10160.
2017-12-02:
  • Fiber source stage loose wire at Galil controller.
    Trouble: 11187.
2017-11-29:
  • 2-inch, wider passband Sodium TT filter installed (replaces 1-inch filter, which appeared to introduce severe focus aberration).
  • For LGS operations, in slinelamp_fe GUI activate Spare1 outlet (formerly Spare2 was used, but was found to be unserviceable during the Puetter / AeroSpOpIR visitor instrument run 2017-10-06--2017-10-11).
2017-10-30:
  • Hybrid acquisition procedure and scripts for Hennawi program introduced.
2017-09-28:
  • /local/u/user/observers/lgs/scripts/on-axis script introduced.
2017-08-04:
  • LGS power of 2.3 W compromises observations.
2017-07-10:
  • Limit switched on various ShaneAO motor stages lubricated.
2017-06-30--2017-07-03:
  • ShARCS visits UCSC campus for ion pump replacement.
2017-05-26:
  • Time-critical AO system safety software (e.g. laser shutdown monitor, LSM) is now running on the virtual machine shimmy.
    Previous concerns about running time-critical software on virtual machines have been circumvented by:
    1. Installation of new/replacement 100 GB switch.
    2. Debugging of some of A. Rudy's code to resolve the former hogging by some of those processes.
2017-05-12:
  • The machine formerly known as shard was re-tasked as a replacement for the then-flaky old kaready.
    Trouble: 10919.
2017-03-14--2017-03-15:
2017-03-13:
  • Temporarily run LSM.tcl on shard.ucolick.org instead of virtual machine shred.ucolick.org to circumvent sluggishness issues.
    Trouble: 10812.
2017-03-12:
  • Low return from Sodium layer?
  • Shane line lamp controller toggle switch OFF:
    Trouble: 10810.
  • shred sluggish:
    Trouble: 10811; Trouble: 10812.
2017-03-10:
  • ShARCS USB cable replaced:
    Trouble: 10804.
2017-01-16: User turk consuming CPU:
  • Legitimate user turk has runaway python process: sharcstherm, consuming (100 per cent CPU) resources on shred.ucolick.org, causing safety systems (e.g. Laser Shutdown Monitor, LSM.tcl) to run sluggishly.
    Proposed: Additional host shredly.ucolick.org for dedicated time-critical software.
2017-01-14: Sodium (Na) Filter replaced:
  • When the Na filter is inserted, set cx=-0.03 and cy=-0.03 in the fieldsteer.tcl GUI to ensure centering is correct. Remember to go Back On-Axis after removing the Na filter.
2016-12-18: ShARCS FITS header dates/times:
  • Use the DATE-BEG FITS header keyword in preference to unreliable DATE-OBS, TIME-OBS, TIME-END keywords, as documented in the FITS Header Keyword of ShARCS user manual
    Trouble: 10682
2016-11-30: Miscellaneous updates:
  • Woofer heater thermostat installed: Whenever the temperature in the AO system is less than 55 F, it will turn on the heater.
  • 1-inch Sodium filter replaced 2-inch Sodium filter. On-sky tests pending.
  • Reconnected large number of loose connections on screw terminals.
2016-11-17--2016-11-23: SideCar ASIC IDE Failure:
Cause: Teledyne Scientific Sidecar ASIC IDE software corrupted, possibly resulting from inelegant reboots (via power-cycle) of Windows XP host.
Solution: Back-up files. Uninstall Sidecar Server software. Copy Sidecar Server installer from CVS repositories. Re-install Sidecar software via installer. Copy relevant files (e.g. Test.soln) from back-up directory.
Trouble: 10629
2016-11-22: Spontaneous partial recovery of Tip-Tilt Camera Malfunction:
2016-11-19: VNC Display (1) VNC: AO Crash:
Cause: Interaction with ShaneAO loop control GUI (saocon_gui).
Solution: Re-start 1) shaneao real-time code, 2) ShaneAO loop control GUI (saocon_gui).
Trouble: 10628
2016-11-02: All ShaneAO services migrated to virtual host shred.ucolick.org:
ShaneAO services formerly running on shade.ucolick.org have been moved to shred.ucolick.org. All Shane hosts, including shade.ucolick.org, have had their services files updated to refer to shred.ucolick.org instead of shade.ucolick.org.

Only peeko and ttpeeko remain running on shade.ucolick.org under the gavel account. Those should NOT be moved to shred.ucolick.org at this time.

shade.ucolick.org will no longer start any services at boot time.

2016-10-27: Internet connectivity to ShaneAO system lost, resulting in loss of connection of the sharcs server:
Caused shaneao software to crash.
Solution: Re-boot of real.ucolick.org and re-start of shaneao software.
2016-10-26: Tip-Tilt Camera Malfunction at 250 Hz, 500 Hz, 1000 Hz:
Tip-tilt camera is not functioning at faster rates, with not all quadrants working. However, Tip-tilt to continue functioning at 40 Hz and 125 Hz.
Workaround: use 125 Hz for acquistion of targets and monitoring in NGS. Spontaneous partial recovery (of 250 and 500 Hz) 2016-11-22?
2016 Q3: Modified alignment parameters:
  • Alignment windowing: start x = 1061, end x = 1160, start y = 710, end y = 809
  • Alignment 16x parameters: WFSCam rate (Hz): 250; Tweeter+Woofer gain: 0.2; Tweeter bleed: 0.900; Woofer bleed: 0998
  • Alignment 8x parameters: WFSCam rate (Hz): 250; Tweeter+Woofer gain: 0.1; Tweeter bleed: 0.950; Woofer bleed: 0995
2016-07-XX/2016-08-XX: Intervention to replace motor(s):
WFS Steering Mirror 2 stage did not go back in to the same position. Thus in NGS mode, when dithering by scales larger than 2.5 arcseconds using the conventional scriptproc.tcl, bxy script, waffle patterns are manifested in the tweeter display.
Workaround: The following temporary dithering script can be used:

user@shred.ucolick.org:/observers/lgs/scripts/saoBxy4tmpNGS

2016-01-21: PAM and PRM file naming convention modifications:
  1. Laser Clearing House (LCH) has recently upgraded software to generate PAMs. This upgrade contains several new features, one requiring a simple change on the part of the laser owner/operators.

    As of January 2016, use the following naming convention for all PRM files posted to space-track.org for laser predictive avoidance. This naming convention is required for import into the 14-2 system in this EXACT configuration:

    PRM_LaserOwner_A_B_##Mon####_For_JDAY###_TargetType.txt

    To clarify:

    This naming convention is case sensitive and all fields are required, including a total of 7 underscores. No spaces are allowed, use only the 7 underscores.

    The Laser Owner field is self-explanatory and unchanged from current/previous guidance.

    Fields A and B should be unique to the program, laser, mission, etc. Whatever characters are in between the underscores for A and B will be returned to the laser owner/operator in the PAM file (as shown below). The intent was to provide space for Laser Name and unique ID in these spaces but there is no specific requirement.

    Example: PRM_Lick_589nm_15W_1urad_13kHz_26Jan2016FORJDay_026_AZEL.txt

    Example: PRM_Lick_589nm_15W_1urad_13kHz_26Jan2016FORJDay_026_RADEC.txt

    Date (##Mon####) is date submitted and For_JDAY### is date of activity.

    The 14-2 system will then run the PAM files to be returned to the laser owner/operators in the following naming convention:

    PAM_LaserOwner_A_B_T+000_##Mon####_For_JDAY###_TargetType.txt

    Date (##Mon####) is date run by DECON, T-### is how many days from JDAY window requested by laser owner/operator.

    Example: PAM_Lick_589nm_15W_T-001_25JAN2016_For_JDAY026_AZEL-1.txt

    Example: PAM_Lick_589nm_15W_T-001_25JAN2016_For_JDAY026_RADEC-1.txt

2016-01-04: sharcs_fe modifications:
  1. In Window GUI, Center Zoom sets windowing to zoom region for alignment.
  2. Config and Scripts elimiated from menus.
2015-12-21: sharcsdisplay modifications:
fwhm tool modified from former Lick AO system to:
  1. Make improved (but still not true) Strehl ratio estimates for each wavelength. Strehl ratio ≤ 0.65 (initially 0.65) is the threshold for attemting image sharpening.
  2. Determine filter from image header. If filter is missing from header (sometimes happens with sub-images), FeII is assumed.
2015Q4: Low temperature (≤ 44°F) lenslet array issue.
2015-12-07 (Notification 2015-11-24): space-track.org modification:
On 7 December 2015, Space-Track will change how long files are retained on the FILES Panel. Each folder will have a setting that determines how many days its files are retained before they are deleted. To see files you have access to and their scheduled deletion dates please go to the following URL:

Space-Track: File Deletions

Any file with a deletion date prior to 7 December 2015 will actually be deleted on 7 December 2015.

2015-11-23: space-track.org modification:
Support added for uploading multiple files via a single API command. Visit the Beta site's How To section using the following URL to see examples of uploading/downloading files through the API.

Space-Track: How To Upload Multiple Files

2015-02-17: Laser Shutdown Monitor modification:
Enhancing laser safety. PAM files are processed in the same way (using lsmParse, lsmCheck, and lsmNames) and the new monitor software is:

user@shard:/usr/local/lick/bin/LSM.tcl

The new monitor checks the position listed in the .lsm file (and type, AZEL or RADEC) with the telescope state (position and tracking) in addition to the satellite shutdowns.

If there are problems, use the old software:

user@shard:/usr/local/lick/bin/lsm.tcl

and edit the target .lsm file to remove the first two lines (starting with Type: and Target:, respectively).

The required format for LGS target lists submitted by observers has also been modified. The modified format is described at the following URL:

http://mtham.ucolick.org/techdocs/instruments/ShaneAO/prep/

Observers can use either the slightly revised Lick format or Keck format target lists.

Instructions for how to deal with target lists using targetListParse.tcl are available at the following URL:

http://mtham.ucolick.org/techdocs/techref/ShaneAO/LGStargetLists.html

2015-01-01--2015-01-02: Woofer low temperature (≤ 50°F) issue
Woofer heater fabricated.
2014-09-05--2014-09-09: Detector Controller, stealthie trouble.
2014-05-15--2014-05-18: First science run.
2014-05-09: Draft Operations HTML document structure created (P. Lynam).
2014-05-06--2014-05-11: Second commissioning run
2014-04-09--2014-04-15: First commissioning run
Including first light image of Phi Geminorum.

This document last updated (UTC): Tuesday 04 March 2025