Shane AO  

They will still run on karnak host machine if shimmynova exhibits anomalies.

On shimmynova, the commands ta and sa automatically invoke the shaneao conda environment (thus circumventing the former intializing commmand conda activate shaneao).

• Adjusted the TT camera lens a small amount in Z to focus and things now look very good on the TT Camera.

• Bottom row of subapertures in 8x was under-illuminated when on sky but not in the lab. Adjusting Fold 1 mirror gave better Strehl for faint objects. For both 8x and 16x, adjusted Fold 1Ry = 67.50 (old value 68.2) to fine tune left/right centering. Fold 1Rx = 7.50 for best performance on sky for 8x NGS and LGS operations (note, the 16x fold mirror positions work better for the 8x LGS with Na filter). The change in 1Ry position introduces about a 5 arcseconds E/W shift between 8x and 16x. 3 new saomot setups:
  1. default_setup-2023-08-26 with the new default Fold 1 Ry position (otherwise identical to the previous default setting).
  2. 2023-16x-fold1 which only changes Lenslet and Fold 1 positions for 16x operations.
  3. 2023-8x-fold1 which only changes Lenslet and Fold 1 positions for 8x operations>
  The two *fold1 setups are to help to easily switch the Fold1Rx values when switching between 16x and 8x modes.

  While things will work using the default_setup for 16x and 8x, better performance should result in 8x (no Na filter) with the new 8x Fold 1Rx position.

  Performance OK with the laser in 8x mode, no Na filter, with no tweaking of Hartmann modes. Unable to get good performance with the laser with Na filter installed (suspect count levels too low).

  40 Hz on the TT sensor is useless. Better S/N at 125Hz. Recommend using 125Hz instead of 40Hz.

• shadow formerly known as shaman.

Contains the following bullet point regarding Starlink satellites and the LCH:

SpaceX agreed to analyze the impact of astronomical facility lasers on its satellites. Following this analysis, the Laser Clearinghouse removed the coordination requirements for these lasers. Therefore, adaptive optics lasers at ground-based facilities will no longer undergo multiple closures every time the SpaceX satellites pass nearby.

• Refer to: SHANEAO WFS focus, WFS Y, WFS Y / Galil 7 Amplifier Fault.
• 2022-07-08: Recurrence:
  1. In ShaneAO motor control GUI: Actions, Lock stages...; Unselect all.

     Select WFS focus, WFS X & WFS Y.

     Add entry to explanation field (e.g. "Trouble report 12779").

     Select Lock.

  2. Wait a judicious amount of time (e.g. several hours) for locked motors to cool.

     For the interim, use workaround script on shimmy:

     /u/user/observers/lgs/scripts/objsky.tcl <Nexposures> <Xloops>

     This script takes N exposures on axis with the loops closed, opens the loops, moves the telescope 30 arcseconds and takes either N or 3 (whichever is less) sky frames, then goes back on axis and closes loops, repeats X times.

  3. In ShaneAO Power Controls GUI: Power-cycle Galil 7 (C7), with an unpowered phase of 30--60 seconds.

     Following restoration of power, re-calibrate WFS X & WFS Y.

     Following recalibration, set:

     WFS X to -1290

     WFS Y to -7725

• 2022-06-27: WFS X & WFS Y seem to move fine for NGS field steering as long as WFS focus (WFS Z) is off. Therefore, for NGS, execution of pre-ordained scripts via ScriptProc GUI can be resumed.

  Further investigation required to determine the actual issue causing the amplifier faults.

• 2022-06-27: WFS X & WFS Y seem to move fine for NGS field steering as long as WFS focus (WFS Z) is off. Therefore, for NGS, execution of pre-ordained scripts via ScriptProc GUI can be resumed.

  Further investigation required to determine the actual issue causing the amplifier faults.

• Pre-2022-06-27: Use workaround script on shimmy:

  /u/user/observers/lgs/scripts/objsky.tcl <Nexposures> <Xloops>

  This script takes N exposures on axis with the loops closed, opens the loops, moves the telescope 30 arcseconds and takes either N or 3 (whichever is less) sky frames, then goes back on axis and closes loops, repeats X times.

• 2022-05-XX: Recurrence: Galil 7 amplifier fault, affects WFS focus, WFS Y, WFS X motors.

  Trouble issue(s): 12779; 12795.

Trouble issue(s): 12831.

2022-02-19: TT filter wheel removed from ShaneAO and locked out pending further investigation, since a similar problem has been experienced with this device. Repaired and re-installed.

TT filter wheel in SAO needed power cycling (2022-02-17) and service restarting to move. The problem got worse 2022-02-18, when occassionally the filter wheel would move properly, but an error would soon occur, putting the filter wheel in a bad position (half way between filters).

Trouble issue(s): 12831.

/u/user/observers/lgs/scripts/objsky.tcl <Nexposures> <Xloops>

This script takes N exposures on axis with the loops closed, opens the loops, moves the telescope 30 arcseconds and takes either N or 3 (whichever is less) sky frames, then goes back on axis and closes loops, repeats X times.

Trouble issue(s): 12779; 12795.

user@shimmy:∼/observers/lgs/scripts/lgs-ttroi.tcl N

where N is the integer number exposures at each point in the 5-point dither pattern.

Problem: Occasionally, communication to the Sci Filter wheel becomes unresponsive. The recovery is to re-start the saomot5 service.

The saomot GUI has changed slightly to reflect the new working state of the motor and the two WFS Dichroic options (NIR Splitter and Empty). For now we shall always want the NIR Splitter in until such time as we get a visible dichroic beam splitter so we can do I-band (and maybe R-band) imaging with ShARCS.

The following URL has been fully updated:

http://mtham.ucolick.org/techdocs/instruments/ShaneAO/saomot/

Other web pages that refer to this stage and shall be updated.

There is one pending repair to the WFS Dichroic stage, which is to fix the cable loopback signal, which is currently not working. This is not critical for operations, so will likely be fixed in April 2021 or as soon as Gabe can find time to get up to the mountain to look into the SAO wiring.

TTCAMXLCK = Motors removed

TTCAMYLCK = Motors removed

WFSDICHRLCK = Broken Hall sensor

• Re-commissioning of ShaneAO system follwing intervention to replace Wave Front Sensor mechanism.

  The content of some GUIs has been modified as a result.

  Some motors/keywords, formerly used by the pre-2020-10-26 WFS mechanism have been removed (e.g. STEER2RY, STEER2RX, STEER1RX, STEER1RY, WFSFOCUS).

The following services now run on shark:

• saomot1
• saomot2
• saomot3
• saomot4
• saomot5
• saomot7
• saopower
• saomon1
• saomon2
• shawcs
• mako
• sharcstherm (after updating the Agilent code to work with recent changes to the python dispatcher library).

The services file no longer lists shred as a runhost for any of the AO dispatchers. Therefore, if we need to revert to running one or more dispatchers on shred, use:

 
   ao start!   
   

instead of simply:

 
   ao start   
   

• P. Zachary writes:

  I removed the brakes from both the X and Y axis and tried moving the stages by hand, both seemed just fine. Next, I put a load on the stage and in the horizontal orientation things seemed fine but when I raised the stage to vertical I saw 8 inch-Lbs of force needed to hold the stage and about 11 needed to move it. This continued for almost half the stage travel at which point the 8 inch-lbs became zero and the force needed to move it rose very slightly. From this it was obvious that there was a significant amount of drag introduced to the system.

  Disconnecting the lead screw made it obvious the the slide bearings were just fine and the problem was in the motor drive. Loosening the motor drive from the stage showed the ball-nut was free to move and the binding was absolutely in the motor itself.

  Disassembling the motor seems to indicate the front bearings are fine and the problem was the rotor interfering with the windings. There is some chance this contamination originated in the manufacture or handling of the motor or stage since it is not completely sealed but I feel there is a high probability this contamination this material may have originated from within the brake located on the back (and unfortunately above) of the motor. I'm cleaning the motor out now and then we will wire it up for Will to test. I expect to be done with the mechanical part later today but not sure when D. Black might have it wired for W. Deich to test.

  Overall, it feels like neither an endorsement nor condemnation of the Griffin stage but is something to remember should we have problems in future. It seems to me that it might be worth making sure the design wiring allows removal of a motor without too much pain and then we could buy a motor to have in case of trouble.

  D. Cowley and C. Ratcliffe both feel it is not worth reinstalling this stage in any case since the upgrade is just around the corner.

• E. Gates writes:

  I moved the telescope to see how the TTCamY stage moves at different declinations. Movement is improved with the higher torque limit, but we still cannot move the stage properly everywhere on the sky. I believe we can do dithering as normal in the Dec range of +15 degrees to +70 degrees, with the lower declination limit being a little uncertain. Certainly at a Dec of +7 deg (30 degrees south of zenith), the stage could not do moves against gravity successfully. At Dec=+7 degrees, the torque was pegged at its maximum (-6.25V) just holding its position. Our observer (Jon Zink) on Aug 21st only has targets below Dec +7 deg, so his program will be unable to use LGS dithering for any of their targets, so we will have to work around that doing any offsets with the field steer motors with the telescope near zenith, changing the mode to Pos OFF, then acquiring their off-axis guide stars, and doing separate sky background frames.

  I believe we need to further investigate the counterweight option for TTCAMY so we can have proper motion over the entire 45 degree zenith distance range of LGS operations.

  We also have temperature monitoring of the motor housing working, so I will keep track of that through the night to make sure we do not cook the motor. Temperature of the motor housing right now is about 42C with no extra gravity load, which I am going to assume is normal.

  W. Deich writes:

  Yesterday afternoon I did some detailed tests to characterize the motor further. Dale had posited that the Hall sensors might not be reliably showing correct voltage, so I did a check of that, too.

  One of the tests was to power-cycle the Galil -- to ensure consistent starting conditions -- and then do a simple brushless init, carefully monitoring the Galil-reported brushless phase. The init consists of:

   
     BIF=-1		# Tell the Galil to use standard Hall inputs 
     BCF		# Tell Galil to learn the precise phase angle. 
     SHA		# Motor on 
     SPA=500		# Slow speed... 
     PRA=2500	# ...for 2500 counts of motion 
     BGA		# Begin motion. 
     

  Two noteworthy points:

  a) Sometimes the motor would draw excessive current as soon as 'SHA' was given, which suggests the Galil did not get the correct phase from the Halls. One thing I did *not* do, which in hindsight would be useful, is to record the Hall sensor values immediately before and after power-cycling the Galil. That would provide a check on whether it is remaining correct.

  b) I monitored the Hall sensor outputs while the galil moved through circa 120 changes in Hall value. Each time, the transition came at about the expected number of encoder counts (1111.1111), and the Halls showed the expected value.

• If going to Dec < 15 for NGS or LGS, put the TTCamX and TTCamY motors in Pos OFF mode while near zenith before moving the telescope to the lower Dec target.

  Procedures for LGS field steering for lower Decs are on the LickWiki.

  We are monitoring the torque and temperature of the TTCamY motor:

  gshow -s saomot -c ttcamytor

  gshow -s sharcstherm -c testa

  If the temperature of the TTCamY motor gets to 50 C we need to move closer to zenith and set it to Pos OFF mode to let it cool off.

  If the torque is pegged at +6.25 V or -6.25 V it will heat up the motor and will fail to do dither moves correctly. If the torque is close to one of these values, LGS dither move might not work correctly.

We recently encountered an odd error with the TTCAMX and TTCAMY motors: the Galil claimed it was putting out maximum continuous torque (indicated by +-4V on TTCAMX, +-3V on TTCAMY) to try and hold position at angles where much less should be fine.

This was the case on Sunday night (2019-06-16), when TTCAMX was unable to move correctly, and it was the case Sunday morning, when TTCAMY was misbehaving.

The Galil has to know the brushless phase (with respect to the motor windings). At initial power-up, it doesn't know the phase precisely, so it uses the Hall sensors to get a good enough value - it will typically be in error by 30 degrees, but this still is enough to move the motor. We tell it to learn the very accurate phase by watching for the moment at which the on Hall sensors switch state. If the Galil has the wrong phase, it will be range between more difficult and impossible to move the motor.

On the off chance that the Galil somehow lost brushless phase information, I told the Galil to re-learn the phase, and that resolved the problem. Here's how to do this.

(I do not know why TTCAMX and TTCAMY were at incorrect phase. The Galil claimed that the axes were brushless-initialized, yet a re-init corrected their behavior. It would require a combination of the motor encoder not reading out, while the stage drifted by a hundred counts or more.)

1. First, this will not help if the Galil requires the usual amount of 'torque' (voltage to the amplifier), and is simply holding position poorly.

   This might help if the Galil is putting out full power (+- 4V on TTCAMX; +- 3V on TTCAMY) to hold, when you are pretty sure that is not normal.

2. Put the telescope at zenith (more or less).
3. Put the stage into the Halt state, using the GUI.
4. From shred (it has the saomot.io command installed), issue low-level commands to turn on the motor, start a "brushless calibration", and start jogging slowly:

   For TTCAMX:

    
      saomot.io ctrl1aux='SHF' 
      saomot.io ctrl1aux='BCF;JGF=500;BGF' 
      

   Verify that it is moving by observing that the TTCAMX raw position is increasing by about 500 cts/sec, then stop it:

   saomot.io ctrl1aux='STF'

   For TTCAMY:

    
      saomot.io ctrl1aux='SHE' 
      saomot.io ctrl1aux='BCE;JGE=500;BGE' 
      

   Verify that it's moving by observing that the TTCAMY raw position is increasing by about 500 cts/sec, then stop it:

    
      saomot.io ctrl1aux='STE' 
      

   You can now resume normal controls. The torque should no longer be at max output.

Trouble issue(s): 11866; 11819; 11788.

• TTCamX stage won't calibrate:

  It appears to move fine until it gets to the Find Index stage, where it turns the motor off and gives:

   
     error -5413 "Decel or stop due to Off-on-Error - can be positioning error, amplifier error, or an 'abort' input." 
     

  It moves down fine, but has a lot of trouble going up. I can sometimes get it to go up if I give it a helping hand. I'm suspecting we need better tuning (happening tomorrow), but maybe a mechanical help of a stronger spring might make a difference?

  Trouble issue(s): 11866.

• On receiving instrument for night operation, beam path was found to be obstructed by Cal Y stage. Galil motor controller 2 (which controls Cal Y stage via channel F) was found to be in error state. Inspection of cabling to Galil motor controller 2 revealed terminal strip fan-out connector/card that attaches externally to the Galil controller was not properly seated (despite the caputure screws being securely fastened).

  Trouble: 11788; 11819; 11820.

• TTCam stage workaround for the rest of the run:

  TTCamY stage won't move properly at all at Dec < 7 degrees, Hence LGS mode dithering or field steering is not possible at Dec < 7 degrees.

  If observing at a 7 < Dec < 20 degrees, set OFE=-2.5: user@shred: saomot.io ctrl1aux='OFE=-2.5'

  If observing at a Dec > 20 deg, set OFE=0: user@shred: saomot.io ctrl1aux='OFE=0'

  If you have difficulty at Decs between 30 and 37, try using OFE=-2.5. user@shred: saomot.io ctrl1aux='OFE=-2.5'

  If failures continue, employ on-axis LGS observing and nod to sky to get backgrounds instead of dithering.

  Trouble issue(s): 11819; 11788.

1. Whenever the Galils are power-cycled, the saomot dispatchers must be restarted.
2. To calibrate the brushless motor stages (lenslet, TTCamX, and TTCamY) after a Galil power-cycle, it is necessary to issue the calibrate command twice.

• Update of procedure to accommodate USAF Space Command additional/extra shutdowns.

• NGS 8x mode dither script:

  /local/u/user/observers/lgs/scripts/saoBxytmp8xNGS <Number_Exposures> (center)

  where;

  • <Number_Exposures> is the integer number of exposures at each position of a 4-point, or (optionally) 5-point, dither pattern with a fixed scale of 5 arcseconds.
  • center is an optional argument to take an additional set of <nexpo> exposures in the center point of the dither pattern, thus adding a 5th dither position.
• NGS 8x mode on-axis script:

  /local/u/user/observers/lgs/scripts/on-axis8

These scripts introduced in addition to the 16x mode NGS mode dithering scripts (for scales larger than 2.5 arcseconds):

• NGS 16x mode dither script (introduced 2016-09-14):

  /local/u/user/observers/lgs/scripts/saoBxy4tmpNGS <Dither_Scale> <Number_Exposures>

  where;

  • <Dither_Scale> is an integer number of arcseconds. As of 2016-09-14 the value can only be 1, 2, 3 or 4 (the only calibrated distances as of 2016-09-14)
  • <Number_Exposures> is the integer number of exposures at each dither position.
• NGS 16x mode on-axis script (introduced 2017-09-28):

  /local/u/user/observers/lgs/scripts/on-axis

• WFS Steer Motor 1 Ry failing to achieve final dither position.

  Temporary workaround script deployed.

  Trouble: 11537.

• Trouble: 11269; Trouble: .

  ShARCS returned to the mountain after filter wheel repairs. After the first round of ShARCS servicing it was discovered that filter wheel #2 was now repaired, but filter wheel #1 no longer moved. When warmed up, ShARCS #1 moved fine, so there was clearly something wrong when the dewar was cold. Over the (2018-05-19, 2018-05-20) weekend Dale Sandford took things apart, rebuilt wheel #1 with the correct bearing, and cooled and tested. ShARCS is now (2018-05-22) cooling and just got cold enough to test motor motion.

  Right now everything looks like it is working, need to do the following:

  1. Check pupil alignment
  2. Check repeatability of aperture wheel positioning
  3. Check general alignment
  Right now general alignment looks OK, with the white light source coming in close to the position it was in before on ShARCS when loops are closed.

• Investigation/replacement of filter wheel bearings and all other necessary measurements (o-rings, etc.) preparatory to the servicing/overhaul foreseen for June 2018.

  Trouble: 11269.

• ShARCS filter wheel #2 bearing is failing so its motion has been disbled with the filter wheel locked in the open position. Any filters in filter wheel #2 (e.g. Fe II; CH-1.2) will not be available for the AO run between 2018-03-26--2018-04-03.

  For Darks, use alternative script:

  user@shimmy > /u/user/observers/lgs/scripts/darkpos

  The ShARCS dewar is scheduled to be shipped to UCSC campus for investigation during April 2018 and repair during June 2018.

• Issue of new Letter of Determination (LoD), valid for period 2018-04-02--2020-04-02.

• Shane primary mirror support readjustment and migration of LGS parameters.

• For LGS operations slinelamp_fe Spare2 power supply outlet has been repaired and laser TT mirror has been re-connected to the Spare2 power supply. Spare2 should again be used routinely from now on (following 2017-11-29 fall-back to Spare1).

  Laser startup checklist in SAO_ops.html and the shutdown checklist in SAO_shutdown (as root on mtham) have been updated to reflect this change.

• WFS lenslet array fails to move from 16x to 8x mode.
  Trouble: 10159.
• Woofer heater rendered insufficient.
  Trouble: 10160.

• Fiber source stage loose wire at Galil controller.
  Trouble: 11187.

• 2-inch, wider passband Sodium TT filter installed (replaces 1-inch filter, which appeared to introduce severe focus aberration).
• For LGS operations, in slinelamp_fe GUI activate Spare1 outlet (formerly Spare2 was used, but was found to be unserviceable during the Puetter / AeroSpOpIR visitor instrument run 2017-10-06--2017-10-11).

• /local/u/user/observers/lgs/scripts/on-axis script introduced.

• Time-critical AO system safety software (e.g. laser shutdown monitor, LSM) is now running on the virtual machine shimmy.
  Previous concerns about running time-critical software on virtual machines have been circumvented by:
  1. Installation of new/replacement 100 GB switch.
  2. Debugging of some of A. Rudy's code to resolve the former hogging by some of those processes.

• Trouble: 10919.

• White Light Source. No spare, so must order spare.
  Trouble: 10815.

• Low return from Sodium layer?
• Shane line lamp controller toggle switch OFF:
  Trouble: 10810.
• shred sluggish:
  Trouble: 10811; Trouble: 10812.

• ShARCS USB cable replaced:
  Trouble: 10804.

• Legitimate user turk has runaway python process: sharcstherm, consuming (100 per cent CPU) resources on shred.ucolick.org, causing safety systems (e.g. Laser Shutdown Monitor, LSM.tcl) to run sluggishly.
  Proposed: Additional host shredly.ucolick.org for dedicated time-critical software.

• When the Na filter is inserted, set cx=-0.03 and cy=-0.03 in the fieldsteer.tcl GUI to ensure centering is correct. Remember to go Back On-Axis after removing the Na filter.

• Use the DATE-BEG FITS header keyword in preference to unreliable DATE-OBS, TIME-OBS, TIME-END keywords, as documented in the FITS Header Keyword of ShARCS user manual
  Trouble: 10682.

• Woofer heater thermostat installed: Whenever the temperature in the AO system is less than 55 F, it will turn on the heater.
• 1-inch Sodium filter replaced 2-inch Sodium filter. On-sky tests pending.
• Reconnected large number of loose connections on screw terminals.

Only peeko and ttpeeko remain running on shade.ucolick.org under the gavel account. Those should NOT be moved to shred.ucolick.org at this time.

shade.ucolick.org will no longer start any services at boot time.

• Alignment windowing: start x = 1061, end x = 1160, start y = 710, end y = 809
• Alignment 16x parameters: WFSCam rate (Hz): 250; Tweeter+Woofer gain: 0.2; Tweeter bleed: 0.900; Woofer bleed: 0998
• Alignment 8x parameters: WFSCam rate (Hz): 250; Tweeter+Woofer gain: 0.1; Tweeter bleed: 0.950; Woofer bleed: 0995

user@shred.ucolick.org:/observers/lgs/scripts/saoBxy4tmpNGS

1. Laser Clearing House (LCH) has recently upgraded software to generate PAMs. This upgrade contains several new features, one requiring a simple change on the part of the laser owner/operators.

   As of January 2016, use the following naming convention for all PRM files posted to space-track.org for laser predictive avoidance. This naming convention is required for import into the 14-2 system in this EXACT configuration:

   PRM_LaserOwner_A_B_##Mon####_For_JDAY###_TargetType.txt

   To clarify:

   This naming convention is case sensitive and all fields are required, including a total of 7 underscores. No spaces are allowed, use only the 7 underscores.

   The Laser Owner field is self-explanatory and unchanged from current/previous guidance.

   Fields A and B should be unique to the program, laser, mission, etc. Whatever characters are in between the underscores for A and B will be returned to the laser owner/operator in the PAM file (as shown below). The intent was to provide space for Laser Name and unique ID in these spaces but there is no specific requirement.

   Example: PRM_Lick_589nm_15W_1urad_13kHz_26Jan2016FORJDay_026_AZEL.txt

   Example: PRM_Lick_589nm_15W_1urad_13kHz_26Jan2016FORJDay_026_RADEC.txt

   Date (##Mon####) is date submitted and For_JDAY### is date of activity.

   The 14-2 system will then run the PAM files to be returned to the laser owner/operators in the following naming convention:

   PAM_LaserOwner_A_B_T+000_##Mon####_For_JDAY###_TargetType.txt

   Date (##Mon####) is date run by DECON, T-### is how many days from JDAY window requested by laser owner/operator.

   Example: PAM_Lick_589nm_15W_T-001_25JAN2016_For_JDAY026_AZEL-1.txt

   Example: PAM_Lick_589nm_15W_T-001_25JAN2016_For_JDAY026_RADEC-1.txt

1. In Window GUI, Center Zoom sets windowing to zoom region for alignment.
2. Config and Scripts elimiated from menus.

1. Make improved (but still not true) Strehl ratio estimates for each wavelength. Strehl ratio ≤ 0.65 (initially 0.65) is the threshold for attemting image sharpening.
2. Determine filter from image header. If filter is missing from header (sometimes happens with sub-images), FeII is assumed.

Space-Track: File Deletions

Any file with a deletion date prior to 7 December 2015 will actually be deleted on 7 December 2015.

Space-Track: How To Upload Multiple Files

user@shard:/usr/local/lick/bin/LSM.tcl

The new monitor checks the position listed in the .lsm file (and type, AZEL or RADEC) with the telescope state (position and tracking) in addition to the satellite shutdowns.

If there are problems, use the old software:

user@shard:/usr/local/lick/bin/lsm.tcl

and edit the target .lsm file to remove the first two lines (starting with Type: and Target:, respectively).

The required format for LGS target lists submitted by observers has also been modified. The modified format is described at the following URL:

http://mtham.ucolick.org/techdocs/instruments/ShaneAO/prep/

Observers can use either the slightly revised Lick format or Keck format target lists.

Instructions for how to deal with target lists using targetListParse.tcl are available at the following URL:

http://mtham.ucolick.org/techdocs/techref/ShaneAO/LGStargetLists.html