FourStar - Troubleshooting

The Observatories of the Carnegie Institution for Science (Carnegie Observatories), Pasadena, CA


Location of this document: http://instrumentation.obs.carnegiescience.edu/Software/FourStar/trouble.html


  • Alerts
  • SAN/RAID Disk Problems
  • Cloning a DataServer
  • LN2-Processor Parameters


    Alerts

    The follwing table contains a list of email alerts that might be sent out by the FourStar software.
    All 'follow-up' actions include a report to the FourStar team (fourstar@lco.cl).

    Message Trigger Immediate Action Follow Up
    Shroud-Dewar temperature high - Shroud-Dewar might need a top-off dewar temperature greater than 85 K check shroud dewar fill level, refill manually if low investigate why the dewar was not filled [1]
    Shroud-Dewar temperature high - LN2 level might be low dewar temperature greater than 87 K check shroud dewar fill level, refill manually if low investigate why the dewar was not filled [1]
    Last shroud dewar fill more that x.x hours ago No temperature drop detected for more than 12 hours check shroud dewar fill level, refill manually if low investigate why the dewar was not filled [2]
    DetectorTempAlarm: low/high The detector temperature deviates more than 0.1 K from the 'Target' (80 K) TBD
    OpticsTempAlam: low/high The optics temperature deviates more than 0.5 K from the 'Target' (200 K) TBD
    Detector-Heater power low - LN2 level might be low The detector heater power is below 0.1 W check both dewars, fill if low investigate why the dewar was not filled [3]
    LN2-Processor Alarm LN2 level is below "empty" refill the shroud dewar
    LN2-Processor Error LN2 processor malfunction reboot (power cycle) LN2 processor [7] replace LN2 processor if problem persists
    E-rack temperature is xx.x [C] E-rack temperature above 30 C Power off the 2 JADEs, 4 PCs and motor controller inside the electronics racks [4]
    LN2-Level (Detector/Shroud) at x.x % LN2 level below 10% refill dewar manually investigate why the dewar was not filled
    Dewar pressure high (x.x Torr) - LN2 level might be low vacuum worse than 1.5e-5 Torr check shroud dewar level, refill manually if low If the pressure does not go down right away, put the instrument on the turbo-pump [5]
    LN2-Solenoid open for X minutes The valve of the LN2 autofill system stays open for more that 40 minutes check the re-fill tank, replace if empty [8]
    LN2-Solenoid closed for x.x hours The valve of the LN2 autofill system has not opened for more than 12 hours check status of LN2 processor, reboot [7] [6]

    Notes:

    1. If the shroud dewar temperature rises above 90 K the vacuum will quickly deteriorate. This may cause damage to the the instrument. If this happens turn off the ion-pump, put the instrument on the turbo-pump and contact the FourStar team immediately.
    2. This alert is only created in dead-reckoning mode.
    3. If this alert was not caused by a low LN2 level this needs to be investigated.
    4. Please note that opening the E-rack doors does not improve the cooling; it actually raises the temperature inside the racks because it impedes the airflow inside the rack.
    5. If the dewar was full then FourStar might have a leak. Contact the FourStar team immediately.
    6. This alert is only created in dead-reckoning mode.
    7. Please note that the LN2-procssor (auto-fill system) is in pass-through mode after a reboot. It is necessary to set the mode to Automatic in the 'LN2 & Sensors" GUI even though the GUI might indicate otherwise.
    8. Automatic fill mode only.


    SAN/RAID Disk Problems

    If the SAN/RAID disk fails (or the pipeline computers that manage the SAN) the FourStar software will become unable to access the SAN-disk and may become unresponsive.
    Please clear the /Volumes/FourStarSAN/... field in the menu Camera - Set DataPaths:

    If the FourStar software is already unresponsive you might have to terminate (Force Quit) the FourStar-GUI and edit its preferences file (keyword datapath2) at

    ~/Library/Preferences/edu.carnegiescience.obs.fourstar.plist 
    
    before restarting the FourStar software.

    For the proper shutdown/re-start sequence of the SAN/RAID system please read this article
    Xsan: Shutdown and startup order for network devices
    on the Apple website.


    Cloning a DataServer

    Copy Source

    Create Target

    Setup Target

    Install driver for fast network (Intel Pro 1000)
    ... (call David Murphy)
    Change IP number
    192.168.1.x
    Change Host name
    ControlPanel - System - ComputerName 'FS-DA-X'


    2009-10-08, Christoph C. Birk, Carnegie Observatories (Last Update: 2015-12-10)