Andor Software Documentation - Troubleshooting

The Observatories of the Carnegie Institution for Science (Carnegie Observatories), Pasadena, CA

Location of this document: http://instrumentation.obs.carnegiescience.edu/Software/Andor/trouble.html

Note: This document currently serves me as a collection place of common problems during operations at Clay and their solutions.
I also keep track of which problems will be avoided by the new design.

Failure Modes, New Design, TroubleShooting

List of Failure Modes

  • ssh (X11-forwarding) crashes
    there is no fix -- need to restart the andorgui and possibly reboot the NUC if the USB driver does not recover.
    The new design avoids X11-forwarding -- the GUI is running locally on the NUC in the control room.
  • VNC lockfile(s) in /tmp after improper shutdown/crash of the system:
    rm -rf /tmp/.X*
  • VNC port changes (unexplicably)
    The new design will use VNC only for displaying the GUI on the observer workstation (guanaco)
    VNC is not critical for the operation of the guider.
  • AndorGUI was started as 'root'; later incarnations will fail to write to existing logfiles or access some resources, in particular the VNC system.
    sudo chown andor /tmp/andor*.log
    sudo rm -rf /tmp/.X*
    The new design should make it un-neccessary to ever use any root privileges (sudo) as there are no drivers on the NUC.
    -- Maybe VNC will require root ?
  • the Andor camera is connected on the USB2 bus (should be USB3)
    -- the 'lsusb' command in a terminal should return something like:
    Bus 002 Device 007: ID 136e:0014 Andor Technology Ltd. Zyla 5.5 sCMOS camera
    Bus 002 Device 001: ID 1d6b:0003 Linux Foundation 3.0 root hub
    Ensure that the 'Andor' camera is on the same bus as the USB3 root hub.

    Advantages of the New Design

  • Only one computer (NUC) per telescope needs to be added to the TCSIS whitelist (allowed to send corrections to the TCS).
  • Each andorserver running on its "minnowXX" only connects to one Andor camera (via USB3).
    It only interacts via TCP/IP (network) with the AndorGUI.
  • A reboot of the NUC should be extremely rare.
    Since there are no special driver or hardware connected to it the AndorGUI should be very stable.
  • Each andorserver (MinnowBoard) can be rebooted remotely from the AndorGUI.
    The GUI (and other guiders) are not affected by a reboot of any andorserver (MinnowBoard).
  • No VNC necessary
    VNC may be used to display the AndorGUI on the observer workstation (guanaco) -- but is not critical the the guider operation.

    The test setup in the lab using 3 Andor camera (right), 3 MinnowBoard computers (blue boxes to the left of the Andors), network switch, copper-fiber converter (just left of the monitor, to the right of the fiber bundle) fiber bundle, fiber-copper converter (left of the fiber bundle), network switch, NUC with monitor, keyboard and mouse:

    Everything up-stream (mostly to the right) of the fiber bundle will be on the platform, while the 2nd converter, switch and the NUC will be in the control room.

    TroubleShooting the Client/Server System (AndorGUI v0.2+)

    no data , stripes , incomplete frame

    No Data

    If the Andor system (usually the USB driver) fails the screen of the andorgui will look like this:

    The first thing you should try is a "reset" command, this will:

    If this works the screen will look like this:

    If the "reset" does not work (eg. "minnowXX not connected to Andor" message),

    please try a "reboot":

    If this fails, please try another "reset".

    If this fails again, it's time to go up to the platform and power-cycle both, the MinnowBoard and the Andor camera:

    1. Turn off the Andor camera and disconnect the "MinnowBoard" from its power supply
    2. Turn on the Andor camera
    3. Connect the "MinnowBoard" to its power supply

    Stripes

    Occasionally the images from the Andor camera might look like this:

    This means the Andor camera itself is in a bad state; this cannot be recorvered from without a power cycle of the Andor camera.

    Incomplete Frame

    If the andorgui connects to the andorserver but no data, or only intermittent data arrives at the GUI, please check on the computer that is connected to the Andor camera via USB3; in a terminal type 
    xterm> lsusb

Bus 002 Device 007: ID 136e:0014 Andor Technology Ltd. Zyla 5.5 sCMOS camera
Bus 002 Device 001: ID 1d6b:0003 Linux Foundation 3.0 root hub
    and make sure that the Andor camera is connected to the USB3 bus (on the same bus as the USB3 root hub).

    TODO: "cannot connect to minnowXX" error message

    2018-07-20, Christoph C. Birk, Carnegie Observatories