FourStar - Troubleshooting

The Observatories of the Carnegie Institution for Science (Carnegie Observatories), Pasadena, CA

Location of this document: http://instrumentation.obs.carnegiescience.edu/Software/FourStar/trouble.html

Alerts

The follwing table contains a list of email alerts that might be sent out by the FourStar software.
All 'follow-up' actions include a report to the FourStar team (fourstar@lco.cl).

Message Trigger Immediate Action Follow Up

Shroud-Dewar temperature high - Shroud-Dewar might need a top-off dewar temperature greater than 85 K check shroud dewar fill level, refill manually if low investigate why the dewar was not filled [1]

Shroud-Dewar temperature high - LN2 level might be low dewar temperature greater than 87 K check shroud dewar fill level, refill manually if low investigate why the dewar was not filled [1]

Last shroud dewar fill more that x.x hours ago No temperature drop detected for more than 12 hours check shroud dewar fill level, refill manually if low investigate why the dewar was not filled [2]

DetectorTempAlarm: low/high The detector temperature deviates more than 0.1 K from the 'Target' (80 K) TBD

OpticsTempAlam: low/high The optics temperature deviates more than 0.5 K from the 'Target' (200 K) TBD

Detector-Heater power low - LN2 level might be low The detector heater power is below 0.1 W check both dewars, fill if low investigate why the dewar was not filled [3]

LN2-Processor Alarm LN2 level is below "empty" refill the shroud dewar

LN2-Processor Error LN2 processor malfunction reboot (power cycle) LN2 processor [7] replace LN2 processor if problem persists

E-rack temperature is xx.x [C] E-rack temperature above 30 C Power off the 2 JADEs, 4 PCs and motor controller inside the electronics racks [4]

LN2-Level (Detector/Shroud) at x.x % LN2 level below 10% refill dewar manually investigate why the dewar was not filled

Dewar pressure high (x.x Torr) - LN2 level might be low vacuum worse than 1.5e-5 Torr check shroud dewar level, refill manually if low If the pressure does not go down right away, put the instrument on the turbo-pump [5]

LN2-Solenoid open for X minutes The valve of the LN2 autofill system stays open for more that 40 minutes check the re-fill tank, replace if empty [8]

LN2-Solenoid closed for x.x hours The valve of the LN2 autofill system has not opened for more than 12 hours check status of LN2 processor, reboot [7] [6]

Notes:

If the shroud dewar temperature rises above 90 K the vacuum will quickly deteriorate. This may cause damage to the the instrument. If this happens turn off the ion-pump, put the instrument on the turbo-pump and contact the FourStar team immediately.
This alert is only created in dead-reckoning mode.
If this alert was not caused by a low LN2 level this needs to be investigated.
Please note that opening the E-rack doors does not improve the cooling; it actually raises the temperature inside the racks because it impedes the airflow inside the rack.
If the dewar was full then FourStar might have a leak. Contact the FourStar team immediately.
This alert is only created in dead-reckoning mode.
Please note that the LN2-procssor (auto-fill system) is in pass-through mode after a reboot. It is necessary to set the mode to Automatic in the 'LN2 & Sensors" GUI even though the GUI might indicate otherwise.
Automatic fill mode only.

SAN/RAID Disk Problems

If the SAN/RAID disk fails (or the pipeline computers that manage the SAN) the FourStar software will become unable to access the SAN-disk and may become unresponsive.
Please clear the /Volumes/FourStarSAN/... field in the menu Camera - Set DataPaths:

If the FourStar software is already unresponsive you might have to terminate (Force Quit) the FourStar-GUI and edit its preferences file (keyword datapath2) at

~/Library/Preferences/edu.carnegiescience.obs.fourstar.plist

before restarting the FourStar software.

For the proper shutdown/re-start sequence of the SAN/RAID system please read this article
Xsan: Shutdown and startup order for network devices
on the Apple website.

Cloning a DataServer

Copy Source

Connect the external USB disk to source PC
Boot from Acronis CDROM (check BIOS setting)
Select 'Full Acronis Version'
Task: 'Clone Disk'
Select source: primary disk
Select destination: USB
Select 'Delete Partitions on destination disk'
ok .. ok .. ok
Exit
remove CDROM and reboot

Create Target

Connect the external USB disk to target PC
Boot from Acronis CDROM (check BIOS setting)
Select 'Full Acronis Version'
Task: 'Clone Disk'
Select source: USB
Select destination: primary disk
Select 'Delete Partitions on destination disk'
ok .. ok .. ok
Exit
remove CDROM and reboot

Setup Target

Install driver for fast network (Intel Pro 1000): ... (call David Murphy)
Change IP number: 192.168.1.x
Change Host name: ControlPanel - System - ComputerName 'FS-DA-X'

2009-10-08, Christoph C. Birk, Carnegie Observatories (Last Update: 2015-12-10)

Message	Trigger	Immediate Action	Follow Up
Shroud-Dewar temperature high - Shroud-Dewar might need a top-off	dewar temperature greater than 85 K	check shroud dewar fill level, refill manually if low	investigate why the dewar was not filled [1]
Shroud-Dewar temperature high - LN2 level might be low	dewar temperature greater than 87 K	check shroud dewar fill level, refill manually if low	investigate why the dewar was not filled [1]
Last shroud dewar fill more that x.x hours ago	No temperature drop detected for more than 12 hours	check shroud dewar fill level, refill manually if low	investigate why the dewar was not filled [2]
DetectorTempAlarm: low/high	The detector temperature deviates more than 0.1 K from the 'Target' (80 K)	TBD
OpticsTempAlam: low/high	The optics temperature deviates more than 0.5 K from the 'Target' (200 K)	TBD
Detector-Heater power low - LN2 level might be low	The detector heater power is below 0.1 W	check both dewars, fill if low	investigate why the dewar was not filled [3]
LN2-Processor Alarm	LN2 level is below "empty"	refill the shroud dewar
LN2-Processor Error	LN2 processor malfunction	reboot (power cycle) LN2 processor [7]	replace LN2 processor if problem persists
E-rack temperature is xx.x [C]	E-rack temperature above 30 C	Power off the 2 JADEs, 4 PCs and motor controller inside the electronics racks	[4]
LN2-Level (Detector/Shroud) at x.x %	LN2 level below 10%	refill dewar manually	investigate why the dewar was not filled
Dewar pressure high (x.x Torr) - LN2 level might be low	vacuum worse than 1.5e-5 Torr	check shroud dewar level, refill manually if low	If the pressure does not go down right away, put the instrument on the turbo-pump [5]
LN2-Solenoid open for X minutes	The valve of the LN2 autofill system stays open for more that 40 minutes	check the re-fill tank, replace if empty	[8]
LN2-Solenoid closed for x.x hours	The valve of the LN2 autofill system has not opened for more than 12 hours	check status of LN2 processor, reboot [7]	[6]