Troubleshooting

CCD starts warming up

If you see one of the CCDs struggling to hold its set point temperature (-90,-89,-88,-90,-90 for CCDs 1-5, respectively) but it remains within a degrree or so of its set point, this is nothing to worry about. Ask the TO what the GTC chiller temperature is set to and ensure that it is no more than +5 degC. Sometimes, the ambient temperature is so high, or there is so much demand on the GTC chiller from the various cabinets in the telescope, that although the temperature of the water exiting the chiller is +5 degC, by the time it gets to HiPERCAM it could be a few degrees warmer than this. This is the likely cause of minor CCD temperature excursions and can be safely ignored - dark current only becomes a problem above a temperature of about -83 degC.

For more serious CCD temperature excursions, the Grafana server will issue a Slack alert, and the Grafana display boxes will turn red or amber under the following conditions:

  • Flow Rate: 0.8 l/min (amber/warning), 0.5 l/min (red/error).

  • Pressure: 5e-3 mbar (amber/warning), 8e-3 mbar (red/error).

  • Temperatures (amber warning/red error):

    • CCD1: -89 / -86 degC

    • CCD2: -88 / -86 degC

    • CCD3: -87 / -86 degC

    • CCD4: -89 / -86 degC

    • CCD5: -89 / -86 degC

If just one CCD starts to warm up, this is probably due to a flow sensor failure, causing the peltier to power off automatically to protect the CCD from overheating. You can check this is the case by inspecting the flow rate of the CCD in question. If it is zero, and the other CCD flow rates are all ok, it is almost certain that the flow is actually ok but the sensor has failed.

It will take about an hour to fix this problem, so if it occurs whilst observing, you need to decide whether to continue with just 4 CCDs for the rest of the night and fix it in the morning. If you decide to leave it until the morning, then change the temperature set point of the affected CCD to +10 degC using hdriver, as described in pumping and cooling. This protects the CCD against being cooled back down to -90 degC if the flow sensor starts working again for some reason, which could then cause contamination of the CCD if the head pressure gets too high.

To fix the problem, get the spare flow sensor from the crates and go up to the elevation platform, taking all of the safety precautions described in pumping and cooling.

Open the cabinet door and check on the Honeywell data recorder that the flow sensor in question is indeed reading zero. Next, check the CCD temperature and head pressure. It is likely that the CCD temperature will have risen substantially from -90 degC, in which case the CCD head pressure will probably have risen substantially as well. If the pressure is above the low e-3 mbar level, then it is dangerous to cool the CCD back down to -90 degC without pumping it, as the detector could become contaminated. In this case, you will have to start pumping the CCD in question, as described in pumping and cooling, being careful to ensure that the valves to the other CCD heads remain closed.

Now swap the faulty flow sensor with the spare one. First, unscrew the connector, being careful with this plastic, potted fitting, as it is very easy to snap off the connector or strip the thread. There is no need to power on/off the flow sensors before removing/connecting this cable. Then pop out the faulty flow sensor and replace it with the spare. It can be difficult to reconnect the water pipe - make sure you have your body in the correct position so that you are not pushing the connector on at an angle. Reattach the cable and confirm that the flow sensor is working by inspecting the Honeywell. The CCD will automatically start cooling to -90 degC again. If the pressure is below the low e-3 mbar range, then this is not a problem, although it would be best to pump the CCD if you have the time. If the pressure is above the low e-3 mbar range, then you should immediately change the temperature set point of the CCD to +10 degC to prevent further cooling and start pumping. When the head pressure is in the low e-3mbar range again, you can being cooling.

You should now repair the failed flow sensor, so that you have a working spare. For instructions, please contact Vik Dhillon.

If more than one flow sensor fails, then a quick solution is to use the flow sensor in the NGC circuit as a second spare. Although the NGC does need flow, there is no interlock to the peltier powers, so putting the faulty one in the NGC circuit is not a problem.

CCDs will not power on

When clicking power on on hdriver, the GUI can sometimes return an error of the form: CLDC module 1, write failed, device is not open, unable to power on CLDC. This is most probably because the system was previously shut down without doing a power off. The problem can usually be rectified by hitting the power on button on the GUI once or twice more.

Another error on power on on hdriver, the GUI can sometimes return an error of the form: CLDC module 1, write failed, device is not open, unable to power on CLDC. This is most probably because the system was previously shut down without doing a power off. The problem can usually be rectified by hitting the power on button on the GUI once or twice more.

Another error on power on can be cannot determine if CLDC is already on, sPoweredOn error: cannot read run number. If multiple attempts to power on and stop_hicam/start_hicam fail, rebooting the rack PC should fix this (and may still require multiple attempts to power on).

Power cut

HiPERCAM is connected to the GTC UPS and hence should not crash when there is a mains power failure. However, the GTC coolant supply is not on the UPS, and hence fails for approximately 1 minute whilst the GTC generator gets up to speed. This causes the flow rate to drop to zero for a minute and hence the peltier powers will turn off for a minute. The CCDs will therefore warm up, but not much above -50 degC. This will cause the CCD head pressures to rise, but not much above the low e-3 mbar range. When the coolant flow is restored a minute later, the CCDs will automatically begin cooling to -90 degC, but this is not a problem as the head pressures are low enough to prevent CCD contamination. The CCD heads should be pumped as soon as possible to reduce the pressures to 1e-5 mbar when cold, but this can wait until the morning if the power cut occurs whilst observing.

So, in summary, don’t panic. If you’re observing, and the flow is restored within a minute or so, you need do nothing until the morning, when you should pump the CCD heads. If the power cut occurs during the day, then connect the pump as soon as you can. In both cases, just let the CCDs cool down automatically. If, however, the coolant supply is disrupted for more than minute, and the CCD head pressures rise above the low e-3 mbar range due to the warming CCDs, then you should follow the procedure outlined in the CCD starts warming up section above.

Focal-plane slide doesn’t work

First, check that there is power to the focal-plane slide by lifting the black tape covering the green LED on the motor. If it is not lit, then there must be a fault with the cable or power supply to the slide. There are spare cables and powers supplies in the spare focal-plane slide box in the HiPERCAM crates, so try connecting these to isolate the problem.

If the green LED is lit, but the slide still does not respond, then it is possible that the connector in the foam-covered cable bundle leading up to the electronics cabinet has become partially separated. Try reseating the connector.

All server windows display “connection refused” messages

If the various server windows (HServer, HWServer, GTCServer) all display connection refused messages, then this means that the WAMP server used for communication between the HiPERCAM hardware (vacuum gauges, Meerstetter peltier controllers, focal-plane slide, COMPO) is not running. You can restart the WAMP server by typing sudo systemctl restart crossbar on the rack PC. Note that you can enter the password for insuser when prompted here.

HServer window displays “connection refused” messages

A different issue is when only the Hserver window displays the message: Control-Server (ngclin7:8001) is not running - cannot connect to server - connection refused. When this happens, the Power On button will not work.

This arises because the NGC Control software did not start up correctly. It is fixed by following the instructions below for Cannot start exposure - system in error state.

Cannot start exposure - system in error state

If you get errors that say cannot start exposure - system in error state you need to reboot the NGC software. Do this by running start_hicam again in an xterm on the rack PC. You don’t need to shut down any windows beforehand. You will, however, need to click the Power on button again before taking any data. If Power on fails for any reason, try it again a couple of times, before contacting the HiPERCAM team for advice.

Exposure starts, but no frames are written

Some setups can trigger a bug where a run can be successfully started, but no frames are written. The same bug can also result in frames being written much less frequently than expected - for example a setup with a cadence of 3 seconds only writes frames every minute.

In this case, the solution is to change the setup very slightly. Usually the easiest thing to do is to add a very small amount (perhaps 1 millisecond) to the exposure delay.

Cannot start exposure - system in offline state

If you see error messages such as standby not in [offline] together with errors that indicate that ccsInit has failed, this means that the messaging system used by the ESO-VLT NGC control software is not running.

To fix this, first shut down any HiPERCAM software components running by typing stop_hicam in a terminal on the rack PC.

Next, open a terminal on the rack PC and type vccEnv. In the GUI that appears click the plus icon to the right of the Environment box and choose wngcli. Then click Apply. Finally, using the row of buttons at the bottom of the GUI, click Start. If everything works, the Status box in the GUI should show OK in green.

Sometimes this won’t start up correctly because there is a lock file present. You will get an error message in the GUI telling you the location of the lock file. Remove this file, and try the procedure above again.

Once the wngcli environment is running, you can use start_hicam as usual in the rack PC window to start everything up.

No CCD temperatures, pressures or flow rates available

If you are unable to see the CCD temperatures, pressures or flow rates in hdriver or Grafana, it is probably because HWServer and/or hwlogger are not running.

First, on the DRPC, check if the hwlogger process is already running with a ps ax | grep hwlogger command. If it isn’t, either type hwlogger & in an xterm on the DRPC, or double-click the hwlogger desktop icon on the DRPC.

If this doesn’t work, open an xterm on the rack PC and check if the HWServer process is already running with a ps ax | grep hwserver command. This should come back with a number of responses, one for each piece of HiPERCAM hardware. You can kill all of these processes by typing stop_hicam hwserver and then type hwserver & to restart them again.

No TCS info on hdriver, and dithering and gtc_tweak_pointing fail

This is most probably because the link between HiPERCAM and the GTC is not working. The problem could either be on the GTC side - you should ask the TO to restart their GTC Server - or on the HiPERCAM side. If the latter, you can restart the HiPERCAM GTC Server on the rack PC by typing gtcserver /insroot/insuser/SYSTEM/DETDATA.

Drift mode run fails to start, after which no runs will start

Drift mode runs often take a few seconds to start, which is normal behaviour and not a sign of concern. Occasionally the drift mode run will not start, and subsequently no runs can be started successfully.

When this occurs, first attempt to restart the ESO messaging software following the instructions in Cannot start exposure - system in offline state. If this fails to work, it is likely the NGC controller itself has crashed and you will need to power cycle the NGC controller using the Remote Power Management (RPM) strip in the electronics rack.

To do this, open a firefox browser on the DRPC (either click on the globe icon on the lower menu bar or right-click on the desktop and choose Applications -> Web Browser). Use the RPM bookmark in the menubar of the firefox browser and power cycle the NGC, either by manually turning on and off, or by clicking the Reset icon located under the power toggle.

No light in flats, or odd pattern in flats

Check that the focal-plane slide is not in the wrong position. It should be at pixel 1100. If it is not, you can move it using the Unblock button on the hdriver GUI.

Also check that the COMPO arms are not in the light path to the CCDs. If they are, park them using the instructions in Afternoon Activities.