.. _archiving:


Archiving data
==============

At the end of each night, the following tasks should be performed to archive
the data:

1. Ensure that data taking and data reduction have completely finished.
   
2. Ensure that ``hlogger`` has been correctly shut down.
 
3. Tidy up the contents of the relevant
   ``/home/observer/reduce/yyyy_mm_dd`` directory on the DRPC,
   deleting any superfluous files. All data reduction files of the
   form ``run0***.ape``, ``run0***.hcm``, ``run0***.red`` and
   ``run0***.log`` in this directory will be archived.

4. Open an xterm on the DRPC in an empty virtual desktop and run the
   script ``end_of_night_tasks_hipercam``. The script is
   self-explanatory - it only requires you to confirm the start
   date of the night. The script then:

   * Checks how much data has been generated on that night, and the
     current capacities of the various archiving disks.

   * Checks that no existing directories with the same night start date
     already exist.

   * Creates an MD5SUM file of the raw data files to ensure data integrity.

   * Copies the MD5SUM file, the raw data files, the ``hlogger`` files, and
     the ``reduce`` files to three locations: the GTC disk ``rack:/gtcnfs``,
     two portable USB disk drives connected to the DRPC in the control room,
     and a subdirectory of the ``/data`` disk of the form ``/data/yyyy_mm_dd``.

   * Uses the MD5SUM files to verify that the data were copied without error
     to the USB drives.

   * Removes the raw data from the ``/data`` directory on the rack PC in
     preparation for the next night of observation.
    
If the ``rack:/gtcnfs`` disk is not mounted on the rack PC, the
``end_of_night_tasks_hipercam`` script will skip this step of the
archiving process.  If this happens, do not attempt to run the
``end_of_night_tasks_hipercam`` again. Instead, do the following:

1. NFS mount the ``/gtcnfs`` disk on the rack PC. To do this, first
   type ``su`` in an xterm on the rack PC. You will then need to enter
   the root password - if you don't know the password, please
   :ref:`contact <contacts>` one of the HiPERCAM team members. Then
   type ``mount /gtcnfs``.

3. In an xterm on the DRPC, run the
   ``/home/observer/scripts/archive_to_gtc_disk`` script to copy the
   data to the GTC disk.