Introduction

The summary pages are a website showing plots of useful channels updated every ~30min. They can be found here:

https://nodus.ligo.caltech.edu:30889/detcharsummary/

Different kinds of plots are can be produced: time series, spectra, spectrograms, Rayleigh gaussianity statistics, etc. This is all produced using the GWsumm software (https://github.com/gwpy/gwsumm) developed for the big detectors (https://ldas-jobs.ligo.caltech.edu/~detchar/summary/).

Configuration

The content of the pages is controled by configuration files found in:

nodus:/cvs/cds/caltech/chans/GWsummaries/

These are synced to the LDAS computer cluster, where the data are processed. The filenames must begin with the characters c1 and have the .ini extension. A special case is the defaults.ini file, which contains HTML and other general information. Although this file is always loaded, its settings can be overwritten by custom files (e.g. if the same property is defined in defaults.ini and c1-lsc.ini, the latter will take precedence).

The remote LDAS folder mirrors the local one in nodus, where the files are version controlled (note that there no longer is version control on the LDAS side, as this was redundant).

For information on the INI format itself see: https://ldas-jobs.ligo.caltech.edu/~duncan.macleod/gwsumm/latest/ or http://www.ligo.caltech.edu/~misi/iniguide.pdf

Workflow

Overview

The central part of the process are cron-like gwsumm jobs executed on the cluster every 30min. This is the chain of events:

1. A cron-type Condor (http://research.cs.wisc.edu/htcondor/manual/) job wakes up and executes the gw_daily_summary bash script which:

• Sets up Python environment;

• Rsyncs nodus and LDAS directories containing config files;

• Lists the config files present;

• Executes a gw_summary_pipe job with proper options and waits for it to finish.

2. Files are processed in parallel in the cluster:

• gw_summary_pipe spawns multiple gw_summary jobs (one per config file);

• Each gw_summary job corresponds to a node in the Condor DAG;

• Condor jobs are processed in the local universe, maximum two at a time (default).

3. Output is synced back to nodus:

• Handled by a regular cron tab running every 15 minutes;
• Only HTML from local time "yesterday," "today" and "tomorrow" (if exists) are rsynced.

Note that this whole process depends on the 40m frames being available in the cluster. The processes responsible for that are handled by Dan Kozak.

Technical details

Several independent pieces come into play:

1. GWpy (https://gwpy.github.io/): python module to handle LIGO data developed by Duncan Macleod;

2. GWsumm (https://github.com/gwpy/gwsumm and https://ldas-jobs.ligo.caltech.edu/~duncan.macleod/gwsumm/latest/index.html): python module and associated executables (gw_summary and gw_summary_pipe) that produced detector summary pages, based on GWpy;

3. Configuration files: stored locally in nodus and VCS version controlled;

4. 40m-specific scripts: git repository containing bash scripts and condor submit files that make the necessary preparations to run GWsumm jobs for the 40m, also take care of syncing HTML, plots and config files between nodus and LDAS;

5. Data transfer: cron jobs that sync frames from nodus to LDAS, managed by Dan Kozak.