• Computer_Restart_Procedures

Computer Restart Procedures

Here is where we should keep information on how to restart the computers that periodically need restarting.

Out of Date Ethernet network connection diagram as of Oct 7, 2008: 40m_network_10-07-08.pdf

Martian_Host_Table

Here you can find a map of the computers around the lab.

• Useful links

Which models run on which machines?

• Answer: Electronics/Existing RCG DCUID and gds ids

• fb and DAQ issues

To restart the frame builder daqd process, simply do the following from a control room machine:

• 1. telnet fb 8087 8083 (the address has changed since the 2017 RCG overdo)

• 2. shutdown

The init process running on the fb machine will then automatically restart daqd.

Alternatively, one can ssh into FB1 (192.168.113.201), and run sudo systemctl restart daqd_*

Generally after restarting the frame builder process, the front ends will not be talking to the fb properly (0x2bad and red lights). The easiest solution is to reboot the front ends.

For dataviewer to get data you need to make sure "daqd" and "nds pipe" are running on the fb machine.

daqd and nds have been added to the /etc/inittab file on the fb machine. These will automatically restart when killed or the machine is restarted.

However, if either process fails to start several times in rapid succession, the init process will stop trying.

• 1. Fix the underlying problem. Try looking at /opt/rtcds/caltech/c1/target/fb/logs/daqd.log.XXXX for error messages. Make sure the master and daqdrc files are correct in the fb directory.
• 2. ssh fb
• 3. "sudo /sbin/init q" to restart the init process or restart the fb machine with "sudo shutdown -r now"

The code which is called by the init process lives in /opt/rtcds/caltech/c1/target/fb/.

For testpoints to be available for a given front end, you need running on the correct front end computer:

• The IOP needs to be running, since it handles, it is genamed something like c1x00, c1x01, etc.
• The front end module needs to be loaded and running.
• The awgtpman process since it handles the arbitrary waveform generator and test points for the front end.
• The IOC needs to be running since it hands all the EPICs communication
• mx_streams running (use "sudo /etc/restart_streams") this should start a mx_stream for each front end system and is needed to talk to the fb

To confirm the necessary codes are running on a front end, you can:

• To check if the front ends are loaded, you can use "lsmod" on the front end machine, looking for c1SYSNAMEfe entries
• To check if the IOCs are running, you can "ps -ef | grep epicsC1.cmd" - there should be 1 per model
• To check if mx_stream is running, you can "ps -ef | grep mx_stream" - there should be only 1 that has all the models as command line inputs
• To check if awgtpman are running, you can "ps -ef | grep awgtpman" - there should be 1 per model

Cold start order is:

0. If necessary, stop everything (fb, front ends, mx_streams, etc)
1. Start (or restart) fb services (open-mx, mx, nds, daqd, nds):

controls@fb1:~ 0$ sudo modprobe -r gpstime
controls@fb1:~ 0$ sudo modprobe gpstime
controls@fb1:~ 0$ sudo systemctl start open-mx
controls@fb1:~ 0$ sudo systemctl start mx
controls@fb1:~ 0$ sudo systemctl start nds
controls@fb1:~ 0$ sudo systemctl start daqd*

• If removal of gpstime module command gives error that it is in use, make sure the daqd_* processes are stopped.

1. Start the front ends. They should automatically start all the necessary processes on boot up.
   1. Be patient, it can take over a minute for all the modules to finally load. You can type "dmesg" to see how far its gotten.

• General FE Instructions

• Trouble Shooting

• First, try running /opt/rtcds/caltech/c1/scripts/cds/rebootC1LSC.sh (elog13935).
• If all the fb-related lights on a single machine are red, try restarting the mx_stream process.
• If just one model is stuck, ssh to that computer and try "rtcds restart c1SYS"
• If multiple models are stuck, ssh to that computer and try "rtcds restart all", which will stop the models, and restart them in order (IOP first)

• Look at the troubleshooting document T1100625

• Next thing to try is a reboot. All the front ends should automatically start their FE processes on a reboot.

• Also see the fb/DAQ section.

• The following assume you are running a terminal on the FE computer having problems, i.e. ssh -X computer_name
• Try "dmesg | grep c1SYS", where c1SYS is the front end model name, such as c1lsc, c1x02, c1us, c1rfm, c1scy, and so on.
  1. Look for things like ADC timeouts, not finding ADC/DAC or BO cards or other error messages.
• If the EPICS process is running (you see non-white channels in medm screens, but nothing is responding or changing), try restarting the FE itself
  1. Be aware that BURT Restore needs to be set to 1 on the corresponding GDS_TP screen. These can be accessed from the sitemap with the GDS FE and GDS IOP buttons.
  2. The IOP (usually named c1xYY where YY is a number like 01 or 02) for a given computer needs to be running for any other FE processes to run properly on that computer.
  3. Check to see if the FE module is loaded. Type "lsmod" and look for c1SYSfe, where SYS is the 3 letter name of the model.
  4. If its running but not responding, kill it with "sudo rmmod c1SYSfe", again replacing SYS with the 3 letter name of the model.
  5. To restart first go to the directory where the module lives. Type "target", then "cd c1SYS/bin/". Then type "sudo insmod c1SYSfe.ko", with the correct replacement for SYS.
• To restart just one FE including its IOC and awgtpman processes, go to the scripts directory (type "scripts") then run the startc1SYS script, where SYS is the FE model you want to start.
  1. Do not use sudo to run the script. If you get an error indicating a log file cannot be opened due to permissions, go to the log file and run "sudo chown controls:controls LOGFILE", where LOGFILE is the file name.

• c1ioo

This machine runs the c1x03, c1ioo, and c1gpt FE models. It controls mode cleaner wavefront sensors, mode cleaner length, and green locking.

On reboot, these models should automatically start up. See also the fb/DAQ and the General FE Instructions sections.

It is connected via GE Fanuc VMIC 5565 reflected memory card to the c1sus, c1scx, and c1scy computers.

c1ioo is a Sun X4600 machine. As such for a complete shutdown (not normally necessary but sometimes), do the following:

Shutdown the computer normally. (Power button or "shutdown -h now").

Go out to the rack and unplug all 4 power supply cables on the back of the machine.

Wait for a bit for the machine to completely stop (30 seconds or so).

Plug all the cables back in, and press the power button.

• c1sus

This machine runs the c1x02, c1sus, c1mcs, c1rfm, c1pem FE models. It controls the BS, ITMX, ITMY, PRM,SRM,MC1,MC2, and MC3 optics.

On reboot, these models should automatically start up. See also the fb/DAQ and the General FE Instructions sections.

It is connected by a GE Fanuc VMIC 5565 RFM card to the c1scx, c1scy, and c1ioo machines.

It is connected by a Dolphin PCIE reflected memory card to the c1lsc machine.

• c1lsc

This machine runs the c1x04 and c1lsc FE models.

On reboot, these models should automatically start up. See also the fb/DAQ and the General FE Instructions sections.

It is connected by a Dolphin PCIE reflected memory card to the c1sus machine.

• c1iscex

This machine runs the c1x01 and c1scx FE models.

On reboot, these models should automatically start up. See also the fb/DAQ and the General FE Instructions sections.

It is connected by a GE Fanuc VMIC 5565 reflected memory card to the c1sus, c1ioo, and c1scy computers.

• c1iscey

This machine runs the c1x05 and c1scy FE models.

On reboot, these models should automatically start up. See also the fb/DAQ and the General FE Instructions sections.

It is connected by a GE Fanuc VMIC 5565 reflected memory card to the c1sus, c1ioo, and c1scy computers.

• Acromag Slow Controls

Newer (circa 2018) slow controls systems consist of a Debian rackmount server connected to an Acromag chassis. As of June 2021, all of the slow controls systems have been replaced, except for c1auxey. The new Acromag systems currently include c1vac, c1psl, c1susaux, and c1auxex.

If the EPICS channels are unresponsive, first try connecting via ssh from another computer and remotely rebooting:

•  ssh c1psl
   sudo reboot now

and then burt restore. If you are unable connect to the server via ssh, walk out to the rack and manually power cycle the computer.

If some channels remain unresponsive or show zero values after rebooting, power cycle the Acromag chassis and reboot the server again.

• c1pem1

Sometimes you can just do this guy by doing:

•  telnet c1pem1
   reboot

then burt restore this guy.

If it still doesn't come back then sing this link.

• c1iool0

At the command prompt, type:

•  telnet c1iool0

Try CTRL+x

It should reboot c1iool0

This computer automatically executes startup.cmd. So there is no need to run it manually.

If for some reason it does not load the startup script automatically, try this:

At the telnet prompt, type

•  < /cvs/cds/caltech/target/c1iool0/startup.cmd

Then, after the main loop is started, type CTRL-], followed by

•  quit

• c1omc

-1) Make sure the c1omc is powered on--it doesn't power up automatically following a power outage. First find the OMC, then press its power button.

0) Make sure the previous incarnation of the code is no longer running. See Appendix A for details.

1) while logged in as controls, run the script startupC1 in the c1omcepics target directory.

2) Log in as root. Start the real-time code by running the omcfe.rtl script in the c1omc

• target directory.

2.5) Now the process will wait for a BURT restore. Find the appropriate autoburt snapshot file, and restore it.

3) Also, as root, run the command /opt/gds/awgtpman -2 in the background.

Note that c1omc has two ethernet ports. Use the bottom one.

If nothing works, check the mount tables and make sure that linux1:/home/cds is mounted as /cvs/cds. If it's not, sudo mount -a.

A) To stop the front end code, first press the red FE RESET button on the C1OMC_GDS screen. Then,

• i) log in to c1omc. become root.
• ii) kill epics with a 'pkill omcepics'
• iii) kill the test-point manager with a 'pkill awgtpman'
• iv) remove the front end kernel module with '/sbin/rmmod omcfe'
• v) check that the [omcfe] kernel module is gone with a '/sbin/lsmod'

• c1ass

• Currently the procedure for restarting C1ASS seems to be the same as for C1OMC above except that the ass test point manager doesn't need the "-2" flag.

• op440m

Reboot as usual. If its acting weird or slow just hit the moon button. Pick the shutdown option. After a few minutes it will turn off. The hit the on button on the front of the machine. Wait for the login prompt. Then log in as controls.

• op340m

Reboot as usual. It's headless, so you'll need to ssh in and type 'reboot'.

Restart the following scripts:

• conlog

• nodus

Nodus is a Solaris box in the rack in the office. Here are some of the things that it runs that you will want to restart:

• EPICS_gateway

• ndsproxy

• Apache (Required for SVN remote access)

• elog (https://wiki-40m.ligo.caltech.edu/How_To/Restart_the_elog)

• megatron

Megatron is a Ubuntu box that is used for running our MC autolocker and FSS Slow control scripts. These scripts are enabled to start at boot and restart on failure. So you would not need to start them ideally. If the are not running, check their status by:

 sudo systemctl status MCautolocker -l
 sudo systemctl status FSSslowPy -l

Make sure the nfs mount is still working by checking:

 mount -l | grep nfs

If nfs mount isn't working, check if the nameserver resolution is working. 192.168.113.104 (chiara) should be listed as nameserver in the file, /etc/resolv.conf. If that is not the case, follow the steps in this elog 40m/16479. Once you have nameserver resolution, rebooting the computer again should fix everything.

• SLOW controls computers

When rebooting the FE machines, it is necessary to BURTgooey to the SLOW controls machines to restore the epics settings (restore using snapshot files from directory /opt/rtcds/caltech/c1/burt/autoburt/today/).

Since these are really OLD machines, we cannot ssh into these. Ping the targets listed in /cvs/cds/caltech/target/. You can connect to the machines by 'telnet <computer name>'.

To check the status of the computer, type 'i' at the command prompt. This will output a table of the processes running on the computer. To exit the machine type 'Ctrl+Shift+]' which will take you back to telnet. Then type 'quit' to safely exit.

If the computers do not respond (in cases where you cannot telnet into them), it is ok to hard reboot them. This can be done using the key on the crate. If no key, press the reset button or keep the ON/OFF button pressed until the machine powers down. Then, press ON/OFF again.

List of SLOW computers that are alive (as of Jul 18 2013):

• c1pem1
• c1aux
• c1auxex
• c1auxey
• c1susaux
• c1iool0
• c1iscaux (no button)
• c1iscaux2
• c1iscex
• c1iscey
• c1psl
• c1vac1
• c1vac2

SLOW computers not present or dead (as of Jul 18 2013):

• c1dcuepics
• c0daqawg
• c0daqctrl
• c0dcu1
• c1iscepics
• c1loepics