Computer Restart Procedures

Here is where we should keep information on how to restart the computers that periodically need restarting.

Out of Date Ethernet network connection diagram as of Oct 7, 2008: attachment:40m_network_10-07-08.pdf

Martian Host Table

[#Electronics Here] you can find a map of the computers around the lab.

• Useful links

Which models run on which machines?

• Answer: Electronics/Existing RCG DCUID and gds ids

• fb and DAQ issues

To restart the frame builder process, simply do the following from a control room machine:

• 1. telnet fb 8088
• 2. shutdown

The init process running on the fb machine will then automatically restart daqd.

Generally after restarting the frame builder process, the front ends will not be talking to the fb properly (0x2bad and red lights). The easiest solution is to reboot the front ends.

For dataviewer to get data you need to make sure "daqd" and "nds pipe" are running on the fb machine.

daqd and nds have been added to the /etc/inittab file on the fb machine. These will automatically restart when killed or the machine is restarted.

However, if either process fails to start several times in rapid succession, the init process will stop trying.

• 1. Fix the underlying problem. Try looking at /opt/rtcds/caltech/c1/target/fb/logs/daqd.log.XXXX for error messages.
• 2. ssh fb
• 3. "sudo /sbin/init q" to restart the init process or restart the fb machine with "sudo shutdown -r now"

The code which is called by the init process lives in /opt/rtcds/caltech/c1/target/fb/.

For testpoints to be available for a given front end, you need running on the correct front end computer:

• The IOP needs to be running, since it handles, it is genamed something like c1x00, c1x01, etc.
• "sudo /opt/rtcds/caltech/c1/scripts/startc1SYSNAME" where SYSNAME is something like sus or ioo or x02
• The above starts the IOCs, awgtpman, and loads the front end module.
• mx_streams running (use "sudo /etc/restart_streams") this should start a mx_stream for each front end system and is needed to talk to the fb

To confirm the necessary codes are running on a front end, you can:

• To check if the front ends are loaded, you can use "lsmod" on the front end machine, looking for c1SYSNAMEfe entries
• To check if the IOCs are running, you can "ps -ef | grep epicsC1.cmd" - there should be 1 per model
• To check if mx_streams are running, you can "ps -ef | grep mx_stream" - there should be 1 per model
• To check if awgtpman are running, you can "ps -ef | grep awgtpman" - there should be 1 per model

Cold start order is:

• 0. If necessary, stop everything (fb, front ends, mx_streams, etc)
• 1. Start fb codes (daqd, nds, dhcpd)
• 2. Start the front ends.
• 3. Start IOP on front ends (startc1x##)
• 4. Start FE models (startc1SYSNAME)
• 5. Start mx_streams (/etc/restart_streams)
• If the DAQ is in a bad state, try to start fresh in this order. It seems to usually work.
• The frame builder for an unknown reason is flaky, and generally I find I have to restart it 2 or 3 times before it doesn't die within the first 60 seconds or so.
• If you get past that first minute or so, it tends to be stable from then on

Restarting the Nightly Backup of Frames

• The following instructions are valid for the old fb40m Frame Builder. They're here for reference, until the backup scripts are fixed, at which time new instructions will be posted. ---JD 18Oct2010

Restart the nightly BACKUP of /cvs/cds and our trend-frames by following the instructions in Restarting the backup script Summary of how to restart the backup script: The steps are as follows (copy everything after each of the numbered steps verbatim):

1. ssh fb40m
2. cd /cvs/cds/caltech/scripts/backup

3. ssh-agent > .agent

4. awk '/setenv/' .agent > .agent.edit

5. mv .agent.edit .agent
6. source .agent
7. ssh-add ~/.ssh/id_rsa
   • (This one will not ask for a passphrase)
8. ssh-add ~/.ssh/backup2PB
   • ( This one requires a passphrase. Read the README: ..../scripts/backup/000README.txt )
9. ssh-add -l
   • (This verifies that both the id_rsa and backup2PB are there. If it also picks up the wrong one (id_dsa), remove it by typing "ssh-add -d" )

1. ssh 40m@ldas-cit.ligo.caltech.edu /bin/ls /archive/frames/trend/minute-trend/40m

   • (This should do a test ssh, and list the archived frame folders. You can open the last one, and then look at the gps time of the last .gwf file, and it should be sometime in the middle of the previous night.)

• c1ioo

This machine runs the c1x03, c1ioo, and c1gpt FE models. It controls mode cleaner wavefront sensors, mode cleaner length, and green locking.

On reboot, these models should automatically start up. See also the [#fb fb/DAQ] section.

c1ioo is a Sun X4600 machine. As such for a complete shutdown (not normally necessary but sometimes), do the following:

Shutdown the computer normally. (Power button or "shutdown -h now").

Go out to the rack and unplug all 4 power supply cables on the back of the machine.

Wait for a bit for the machine to completely stop (30 seconds or so).

Plug all the cables back in, and press the power button.

• c1sus

This machine runs the c1x02, c1sus, c1mcs, c1rms FE models. It controls the BS, ITMX, ITMY, PRM,SRM,MC1,MC2, and MC3 optics.

On reboot, these models should automatically start up. See also the [#fb fb/DAQ] section.

• c1psl

Sometimes you can just do this guy by doing:

•  telnet c1psl
   reboot

then burt restore this guy.

But often, this just makes it upset and the screens go white but it never comes back. When that happens go out to the rack (the one next to the one with the MC servo) and turn off the crate (on the bottom) which has the c1psl processor. After ~3.14 seconds, turn it back on. c1psl ought to come back now.

If it still doesn't come back then sing [http://www.amazon.com/gp/music/clipserve/B000002W9Q001005/1/ref=mu_sam_ra001_005/002-7727484-0862420 this] link.

• c1pem1

Sometimes you can just do this guy by doing:

•  telnet c1pem1
   reboot

then burt restore this guy.

If it still doesn't come back then sing [http://www.amazon.com/gp/music/clipserve/B000002W9Q001005/1/ref=mu_sam_ra001_005/002-7727484-0862420 this] link.

• c1iool0

At the command prompt, type:

•  telnet c1iool0

Try CTRL+x

It should reboot c1iool0

This computer automatically executes startup.cmd. So there is no need to run it manually.

If for some reason it does not load the startup script automatically, try this:

At the telnet prompt, type

•  < /cvs/cds/caltech/target/c1iool0/startup.cmd

Then, after the main loop is started, type CTRL-], followed by

•  quit

• c1omc

-1) Make sure the c1omc is powered on--it doesn't power up automatically following a power outage. First find the OMC, then press its power button.

0) Make sure the previous incarnation of the code is no longer running. See Appendix A for details.

1) while logged in as controls, run the script startupC1 in the c1omcepics target directory.

2) Log in as root. Start the real-time code by running the omcfe.rtl script in the c1omc

• target directory.

2.5) Now the process will wait for a BURT restore. Find the appropriate autoburt snapshot file, and restore it.

3) Also, as root, run the command /opt/gds/awgtpman -2 in the background.

Note that c1omc has two ethernet ports. Use the bottom one.

If nothing works, check the mount tables and make sure that linux1:/home/cds is mounted as /cvs/cds. If it's not, sudo mount -a.

A) To stop the front end code, first press the red FE RESET button on the C1OMC_GDS screen. Then,

• i) log in to c1omc. become root.
• ii) kill epics with a 'pkill omcepics'
• iii) kill the test-point manager with a 'pkill awgtpman'
• iv) remove the front end kernel module with '/sbin/rmmod omcfe'
• v) check that the [omcfe] kernel module is gone with a '/sbin/lsmod'

• c1ass

• Currently the procedure for restarting C1ASS seems to be the same as for C1OMC above except that the ass test point manager doesn't need the "-2" flag.

• op440m

Reboot as usual. If its acting weird or slow just hit the moon button. Pick the shutdown option. After a few minutes it will turn off. The hit the on button on the front of the machine. Wait for the login prompt. Then log in as controls.

• op340m

Reboot as usual. It's headless, so you'll need to ssh in and type 'reboot'.

Restart the following scripts:

• conlog

• nodus

Nodus is a Solaris box in the rack in the office. Here are some of the things that it runs that you will want to restart:

• EPICS gateway

• ndsproxy

• Apache (Required for SVN remote access)

• elog