$/!\$ Generally after restarting the frame builder process, the front ends will not be talking to the fb properly (0x2bad and red lights). The easiest solution is to reboot the front ends.

For dataviewer to get data you need to make sure "daqd" and "nds pipe" are running on the fb machine.

daqd and nds have been added to the /etc/inittab file on the fb machine. These will automatically restart when killed or the machine is restarted.

However, if either process fails to start several times in rapid succession, the init process will stop trying.

1. Fix the underlying problem. Try looking at /opt/rtcds/caltech/c1/target/fb/logs/daqd.log.XXXX for error messages. Make sure the master and daqdrc files are correct in the fb directory.
2. ssh fb
3. "sudo /sbin/init q" to restart the init process or restart the fb machine with "sudo shutdown -r now"

The code which is called by the init process lives in /opt/rtcds/caltech/c1/target/fb/.

For testpoints to be available for a given front end, you need running on the correct front end computer:

The IOP needs to be running, since it handles, it is genamed something like c1x00, c1x01, etc.
The front end module needs to be loaded and running.
The awgtpman process since it handles the arbitrary waveform generator and test points for the front end.
The IOC needs to be running since it hands all the EPICs communication
mx_streams running (use "sudo /etc/restart_streams") this should start a mx_stream for each front end system and is needed to talk to the fb

To confirm the necessary codes are running on a front end, you can:

To check if the front ends are loaded, you can use "lsmod" on the front end machine, looking for c1SYSNAMEfe entries
To check if the IOCs are running, you can "ps -ef | grep epicsC1.cmd" - there should be 1 per model
To check if mx_stream is running, you can "ps -ef | grep mx_stream" - there should be only 1 that has all the models as command line inputs
To check if awgtpman are running, you can "ps -ef | grep awgtpman" - there should be 1 per model

Cold start order is:

If necessary, stop everything (fb, front ends, mx_streams, etc)
Start (or restart) fb codes (daqd, nds, dhcpd)
Start the front ends. They should automatically start all the necessary processes on boot up.
1. Be patient, it can take over a minute for all the modules to finally load. You can type "dmesg" to see how far its gotten.

General FE Instructions
Trouble Shooting
If all the fb-related lights on a single machine are red, try restarting the mx_stream process.
If just one model is stuck, ssh to that computer and try "rtcds restart c1SYS"
If multiple models are stuck, ssh to that computer and try "rtcds restart all", which will stop the models, and restart them in order (IOP first)
Look at the troubleshooting document T1100625
Next thing to try is a reboot. All the front ends should automatically start their FE processes on a reboot.
Also see the fb/DAQ section.
The following assume you are running a terminal on the FE computer having problems, i.e. ssh -X computer_name
Try "dmesg | grep c1SYS", where c1SYS is the front end model name, such as c1lsc, c1x02, c1us, c1rfm, c1scy, and so on.
1. Look for things like ADC timeouts, not finding ADC/DAC or BO cards or other error messages.
If the EPICS process is running (you see non-white channels in medm screens, but nothing is responding or changing), try restarting the FE itself
1. Be aware that BURT Restore needs to be set to 1 on the corresponding GDS_TP screen. These can be accessed from the sitemap with the GDS FE and GDS IOP buttons.
2. The IOP (usually named c1xYY where YY is a number like 01 or 02) for a given computer needs to be running for any other FE processes to run properly on that computer.
3. Check to see if the FE module is loaded. Type "lsmod" and look for c1SYSfe, where SYS is the 3 letter name of the model.
4. If its running but not responding, kill it with "sudo rmmod c1SYSfe", again replacing SYS with the 3 letter name of the model.
5. To restart first go to the directory where the module lives. Type "target", then "cd c1SYS/bin/". Then type "sudo insmod c1SYSfe.ko", with the correct replacement for SYS.
To restart just one FE including its IOC and awgtpman processes, go to the scripts directory (type "scripts") then run the startc1SYS script, where SYS is the FE model you want to start.
1. Do not use sudo to run the script. If you get an error indicating a log file cannot be opened due to permissions, go to the log file and run "sudo chown controls:controls LOGFILE", where LOGFILE is the file name.

c1ioo

This machine runs the c1x03, c1ioo, and c1gpt FE models. It controls mode cleaner wavefront sensors, mode cleaner length, and green locking.

On reboot, these models should automatically start up. See also the fb/DAQ and the General FE Instructions sections.

It is connected via GE Fanuc VMIC 5565 reflected memory card to the c1sus, c1scx, and c1scy computers.

c1ioo is a Sun X4600 machine. As such for a complete shutdown (not normally necessary but sometimes), do the following:

Shutdown the computer normally. (Power button or "shutdown -h now").

Go out to the rack and unplug all 4 power supply cables on the back of the machine.

Wait for a bit for the machine to completely stop (30 seconds or so).

Plug all the cables back in, and press the power button.

c1sus

This machine runs the c1x02, c1sus, c1mcs, c1rfm, c1pem FE models. It controls the BS, ITMX, ITMY, PRM,SRM,MC1,MC2, and MC3 optics.

On reboot, these models should automatically start up. See also the fb/DAQ and the General FE Instructions sections.

It is connected by a GE Fanuc VMIC 5565 RFM card to the c1scx, c1scy, and c1ioo machines.

It is connected by a Dolphin PCIE reflected memory card to the c1lsc machine.

c1lsc

This machine runs the c1x04 and c1lsc FE models.

On reboot, these models should automatically start up. See also the fb/DAQ and the General FE Instructions sections.

It is connected by a Dolphin PCIE reflected memory card to the c1sus machine.

c1iscex

This machine runs the c1x01 and c1scx FE models.

On reboot, these models should automatically start up. See also the fb/DAQ and the General FE Instructions sections.

It is connected by a GE Fanuc VMIC 5565 reflected memory card to the c1sus, c1ioo, and c1scy computers.

c1iscey

This machine runs the c1x05 and c1scy FE models.

On reboot, these models should automatically start up. See also the fb/DAQ and the General FE Instructions sections.

It is connected by a GE Fanuc VMIC 5565 reflected memory card to the c1sus, c1ioo, and c1scy computers.

c1psl

Sometimes you can just do this guy by doing:

```
 telnet c1psl
 reboot
```

then burt restore this guy.

But often, this just makes it upset and the screens go white but it never comes back. When that happens go out to the rack (the one next to the one with the MC servo) and turn off the crate (on the bottom) which has the c1psl processor. After ~3.14 seconds, turn it back on. c1psl ought to come back now.

If it still doesn't come back then sing this link.

c1pem1

Sometimes you can just do this guy by doing:

```
 telnet c1pem1
 reboot
```

then burt restore this guy.

If it still doesn't come back then sing this link.

c1iool0

At the command prompt, type:

```
 telnet c1iool0
```

Try CTRL+x

It should reboot c1iool0

This computer automatically executes startup.cmd. So there is no need to run it manually.

If for some reason it does not load the startup script automatically, try this:

At the telnet prompt, type

 < /cvs/cds/caltech/target/c1iool0/startup.cmd

Then, after the main loop is started, type CTRL-], followed by

```
 quit
```

c1omc

-1) Make sure the c1omc is powered on--it doesn't power up automatically following a power outage. First find the OMC, then press its power button.

0) Make sure the previous incarnation of the code is no longer running. See Appendix A for details.

1) while logged in as controls, run the script startupC1 in the c1omcepics target directory.

2) Log in as root. Start the real-time code by running the omcfe.rtl script in the c1omc

target directory.

2.5) Now the process will wait for a BURT restore. Find the appropriate autoburt snapshot file, and restore it.

3) Also, as root, run the command /opt/gds/awgtpman -2 in the background.

Note that c1omc has two ethernet ports. Use the bottom one.

If nothing works, check the mount tables and make sure that linux1:/home/cds is mounted as /cvs/cds. If it's not, sudo mount -a.

A) To stop the front end code, first press the red FE RESET button on the C1OMC_GDS screen. Then,

i) log in to c1omc. become root.
ii) kill epics with a 'pkill omcepics'
iii) kill the test-point manager with a 'pkill awgtpman'
iv) remove the front end kernel module with '/sbin/rmmod omcfe'
v) check that the [omcfe] kernel module is gone with a '/sbin/lsmod'

c1ass
Currently the procedure for restarting C1ASS seems to be the same as for C1OMC above except that the ass test point manager doesn't need the "-2" flag.

op440m

Reboot as usual. If its acting weird or slow just hit the moon button. Pick the shutdown option. After a few minutes it will turn off. The hit the on button on the front of the machine. Wait for the login prompt. Then log in as controls.

op340m

Reboot as usual. It's headless, so you'll need to ssh in and type 'reboot'.

Restart the following scripts:

conlog

nodus

Nodus is a Solaris box in the rack in the office. Here are some of the things that it runs that you will want to restart:

megatron

Megatron is a Ubuntu box that is running the new conlog, here is what needs to be done to restart it:

service mysql start
service apache2 start
screen (conlog is run from within a screen)
tcsh # just because the environment script below assumes csh instead of bash)
cd /cvs/cds/rtcds/caltech/c1/target/conlog/conlogepics
source conlog_environment.txt
./starup.c1conlog
Enter: Ctrl+a , d to exit the screen and leave the conlog running

Megatron also runs our MC autolocker and FSS Slow control scripts. To start these, ssh into megatron and run the following commands:

 sudo initctl start MCautolocker
 sudo initctl start FSSslowPy

SLOW controls computers

When rebooting the FE machines, it is necessary to BURTgooey to the SLOW controls machines to restore the epics settings (restore using snapshot files from directory /opt/rtcds/caltech/c1/burt/autoburt/today/).

Since these are really OLD machines, we cannot ssh into these. Ping the targets listed in /cvs/cds/caltech/target/. You can connect to the machines by 'telnet <computer name>'.

To check the status of the computer, type 'i' at the command prompt. This will output a table of the processes running on the computer. To exit the machine type 'Ctrl+Shift+]' which will take you back to telnet. Then type 'quit' to safely exit.

If the computers do not respond (in cases where you cannot telnet into them), it is ok to hard reboot them. This can be done using the key on the crate.

List of SLOW computers that are alive (as of Jul 18 2013):

c1pem1
c1aux
c1auxex
c1auxey
c1susaux
c1iool0
c1iscaux
c1iscaux2
c1iscex
c1iscey
c1psl
c1vac1
c1vac2

SLOW computers not present or dead (as of Jul 18 2013):

c1dcuepics
c0daqawg
c0daqctrl
c0dcu1
c1iscepics
c1loepics

-  ⇤ ← Revision 153 as of 2017-12-15 19:35:12 → 
  Size: 14604
  Editor: GautamvenugopalanATligoDOTorg
  Comment:
+   ← Revision 154 as of 2017-12-15 19:36:39 → ⇥
  Size: 14712
  Editor: GautamvenugopalanATligoDOTorg
  Comment:
-Deletions are marked like this.
+Additions are marked like this.
 Line 48:
+Alternatively, one can {{{ssh}}} into FB1 (192.168.113.201), and run {{{sudo systemctl restart daqd_*}}}