Complete power shutdown and start-up plan

There will be a power shutdown for 40m lab completely on for Sep 16th (Fri), 2022 from 1pm. Following is the detailed plan for how to shutdown the whole lab and bring it back up online.

1. Shutdown

1.1. Preliminary

• Unplug all wifi routers
• center all the oplevs

1.1.1. Creating backups

• Ensure all SDF tables in all FE models are saved to keep the changes persistent through power cycle.
• Create burtgooey backups of all available req files or ensure that autoburt has successfully run in past some time.
  • Save the burt backup at /opt/rtcds/caltech/c1/Git/40m/measurements/cds/backup_YYYY_MM_DD

1.2. Vacuum shutdown

1.2.1. Vacuum

On the Asus computer, open editable version of vacuum system medm by:

cd /cvs/cds/caltech/target/c1vac/medm
medm -x C0VAC_CONTROL.adl

Then perform following steps:

• Isolate the main volume
  • Close V1
• Isolate all the annular volume
  • Close VASE, VASV, VABSSCI, VABS, VABSSCO, VAEV, VAEE
• Isolate RGA volume by doing following in order:
  • Close VA6
  • Close VM3
• Turn off TP1 and monitor RPM on TP1 medm screen and then on the Turbo Controller Module on the rack.
• Close V5
• Turn off TP3 and wait 2 minutes because there is no way to confirm if the pump actually stopped.
• Close V4. This should change the System Status to "All pneumatic valves are closed".
• Turn off TP2 and wait 2 minutes because there is no way to confirm if the pump actually stopped.
• Close the manual valves on TP1, TP2 and TP3 which are located below them. Turn clockwise to close.
• Turn off the AUX pump from the yellow breaker near the wall. The test switch breaks the circuit and functions as an OFF button.
• Unplug power connections for TP2 and TP3. We heard these ramp down in speed afterward, so it looks like we don't have control of TP2 and TP3. This is bad!

1.2.2. Shutdown Electronics and computer

Note the power supply readings:

Shutdown c1vac by (from Asus computer):

ssh c1vac
sudo /sbin/shutdown -h now

Switch off following on the rack:

• GP136a
• GP136b
• GP307
• MKS937a
• MKS937b
• XGS-600 (unplugged on back)
• Serial Device Server
• Vacuum acromag chassis (from back)
• PTP3
• PTP2
• TP1
• TP2 (unplugged on back)
• TP3 (unplugged on back)
• Turn off UPS supply at the bottom. This should remove power supply to c1vac.

1.3. CDS Shutdown

1.3.1. Slow computers

ssh into the following computers:

• c1susaux
• c1auxex
• c1auxey1
• c1psl
• c1iscaux
• c1susaux2

And do the following:

sudo systemctl stop modbusIOC.service
sudo shutdown -h now

1.3.2. Fast computers

ssh into following computers:

• c1lsc
• c1sus
• c1ioo

sudo /sbin/shutdown -h now

ssh into following computers:

• c1iscex
• c1iscey

And do the following:

rtcds stop --all
sudo /sbin/shutdown -h now

For

• c1sus2

rtcds stop c1bac c1hpc c1su2 c1x07
sudo /sbin/shutdown -h now

1.3.3. Teststand computers

First ssh into c1teststand from pianosa

ssh c1teststand

then ssh into the following computers in order and do the following:

• c1bhd
• c1sus (clone)
• c1lsc (clone)
• c1ioo (clone)
• fb1 (clone)

sudo shutdown -h now

And then shutdown chiara (clone) computer itself.

Switch off the chassis power buttons, AA/AI power buttons, and then switch off the power supply at the bottom of the stand.

1.3.4. Martian computers

1.3.4.1. Megatron

ssh megatron
sudo shutdown -h now

1.3.4.2. Optimus

ssh optimus
cd /home/controls/Git/40m/scripts/MC
sudo docker-compose down
cd /home/controls/Git/40m/softchansmodbus
sudo docker-compose down
sudo shutdown -h now

1.3.4.3. viviana

ssh viviana
sudo shutdown -h now

1.3.4.4. paola

This is dead right now.

ssh paola
sudo shutdown -h now

1.3.4.5. fb1

ssh fb1
sudo shutdown -h now

Do not shutdown chiara or nodus until you have finished end of shutdown.

1.4. Electronics racks and in-air tables

1.4.1. Y-AUX Rack

If you switch off any of the power buttons on the rack, list them down here.

• SR560, gain: 10, DC, DC coupled, Low Noise State. Turned off.
• Signal Generator DS345: Settings: Freq: 213120.000, Amp: 2.00 Vpp, Offset: 0, Phase 0
• Turned off white power strip in the bottom

1.4.2. Under YEND table

If you switch off any of the power buttons on the rack, list them down here.

• Oven temp controller, TC200L Set temperature was 35.7 degrees Celsius.Switched off from front button.
• Uniblitz shutter driver: NC, Remote. Turned off from power switch on back.
• JDS Uniphase (opLev laser_, turned key to 0 to turn off.
• Lightwave 125/126 laser controller. Turn key to off.
• Power strips: 3 white power strips under the table.
• Switched off CRT MON4 at the end.

1.4.3. 1Y4

Computers that should be shutdown by now:

• c1iscey
• c1auxey1

If you switch off any of the power buttons on the rack, list them down here.

• I/O Expansion chassis, turned power button on front.
• Power strips: 1 black, 1 white, in the back was turned off.

1.4.4. 1Y3

Computers that should be shutdown by now:

• c1iscaux

If you switch off any of the power buttons on the rack, list them down here.

• I/O Expansion chassis, turned power button on the front.
• Powerstrip in the back was turned off.

1.4.5. 1Y2

If you switch off any of the power buttons on the rack, list them down here.

• Powerstrip in the back was turned off.
• Turn off all AUX-1Y2 power supplies from front button.

1.4.6. 1Y1

If you switch off any of the power buttons on the rack, list them down here.

• C1SU2 I/O chassis on front
• Turned off the power strip

1.4.7. 1Y0

If you switch off any of the power buttons on the rack, list them down here.

• Turned off the power strip

1.4.8. Under ITMY table

If you switch off any of the power buttons on the rack, list them down here.

• JDSU (oplev laser), turned key to off
• Power strips: 2 white strips under table turned off

1.4.9. Under BS table

If you switch off any of the power buttons on the rack, list them down here.

• Uniphase key flipped

1.4.10. Under AS table

If you switch off any of the power buttons on the rack, list them down here.

• 2 long grey powers strips turned off. Label L#36.
• CRT Monitor on injection chamber turned off.

1.4.11. Under ITMX table

If you switch off any of the power buttons on the rack, list them down here.

• JDS Uniphase (oplev laser), turned key to off
• Uniblitz at NP, remote. Used back button to switch off.
• Power strips: 1 white strip under table turned off

1.4.12. PSL table

Laser current driver showed 2.098 A of current. If you switch off any of the power buttons on the rack, list them down here.

• Switched off laser controller by pressing off and turning key.
• Turned off Uniblitz shutter driver under the table from back buttton. In NC, remote mode.

1.4.13. 1X1

If you switch off any of the power buttons on the rack, list them down here.

• I/O Expansion chassis turned off at front
• Turned off power strip in the back top.
• 1X1-9-1 right most Kepco High Voltage PS powered down by button on front panel.

1.4.14. AUX OMC SOUTH (Small rack south of 1X2)

If you switch off any of the power buttons on the rack, list them down here.

• 1 Marconi FG on top at 11.066195 MHz:
  • Freq: 11.066195 MHz
  • RF Level: 13 dBm
  • MOD Disable
  • FM2 AC OFF
  • FM1 0, AC, OFF
  • MOdF AC 1.0000 kHz
• Turn off white small power strip exted out from the back of this rack for bottom 4 power supplies.

1.4.15. 1X2

Computers that should be shutdown by now:

• c1psl
• c1ioo

If you switch off any of the power buttons on the rack, list them down here.

• Power strip in back bottom

1.4.16. 1YNot

• Empty unused rack

1.4.17. 1X4

Computers that should be shutdown by now:

• c1lsc
• c1sus
• c1susaux

If you switch off any of the power buttons on the rack, list them down here.

• Turned off 1X4 I/O Chassis power buttons front
• Turned off power strip in the back middle region.

1.4.18. 1X5

If you switch off any of the power buttons on the rack, list them down here.

If you switch off any of the power buttons on the rack, list them down here.

• Turned off power strip at the bottom.
• Turned off power stip in the top but it has nothing connected to it.

1.4.19. Teststand

• Turned off everything by the 2 power strips below the teststand.

1.4.20. 1X9

Computers that should be shutdown by now:

• c1auxex
• c1iscex

If you switch off any of the power buttons on the rack, list them down here.

• Turned off I/O chassis in the front
• Turned off ac power strip in the back.
• Remove power cable of c1iscex from socket as it is directly powered

1.4.21. X-AUX Rack

Power to this rack got turned off when we turned off 1X9 power.

1.4.22. Under XEND table

If you switch off any of the power buttons on the rack, list them down here.

• Oven temp controller, TC200L Set temperature was 37.1 degrees Celsius. Switched off from front button.
• Uniblitz shutter driver: NC, Remote. Turned off from power switch on back.
• Innolight laser controller 1.985 A. Turn key to off.
• Power strips: 3 white power strips under the table, 1 on east side, 1 on west side, 1 under ETMX chamber on west side.
• Switched off CRT MON at the end.

1.5. Control room equipment

• Shutdown following computers:
  • zita
  • donatella
  • pianosa
  • rossa
  • Giada
  • c1sim
  • CAD (windows machine)
• Turned off moku
• Turned off all monitors and TVs

1.6. Last steps

Now shutdown the following computers

1.6.1. nodus

ssh nodus
sudo /sbin/shutdown -h now

1.6.2. chiara

ssh Chiara
sudo /sbin/shutdown -h now

1.6.3. 1X6

Computers that should be shutdown by now:

• fb1
• megatron
• FB (old) already off
• Jetstor powered at the bottom most location.

If you switch off any of the power buttons on the rack, list them down here.

• Turned off the rear (upper) power strip
• Turn off the VMI RFM hub switch on the back near the top of the rack.

1.6.4. 1X7

Computers that should be shutdown by now:

• nodus
• chiara
• optimus

• UPS at the bottom of the rack

If you switch off any of the power buttons on the rack, list them down here.

• Both UPSs in the bottom turned off
• Power strips in the back, top and bottom.

1.6.5. Office network rack

• Network switches for Martian and GC network were turned off
• Turn off printer

2. Startup

2.1. Initial Steps

2.1.1. Office Area

• Turned on network switches for martian network, and WiFi routers.

• Turned on windows (CAD) machine

2.1.2. 1X7

Switch on following (check the shutdown section for any extra buttons that were switched off):

• Both UPSs in the bottom turned on
• Power strips in the back, top and bottom.

2.1.3. 1X6

Switch on following (check the shutdown section for any extra buttons that were switched off):

• Turned on the rear (upper) power strip
• Turn on the VMI RFM hub switch on the back side, near the top of the rack.

2.1.4. Turn on chiara and nodus

• Power on chiara on 1X7.
• Reboot nodus which would have started automatically as soon as you turned on the power to the rack.
• Turn on pianosa, and rossa.
• On pianosa and rossa, check if nfs mount has happened at:

  •    cd /opt/rtcds
       ls

• Check if elog started running by going to nodus.ligo.caltech.edu:8080/. If not, do following in nodus:

  •    cd /home/controls/elog
       ./startELOGD.sh

• Check if nameserver is working porperly on chira, rossa, pianosa, and nodus, by pinging a few computer names and checking if the IP was found correctly.
• Also, on chiara, check the status of nameserver and dhcp server by:

  •    sudo service isc-dhcp-server status
       sudo service bind9 status

    If any of the services give status "stop/waiting", restart nameserver by:

       sudo service bind9 reload

    To restart dhcp by:

       sudo service isc-dhcp-server restart

2.1.5. Turn on FB1, optimus, megatron

• Power on fb1, and megatron on 1X6, and optimus on 1X7.
• ssh into fb1 and ping a few lab computers (nodus, chiara, rossa, c1lsc) to see if nameserver is working for it.
• Check NTP status on fb1 by:

  •    ntpq -p
         remote           refid      st t when poll reach   delay   offset  jitter
    ==============================================================================
    +tor.viarouge.ne 85.199.214.102   2 u  244 1024  377  144.478    0.761   0.566
    *ntp.exact-time. .GPS.            1 u   93 1024  377  174.450   -1.741   0.613
     time.nullrouten .STEP.          16 u    - 1024    0    0.000    0.000   0.000
    +ntp.as43588.net 129.6.15.28      2 u  39m 1024  314  189.152    4.244   0.733
     192.168.123.255 .BCST.          16 u    -   64    0    0.000    0.000   0.000 

    If you do not get a output like above with a few servers with + sign and atleast one server with * sign and a .BCST. server, then NTP is not working properly. Check back dhcp status and namerserver at chiara and restart fb1.

• Note: It takes about 5 minutes for ntp server to start functioning after fb1 comes online.

• After fb1 shows nfs mounts, pinging ability to computers, and is running ntp server, then check megatron.
• ssh into megatron check for cron job:

  •   sudo systemctl status cron -l

• ssh into optimus, and do following:

  •   cd /opt/rtcds/caltech/c1/Git/40m/softchansmodbus
      sudo docker-compose up -d
      cd /opt/rtcds/caltech/c1/Git/40m/scripts
      sudo docker-compose up -d

2.1.6. Control room equipment

• Power on zita
• Power on giada
• Power on donatella
• power on the monitors and the tv.
• On zita, run:

   ./startStrip.sh

2.2. Turn on racks

2.2.1. Under XEND table

Switch on following (check the shutdown section for any extra buttons that were switched off):

• Power strips: 3 white power strips under the table, 1 on east side, 1 on west side, 1 under ETMX chamber on west side.
• Switched on CRT MON at the end.
• Oven temp controller, TC200L Set temperature to 37.1 degrees Celsius. Switched ON from front button, press Enable.
• Turn on oplev laser key.
• Uniblitz shutter driver: NC, Remote. Turned on from power switch on back.
• Innolight laser controller 1.985 A. Turn key to on.

2.2.2. 1X9

Switch on following (check the shutdown section for any extra buttons that were switched off):

• Turned on ac power strip in the back.
• Turned on I/O chassis in the front
• Check power supply status from shutdown notes above.
• Connect power of c1iscex which is directly connected to wall socket on rack.
• Turn on c1iscex
• Turn on c1auxex

2.2.3. Teststand

• Turned on everything by the 2 power strips below the teststand.

2.2.4. 1X5

Switch on following (check the shutdown section for any extra buttons that were switched off):

• Turned on power strip at the bottom.
• Turned on power stip in the top but it has nothing connected to it.
• Check power supply status from shutdown notes above.

2.2.5. 1X4

Switch on following (check the shutdown section for any extra buttons that were switched off):

• Turned on 1X4 I/O Chassis power buttons front
• Turned on power strip in the back middle region.
• Turn on c1susaux
• c1lsc and c1sus should have turned on by themselves. Turn them on if not.

2.2.6. AUX OMC SOUTH (Small rack south of 1X2)

Switch on following (check the shutdown section for any extra buttons that were switched off):

• Turn on white small power strip exted out from the back of this rack for bottom 4 power supplies.
• Check power supply status from shutdown notes above.
• 1 Marconi FG on top at 11.066195 MHz:
  • Freq: 11.066195 MHz
  • RF Level: 13 dBm ON
  • MOD Disable
  • FM2 AC OFF
  • FM1 0, AC, OFF
  • MOdF AC 1.0000 kHz

2.2.7. 1X2

Switch on following (check the shutdown section for any extra buttons that were switched off):

• Power strip in back bottom
• Turn on c1psl
• Turn on c1ioo

2.2.8. 1X1

Switch on following (check the shutdown section for any extra buttons that were switched off):

• Turned on power strip in the back top.
• 1X1-9-1 right most Kepco High Voltage PS powered down by button on front panel.
• I/O Expansion chassis turned on at front
• Check power supply status from shutdown notes above.

2.2.9. Under ITMX table

Switch on following (check the shutdown section for any extra buttons that were switched off):

• Power strips: 1 white strip under table turned on
• JDS Uniphase (oplev laser), turned key to on
• Uniblitz at NP, remote. Used back button to switch on.

2.2.10. PSL table

Switch on following (check the shutdown section for any extra buttons that were switched off):

• Power strips: 1 white strip under table turned on
• Switched on laser controller by turning key to on and pressing on.
• Laser driver current should be around 2.098 A.
• Turned on Uniblitz shutter driver under the table from back buttton. In NC, remote mode.

2.2.11. Under AS table

Switch on following (check the shutdown section for any extra buttons that were switched off):

• 2 long grey powers strips turned om. Label L#36.
• CRT Monitor on injection chamber turned om.

2.2.12. Under BS table

Switch on following (check the shutdown section for any extra buttons that were switched off):

• Uniphase key flipped (not the light indicator does not work). check if oplev turned on by opening the table.

2.2.13. 1Y1

Switch on following (check the shutdown section for any extra buttons that were switched off):

• Turned on the power strip
• Check power supply status from shutdown notes above.
• C1SU2 I/O chassis on front
• Turn on c1sus2

2.2.14. 1Y0

Switch on following (check the shutdown section for any extra buttons that were switched off):

• Turned on the power strip
• Check power supply status from shutdown notes above.
• Turn on c1susaux2

2.2.15. Under ITMY table

Switch on following (check the shutdown section for any extra buttons that were switched off):

• Power strips: 2 white strips under table turned on
• JDSU (oplev laser), turned key to on

2.2.16. 1Y2

Switch on following (check the shutdown section for any extra buttons that were switched off):

• Powerstrip in the back was turned on
• Turn on AUX-1Y1 poweer supplies on the small rack.
• Check power supply status from shutdown notes above.

2.2.17. 1Y3

Switch on following (check the shutdown section for any extra buttons that were switched off):

• Powerstrip in the back was turned on
• Check power supply status from shutdown notes above.
• I/O Expansion chassis, turned power button on the front.
• Turn on c1iscaux

2.2.18. 1Y4

Switch on following (check the shutdown section for any extra buttons that were switched off):

• Power strips: 1 black, 1 white, in the back was turned on,
• Check power supply status from shutdown notes above.
• I/O Expansion chassis, turned power button on front.
• Turn on c1iscey
• Turn on c1auxey1

2.2.19. Y-AUX Rack

Switch on following (check the shutdown section for any extra buttons that were switched off):

• Turned on white power strip in the bottom
• SR560, gain: 10, DC, DC coupled, Low Noise State. Turned on.
• Signal Generator DS345: Settings: Freq: 213120.000, Amp: 2.00 Vpp, Offset: 0, Phase 0

2.2.20. Under YEND table

Switch on following (check the shutdown section for any extra buttons that were switched off):

• Power strips: 3 white power strips under the table.
• Oven temp controller, TC200L Set temperature was 35.7 degrees Celsius.Switched on from front button. Press Enable.
• Uniblitz shutter driver: NC, Remote. Turned on from power switch on back.
• JDS Uniphase (opLev laser_, turned key to 0 to turn on.
• Lightwave 125/126 laser controller. Turn key to on.
• Switched on CRT MON4 at the end.

2.3. CDS Startup

2.3.1. Slow computers

ssh into the following computers:

• c1susaux
• c1auxex
• c1auxey1
• c1psl
• c1iscaux

And do the following:

• Check status of modbusIOC service:

sudo systemctl status modbusIOC.service -l

• If the service is not running, try:

  sudo systemctl restart modbusIOC.service

• If you get "Unit modbusIOC.service could not be found." error, ensure that nfs mount is working and then do:

   sudo systemctl enable modbusIOC.service
   sudo systemctl restart modbusIOC.service

Doing same for (extra steps):

• c1susaux2

c1susaux2 might take upto 5 min to completely come online, so be patient. Once it is online, check if eth1 is up or not by doing:

ip a

If eth1 is not up, do following:

sudo /sbin/ifdown eth1
sudo /sbin/ifup eth1

Finally, do:

sudo systemctl daemon-reload (this is sometimes required before modbusIOC.service)
sudo systemctl start modbusIOC.service
sudo systemctl status modbusIOC.service -l

• c1vac

Contact Koji and JC. For restarting the computer part, it should be same steps as above.

2.3.2. Fast computers

Try to SSH into all fast computers. If there is an issue, try this line on Chiara,

sudo service isc-dhcp-server restart

Run following command on rossa:

cd /opt/rtcds/caltech/c1/Git/40m/scripts/cds
./startAllModels.sh /opt/rtcds/caltech/c1/Git/40m/measurements/cds/backUp_2022_08_12

This should restore the lab to 12th of August 1:19 pm.

2.4. Vacuum pump down

2.4.1. Shutdown Electronics and computer

Tun on following on the rack:

• GP136a
• GP136b
• GP307
• MKS937a
• MKS937b
• XGS-600 (unplugged on back)
• Serial Device Server
• Vacuum acromag chassis (from back)
• PTP3
• PTP2
• TP1
• TP2 (unplugged on back)
• TP3 (unplugged on back)

Turn on c1vac, then ssh to c1vac and do:

sudo systemctl status modbusIOC -l

If the service is not running, try:

sudo systemctl restart modbusIOC.service

If you get "Unit modbusIOC.service could not be found." error, ensure that nfs mount is working and then do:

sudo systemctl enable modbusIOC.service
sudo systemctl restart modbusIOC.service

2.4.2. Vacuum

On the Asus computer, open editable version of vacuum system medm by:

cd /cvs/cds/caltech/target/c1vac/medm
medm -x C0VAC_CONTROL.adl

Then perform following steps:

• Turn on the AUX pump from the yellow breaker near the wall. The reset switch as an ON button.
• Open the manual valves on TP2 and TP3 which are located below them. Turn anti-clockwise to open. Leave them at a point where the vales is free to move both sides, so easily can be determined to be open.
• Turn on TP2 and TP3.
• Wait for atleast 5 minutes, and following conditions to be met:
  • The RPM onf TP2 and TP3 is above the minim marked next to Normal on the window. The status should be "Normal".
  • PTP3 and PTP2 both show pressure less than 2e-2 torr.
• Open V4 and V5. This connects TP2 and TP3 to upstream volumes.
• Turn on TP1 (which is backed by TP2) and monitor RPM on TP1 medm screen and on the Turbo Controller Module on the rack.
• Wait for atleast 2 minurs, and following conditions to be met:
  • Rotating marked X.
  • Accelerating is not marked X.
  • Normal marked X.
  • Speed is around 33.6 kRPM.
• Merge RGA volume by doing following in order:
  • Open VM3
  • Open VA6
• Merge all the annular volume
  • Open VASE, VASV, VABSSCI, VABS, VABSSCO, VAEV, VAEE
• Merge the main volume
  • Open V1