• Complete_power_shutdown_2022_07

Complete power shutdown and start-up plan

There will be a power shutdown for 40m lab completely on Feb 5th (Sat), 2022 from 10am to 2pm. Following is the detailed plan for how to shutdown the whole lab and bring it back up online.

1. Shutdown

* Unplug all wifi routers

1.1. Vacuum shutdown

elog: http://nodus.ligo.caltech.edu:8080/40m/16980

1.1.1. 1X8 (Vacuum Rack)

• Computers that should be shutdown by now:
  • c1vac
  • Lenovo laptop

If you switch off any of the power buttons on the rack, list them down here.

1.1.1.1. Power supplies

1.2. CDS Shutdown

1.2.1. Creating backups

• Ensure all SDF tables in all FE models are saved to keep the changes persistent through power cycle.
• Note that ADC, DAC index orders might change in C1SUS2 I/O chassis and maybe others.
• Create burtgooey backups of all available req files or ensure that autoburt has successfully run in past some time.

1.2.2. Slow computers

1.2.2.1. Old computers

telnet into following computers:

• c1auxey

shutdown

If the computers do not respond (in cases where you cannot telnet into them or maybe the above shutdown command does not work through telnet, it is untested), it is ok to hard reboot them. This can be done using the key on the crate. If no key, press the reset button or keep the ON/OFF button pressed until the machine powers down.

1.2.2.2. New computers

ssh into the following computers:

• c1susaux
• c1auxex
• c1auxey1
• c1psl
• c1iscaux
• c1susaux2

And do the following:

sudo systemctl stop modbusIOC.service
sudo shutdown -h now

• c1vac

I suppose shutting down c1vac changes the state of the gate valves. But shutting down (power down) of the acromag targets will change the relay (or signal) states. It may close the valves.

• Take a screenshot of the valve states before the action
• Close the automated valves as many as possible.
• Shutdown c1vac and power down the acromag targets

1.2.3. Fast computers

ssh into following computers:

• c1lsc
• c1sus
• c1ioo

sudo /sbin/shutdown -h now

ssh into following computers:

• c1iscex
• c1iscey
• c1su2

And do the following:

rtcds stop --all
sudo /sbin/shutdown -h now

For

• c1su2

And do the following:

rtcds stop c1bac c1hpc c1su2 c1x07
sudo /sbin/shutdown -h now

1.2.4. Teststand computers

First ssh into c1teststand

ssh c1teststand

then ssh into the following computers in order and do the following:

• c1bhd
• fb1 (clone)
• chiara (clone)

sudo shutdown -h now

And then shutdown c1teststand computer itself.

Switch off the chassis power buttons, AA/AI power buttons, and then switch off the power supply at the bottom of the stand.

1.2.5. Martian computers

1.2.5.1. Megatron

ssh megatron
sudo shutdown -h now

1.2.5.2. Optimus

ssh optimus
cd /home/controls/Git/40m/scripts/MC
sudo docker-compose down
cd /home/controls/Git/40m/softchansmodbus
sudo docker-compose down
sudo shutdown -h now

1.2.5.3. viviana

ssh viviana
sudo shutdown -h now

1.2.5.4. paola

This is dead right now.

ssh paola
sudo shutdown -h now

1.2.5.5. fb1

ssh fb1
sudo shutdown -h now

Do not shutdown chiara or nodus until you have finished end of shutdown.

1.3. Electronics racks and in-air tables

1.3.1. Y-AUX Rack

If you switch off any of the power buttons on the rack, list them down here.

• SR560, gain: 10, DC, DC coupled, Low Noise State
• Signal Generator DS345: Settings: Freq: 213120.000, Amp: 2.00 Vpp, Offset: 0, Phase 0

1.3.2. Under YEND table

If you switch off any of the power buttons on the rack, list them down here.

• Oven temp controller, TC200L Set temperature was 35.7 degrees Celsius.Switched off from front button.
• Uniblitx shutter driver: NC, Remote. Turned off from power switch on back.
• JDS Uniphase (opLev laser_, turned key to 0 to turn off.
• Lightwave 125/126 laser controller. Turn key to off.
• Power strips: 3 white power strips under the table.
• Switched off MON4 at the end.

1.3.3. 1Y4

• Computers that should be shutdown by now:
  • c1iscey
  • paola

If you switch off any of the power buttons on the rack, list them down here.

• I/O Expansion chassis, turned power button on front.
• Power strips: 1 black, 1 white, in the back was turned off.

1.3.3.1. Sorenson power supplies

1.3.4. 1Y3

• Computers that should be shutdown by now:
  • c1iscaux

If you switch off any of the power buttons on the rack, list them down here.

• I/O Expansion chassis, turned power button on the front.
• Powerstrip in the back was turned off.

1.3.4.1. Sorenson power supplies

1.3.5. 1Y2

If you switch off any of the power buttons on the rack, list them down here.

• Powerstrip in the back was turned off.

1.3.5.1. Sorenson power supplies

• Power buttons on Kepco HV supply are on the front bottom left corner.
• What is the AUX-1Y2-bottom Sorenson p[owering that takes 6 amperes of current?

1.3.6. 1Y1

If you switch off any of the power buttons on the rack, list them down here.

• C1SU2 I/O chassis on front
• Turned off the power strip

1.3.7. 1Y0

If you switch off any of the power buttons on the rack, list them down here.

1.3.7.1. Sorenson power supplies

==== Under ITMY table====

If you switch off any of the power buttons on the rack, list them down here.

• JDSU (oplev laser), turned key to off
• Power strips: 2 white strips under table turned off

==== Under BS table====

If you switch off any of the power buttons on the rack, list them down here.

• Uniphase key flipped

==== Under AS table====

If you switch off any of the power buttons on the rack, list them down here.

• 2 long grey powers strips turned off. Label L#36.\
• Monitor on injection chamber turned off.

==== Under ITMX table====

If you switch off any of the power buttons on the rack, list them down here.

• JDS Uniphase (oplev laser), turned key to off
• Uniphase shutter driver key flipped
• Power strips: 1 white strip under table turned off

==== PSL table====

If you switch off any of the power buttons on the rack, list them down here.

• Laser was already off, maybe through interlock.
• Turned off uniphase shutter driver under the table from back buttton.

1.3.8. 1X1

If you switch off any of the power buttons on the rack, list them down here.

• I/O Expansion chassis turned off at front
• Turned off power strip in the back top.

1.3.8.1. Power supplies

1.3.9. 1X2

• Computers that should be shutdown by now:
  • c1psl
  • c1ioo

If you switch off any of the power buttons on the rack, list them down here.

• Power strip in back bottom

1.3.10. AUX OMC SOUTH (Small rack south of 1X2)

If you switch off any of the power buttons on the rack, list them down here.

• 1 Marconi FG on top at 11.066195 MHz:
  • Freq: 11.066195 MHz
  • RF Level: 13 dBm
  • MOD Disable
  • FM2 AC OFF
  • FM1 0, AC, OFF
  • MOdF AC 1.0000 kHz

1.3.11. 1YNot

• Empty unused rack

1.3.12. 1X4

• Computers that should be shutdown by now:
  • c1lsc
  • c1sus
  • c1susaux

If you switch off any of the power buttons on the rack, list them down here.

• Turned off 1X4 I/O Chassis power buttons front

1.3.13. 1X5

If you switch off any of the power buttons on the rack, list them down here.

1.3.13.1. Power supplies

1.3.14. Teststand

• Turned off everything by power strip below the teststand.

1.3.15. 1X9

• Computers that should be shutdown by now:
  • c1auxex

If you switch off any of the power buttons on the rack, list them down here.

• Turned off I/O chassis in the front
• Turned off ac power strip in the back.

1.3.15.1. Power supplies

1.3.16. X-AUX Rack

Power to this rack got turned off when we turned off 1X9 power.

1.3.17. Under XEND table

If you switch off any of the power buttons on the rack, list them down here.

• Oven temp controller, TC200L Set temperature was 37.1 degrees Celsius. Switched off from front button.
• Uniblitx shutter driver: NC, Remote. Turned off from power switch on back.
• Innolight laser controller 1.985 A. Turn key to off.
• Power strips: 3 white power strips under the table, 1 on east side, 1 on west side, 1 under ETMX chamber on west side.
• Switched off MON at the end.

1.4. Control room equipment

• Shutdown following computers:
  • zita
  • donatella
  • pianosa
  • rossa
  • Giada
  • c1sim
• Turned off moku
• Turned off all monitors and TVs

1.5. Last steps

Now shutdown the following computers

1.5.1. nodus

ssh nodus
sudo /sbin/shutdown -h now

1.5.2. chiara

ssh Chiara
sudo /sbin/shutdown -h now

1.5.3. 1X6

• Computers that should be shutdown by now:
  • fb1
  • megatron
  • FB (old) already off
• Jetstor powered at the bottom most location. Needs to be checked.

If you switch off any of the power buttons on the rack, list them down here.

• Network switch (unplugged in the back)

• Jetstor turned off in the back by little black button

• Master Timer Sequencer in the back by button

• Turned off the rear (upper) power strip

1.5.4. 1X7

• Network Switch 1
• Network switch 2
• 19 AC power connections
• Computers that should be shutdown by now:
  • nodus
  • chiara
  • optimus
• UPS at the bottom of the rack

If you switch off any of the power buttons on the rack, list them down here.

• Network switch 1

• Network switch 2

• GPS TimeSource

• UPS turned off
• Power strips in the back, top and bottom.

1.5.5. Office network rack

* Network switches for Martian and GC network were turned off * Turn off printer

2. Startup

2.1. Office space

• Turned on network switches for martian network, and WiFi routers.

• Turned on windows (CAD) machine

2.2. Main network stations

• Power on network switches,
• Power on chiara (main nameserver)
• Power on nodus, try ping, try ssh.
• Power on fb1, try ssh.
• Power on paola (xend laptop), viviana (yend laptop), optimus, megatron.
• Restart PSL FSSSlow.service on megatron
• Restart elog on nodus
• Restart IMC autolocker on optimus

2.3. Control room equipment

• Power on zita
• Power on giada
• Power on donatella
• Power on allegra
• Power on pianosa
• Power on rossa

2.4. PSL + Vertex instruments

• Turn on newport PD power supplies on PSL table.

• Turn on TC200 temp controller on (setpoint --> 36.9 C)

• Turn on two oscilloscopes in PSL table.

• Turn on PSL (current setpoint --> 2.1 A, other settings seem nominal)

• Turn on Thorlabs HV pzt supply.

2.5. YEND and XEND instruments

• Turn on XEND AUX pump on (current setpoint --> 1.984 A)

• Turn on XEND AUX SHG oven on (setpoint --> 37.1 C), wait until see green beam.

• Turn on XEND AUX shutter controller on.

• Turn on DCPD supplies, OpLev supplies, and/or AC strips on.

• Turn on YEND AUX pump on (current setpoint --> )

• Turn on YEND AUX SHG oven on (setpoint -->35.7 C), wait until see green beam.

• Turn on YEND AUX shutter controller on.

• Turn on DCPD supplies, OpLev supplies, and/or AC strips on.

2.6. Other instruments

• Turn on ITMX OpLev / laser instrument AC strip.

2.7. CDS Startup

2.7.1. Slow computers

2.7.1.1. Old computers

2.7.1.2. New computers

ssh into the following computers:

• c1susaux
• c1auxex
• c1auxey1
• c1psl
• c1iscaux

And do the following:

sudo systemctl start modbusIOC.service
sudo systemctl status modbusIOC.service -l

• c1susaux2

c1susaux2 might take upto 5 min to completely come online, so be patient. Once it is online, check if eth1 is up or not by doing:

ip -a

If eth1 is not up, do following:

sudo /sbin/ifdown eth1
sudo /sbin/ifup eth1

Finally, do:

sudo systemctl daemon-reload (this is sometimes required before modbusIOC.service)
sudo systemctl start modbusIOC.service
sudo systemctl status modbusIOC.service -l

• c1vac

Contact Koji and JC. For restarting the computer part, it should be same steps as above.

2.7.2. Fast computers

Try to SSH into all fast computers. If there is an issue, try this line on Chiara,

sudo service isc-dhcp-server restart

Run following command on rossa:

cd /opt/rtcds/caltech/c1/Git/40m/scripts/cds
./startAllModels.sh /opt/rtcds/caltech/c1/Git/40m/measurements/cds/backUp_2022_07_08

This should restore the lab to 8th of July 1:19 pm.

2.8. Wifi routers

Put back wifi routers to wall in sequence, goping away from control room, one at a time and waiting atleast 2 minutes between each installation.