Slow Controls Upgrade Procedure

Updated Mar 2019 Jon Richardson

Updated Jul 2019 gautam

Hardware

Auxiliary DAQ Modules

The new auxiliary DAQ systems use the Acromag XT series DIN-rail mounted ADC, DAC and BIO modules. The model numbers are in the format XTYYY1, where "XT" is a static prefix, "YYY" determines the function, and the "1" as the last digit denotes the ModBus/TCP version of the model. The 1 at the end is important, as the other variations will not be able to interface with CDS through EPICS.

Rackmount Chassis

The Acromag units are mounted on a DIN rail inside a custom 6U rackmount chassis. To mount the DIN rails two screw holes need be drilled on each side of the Acromag chassis. The Din rails are then screwed to the sides using spacers as shown below. A 24 V DC power source connected through the rear of the chassis provides power for the Acromag units. A second 15 V DC source connected in the same way provides the excitation source for Acromag outputs. Note that the XT1121 model can get the excitation voltage only from the terminals so additional wiring is needed.

A diagram of the internal chassis wiring is shown below. Note:

- Although not shown in the diagram, all the LEDs should be connected in series to 1kOhm resistors in order to work properly.

- A typical Acromag module draws ~ 0.1A. Based on the number of modules to be installed choose wires that can handle the expected total amount of current.

Host Computer

1. Fresh install of Debian 8 (or most recent stable release) - you can use the USB stick labeled "Debian 8.5 boot disk"

   • Hostname: same as the old system. For testing purposes, you may want to append a numeral to the old hostname (e.g. c1susaux1) to permit the existing machine to run while the new one is being bench-tested.

   • Username: controls

   • Optional: lightweight desktop environment LXDE (recommended)

   • Hard-disk partitioning: Guided - use entire disk , choose All files in one partition option.

   • During the install, the installer will ask you about the default network interface to use (since the SuperMicro has 2 network cards). Choose eth0. Note that attempting to set up a DHCP connection to the internet for the installation process, you will have to plug into the correct physical ethernet jack - eth0 is the one on the left, with the SuperMicro seen from the rear. Also be sure to plug into the general-computing network switch, not the martian network switch. The static IP setup can be done later.

2. Give sudo privileges to user controls

   • $ su
     $ usermod -aG sudo controls
     $ exit

     Log out and log back in for the change to take effect.

3. Assign the host machine an available IP address on the martian network, 192.168.113.<XXX>

   • <XXX> is an integer in the range 0-255. Do not use 0, 10, or 255 (reserved)

   • See here for the list of in-use addresses

4. Register the host machine in the DNS lookup table on chiara

   • Connect to chiara

     $ ssh -X 192.168.113.104

     Add the following line to /var/lib/bind/martian.hosts

     <HOSTNAME> A 192.168.113.<XXX>

     • <HOSTNAME> is a placeholder for the assigned hostname

     • Search the file to make sure <XXX> is not already in use

     Add the following line to /var/lib/bind/rev.113.168.192.in-addr.arpa

     <XXX> PTR <HOSTNAME>

     Restart the DNS server and log off

     $ sudo service bind9 restart
     $ logout

5. Configure the martian and local ethernet adapters

   • Edit /etc/network/interfaces as follows

     allow-hotplug eth0
     iface eth0 inet static
     address 192.168.113.<XXX>
     netmask 255.255.255.0
     gateway 192.168.113.2
     dns-nameservers 192.168.113.104 131.215.125.1 131.215.139.100
     dns-search martian
     
     allow-hotplug eth1
     iface eth1 inet static
     address 192.168.114.<YYY>
     netmask 255.255.255.0

     • <YYY> is a placeholder for the IP address of the host machine on the local subnet

6. Enable DNS lookup on the martian network

   • Add the following lines to /etc/resolv.conf. Comment out all of the existing lines.

     search martian
     nameserver 192.168.113.104

7. Optional: If any devices on the subnet will need to access the martian network, follow these steps to set this up. Note that Acromags do not need martian access, as they will only be communicating with the local host.

   • Enable IP forwarding from eth1 to eth0

     $ su
     $ echo 1 > /proc/sys/net/ipv4/ip_forward
     $ exit

     Configure IP tables to allow outgoing connections, while keeping the subnet invisible from the outside

     $ sudo iptables -t nat -A POSTROUTING -o eth0 -j MASQUERADE
     $ sudo iptables -A FORWARD -i eth0 -o eth1 -m state --state RELATED,ESTABLISHED -j ACCEPT
     $ sudo iptables -A FORWARD -i eth1 -o eth0 -j ACCEPT

8. NOTE: THIS PREVENTS EPICS FROM AUTO-DETECTING ALL THE OTHER IOCS

   • Optional: If an EPICS client application will be running on the host machine (e.g., a Python interlock/watchdog process), we must specify the EPICS server address. This is because the EPICS server automatically binds to all detected network interfaces, so local clients will see two server instances, one at 192.168.113.<XXX> and one at 192.168.114.<YYY> Add the following lines to /home/controls/.bashrc

     EPICS_CA_AUTO_ADDR_LIST=NO
     EPICS_CA_ADDR_LIST=192.168.114.<YYY>

     Load the changes into the bash shell

     $ source /home/controls/.bashrc

9. Finally restart the networking service

   • $ sudo /etc/init.d/networking restart

10. Update the Debian package manager

    • $ sudo apt update
      $ sudo apt upgrade
      $ sudo apt update

11. Set up incoming SSH access (second line is critical, check status of the ssh service using sudo service ssh status):

    • $ sudo apt -y install openssh-client
      $ sudo apt -y install openssh-server

12. Optional: Set up incoming remote desktop access

    • $ sudo apt -y install xrdp

13. Set up access to the 40m network drive

    • $ sudo apt -y install nfs-common
      $ sudo mkdir /cvs/
      $ sudo mkdir /cvs/cds

      Add the following line to /etc/fstab

      chiara:/home/cds /cvs/cds nfs rw,bg,nfsvers=3

      Mount the network drive

      $ sudo mount -a

Configure Acromag DAQ Modules

The second screen varies by module type. Non-default options are circled. Note that the non-default options have to be set for INDIVIDUAL channels! Once the non-default options of all the channels of one device are set, they can be easily set for the next devices by not allowing the utility to refresh the I/O configuration screen when a new device is connected.

Install Modbus

There are three packages needed to get modbus over TCP/IP (Acromags) working with an EPICS IOC process: the EPICS base and two modules that extend its function, asyn and modbus. The latest modbus package and its required modules must be downloaded from the official source. It is best to use the latest stable release of modbus and match with the recommend base and asyn versions. For example, modbus 2.11 recommends EPICS base 7.0.1 and asyn 4.33. To install you must download the source, install build tools, set some environment variables correctly, and then compile. Depending on the speed of the machine this might take 15-60 minutes.

1. Install the usual basic build tools

   • $ sudo apt -y install curl g++ make libperl-dev libreadline-dev wget

2. Now we need to add some environment variables so the make process knows where to look for and put things.

   • Add the following lines to /home/controls/.bashrc

     export EPICS_HOST_ARCH=linux-x86_64
     export EPICS_ROOT=/opt/epics
     export EPICS_BASE=${EPICS_ROOT}/base
     export EPICS_BASE_BIN=${EPICS_BASE}/bin/${EPICS_HOST_ARCH}
     export EPICS_BASE_LIB=${EPICS_BASE}/lib/${EPICS_HOST_ARCH}

     Then load the changes into the bash shell

     $ source /home/controls/.bashrc

3. Download the source files from the official websites (linked above). Here we use the recommended stable combination of modbus-R2-11, asyn4-33, and base-7.0.1 (as of March 2019).

   • $ cd /home/controls/Downloads
     $ wget https://epics.anl.gov/download/base/base-7.0.1.1.tar.gz
     $ wget https://epics.anl.gov/download/modules/asyn4-33.tar.gz
     $ wget https://github.com/epics-modules/modbus/archive/R2-11.tar.gz -O modbus-R2-11.tar.gz

4. Now make the directories to build epics and its modules into

   • $ sudo mkdir /opt/epics
     $ sudo mkdir /opt/epics/modules

5. Untar the downloaded files into these directories

   • $ sudo tar xvzf base-7.0.1.1.tar.gz -C /opt/epics/  
     $ sudo tar xvzf asyn4-33.tar.gz -C /opt/epics/modules/  
     $ sudo tar xvzf modbus-R2-11.tar.gz -C /opt/epics/modules/

     • You can delete the original tar files after this point if you like.
6. Symlink these directories to simple (unversioned) names. This makes navigating easier and makes the final build version-agnostic.

   • $ sudo ln -s /opt/epics/base-7.0.1.1 /opt/epics/base
     $ sudo ln -s /opt/epics/modules/asyn4-33 /opt/epics/modules/asyn
     $ sudo ln -s /opt/epics/modules/modbus-R2-11 /opt/epics/modules/modbus

7. Tweak the configuration files for the asyn and modbus modules so that they know where and how the EPICS base was built.

   • For asyn, open the file /opt/epics/modules/asyn/configure/RELEASE and comment out the lines for IPAC and SNCSEQ. Then change the line for SUPPORT to SUPPORT=/opt/epics/modules and the line for EPICS_BASE to EPICS_BASE=/opt/epics/base. The modified RELEASE file for asyn should look like

     #RELEASE Location of external products
     
     SUPPORT=/opt/epics/modules
     -include $(TOP)/../configure/SUPPORT.$(EPICS_HOST_ARCH)
     
     #  IPAC is only necessary if support for Greensprings IP488 is required
     #  IPAC release V2-7 or later is required.
     #IPAC=$(SUPPORT)/ipac-2-14
     
     # SEQ is required for testIPServer
     #SNCSEQ=$(SUPPORT)/seq-2-2-4
     
     #  EPICS_BASE 3.14.6 or later is required
     EPICS_BASE=/opt/epics/base
     -include $(TOP)/../configure/EPICS_BASE.$(EPICS_HOST_ARCH)

     For modbus, open the file /opt/epics/modules/modbus/configure/RELEASE and change the SUPPORT and EPICS_BASE lines in the same way as above. Also change the ASYN line to ASYN=$(SUPPORT)/asyn. The final RELEASE file for modbus should look like

     #RELEASE Location of external products
     # Run "gnumake clean uninstall install" in the application
     # top directory each time this file is changed.
     #
     # NOTE: The build does not check dependancies on files
     # external to this application. Thus you should run
     # "gnumake clean uninstall install" in the top directory
     # each time EPICS_BASE, SNCSEQ, or any other external
     # module defined in the RELEASE file is rebuilt.
     
     TEMPLATE_TOP=$(EPICS_BASE)/templates/makeBaseApp/top
     
     SUPPORT=/opt/epics/modules
     -include $(TOP)/../configure/SUPPORT.$(EPICS_HOST_ARCH)
     
     ASYN=$(SUPPORT)/asyn
     
     # If you don't want to install into $(TOP) then
     # define INSTALL_LOCATION_APP here
     #INSTALL_LOCATION_APP=<fullpathname>
     
     # EPICS_BASE usually appears last so other apps can override stuff:
     EPICS_BASE=/opt/epics/base
     -include $(TOP)/../configure/EPICS_BASE.$(EPICS_HOST_ARCH)
     
     #Capfast users may need the following definitions
     #CAPFAST_TEMPLATES=
     #SCH2EDIF_PATH=

8. Now you're ready to compile the source code
   • First build the EPICS base

     $ cd /opt/epics/base
     $ sudo make

     Next build the asyn module

     $ cd /opt/epics/modules/asyn
     $ sudo make

     Finally build the modbus module

     $ cd /opt/epics/modules/modbus
     $ sudo make

9. Add the build location to the system path

   • Add the following line to /home/controls/.bashrc

     export PATH=/opt/epics/base/bin/linux-x86_64:/opt/epics/extensions/bin/linux-x86_64:/opt/epics/modules/modbus/bin/linux-x86_64:/opt/epics/modules/asyn/bin/linux-x86_64:$PATH

     Load the changes into the bash shell

     $ source /home/controls/.bashrc

Configure Modbus Driver

epicsEnvSet("IOC", "c1<HOSTNAME>_iocconfig")   # insert proper hostname
epicsEnvSet("ARCH","linux-x86_64")
epicsEnvSet("TOP","/opt/epics/modules/modbus")
epicsEnvSet("MDBTOP","$(TOP)")
dbLoadDatabase("$(MDBTOP)/dbd/modbus.dbd")
modbus_registerRecordDeviceDriver(pdbbase)

drvAsynIPPortConfigure(const char *portName,  # user-defined: used for subsequent referencing, 
                       const char *hostInfo,  # format: "IP-Address:Port". Standard port for Modbus is 502
                       unsigned int priority, # 
                       int noAutoConnect,     # 
                       int noProcessEos);     # 

modbusInterposeConfig(const char *portName,    # reference to portName created with drvAsynIPPortConfigure command
                      modbusLinkType linkType, # 
                      int timeoutMsec,         # define timeout for waiting for response from unit
                      int writeDelayMsec)      #

drvModbusAsynConfigure(portName,                # used by channel definitions in .db file to reference this unit)
                       tcpPortName,             # reference to portName created with drvAsynIPPortConfigure command
                       slaveAddress,            # 
                       modbusFunction,          # defines driver function for the unit (read register = 4, write register = 6, write single coil = 5) - see examples in next section
                       modbusStartAddress,      # ADC and binary channel numbering starts with 0, DAC channel numbering with 1
                       modbusLength,            # length in dataType units - see examples in next section
                       dataType,                # 4 = 16-bit signed integers (for A/D and D/A), 0 = binary (for BIO, duh)
                       pollMsec,                # how frequently to request a value in [ms]
                       plcType);                #

drvAsynIPPortConfigure("<TCP_PORT_NAME>","<UNIT_IP_ADDRESS>:502",0,0,1)
modbusInterposeConfig("<TCP_PORT_NAME>",0,5000,0)
drvModbusAsynConfigure("<PORT_NAME>","<TCP_PORT_NAME>",0,<modbusFunction>,<modbusStartAddress>,<modbusLength>,<dataType>,<pollMsec>,"Acromag")

drvAsynIPPortConfigure("c1susaux_adc00","192.168.115.20:502",0,0,1)
modbusInterposeConfig("c1susaux_adc00",0,5000,0)
drvModbusAsynConfigure("C1SUSAUX_ADC00","c1susaux_adc00",0,4,0,8,4,32,"Acromag")

drvAsynIPPortConfigure("c1susaux_dac00","192.168.115.40:502",0,0,1)
modbusInterposeConfig("c1susaux_dac00",0,5000,0)
drvModbusAsynConfigure("C1SUSAUX_DAC00","c1susaux_dac00",0,6,1,8,4,1,"Acromag")
drvModbusAsynConfigure("C1SUSAUX_DAC00_BIO","c1susaux_dac00",0,5,0,4,0,1,"Acromag")   # Separate instruction for integrated BIO channels

drvAsynIPPortConfigure("c1susaux_bio00","192.168.115.60:502",0,0,1)
modbusInterposeConfig("c1susaux_bio00",0,5000,0)
drvModbusAsynConfigure("C1SUSAUX_BIO00","c1susaux_bio00",0,6,0,4,0,1,"Acromag")

dbLoadDatabase("/cvs/cds/caltech/target/<HOSTNAME>/<SUBSYSTEM>.db")

A good resource for some example/template ways of representing various types of EPICS channels in modbus parlance is given in the modbus installation itself. If you followed the instructions on this page and installed a local copy of EPICS, you may find the example files in /opt/epics/modules/modbus/modbusApp/Db. Otherwise, go look in the modbusApp/Db directory wherever you installed it. The examples are great!

The EPICS records in the .db files for the different DAQ channels are defined as follows (only fields relevant for proper addressing and conversion are shown):

XT1221 ADC Channels:

record(ai,"C1:<CHANNEL_NAME>")
{
        field(DTYP,"asynInt32")
        field(INP, "@asynMask(<PORT_NAME> <X> -16)MODBUS_DATA")   # <X>: 0-7 physical channel on Acromag unit
        field(LINR,"LINEAR")    # For count to float conversion: The ADC module converts +/-10V inputs to +/-30,000 counts (if legacy support is disabled, +/- 20,000 counts otherwise)
        field(ALSO,"1.09225")   # Rescales the raw counts from +/-30,000 to +/-32,767 BEFORE applying the linear conversion
        field(EGUF,"10")        # Corresponds to +32767 after rescale
        field(EGUL,"-10")       # Corresponds to -32767 after rescale
}

XT1541 DAC Channels:

record(ao,"C1:<CHANNEL_NAME>")
{
        field(DTYP,"asynInt32")
        field(OUT, "@asynMask(<PORT_NAME>, <X>, -16)MODBUS_DATA")   # <X>: 0-7 physical channel on Acromag unit
        field(LINR,"LINEAR")    # For count to float conversion: The DAC module converts +/-10V outputs to +/-30,000 counts (if legacy support is disabled, +/- 20,000 counts otherwise)
        field(EGUF,"10")        # Corresponds to +32767 in the above scale
        field(EGUL,"-10")       # Corresponds to -32767 in the above scale
        field(ALSO,"1.09225")   # Finally rescales the raw counts from +/-32,767 to +/-30,000
}

'''XT1541 BIO Channels:'''
record(bo, "<CHANNEL_NAME>")
{
        field(DTYP,"asynInt32")
        field(OUT,"@asynMask(<PORT_NAME>, <X>, -16)MODBUS_DATA")
}

XT1111 & XT1121 BIO Channels:

For both binary input and output:

record(bo, "C1:<CHANNEL_NAME>")
{
        field(DTYP,"asynUInt32Digital")
        field(OUT,"@asynMask(<PORT_NAME>, <X>, 0x<Y>)")   # <X>: 0-3    <Y>: 1,2,4,or 8
}

Note: The addressing scheme used to map the physical BIO channels to the above <X>, <Y> notation is as follows:

The above mapping can be used for the mbboDirect record type as well. This datatype basically maps multiple hardware BIO channels into a "BIT"/control word. An important specifier is the EPICS variable NOBT, which allows one to specify the number of bits that are controlled in tandem. An example mbboDirect record is:

grecord(mbboDirect,"C1:LSC-CM_BOOST2_BITS")
{
    field(DESC,"AO gain bits")
  field(DTYP,"asynUInt32Digital")
  field(OUT,"@asynMask(C1ISCAUX_BIO03 0 0x8)")
    field(NOBT,"2")
#   field(PHAS,"5")
#   field(SCAN,".1 second")
    field(OMSL,"closed_loop")
    field(DOL,"C1:LSC-CM_BOOST2_SET")
  field(PINI,"YES")
}

1. Install procserv for proper detaching of Modbus IOC systemd service

   $ sudo apt -y install procserv

2. Create a new file /etc/systemd/system/modbusIOC.service with the following content. The field <HOSTNAME> is a placeholder for the system name.

   • [Unit]
     Description=ModbusIOC Service via procServ
     Requires=network.target
     After=syslog.target network.target
     ConditionFileIsExecutable=/usr/bin/procServ
     ConditionFileIsExecutable=/opt/epics/modules/modbus/bin/linux-x86_64/modbusApp
     ConditionFileIsExecutable=/cvs/cds/caltech/target/<HOSTNAME>/<HOSTNAME>.cmd
     
     [Service]
     User=controls
     Group=controls
     WorkingDirectory=/cvs/cds/caltech/target/<HOSTNAME>
     
     # Set environment variables for EPICS
     EnvironmentFile=/cvs/cds/caltech/target/<HOSTNAME>/<HOSTNAME>.env
     
     # First line enables logging to local file for debugging. The logfile tends to grow large when there are problems. Use second line for normal operation.
     #ExecStart=/usr/bin/procServ -f -L /cvs/cds/caltech/target/<HOSTNAME>/modbusIOC.log -p /run/modbusioc.pid 8008 /opt/epics/modules/modbus/bin/linux-x86_64/modbusApp /cvs/cds/caltech/target/<HOSTNAME>/<HOSTNAME>.cmd
     ExecStart=/usr/bin/procServ -f -p /run/modbusioc.pid 8008 /opt/epics/modules/modbus/bin/linux-x86_64/modbusApp /cvs/cds/caltech/target/<HOSTNAME>/<HOSTNAME>.cmd
     
     Restart=always
     RestartSec=30
     KillMode=process
     
     [Install]
     WantedBy=multi-user.target

3. Create an environment file /cvs/cds/caltech/target/<HOSTNAME>/<HOSTNAME>.env with the following content. Any environment variables, including PATH, must be supplied through this file, as systemd does not have access to the bash shell environment.

   • export PATH=/opt/epics/base/bin/linux-x86_64:/opt/epics/extensions/bin/linux-x86_64:/opt/epics/modules/modbus/bin/linux-x86_64:/opt/epics/modules/asyn/bin/linux-x86_64:$PATH

4. Activate the new service

   • $ sudo systemctl enable modbusIOC.service
     $ sudo systemctl start modbusIOC.service

Optional: installation of CDS Workstation tools