Slow Controls Upgrade Procedure

Updated Mar 2019 Jon Richardson

Hardware

Auxiliary DAQ Modules

The new auxiliary DAQ systems use the Acromag XT series DIN-rail mounted ADC, DAC and BIO modules. The model numbers are in the format XTYYY1, where "XT" is a static prefix, "YYY" determines the function, and the "1" as the last digit denotes the ModBus/TCP version of the model. The 1 at the end is important, as the other variations will not be able to interface with CDS through EPICS.

Auxiliary DAQ Chassis

Auxiliary DAQ Host Machine

The Acromag modules communicate with a central host computer on a local network. The computer runs an EPICS IOC which interfaces with the Acromag modules via Modbus/TCP and hosts the EPICS records for analog and binary auxiliary channels. For these systems we've chosen to use Supermicro SYS-5017A-EP 1U rackmount servers, which have two Ethernet interfaces. Having two network interfaces is critical for creating a closed local network between Acromags and host.

HowTo

Configure Host Machine

The following has proven to work with Debian 8 (x64) on a Supermicro SYS-5017A-EP.

Initial Set Up

Setting up the machine initially requires a monitor, keyboard, and mouse connected locally.

1. Fresh install of Debian 8 (or most recent stable release)
   • Hostname: same as old system

   • Username: controls

   • Optional: lightweight desktop environment LXDE (recommended)

2. Give sudo privileges to user controls

   • $ su
     $ usermod -aG sudo controls
     $ exit

Set Up Networking

1. Assign the host machine an available IP address on the martian network, 192.168.113.XXX

   • XXX is an integer in the range 0-255. Do not use 0, 10, or 255 (reserved)

   • See here for the list of in-use addresses

2. Register the host machine in the DNS lookup table on chiara

   • Connect to chiara

     $ ssh -X 192.168.113.104

     Add the following line to /var/lib/bind/martian.hosts

     HOSTNAME A 192.168.113.XXX

     • HOSTNAME is a placeholder for the assigned hostname

     • Search the file to make sure XXX is not already in use

     Add the following line to /var/lib/bind/rev.113.168.192.in-addr.arpa

     XXX PTR HOSTNAME

     Restart the DNS server and log off

     $ sudo service bind9 restart
     $ logout

3. Configure the martian and local ethernet adapters

   • Edit /etc/network/interfaces as follows

     allow-hotplug eth0
     iface eth0 inet static
     address 192.168.113.XXX
     netmask 255.255.255.0
     gateway 192.168.113.2
     dns-nameservers 192.168.113.104 131.215.125.1 131.215.139.100
     dns-search martian
     
     allow-hotplug eth1
     iface eth1 inet static
     address 192.168.114.YYY
     netmask 255.255.255.0

     • YYY is a placeholder for the IP address of the host machine on the local subnet, 192.168.114.YYY

4. Enable DNS lookup on the martian network

   • Add the following lines to /etc/resolv.conf

     search martian
     nameserver 192.168.113.104

5. Enable IP forwarding from eth1 to eth0

   • $ su
     $ echo 1 > /proc/sys/net/ipv4/ip_forward
     $ exit

6. Configure IP tables to allow outgoing connections, while keeping the subnet invisible from the outside

   • $ sudo iptables -t nat -A POSTROUTING -o eth0 -j MASQUERADE
     $ sudo iptables -A FORWARD -i eth0 -o eth1 -m state --state RELATED,ESTABLISHED -j ACCEPT
     $ sudo iptables -A FORWARD -i eth1 -o eth0 -j ACCEPT

7. Optional: If an EPICS client application will be running on the host machine, we must specify which server address it should use

   • Add the following lines to /home/controls/.bashrc

     EPICS_CA_AUTO_ADDR_LIST=NO
     EPICS_CA_ADDR_LIST=192.168.114.YYY

     • The EPICS server automatically binds to both network interfaces, so local clients will see two server instances, one at 192.168.113.XXX and one at 192.168.114.YYY
     Load the changes into the bash shell

     $ source /home/controls/.bashrc

8. Restart the networking service

   • $ sudo /etc/init.d/networking restart

Set Up Remote Access Protocols

1. Update apt, the Debian package manager

   • $ sudo apt update
     $ sudo apt upgrade
     $ sudo apt update

2. Set up SSH access

   • $ sudo apt -y install openssh-client

3. Optional: Set up remote desktop access

   • $ sudo apt -y install xrdp

4. Set up access to the 40m network drive

   • $ sudo apt -y install nfs-common
     $ mkdir /cvs/cds

     Add the following line to /etc/fstab

     chiara:/home/cds /cvs/cds nfs rw,bg,nfsvers=3

     Mount the network drive

     $ sudo mount -a

Set Up Modbus IOC as System Service

Once ssh and/or rdesktop access are set up, the remaining steps can be done remotely.

1. Install procserv for proper detaching of Modbus IOC systemd service

   $ sudo apt -y install procserv

Install Modbus

For reasons discussed in elog 14495, the new Acromag systems cannot be reliably run using the modbus executable on the network drive. It must be installed locally. The following installation instructions are largely copied from an earlier note by Andrew Wade.

There are three packages needed to get modbus over TCP/IP (Acromags) working with an EPICS IOC process: the EPICS base and two modules that extend its function, asyn and modbus. The latest modbus package and its required modules must be downloaded from the official source. It is best to use the latest stable release of modbus and match with the recommend base and asyn versions. For example, modbus 2.11 recommends EPICS base 7.0.1 and asyn 4.33. To install you must download the source, install build tools, set some environment variables correctly, and then compile. Depending on the speed of the machine this might take 15-60 minutes.

Configure Acromag DAQ Modules

Initial configuration via USB is required before the units can be used on the network. This requires:

• USB-to-miniUSB cable
• Microsoft Windows (virtual) machine

• Configuration Utilities: 9500465D.zip

The above zip file contains the setup executable to install the configuration software for all different modules. Depending on the module, a different version of the utility needs to be launched after install. As during operation, the to-be-configured device needs to be powered by a 12-32V DC voltage source, either through the designated blue plug-in terminal on the bottom, or the clip-on dock on the rail.

On a virtual machine it may be necessary to open some ports to allow the guest OS to access the physical USB device.

In USB Settings:

add a corresponding filter:

The vendor ID and product ID can be found via lsusb

Network Configuration

The primary screen of the configuration utilities looks identical for all models:

Once the device is connected to the host via USB and the correct application has been launched, it will appear in the drop down menu. On the right the device IP and other network parameters can be set. Every host machine of the auxiliary DAQ acts as a node between the martian network and a local subnet to which the Acromag modules belong and which is not directly accessible from other machines on the martian network.

• For consistency, the ADC, DAC, and BIO modules occupy different IP address spaces:

  Module	Local IP Address Space
  ADC	20-39
  DAC	40-59
  BIO	60-79

• Hostname and IP on the local network are assigned according to the following scheme:

  Hostname	Local IP
  C1XXX-ADCYY	192.168.11X.YY+20
  C1XXX-DACYY	192.168.11X.YY+40
  C1XXX-BIOYY	192.168.11X.YY+60

• Other settings (of no practical significance) are
  • Gateway is the local IP of the node machine
  • DNS is that of Chiara on the martian network: 192.168.113.104

Configure XT unit Modbus driver

The EPICS binary modbusApp which launches the EPICS IOCs for the auxiliary DAQ system requires a .cmd instruction file which sets up the communication between the Acromag units and the EPICS IOC. The .cmd file is placed in /cvs/cds/caltech/target/HOSTNAME/HOSTNAME.cmd. It is largely similar to other EPICS binaries which launch an IOC, such as softIOC, with the addition that it can load the appropriate drivers for the modbus protocol prior to launching the IOC.

The file header consists of path definitions. At the current time there are no system variables pointing to these paths globally set, so the paths are explicitly defined.

epicsEnvSet("IOC", "c1XXX_iocconfig")   # insert proper hostname for XXX
epicsEnvSet("ARCH","linux-x86_64")
epicsEnvSet("TOP","/cvs/cds/rtapps/epics-3.14.12.2_long/modules/modbus")
epicsEnvSet("MDBTOP","/cvs/cds/rtapps/epics-3.14.12.2_long/modules/modbus")
dbLoadDatabase("$(MDBTOP)/dbd/modbus.dbd")
modbus_registerRecordDeviceDriver(pdbbase)

Each Acromag unit requires a set of three instructions which define the driver and Modbus parameters for the handshake during communication.

drvAsynIPPortConfigure(const char *portName,  # user-defined: used for subsequent referencing, 
                       const char *hostInfo,  # format: "IP-Address:Port". Standard port for Modbus is 502
                       unsigned int priority, # 
                       int noAutoConnect,     # 
                       int noProcessEos);     # 

modbusInterposeConfig(const char *portName,    # reference to portName created with drvAsynIPPortConfigure command
                      modbusLinkType linkType, # 
                      int timeoutMsec,         # define timeout for waiting for response from unit
                      int writeDelayMsec)      #

drvModbusAsynConfigure(portName,                # used by channel definitions in .db file to reference this unit)
                       tcpPortName,             # reference to portName created with drvAsynIPPortConfigure command
                       slaveAddress,            # 
                       modbusFunction,          # defines driver function for the unit (read register = 4, write register = 6, write single coil = 5) - see examples in next section
                       modbusStartAddress,      # ADC and binary channel numbering starts with 0, DAC channel numbering with 1
                       modbusLength,            # length in dataType units - see examples in next section
                       dataType,                # 4 = 16-bit signed integers (for A/D and D/A), 0 = binary (for BIO, duh)
                       pollMsec,                # how frequently to request a value in [ms]
                       plcType);                #

Examples

A generic example sequence would be

drvAsynIPPortConfigure("<TCP_PORT_NAME>","<UNIT_IP_ADDRESS>:502",0,0,1)
modbusInterposeConfig("<TCP_PORT_NAME>",0,5000,0)
drvModbusAsynConfigure("<PORT_NAME>","<TCP_PORT_NAME>",0,<modbusFunction>,<modbusStartAddress>,<modbusLength>,<dataType>,<pollMsec>,"Acromag")

Note that the default values for the last instruction differ between the different XT DAQ modules.

ADC Module XT1221:

drvAsynIPPortConfigure("c1susaux_adc00","192.168.115.20:502",0,0,1)
modbusInterposeConfig("c1susaux_adc00",0,5000,0)
drvModbusAsynConfigure("C1SUSAUX_ADC00","c1susaux_adc00",0,4,0,8,4,32,"Acromag")

DAC (+4x BIO) Module XT1541:

drvAsynIPPortConfigure("c1susaux_dac00","192.168.115.40:502",0,0,1)
modbusInterposeConfig("c1susaux_dac00",0,5000,0)
drvModbusAsynConfigure("C1SUSAUX_DAC00","c1susaux_dac00",0,6,1,8,4,1,"Acromag")
drvModbusAsynConfigure("C1SUSAUX_DAC00_BIO","c1susaux_dac00",0,5,0,4,0,1,"Acromag")   # Separate instruction for integrated BIO channels

BIO Modules XT1111 & XT1121:

drvAsynIPPortConfigure("c1susaux_bio00","192.168.115.60:502",0,0,1)
modbusInterposeConfig("c1susaux_bio00",0,5000,0)
drvModbusAsynConfigure("C1SUSAUX_BIO00","c1susaux_bio00",0,6,0,4,0,1,"Acromag")

After loading the drivers and setting up the modbus communication, the .cmd file instructs to load EPICS database files, just like other EPICS IOC starters. Like the .cmd file, the database files are located in /cvs/cds/caltech/target/HOSTNAME.

dbLoadDatabase("/cvs/cds/caltech/target/HOSTNAME/SUBSYSTEM.db")

Define EPICS channels

Note: Modbus/TCP is simply a protocol for sending commands via TCP that the XT units can interpret, as in read/write the correct register values to/from the channels. There may be multiple ways to define the channels have the same effect. The settings reported here have been found to work as intended.

The EPICS records in the .db files for the different DAQ channels are defined as follows (only fields relevant for proper addressing and conversion are shown):

XT1221 ADC Channels:

record(ai,"C1:<CHANNEL_NAME>")
{
        field(DTYP,"asynInt32")
        field(INP, "@asynMask(<PORT_NAME> <X> -16)MODBUS_DATA")   # <X>: 0-7 physical channel on Acromag unit
        field(LINR,"LINEAR")    # For count to float conversion: The ADC module converts +/-10V inputs to +/-30,000 counts (if legacy support is disabled, +/- 20,000 counts otherwise)
        field(EGUF,"10.923")    # Corresponds to +32767 in the above scale
        field(EGUL,"-10.923")   # Corresponds to -32767 in the above scale
}

XT1541 DAC Channels:

record(ao,"C1:<CHANNEL_NAME>")
{
        field(DTYP,"asynInt32")
        field(OUT, "@asynMask(<PORT_NAME>, <X>, -16)MODBUS_DATA")   # <X>: 0-7 physical channel on Acromag unit
        field(LINR,"LINEAR")    # For count to float conversion: The ADC module converts +/-10V inputs to +/-30,000 counts (if legacy support is disabled, +/- 20,000 counts otherwise)
        field(EGUF,"10.923")    # Corresponds to +32767 in the above scale
        field(EGUL,"-10.923")   # Corresponds to -32767 in the above scale
}

'''XT1541 BIO Channels:'''
record(bo, "<CHANNEL_NAME>")
{
        field(DTYP,"asynInt32")
        field(OUT,"@asynMask(<PORT_NAME>, <X>, -16)MODBUS_DATA")
}

XT1111 & XT1121 BIO Channels:

For binary input: Not yet confirmed

For binary output:

record(bo, "C1:<CHANNEL_NAME>")
{
        field(DTYP,"asynUInt32Digital")
        field(OUT,"@asynMask(<PORT_NAME>, <X>, 0x<Y>)")   # <X>: 0-3    <Y>: 1,2,4,or 8
}