Alarm System for EPICS

From GlueXWiki
Revision as of 17:48, 7 March 2015 by Hovanes (Talk | contribs) (Importing an XML into BEAST)

(diff) ← Older revision | Latest revision (diff) | Newer revision → (diff)
Jump to: navigation, search


One of the most important components of the Hall D online monitoring will be the alarm system. Such a system should be configurable to annunciate in the control room and send e-mail messages in case EPICS variables severity changes. It is also very desirable to have an alarm system which can be integrated with the Controls System Studio (CSS).


Alarm Handler (ALH) is a standalone program that is part of the EPICS extensions that was developed for monitoring and reacting to the alarm condition of the EPICS records. ALH can sound an audible alarm, displays suggested actions for various situations, and can execute commands for different alarm states and severities. It has been extensively used at Jefferson Lab since mid-nineties, and in principle can be used in the 12 GeV era.


EPICS community developed an alarm system called Best Ever Alarm System Toolkit (BEAST) providing integration with CSS. This system consists of multiple components, and each component is built and configured from Eclipse RCP environment. Some of the component should run standalone, some of them are supposed to be built-in into the CSS executable, and some can used in both ways (Annunciator).

Compiling executables

The compilation of the executable is done in Eclipse RCP. I downloaded SNS source from the SNS web site, unzipped. Then I import the plugins from the source directory as existing project. Since we will be using MySQL databases I edited preferences.ini file in org.csstudio.alarm.beast to have "rdb_schema=" .

Alarm Server

  • Since the setting in the plugin_customization.ini overwrite the setting in preferences.ini I changed plugin_customization.ini in /org.csstudio.alarm.beast.server to have
  • Also change the default root component to Test

Then I used standard procedure with AlarmServer.product to create an executable.

  • I created a file css/3.1.0/AlarmServer/alarm_server.ini to add the the IP addresses to the list of the CA servers the Alarm Server will be trying to access:
. I could have done this when building the executable for the BEAST Server in the /org.csstudio.alarm.beast.server/plugin_customization.ini file. To run the server one needs to issue a command
./AlarmServer  -pluginCustomization /home/hovanes/css/products/sns_epics_w_alarm_3.1.5/AlarmServer/alarm_server.ini
. The manual suggest to always use the full path to the customization file.

Alarm Configuration Tool

  • In plugin_customization.ini for /org.csstudio.alarm.beast.configtool I changed to
the default was some Oracle type setup with different username amd password for SNS.

Alarm Annunciator

  • I compiled this with a two changes in the plugin_customization.ini file:

CSS Product with BEAST

  • In order for the Alarm menu to show up as a submenu of the CSS menu one needs to add org.csstudio.alarm.beast.ui.feature in the CSS .product's Dependencies tab and then recompile the product. The locations of the Alarm submenus are defined in org.csstudio.alarm.beast.ui.feature plugin.
  • It turns out that if one wants to be able to modify the alarm configuration of the BEAST through CSS then the CSS executable will need to be able to authenticate and authorise the user. Before doing this I was trying to compile the CSS based on features. But when I looked at the list of the dependencies with the features I did not find how to add a features to include the authentication and authorisation into the build. So I decided to compile the EPICS CSS based plugins. To do this one needs to
    clicks the radio-button plug-ins in the CSS.product file in the Overview tab.
    Add the following plugins in the Dependencies tab in CSS.product :
    Validate the dependencies. When I did it I got a message that
    org.csstudio.opibuilder.rap<pre> plugin had <pre>Missing Constraint
    and that org.csstudio.rap.core was missing. I just removed this plugin from the dependency list and I was able to compile the executable successfully.
  • Go back to the Overview tab, synchronise and create the executable using product wizard.
  • After launching the newly built CSS go to Edit->Preferences->CSSCore->JAAS Authentication and change the Login Configuration Source to "File" and the Configuration File Entry to "Dummy". This will allow to have dummy authentication allowing everyone to login and be authorised to change the alarm configuration.

Creating RDB

BEAST alarm system utilizes RDB for configuration and maintaining the alarm statuses even when the Alarm Server is down. We will be using MySQL database, so we need to create MySQL users and database for the alarm system.

  • Use org.csstudio.alarm.beast/dbd/MYSQL_USER.sql to create usernames and passwords using
 mysql -h halld-sc -u root < MYSQL_USER.sql


  • Manually create a MySQL database alarm on the MySQL server since the SQL-files provided in the CSS source code did not seem to do it.
mysql> CREATE DATABASE alarm ;
  • Use org.csstudio.alarm.beast/dbd/ALARM_MYSQL.sql to create the tables for the alarm system.
mysql -u username2 -p'password' < ALARM_MYSQL.sql

I had to change all instances of "ALARM." to "alarm." in the ALARM_MYSQL.sql prior to issuing the command above to keep the database name consistent. The username2 user with that simple password was defined in the org.csstudio.alarm.beast/dbd/MYSQL_USER.sql file, and obviously, can be named differently in the production configuration of the Hall D alarm system.

Manipuating RDB with AlarmConfigTool

VEAST has a utility called AlarmConfigTool that allows to manipulate the content of the underlying MySQL DB. In particular, we found it very useful for importing and export the alarm configuration. One can take a snapshot of the current alarm configuration in the form of an XML file, modify it manually or with a script, and then import it back into BEAST.

Exporting Configuration into an XML file

Our policy is to keep track of the alarm configuration via checking the XML file into the SVN repository at Nevertheless, some of us once in a while will modify the content of alarm configuration via CSS GUI. But on a regular basis we need to update the repository to have a good history of the alarm system. Each time we want to change the configuration via an XML file, we need to

  1. export the current configuration file into an XML file,
  2. commit the file into the SVN repository,
  3. modify the XML file
  4. import the new configuration from the CML file into BEAST
  5. commit the new XML file into the SVN repository

In order to export the current alarm configuration simply run the AlarmConfigTool on a gluon-cluster to create an XML file.

 $CSS/linux-x86_64/AlarmConfigTool_3.1.7/AlarmConfigTool -export -file HallD_alarm_configuration.xml -root HallD 

Importing an XML into BEAST

Once the XML file is modified, it can be imported into BEAST using the same AlarmConfigTool

$CSS/linux-x86_64/AlarmConfigTool_3.1.7/AlarmConfigTool -modify -file HallD_alarm_configuration.xml -root HallD


$CSS/linux-x86_64/AlarmConfigTool_3.1.7/AlarmConfigTool -import -file HallD_alarm_configuration.xml -root HallD

commands. One usually need to restart the CSS after the DB has been changed. It may also be necessary to restart the AlarmServer and alarm-notifier processes after a new XML configuration has been imported.

Java Message Server

Install Apache ActiveMQ

  • Download ActiveMQ from, both the tar file and the key file.
  • Copied both files to /usr/local/activemq.
  • Verifying the checksum with PGP did not work on "halld-sc" computer. Ignored it for now.
  • Untared the tar-file in the /usr/local/activemq directory.
  • Go to /usr/local/activemq/apache-activemq-5.5.1 directory and start by issuing
sudo bin/activemq start
  • When checking the port with netstat I get
halld-sc:apache-activemq-5.5.1> netstat -an | fgrep 61616 
tcp        0      0 :::61616                    :::*                        LISTEN      
indicating that ActiveMQ is running.
  • To configure CSS as a proper JMS client, I need to specify the the URL for the JMS server I setup with ActiveMQ
    In /org.csstudio.basic.epics.product/plugin_customization.ini change the URL ti have the following lines:
  • I can check if the client connected to the JMS server by checking http://localhost:8161/admin on the machine that runs the JMS servr (in this case it was Note that this web service was not accessible from some other machines. If asked for credentials, the default is (admin,admin).
  • It appears that the ActiveMQ on freezes once in a while. Then Alarm Panel on CSS indicated Server Timeout. Restarting the ActiveMQ server solves the problem, but the alarm GUIs need to be restarted which might be problematic. This will need to be investigated.

Communicating with ActiveMQ JMS broker from Python scripts

  • If you want to use STOMP for Python interface, then the configuration file conf/activemq.xml needs to be edited and the following line added to it <transportConnector name="stomp" uri="stomp://localhost:61613"/>. For instance, the transportConnectors tag will look something like this:
            <!-- DOS protection, limit concurrent connections to 1000 and frame size to 100MB -->
            <transportConnector name="openwire" uri="tcp://"/>
            <transportConnector name="amqp" uri="amqp://"/>
	    <transportConnector name="stomp" uri="stomp://localhost:61613"/>
  • In order to use an ActiveMQ JMS broker from a Python script, one also needs to install stomppy Linux RPM:
    /apps/bin/sudo yum install stomppy
  • The following example should allow one to send a request to the alarm server notifying it about an update to the alarm configuration. The JMS message should be sent to "*_CLIENT" topic on the ActiveMQ server. Once the alarm server gets the message, the clients displaying the the alarm tree will also get a message to update the views.
>>> import sys,time,stomp
>>> conn = stomp.Connection([('localhost',61613)])
>>> conn.start()
>>> conn.connect()
>>> msg={'TYPE':'alarm','TEXT':'CONFIG','CONFIG':'Test','APPLICATION_ID':'CSS','HOST':'gluon20','USER':'hovanes'}
>>> conn.send(msg,destination='/topic/Test_CLIENT', headers={'transformation' :
   'jms-map-xml'}, ack='auto')

Note that the client script and the ActiveMQ server should be running on the same host for this example, although they do not have to.

  • The following example show how to change the status of an alarm to MAJOR by sending a message to the clients from a Python-based server to change the alarm status of some test PV called testPV.
>>> import sys,time,stomp
>>> conn = stomp.Connection([('localhost',61613)])
>>> conn.start()
>>> conn.connect()
>>> alarm_msg={'TYPE':'alarm','TEXT':'STATE','CONFIG':'Test','NAME':'testPV','CURRENT_SEVERITY':'MINOR','CURRENT_STATUS':'LOW','SEVERITY':'MINOR','STATUS':'LOW','EVENTTIME':'1383833490478', 'APPLICATION_ID':'CSS','HOST':'gluon20','USER':'hovanes'}
>>> conn.send(alarm_msg,destination='/topic/Test_SERVER', headers={'transformation' : 'jms-map-xml'}, ack='auto')
  • There seem to be a lot of ways to use Python for JMS messaging, some of which are abandoned. It would be better not to use Python but write all JMS messaging application in Java.