Difference between revisions of "Archiver for EPICS"

From GlueXWiki
Jump to: navigation, search
(RDB Archiver)
 
(6 intermediate revisions by the same user not shown)
Line 84: Line 84:
 
         at org.eclipse.equinox.launcher.Main.run(Main.java:1410)
 
         at org.eclipse.equinox.launcher.Main.run(Main.java:1410)
 
</pre>
 
</pre>
* First I though the <b>ArchiveConfigTool</b> is broken. The error message about denial of SELECT command to "archive" user for table "smpl_mode" does not make sense because I tested exactly  that in the previous command. I tried to play with it, called Dmitri, but I do not think there is  a problem with the database. I tried it on three different machines. And I also got the same result by checking out the sources from the sourceforge repository. I will have to configure archiving engines manually for now.
+
* First I thought that the <b>ArchiveConfigTool</b> is broken. The error message about denial of SELECT command to "archive" user for table "smpl_mode" does not make sense because I tested exactly  that in the previous command. I tried to play with it, called Dmitri, but I do not think there is  a problem with the database. I tried it on three different machines. And I also got the same result by checking out the sources from the sourceforge repository. I will have to configure archiving engines manually for now.
* It turns out that in the source code there is a variable <i>schema</i> which is probably used for Oracle. When I went into the file <i>/org.csstudio.archive.rdb/preferences.ini</i> and removed the actual assignment to empty string "<i>schema=</i>"  then the code starts to work.
+
* It turns out that in the source code there is a variable <i>schema</i> which is probably used for Oracle databases. We are setting up for MySQL databases. When I went into the file <i>/org.csstudio.archive.rdb/preferences.ini</i> and removed the actual assignment to empty string "<i>schema=</i>"  then the code starts to work.
 
+
  
 
== Configuring the MySQL RDB for this Archiver ==  
 
== Configuring the MySQL RDB for this Archiver ==  
Line 95: Line 94:
  
 
* Then to view variables in the archive select <b>CSS->Trends->Data Browser</b>. If the Archive search panel is not visible, then on Data Browser view <b>RightClick->Open Archive Search Panel</b>. This should pop a panel on the left hand side. There one can use regular patterns to search for the PVs store in the archive and display them on the <b>Data Browser</b>.
 
* Then to view variables in the archive select <b>CSS->Trends->Data Browser</b>. If the Archive search panel is not visible, then on Data Browser view <b>RightClick->Open Archive Search Panel</b>. This should pop a panel on the left hand side. There one can use regular patterns to search for the PVs store in the archive and display them on the <b>Data Browser</b>.
 +
 +
== First Results ==
 +
 +
I was able to configure Archiver Engine to archive "bi" and "bo" records. Addind a single "ai" or "ao" record caused
 +
a problem. No samples were written to the database. I send a message to the author of the <b>RDB Channel Archiver</b>
 +
Kay Kasemir and [[Response from Kay Kasemir regarding analog record problem in RDB Channel Archiver | got a response]] that there is an issue with handling <i>NaN</i> in the older version of the archiver (which was I was using) when
 +
using MySQL RDB. This highlighted a potentially recurring problem in the future coming from the fact that SNS will be using RDB Channel archiver in Oracle RDB, while we probably will prefer MySQL. Compiling a new version of the compiler was giving other error messages.
 +
At this point I gave up fighting with it until we discuss a possibility of using the MyA with CSS DataBrowser.
  
 
== Performance Test ==  
 
== Performance Test ==  
  
We will need to setup an archiving system that can handle on the order of 10K samples per second. Therefore, it is very important to verify that the <b>RDB Archiver</b> engines using a  MySQL database server can handle such a load.
+
We will need to setup an archiving system that can handle on the order of 10K samples per second. Therefore, it is very important to verify that the <b>RDB Archiver</b> engines using a  MySQL database server can handle such a load. I have not done this because of the problems I ran into with analog records.
 +
 
 +
 
  
 
= JLab MYA Archiver =
 
= JLab MYA Archiver =
 +
 +
[http://devweb/controls_web/certified/mya/ MYA archiver] is an EPICS archiver developed at JLab specifically for use with MySQL databases. It is fully supported by the accelerator division of Jefferson Lab. They provide various groups with "deployments" and maintain the data produce by them. There is a simple interface web page for adding PVs to the archiver which basically results in an e-mail to the person in charge of maintaining the archiver (Chris Slominski at this point in time). The accelerator software groups provide tools such as command-line and graphical interfaces for accessing the archived data. The archived data is "live" for certain amount of time, about two years, after which it gets archived on tapes. If one needs data which was already saved on tapes and expunged from the disc one simply send an e-mail to the maintainer of the archiver and the variables are accessible through the usual interfaces. These "features" were used during Hall D solenoid tests and during beam tests in Hall B, and they "worked" as advertised. More description of the MyA archiver can be found on the [http://devweb/controls_web/certified/mya/ MYA web page maintained by the JLAB accelerator group].
 +
 +
MAA Archiver does not  have interface withe the CSS DataBrowser which is one of the desired features for Hall D slow controls framework. We are discussing with the accelerator software group options for integrating MYA with CSS. In such a case MyA would be the best option for Hall D EPICS archiver.

Latest revision as of 12:16, 13 December 2011

RDB Archiver

RDB Archiver is the new new version of the EPICS Channel Archiver which was widely used by the EPICS community. Channel Archiver kept the history in data file and had some kind of indexing scheme. The new RDB Archiver is a part of the Controls System Studio (CSS) and, as the name suggests, it utilizes databases to keep the archive engine configurations and the archived values. RDB Archiver can be used with both MySQL and Oracle (and probably others as well). Folks at ANL tested the RDB Archiver archiver with MySQL and I followed their example, and below you can find what I observed.

  • To create CSS products one needs to have Eclipse RCP which I downloaded from Eclipse download page (version 3.7.0 Indigo) and installed in /usr/local/eclipse/indigo-3.7.0-rcp on my local machines.
  • As CSS sources I used the SNS snapshot 3.0.2 posted on the SNS products web page. I unzipped the file and put it in ~/sns/source directory on my local computers.
  • One also need MySQL server which is included in Linux distributions.


Archive Engine Installation

  • Start Eclipse RCP
    /usr/local/eclipse/indigo-3.7.0-rcp/eclipse
  • Select /home/hovanes/css/source as Workspace since the CSS source was unzipped there.
  • Import the sources into the Workspace by File->Import->General->ExistingProjectsIntoWorkspace->Next, select the /home/hovanes/css/source where the source is located using Browse button, and click Finish.
  • Open org.csstudio.archive.engine/plugin_customization.ini file and remove the schema assignment, it is only needed for Oracle: "org.csstudio.archive.rdb/schema=" . If this is not done then an error similar to what is described in ArchiveConfigTool section will occur because of SQL syntax error, although it will look like an access problem from the error message.
  • Open /org.csstudio.archive.engine/ArchiveEngine.product file.
  • Synchronize and Launch an Eclipse application in the Overview tab. The launch should fail because it requires parameters, but I see the help message, so it runs properly.
  • Export the product using Eclipse Product export wizard in the Overview tab:
    Root directory = ArchiveEngine
    Destination Directory = /home/hovanes/css
uncheck Generate metadata repository, and click Finish button.
  • This created Archive Engine executable /home/hovanes/css/ArchiveEngine/ArchiveEngine
  • If there is a problem with libreadline.so.4 then soft-linking libreadline.so.5 to libreadline.so.4 works, at least it did for me.


Archiver Configurator Tool Installation

  • Here I assume that we are in Eclipse RCP session and the sources are imported when building the Archive Engine.
  • Open /org.csstudio.archive.config.rdb/ArchiveConfigTool.product file.
  • Synchronize and Launch an Eclipse application in the Overview tab. The launch should fail because it requires parameters, but I see the help message, so it runs properly.
  • Export the product using Eclipse Product export wizard in the Overview tab:
    Root directory = ArchiveConfigTool
    Destination Directory = /home/hovanes/css
uncheck Generate metadata repository, and click Finish button.
  • This created Archive Engine executable /home/hovanes/css/ArchiveConfigTool/ArchiveConfigTool.
  • I think the ArchiveConfigTool is defective. Also ANL test document said that they could not use "import" feature and they configured the database manually.


Configuring the MySQL RDB for this Archiver

  • Run
    halld-sc:source> mysql -u root < /home/hovanes/css/source/org.csstudio.archive.rdb/dbd/mysql_schema.txt
    This creates the users and database structure for the RDB archiver.
  • Check if the RDB is fine :
halld-sc:source> mysql -u archive -h localhost -p'$archive' -D archive -e 'select * from smpl_mode;'
+--------------+---------+-----------------------------+
| smpl_mode_id | name    | descr                       |
+--------------+---------+-----------------------------+
|            1 | Monitor | Store every received update |
|            2 | Scan    | Periodic scan               |
+--------------+---------+-----------------------------+
halld-sc:source> 
  • Try to export an XML file for the Demo engine already in there:
halld-sc:source> ../ArchiveConfigTool/ArchiveConfigTool -engine Demo -config engine_Demo.xml -export -host localhost 
Exporting config for engine Demo to engine_Demo.xml
2011-11-02 09:51:51.328 SEVERE [Thread 10] org.csstudio.archive.config.rdb.ArchiveConfigApplication (start) - Exception
com.mysql.jdbc.exceptions.MySQLSyntaxErrorException: SELECT command denied to user 'archive'@'localhost' for table 'smpl_mode'
        at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:936)
        at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:2870)
        at com.mysql.jdbc.MysqlIO.sendCommand(MysqlIO.java:1573)
        at com.mysql.jdbc.MysqlIO.sqlQueryDirect(MysqlIO.java:1665)
        at com.mysql.jdbc.Connection.execSQL(Connection.java:3170)
        at com.mysql.jdbc.Connection.execSQL(Connection.java:3099)
        at com.mysql.jdbc.Statement.executeQuery(Statement.java:1169)
        at org.csstudio.archive.config.rdb.RDBArchiveConfig.loadSampleModes(RDBArchiveConfig.java:109)
        at org.csstudio.archive.config.rdb.RDBArchiveConfig.<init>(RDBArchiveConfig.java:80)
        at org.csstudio.archive.config.rdb.XMLExport.export(XMLExport.java:38)
        at org.csstudio.archive.config.rdb.ArchiveConfigApplication.start(ArchiveConfigApplication.java:120)
        at org.eclipse.equinox.internal.app.EclipseAppHandle.run(EclipseAppHandle.java:196)
        at org.eclipse.core.runtime.internal.adaptor.EclipseAppLauncher.runApplication(EclipseAppLauncher.java:110)
        at org.eclipse.core.runtime.internal.adaptor.EclipseAppLauncher.start(EclipseAppLauncher.java:79)
        at org.eclipse.core.runtime.adaptor.EclipseStarter.run(EclipseStarter.java:344)
        at org.eclipse.core.runtime.adaptor.EclipseStarter.run(EclipseStarter.java:179)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:616)
        at org.eclipse.equinox.launcher.Main.invokeFramework(Main.java:622)
        at org.eclipse.equinox.launcher.Main.basicRun(Main.java:577)
        at org.eclipse.equinox.launcher.Main.run(Main.java:1410)
  • First I thought that the ArchiveConfigTool is broken. The error message about denial of SELECT command to "archive" user for table "smpl_mode" does not make sense because I tested exactly that in the previous command. I tried to play with it, called Dmitri, but I do not think there is a problem with the database. I tried it on three different machines. And I also got the same result by checking out the sources from the sourceforge repository. I will have to configure archiving engines manually for now.
  • It turns out that in the source code there is a variable schema which is probably used for Oracle databases. We are setting up for MySQL databases. When I went into the file /org.csstudio.archive.rdb/preferences.ini and removed the actual assignment to empty string "schema=" then the code starts to work.

Configuring the MySQL RDB for this Archiver

  • To view the data in the saved in RDB Archiver we can use CSS Data Browser. In fact, the SNS Base EPICS CSS build from the SNS site already contains Data Browser plugin. To read PVs from a desired archive one needs to configure the Data Browser. Follow the steps below.
  • Select CSS->Preferences->CSS Applications->Trends->DataBrowser. Then add the MySQL URL, for instance jdbc:mysql://localhost/archive, in the Archive Data Server URLs and the Default Archive Data Sources sections. In the CSS->Preferences->CSS Applications->Trends->RDB Archive one needs to add the username and password which can read the MySQL database. Note, that this user needs to have MySQL access as username@hostname. For isntance, I used archive@localhost. The Database Schema and Stored procedure should stay blank (at least for MySQL).
  • Then to view variables in the archive select CSS->Trends->Data Browser. If the Archive search panel is not visible, then on Data Browser view RightClick->Open Archive Search Panel. This should pop a panel on the left hand side. There one can use regular patterns to search for the PVs store in the archive and display them on the Data Browser.

First Results

I was able to configure Archiver Engine to archive "bi" and "bo" records. Addind a single "ai" or "ao" record caused a problem. No samples were written to the database. I send a message to the author of the RDB Channel Archiver Kay Kasemir and got a response that there is an issue with handling NaN in the older version of the archiver (which was I was using) when using MySQL RDB. This highlighted a potentially recurring problem in the future coming from the fact that SNS will be using RDB Channel archiver in Oracle RDB, while we probably will prefer MySQL. Compiling a new version of the compiler was giving other error messages. At this point I gave up fighting with it until we discuss a possibility of using the MyA with CSS DataBrowser.

Performance Test

We will need to setup an archiving system that can handle on the order of 10K samples per second. Therefore, it is very important to verify that the RDB Archiver engines using a MySQL database server can handle such a load. I have not done this because of the problems I ran into with analog records.


JLab MYA Archiver

MYA archiver is an EPICS archiver developed at JLab specifically for use with MySQL databases. It is fully supported by the accelerator division of Jefferson Lab. They provide various groups with "deployments" and maintain the data produce by them. There is a simple interface web page for adding PVs to the archiver which basically results in an e-mail to the person in charge of maintaining the archiver (Chris Slominski at this point in time). The accelerator software groups provide tools such as command-line and graphical interfaces for accessing the archived data. The archived data is "live" for certain amount of time, about two years, after which it gets archived on tapes. If one needs data which was already saved on tapes and expunged from the disc one simply send an e-mail to the maintainer of the archiver and the variables are accessible through the usual interfaces. These "features" were used during Hall D solenoid tests and during beam tests in Hall B, and they "worked" as advertised. More description of the MyA archiver can be found on the MYA web page maintained by the JLAB accelerator group.

MAA Archiver does not have interface withe the CSS DataBrowser which is one of the desired features for Hall D slow controls framework. We are discussing with the accelerator software group options for integrating MYA with CSS. In such a case MyA would be the best option for Hall D EPICS archiver.