Online Monitoring plugins

From GlueXWiki
Jump to: navigation, search


The Online Monitoring Plugins are used to create histograms that monitor the health of the data in an online environment. The goal is to ensure the quality of the data as it is being read in and to identify any issues quickly so that they can be corrected. The system is designed using the JANA framework and the plugins will have access to the entire set of sim-recon reconstruction libraries.

The system is made up of a set of plugins allowing individual detector groups to maintain control over the monitoring of their systems. Each detector will have one or more monitoring plugins to generate histograms and, in some cases, provide macros for complex displays of those histograms. The plugins for all detectors will be attached to a single multi-threaded process to optimize efficiency. A small farm of monitoring computer nodes will be used, each processing a portion of the incoming data stream. The events will be distributed to the nodes using the same ET software system that the DAQ system in Hall-D uses.

The system is designed to allow the plugins to be developed completely in an offline environment and then used for online monitoring without any special coding requirements. This is due to RootSpy's ability to read histogram definitions from global memory and publish them to the network. A central program will grab all histograms from all nodes and automatically sum them. A set of initial plugins exists that detector groups should modify. If a new plugin is needed, one can make it using instructions found here.

Getting Started

To develop a plugin for use in the online, there a few steps you will want to follow:

1.) Check out and build the sim-recon code (find instructions here)

2.) Simulate some bggen events to produce an hdgeant_smeared.hddm file (find instructions here)

3.) Convert the HDDM file into EVIO format (find instructions here)

4.) Checkout the online monitoring software package and SBMS system:

> mkdir ONLINE
> svn co
> svn co

5.) Build the plugin of interest. e.g.:

> cd monitoring/src/plugins/BCAL_online
> scons -u install

6.) Set your environment to find the plugin and run it along with the DAQ and TTab plugins. e.g.:

> cd ONLINE/monitoring/$BMS_OSNAME/plugins
> hd_root -PPLUGINS=DAQ,TTab,BCAL_online rawevent_000002.evio

Histograms will be written to the file hd_root.root

Using ROOT Macros with RootSpy

RootSpy can plot single histograms easily, but to make more sophisticated plots with things like divided canvases or overlays, one needs a macro. The RootSpy system supports this in allowing server programs (i.e. online monitoring plugins) to provide macros using the same communication channel that it uses for providing histograms and trees. What's more, the build system will automatically compile any macro files it finds in the monitoring plugin's source directory into the plugin itself. For example, in the CDC_online plugin source directory, there is a macro file named "CDC_occupancy.C". This will be made into a string and compiled into the plugin automatically by the build system. Any file with a ".C" suffix will automatically be processed in this way.

The one thing that cannot be easily automated is for RootSpy to know what histograms a macro actually needs. To address this, special comments can be added to the macro to indicate which histograms RootSpy should request so that the macro will have them available. Here is an example from the BCAL_online plugin:


// hnamepath: /bcal/bcal_shower_x
// hnamepath: /bcal/bcal_shower_y
// hnamepath: /bcal/bcal_shower_z   
// hnamepath: /bcal/bcal_shower_t		

	TDirectory *dir = (TDirectory*)gDirectory->FindObjectAny("bcal");
	if(dir) dir->cd();

	TH1I *h_x = (TH1I*)gDirectory->FindObjectAny("bcal_shower_x");
	TH1I *h_y = (TH1I*)gDirectory->FindObjectAny("bcal_shower_y");
	TH1I *h_z = (TH1I*)gDirectory->FindObjectAny("bcal_shower_z");
	TH1I *h_t = (TH1I*)gDirectory->FindObjectAny("bcal_shower_t");
	// Just for testing
	if(gPad == NULL){
		TCanvas *c1 = new TCanvas("c1");
	if(!gPad) return;
	TCanvas *c1 = gPad->GetCanvas();
	c1->Divide(2, 2);
	if(h_x) h_x->Draw();

	if(h_y) h_y->Draw();

	if(h_z) h_z->Draw();

	if(h_t) h_t->Draw();

In this example, the histograms of interest are in the "bcal" directory inside of ROOT's internal directory structure. The lines starting with // hnamepath: are the special comments that tell RootSpy which histograms to request. For example /bcal/bcal_shower_x means to request the histogram named bcal_shower_x from the directory named bcal. The directories can be any depth. When the macro is run, it will be in a directory where the full directory structure is reproduced below it. Histograms will be automatically summed from ones obtained from all servers.

Note that because the histograms may not all be obtained by the time a macro first runs, it should be careful to check pointers before using them as is done in the above example.

The section labelled with the comment "// Just for testing" in the above example is there to ensure a canvas exists if you are running the macro offline. In fact, the system is designed to allow you to generate a ROOT file using your plugin and then develop the macro based on the file.

Testing Plugins with RootSpy

There are multiple options for testing a monitoring plugin with RootSpy. These range from simply using RootSpy to open a ROOT file to setting up a complete ET system and passing events through it. Here are some instructions to how to use different methods to test your plugin.

Using a ROOT file with the RootSpy GUI

This is simplest and probably the most useful method. The RootSpy GUI can open a file and register all histograms and trees it finds there. You can even attach your plugin so that any macros you've made are available for testing. To use this, you will first need to generate a ROOT file with your plugin. For example:

> hd_root -PPLUGINS=DAQ,TTab,FCAL_online rawevent_000002.evio

Next, run RootSpy, telling it the file and the plugin(s) to use:

> RootSpy -f hd_root.root -p FCAL_online

The macros and histograms should be available for plotting in the RootSpy GUI. Note that the histograms, trees, and macros are all internally placed in cMsg objects and then decoded by the RootSpy system using the same code that will be used if the cMsg was obtained from the network. A cMsg server is *not* required for using RootSpy with a ROOT file, but by using cMsg's internally, it is more likely to present in the same that it will in an online environment.

Using rs_filepub

The rs_filepub program that comes with RootSpy can be used to read histograms and trees from an existing ROOT file and present them to the RootSpy GUI via the network. It will not transfer macros over the network though. (You can still use the "-p plugins" option described above for that though.) For this system to work though, one needs a cMsg server running.

Starting the cMsg server

First note that if you are running on the gluon cluster in the Hall-D counting house then a cMsg server should already be running on the gluondb1 node. Just set your ROOTSPY_UDL environment variable to cMsg://gluondb1/cMsg/rootspy

If you are trying to develop code on your own system, you will need to install cMsg. After you have done this, set your CMSGROOT environment variable and then you can run the java server this way:

java -cp $CMSGROOT/jar/cMsg-*.jar org/jlab/coda/cMsg/cMsgDomain/server/cMsgNameServer

The location of the server is specified in the ROOTSPY_UDL environment variable. Thus, you can run it on one node and have the RootSpy servers and clients run on different nodes. Here is how to set it if running the RootSpy processes on the same node as the cMsg server.

> setenv ROOTSPY_UDL cMsg://

Publishing with rs_filepub

Use like this:

> rootspy_filepub file.root

then fire up RootSpy and hit the "Select" button to see the histograms.

Using the rootspy plugin to view "live" histograms as they are being filled

You may use the rootspy plugin with hd_root or hd_ana (or any other JANA program) to publish histograms as they are being filled by attaching the rootspy plugin to the process. This also requires a cMsg server be running so make sure that is running (see above for instructions). Then, start up the program like this:

> hd_ana -PPLUGINS=DAQ,TTab,rootspy,FCAL_online -PEVIO:LOOP_FOREVER=1 rawevent_000002.evio

Next, just start up the RootSpy GUI (with no arguments) and you should be able to see the histograms and macros being published by the process. A couple of things to note about the hd_ana command above:

  1. The rootspy plugin *must* come before any plugins containing macros or the macros won't be published.
  2. The -PEVIO:LOOP_FOREVER option tells the program to continually re-open the input file and read events from it. This is mainly for when you input file has only a few events and the process finishes before you have had an adequate chance to look at its output using RootSpy
  3. You can also use HDDM files here, but there is no "LOOP_FOREVER" option so once the events run out, the server program will quit and RootSpy won't be able to show anything. You will also not have access to the lower level objects such as DBCALDigiHit if using HDDM.

Using an ET system

This is probably overkill for most people and not needed to setup a complete development system for online monitoring plugins. However, for the sake of completeness ...

You need to install ET and have your ETROOT environment variable pointing to it. The DAQ plugin mush be compiled with ETROOT set, otherwise it is built automatically without ET support.

1.) Start up ET server

> $ETROOT/bin/et_start -f /tmp/et_sys_test -s 21000 -n 10

2.) In another window feed events from an EVIO file into the ET system using the file2et program that is part of the Hall-D "etuils" package. You probably already have the packages directory checked out if you've gotten to this point so just go to the etutils directory and build the program with scons

> cd /path/to/online/packages/etutils
> scons -u install ginstall

Then run the program like this:

> file2et -loop -f /tmp/et_sys_test rawevent_000002.evio

The "-loop" option tells the program to repeatedly re-open the file and read events from it so that events flow forever allowing one to more easily focus on testing the plugins.

To read events from the ET system into a DANA program:

> hd_ana -PPLUGINS=DAQ,TTab,BCAL_online ET:/tmp/et_sys_test:MON