Hydra Monitoring

From GlueXWiki
Jump to: navigation, search

Hydra is an A.I. based system deployed in hall-D for near-real time data quality monitoring. It is, at regular intervals, fed images by Rootspy which are analyzed by a set of neural networks to provide basic classifications of the images to aid in monitoring.

Currently the system looks at 7 distinct occupancy plots: the BCAL, CDC, TOF, DIRC (south bar box), SC, FDC, FCAL. To add a new plot please contact tbritton@jlab.org.

Results are available in two different forms, both accessible from off site.

Hydra Runtime for real time inferences

and

Hydra Monitoring Log for trailing 24 hour view of potential problems


How to use Hydra Runtime:

1) go to the above link (you can leave it up and running as it auto updates)

2) glance over at the occupancy plots from time to time (look for plots that are outlined in red)

3) if a box is red look at the plot and compare it to the reference plot. If there is a problem look to the typical diagnostic programs (Rootspy, strip charts, etc) for the appropriate detectors and take appropriate action as necessary. The confidence given can be taken in to account (e.g Bad @ 0.6 likely deserves a little time if the image itself looks fine)

Note: The AI assume standard production, so special tests and runs can cause detector systems to appear bad to the A.I.


The first can be found at the Hydra Runtime page. Initially, and when monitoring is not running or data is not being taken, the page loads as a header block containing "Waiting for data..." or the run number and the last updated time. Below that are a set of empty rectangle blocks which will host the plots. This state should last at most a minute during active data taking. The Last updated time will increment as it waits for data, so rest assured the page is not broken. As data is received you will see the following Hydra RUN.JPG. Each "card" contains the image analyzed, the datetime it was analyzed, the category, and it's "confidence" (normalized to be between 0 and 1). To aid in visibility any plot judged to be "Bad" will have its border changed to red. This page auto-updates and should show every plot seen by Hydra. This, in principle, could replace the "reset" button in Rootspy.


The second page can be found at the Hydra Monitoring Log page. This page shows only the plots deemed bad or needing a second opinion in the trailing 24 hours from the time the page is loaded. When there are images it will look like: HydraLog.JPG

This page's header contains many check boxes as well as a select all and deselect all button. Each check box essentially filters the plots further. Individual run numbers can be shown or hidden and even individual plots. For convenience, the "select all" button will check every plot box and the "Deselect all" will uncheck all plot boxes. Because this page only shows plots labeled as "Bad" and plots which fall below the configurable confidence level the "Filter" section only contains the ability to show the bad plots only or those plots that fall below the confidence threshold. Unchecking both boxes will lead to no images being displayed. In fact, the unchecking of all boxes in a single row will result in no plots being displayed as all plots belong to a single run number, are of a given type, and are either "Bad" or unconfirmed.