Difference between revisions of "Offline Data Processing"

From GlueXWiki
Jump to: navigation, search
(→‎Run Classification)
(→‎Monitoring)
 
(2 intermediate revisions by the same user not shown)
Line 1: Line 1:
−
== Triggering ==
+
== To Include ==
  
−
* [[Level-1_Trigger | Level-1 Trigger]]
+
- Eventstore, grid, university sites, calibration validation, reconstruction validation, skim validation
−
* [[Level-3_Trigger | Level-3 Trigger]]
+
  
−
== Low-Level Monitoring ==
+
- Code versioning, config scripts, software version restoration, database access (web, batch scripts, JANA)
  
−
This is performed by running the monitoring plugins for the different detector systems on data as it arrives. The results are viewable in the counting house with RootSpy.
+
- Documentation, backwards compatibility
−
https://halldweb1.jlab.org/wiki/index.php/Online_Monitoring_plugins
+
  
−
== Calibrations ==
+
== Monitoring ==
  
−
Calibration plugins and scripts are automatically run on the data as it arrives. These scripts are responsible for fitting peaks, checking constants into the calibration database, and iterating as necessary.
+
* Monitoring jobs will be run on new data as it is written to tape. These jobs will do several things:
  
−
Talk to David: Make sure calibrations are integrated into scripts that are automatically run
+
* If calibration / reconstruction-improvement work is ongoing, monitoring jobs will also be rerun on tape data at regular intervals. This may be weekly or bi-weekly, and may run over all of the data for the given run period, or only a subset of it, as the situation demands.
  
−
== Reconstruction Monitoring ==
+
== Calibrations ==
−
 
+
−
This is performed by running the monitoring_hists plugin on data after it is calibrated. The results are viewable in the counting house with RootSpy.
+
  
 
== Run Information ==
 
== Run Information ==
  
−
* The run-control database (RCDB) stores information about each run. The RCDB is located at:
+
== Reconstruction ==
 +
 
 +
REST-format reconstruction data will be saved to tape at JLab. If the disk footprint is small enough, all of the REST reconstruction results will be stored on the /halld/volatile/ disk. If not, then a 2-track skim of the data will be available.
  
−
WHERE AM I?
+
== Run Classification (Good / Junk) ==
  
−
* During run start, information about the run setup are recorded in the RCDB. This includes run type, start time, trigger type, etc. This is performed by the script:
+
== Physical Skimming ==
  
−
https://halldsvn.jlab.org/repos/trunk/online/daq/scripts/run_go
+
* Physical, EVIO-format skims will be made available to the calibration on the /halld/volatile/ disk as needed to assist with calibration studies. No other physical skims will be created.  
  
−
* Since the DAQ crashes sometimes, it cannot be relied on to report run summary information (e.g. run end-time, total # events, scalars, etc.).  Instead, cron jobs on the online system periodically runs a script that crawls through the files and sets this information.  The cron job scripts:
+
== Logical Skimming ==
  
−
https://halldsvn.jlab.org/repos/trunk/online/etc/crontabs/crontab.hdsys.gluonraid1
+
* Logical skims will be stored in EventStore. These will be created by running the
−
https://halldsvn.jlab.org/repos/trunk/online/etc/crontabs/crontab.hdsys.gluonraid2
+
  
−
Launch the RCDB update script at:
+
== Analysis Trains ==
  
−
https://halldsvn.jlab.org/repos/trunk/online/packages/monitoring/src/scripts/update_runinfodb.sh
+
=== Daily, Yesterday's-Runs ===
  
−
== Run Classification ==
+
=== Weekly, Subset-of-all-past-runs ===
  
−
* During run pre-start, shift takers must indicate what the run type is for the run that is about to start. This run type is stored in the RCDB so that it can be used to help analyze the data. Possible run types include:
+
=== Monthly, All-Past-Data ===
  
−
Production Run
+
== Simulation ==
−
DAQ Test
+
−
Cosmic Run
+
−
Pulser Run
+
−
Zero-Field Run
+
−
Empty Target Run
+
−
Misc. Test
+
  
−
* Further classification is performed by analyzing the rates in the various detector systems. If the rates in any system are lower than some threshold, then
+
=== Channel-by-channel Phase-space ===
  
−
* Status Bits:
+
=== bggen ===
  
−
Non-Production Data
+
== Acceptance studies ==
−
Low detector rates
+
−
Acceptance bad
+
−
Flux bad
+
−
Polarization bad
+

Latest revision as of 17:55, 20 January 2015

To Include

- Eventstore, grid, university sites, calibration validation, reconstruction validation, skim validation

- Code versioning, config scripts, software version restoration, database access (web, batch scripts, JANA)

- Documentation, backwards compatibility

Monitoring

  • Monitoring jobs will be run on new data as it is written to tape. These jobs will do several things:
  • If calibration / reconstruction-improvement work is ongoing, monitoring jobs will also be rerun on tape data at regular intervals. This may be weekly or bi-weekly, and may run over all of the data for the given run period, or only a subset of it, as the situation demands.

Calibrations

Run Information

Reconstruction

REST-format reconstruction data will be saved to tape at JLab. If the disk footprint is small enough, all of the REST reconstruction results will be stored on the /halld/volatile/ disk. If not, then a 2-track skim of the data will be available.

Run Classification (Good / Junk)

Physical Skimming

  • Physical, EVIO-format skims will be made available to the calibration on the /halld/volatile/ disk as needed to assist with calibration studies. No other physical skims will be created.

Logical Skimming

  • Logical skims will be stored in EventStore. These will be created by running the

Analysis Trains

Daily, Yesterday's-Runs

Weekly, Subset-of-all-past-runs

Monthly, All-Past-Data

Simulation

Channel-by-channel Phase-space

bggen

Acceptance studies

Retrieved from "https://halldweb.jlab.org/wiki/index.php?title=Offline_Data_Processing&oldid=62823"