Difference between revisions of "Calibration Train"
(→Run Groups) |
(→Run Groups) |
||
Line 8: | Line 8: | ||
! Total # Jobs | ! Total # Jobs | ||
! Notes | ! Notes | ||
+ | ! Total Skim Size (TB) | ||
+ | ! BCAL Skim (TB) | ||
+ | ! FCAL Skim (TB) | ||
+ | ! PS Skim (TB) | ||
|- | |- | ||
| [https://halldweb.jlab.org/calib_challenge/GlueX-CalibRun-2016-02-25 GlueX-CalibRun-2016-02-25] || 10457 - 10529 || || first pass, abandoned | | [https://halldweb.jlab.org/calib_challenge/GlueX-CalibRun-2016-02-25 GlueX-CalibRun-2016-02-25] || 10457 - 10529 || || first pass, abandoned | ||
Line 15: | Line 19: | ||
| [https://halldweb.jlab.org/calib_challenge/GlueX-CalibRun-2016-02-25 GlueX-CalibRun-2016-03-04] || 10649 - 10724 || || first pass, abandoned | | [https://halldweb.jlab.org/calib_challenge/GlueX-CalibRun-2016-02-25 GlueX-CalibRun-2016-03-04] || 10649 - 10724 || || first pass, abandoned | ||
|- | |- | ||
− | | [https://halldweb.jlab.org/calib_challenge/GlueX-CalibRun-2016-03-18/ GlueX-CalibRun-2016-03-18] || 10331 - 10913 || 6570 || second pass | + | | [https://halldweb.jlab.org/calib_challenge/GlueX-CalibRun-2016-03-18/ GlueX-CalibRun-2016-03-18] || 10331 - 10913 || 6570 || second pass || 23 || || || 14 |
|- | |- | ||
− | | [https://halldweb.jlab.org/calib_challenge/GlueX-CalibRun-2016-04-04/ GlueX-CalibRun-2016-04-04] || 11048 - 11145 || 2993 || first pass | + | | [https://halldweb.jlab.org/calib_challenge/GlueX-CalibRun-2016-04-04/ GlueX-CalibRun-2016-04-04] || 11048 - 11145 || 2993 || first pass || 9.4 || || || 3.5 |
|- | |- | ||
− | | [https://halldweb.jlab.org/calib_challenge/GlueX-CalibRun-2016-04-11/ GlueX-CalibRun-2016-04-11] || 11150 - 11312 || 3573 || first pass | + | | [https://halldweb.jlab.org/calib_challenge/GlueX-CalibRun-2016-04-11/ GlueX-CalibRun-2016-04-11] || 11150 - 11312 || 3573 || first pass || 9.7 || || || 2.1 |
|- | |- | ||
− | | [https://halldweb.jlab.org/calib_challenge/GlueX-CalibRun-2016-04-27/ GlueX-CalibRun-2016-04-27] || 11366 - 11668 || 10599 || first pass | + | | [https://halldweb.jlab.org/calib_challenge/GlueX-CalibRun-2016-04-27/ GlueX-CalibRun-2016-04-27] || 11366 - 11668 || 10599 || first pass || 25 || 16 || 2.8 || 5.5 |
+ | |- | ||
+ | | '''TOTAL''' || || || || 48 || || || 24 | ||
|} | |} | ||
Revision as of 15:27, 24 May 2016
Contents
Run Groups
SWIF workflow | Run Range | Total # Jobs | Notes | Total Skim Size (TB) | BCAL Skim (TB) | FCAL Skim (TB) | PS Skim (TB) |
---|---|---|---|---|---|---|---|
GlueX-CalibRun-2016-02-25 | 10457 - 10529 | first pass, abandoned | |||||
GlueX-CalibRun-2016-02-29 | 10531 - 10647 | first pass, abandoned | |||||
GlueX-CalibRun-2016-03-04 | 10649 - 10724 | first pass, abandoned | |||||
GlueX-CalibRun-2016-03-18 | 10331 - 10913 | 6570 | second pass | 23 | 14 | ||
GlueX-CalibRun-2016-04-04 | 11048 - 11145 | 2993 | first pass | 9.4 | 3.5 | ||
GlueX-CalibRun-2016-04-11 | 11150 - 11312 | 3573 | first pass | 9.7 | 2.1 | ||
GlueX-CalibRun-2016-04-27 | 11366 - 11668 | 10599 | first pass | 25 | 16 | 2.8 | 5.5 |
TOTAL | 48 | 24 |
Job Structure
Currently 3 "passes" through the data are performed. The first pass tries to do as many calibrations as possible with one file of data. The second pass tries to do calibrations that need a larger data set. The final pass runs through a full run to use the full statistics in a run and to generate outputs for other calibration procedures that can't be done automatically yet (e.g. pi0 calibrations).
The plugins that are run on each pass are:
- Pass 1
- Step 1 - RF_online
- Step 2 - HLDetectorTiming (coarse timing+ADC/TDC alignment),TOF_TDC_shift
- Step 3 - HLDetectorTiming (track-based timing),BCAL_TDC_Timing (timewalks)
- Pass 2
- Step 1 - TAGH_timewalk,BCAL_attenlength_gainratio,BCAL_TDC_Timing (full)
- Step 2 - st_tw_corr_auto
- Pass 3
- HLDetectorTiming,PSC_TW,BCAL_attenlength_gainratio,BCAL_gainmatrix,FCALgains,FCALpedestals,ST_Tresolution,ST_Propagation_Time,p2gamma_hists,imaging,pedestal_online,BCAL_LEDonline,PS_timing,TOF_calib,pi0fcalskim,pi0fcalskim,ps_skim
Output
The output of the calibration jobs can be found at this webpage or at /volatile/halld/home/gxproj3/calib_jobs . There is a subdirectory for each calibration launch. Each launch has a directory for each run that is processed. For each run, several types of outputs are kept
- The results for the processing of each file are kept in a subdirectory with the same number as the file, in the format NNN
- The summed results for each pass through the data are kept in ROOT files with names of the form" hd_calib_passN_RunRRRRRR.root"
- The processed results for each pass (e.g. constants files and figures) are kept in subdirectories with names of the form "passN/"
The relevant directories are:
- ROOT files & calibration constants: /volatile/halld/home/gxproj3/calib_jobs/[workflow]/output
- EVIO skims: /volatile/halld/home/gxproj3/calib_jobs/[workflow]/skims
Instructions
The software from the calibration train can be obtained from
git clone https://github.com/sdobbs/calibration_train
Job submission is controlled by the following configuration file:
# data.config - example configuration file jobname = GlueX-CalibRun-2016-04-27 # name of the SWIF workflow # data/memory sizes in GB mem_requested = 9 # max vmem for jobs disk_space = 30 # max disk space for jobs nthreads = 8 # number of threads requested # time in hours time_limit = 36 # max run time for jobs # this file is used for debugging ccdb_table_file = configs/calib_tables
Building a SWIF workflow is controlled by the job_manager.py command
ifarm1102> ./job_manager.py usage: job_manager.py init [config_file] job_manager.py build [-z] [config_file] [run file] job_manager.py run [-L] [config_file]
Some descriptions of the sub-commands:
- init - This takes the configuration file as an input, does the basic setup for the job: creates directories, saves configurations, builds CCDB SQLite file
- build - This takes the configuration file and a file with one run number per line as an input, and creates the SWIF workflow
- run - This just starts the SWIF workflow. Standard SWIF commands can be used to deal with the workflow from here.
Currently, there are two major commands to run
Calibration Challenge
Old Planning
Organization
- The jobs will be submitted every Tuesday at noon, JLab time.
- The jobs will be run from the gxproj3 account [parallel use with EventStore jobs]
- The output of the jobs will be stored in ...
Run Ranges
The following runs will be processed:
RunPeriod-2015-03
- 2931, 3079, 3179, 3180, 3183, 3185
Calibrations
Job Requirements
Each calibration process should include the following:
- Plugin stored in standard location
- ROOT/other scripts stored in standard location
- https://halldsvn.jlab.org/repos/trunk/sim-recon/src/scripts/calibrations ?
- Individual git repos?
- Output to files
- QA routines
What is Being Run
The following plugins are currently being run:
- RF_online (RF signal)
- BCAL_TDC_Timing
- HLDetectorTiming
- PSC_TW
Working on adding:
- BCAL_attenlength_gainratio
- BCAL gains - /work/halld/home/wmcginle/Gain_Calib
- TOF calibrations - https://halldsvn.jlab.org/repos/trunk/home/zihlmann/TOF_calib/
Code
The current code can be obtained at
svn co https://halldsvn.jlab.org/repos/trunk/home/sdobbs/calibration_train/
The ROOT library directory can be found at: ...