NOTE: The data for the plots below had to be sewn together from multiple queries since the NERSC slurm is configured to not allow queires spanning more than 1 month. This workflow exceeds that. An issue is clearly seen in the second plot where the blue curve shows a region where 4200 jobs are queued which is double what the workflow is allowed. This is due to some double counting. I don't have time to track down the exact nature of the issue now so instead just put in this disclaimer.
See overall network traffic to/from JLab at https://my.es.net/sites/view/JLAB
(Click image for full resolution)
BAD_RCDB_QUERY_RUNS=[]
BAD_FILE_COUNT_RUNS=[]
BAD_EVNT_COUNT_RUNS={71364L: [24, 25]}
BAD_MSS_FILE_RUNS={}
=================================================
Launch Summary
-----------------------------------------------
RunPeriod: 2019-11
Launch type: recon
Version: 01
batch: 01a
WORKFLOW: recon_2019-11_ver01_batch01a
Origin of run list: RCDB 71350-71365 (query="@is_dirc_production and @status_approved")
Number of runs: 12
Number of files: 1166 (maximum 1001 files/run)
Min. file no.: 0
Max. file no.: 1000
Max. events per job: 175000
events per block: 40
whole file fallback: True
NERSC project: m3120
QOS: low (default)
QOS_SPLIT: flex (only for jobs that can be split)
Time limit per job: 10:50:00 (whole file jobs only)
Time limit split job: 01:35:00 (split jobs only)
Time limit merge jobs: 00:25:00 (used only for split jobs)
Shifter/Docker image: docker:markito3/gluex_docker_devel
halld_recon version: halld_recon/halld_recon-4.19.0 (from CVMFS)
Output directory: mss:/mss/halld/RunPeriod-2019-11/recon/ver01
=================================================
WORKFLOW: recon_2019-11_ver01_batch01a
------------------------------------
Number of runs: 12 (only good runs)
1166/1166 total files submitted (0 files missing from mss)
12687/12687 total jobs submitted
6/6 whole file jobs submitted
11521/11521 split jobs submitted (non-aggregator)
1160/1160 aggregator jobs submitted
252 directories created for output
BAD_RCDB_QUERY_RUNS=[]
BAD_FILE_COUNT_RUNS=[]
BAD_EVNT_COUNT_RUNS={71405L: [0, 1]}
BAD_MSS_FILE_RUNS={}
=================================================
Launch Summary
-----------------------------------------------
RunPeriod: 2019-11
Launch type: recon
Version: 01
batch: 01a
WORKFLOW: recon_2019-11_ver01_batch01a
Origin of run list: RCDB 71366-71405 (query="@is_dirc_production and @status_approved")
Number of runs: 17
Number of files: 1176 (maximum 1001 files/run)
Min. file no.: 0
Max. file no.: 1000
Max. events per job: 175000
events per block: 40
whole file fallback: True
NERSC project: m3120
QOS: low (default)
QOS_SPLIT: flex (only for jobs that can be split)
Time limit per job: 10:50:00 (whole file jobs only)
Time limit split job: 01:35:00 (split jobs only)
Time limit merge jobs: 00:25:00 (used only for split jobs)
Shifter/Docker image: docker:markito3/gluex_docker_devel
halld_recon version: halld_recon/halld_recon-4.19.0 (from CVMFS)
Output directory: mss:/mss/halld/RunPeriod-2019-11/recon/ver01
=================================================
WORKFLOW: recon_2019-11_ver01_batch01a
------------------------------------
Number of runs: 17 (only good runs)
1176/1176 total files submitted (0 files missing from mss)
12774/12774 total jobs submitted
4/4 whole file jobs submitted
11598/11598 split jobs submitted (non-aggregator)
1172/1172 aggregator jobs submitted
357 directories created for output