See overall network traffic to/from JLab at https://my.es.net/sites/view/JLAB
(Click image for full resolution)
NOTE: The values in BAD_RCDB_QUERY_RUNS is informative about what is missing from
the RCDB. An attempt to recover the information from the /mss filesystem
was also made. Values in BAD_FILE_COUNT_RUNS are ones for which that failed.
Thus, only runs listed in BAD_FILE_COUNT_RUNS will not have any jobs submitted
BAD_RCDB_QUERY_RUNS=[61322, 61323, 61324, 61325, 61326, 61327, 61328, 61329, 61330, 61331, 61332, 61333, 61334, 61335, 61336, 61337, 61338, 61339, 61340, 61341, 61342, 61343, 61344, 61345, 61346, 61347, 61348, 61349, 61350, 61351, 61352, 61353, 61354, 61355, 61356, 61357, 61358, 61359, 61360, 61361, 61362, 61363, 61364, 61365, 61366, 61367, 61368, 61369, 61370, 61371, 61372, 61373, 61374, 61375, 61376, 61377, 61378, 61379, 61380, 61381, 61382, 61383, 61384, 61385, 61386, 61387, 61388, 61389, 61390, 61391, 61392, 61393, 61394, 61395, 61396, 61397, 61398, 61399, 61400, 61401, 61402, 61403, 61404, 61405, 61406, 61407, 61408, 61409, 61410, 61411, 61412, 61413, 61414, 61415, 61416, 61417, 61418, 61419, 61420, 61421, 61422, 61423, 61424, 61425, 61426, 61427, 61428, 61429, 61430, 61431, 61432, 61433, 61434, 61435, 61436, 61437, 61438, 61439, 61440, 61441, 61442, 61443, 61444, 61445, 61446, 61447, 61448, 61449, 61450, 61451, 61452, 61453, 61454, 61455, 61456, 61457, 61458, 61459, 61460, 61461, 61462, 61463, 61464, 61465, 61466, 61467, 61468, 61469, 61470, 61471, 61472, 61473, 61474, 61475, 61476, 61477, 61478, 61479, 61480, 61481, 61482, 61483, 61484, 61485, 61486, 61487, 61488, 61489, 61490, 61491, 61492, 61493, 61494, 61495, 61496, 61497, 61498, 61499, 61500, 61501, 61502, 61503, 61504, 61505, 61506, 61507, 61508, 61509, 61510, 61511, 61512, 61513, 61514, 61515, 61516, 61517, 61518, 61519, 61520, 61521, 61522, 61523, 61524, 61525, 61526, 61527, 61528, 61529, 61530, 61531, 61532, 61533, 61534, 61535, 61536, 61537, 61538, 61539, 61540, 61541, 61542, 61543, 61544, 61545, 61546, 61547, 61548, 61549, 61550, 61551, 61552, 61553, 61554, 61555, 61556, 61557, 61558, 61559, 61560, 61561, 61562, 61563, 61564, 61565, 61566, 61567, 61568, 61569, 61570, 61571, 61572, 61573, 61574, 61575, 61576, 61577, 61578, 61579, 61580, 61581, 61582, 61583, 61584, 61585, 61586, 61587, 61588, 61589, 61590, 61591, 61592, 61593, 61594, 61595, 61596, 61597, 61598, 61599, 61600, 61601, 61602, 61603, 61604, 61605, 61606, 61607, 61608, 61609, 61610, 61611, 61612, 61613, 61614, 61615, 61616, 61617, 61618, 61619, 61620, 61621, 61622, 61623, 61624, 61625, 61626, 61627, 61628, 61629, 61630, 61631, 61632, 61633, 61634, 61635, 61636, 61637, 61638, 61639, 61640, 61641, 61642, 61643, 61644, 61645, 61646, 61647, 61648, 61649, 61650, 61651, 61652, 61653, 61654, 61655, 61656, 61657, 61658, 61659, 61660, 61661, 61662, 61663, 61664, 61665, 61666, 61667, 61668, 61669, 61670, 61671, 61672, 61673, 61674, 61675, 61676, 61677, 61678, 61679, 61680, 61681, 61682, 61683, 61684, 61685, 61686, 61687, 61688, 61689, 61690, 61691, 61692, 61693, 61694, 61695, 61696, 61697, 61698, 61699, 61700, 61701, 61702, 61703, 61704, 61705, 61706, 61707, 61708, 61709, 61710, 61711, 61712, 61713, 61714, 61715, 61716, 61717, 61718, 61719, 61720, 61721, 61722, 61723, 61724, 61725, 61726, 61727, 61728, 61729, 61730, 61731, 61732, 61733, 61734, 61735, 61736, 61737, 61738, 61739, 61740, 61741, 61742, 61743, 61744, 61745, 61746, 61747, 61748, 61749, 61750, 61751, 61752, 61753, 61754, 61755, 61756, 61757, 61758, 61759, 61760, 61761, 61762, 61763, 61764, 61765, 61766, 61767, 61768, 61769, 61770, 61771, 61772, 61773, 61774, 61775, 61776, 61777, 61778, 61779, 61780, 61781, 61782, 61783, 61784, 61785, 61786, 61787, 61788, 61789, 61790, 61791, 61792, 61793, 61794, 61795, 61796, 61797, 61798, 61799, 61800, 61801, 61802, 61803, 61804, 61805, 61806, 61807, 61808, 61809, 61810, 61811, 61812, 61813, 61814, 61815, 61816, 61817, 61818, 61819, 61820, 61821, 61822, 61823, 61824, 61825, 61826, 61827, 61828, 61829, 61830, 61831, 61832, 61833, 61834, 61835, 61836, 61837, 61838, 61839, 61840, 61841, 61842, 61843, 61844, 61845, 61846, 61847, 61848, 61849, 61850, 61851, 61852, 61853, 61854, 61855, 61856, 61857, 61858, 61859, 61860, 61861, 61862, 61863, 61864, 61865, 61866, 61867, 61868, 61869, 61870, 61871, 61872, 61873, 61874, 61875, 61876, 61877, 61878, 61879, 61880, 61881, 61882, 61883, 61884, 61885, 61886, 61887, 61888, 61889, 61890, 61891, 61892, 61893, 61894, 61895, 61896, 61897, 61898, 61899, 61900, 61901, 61902, 61903, 61904, 61905, 61906, 61907, 61908, 61909, 61910, 61911, 61912, 61913, 61914, 61915, 61916, 61917, 61918, 61919, 61920, 61921, 61922, 61923, 61924, 61925, 61926, 61927, 61928, 61929, 61930, 61931, 61932, 61933, 61934, 61935, 61936, 61937, 61938, 61939, 61940, 61941, 61942, 61943, 61944, 61945, 61946, 61947, 61948, 61949, 61950, 61951]
BAD_FILE_COUNT_RUNS=[61400, 61401, 61403, 61404, 61406, 61407, 61414, 61415, 61432, 61457, 61604, 61618, 61869]
BAD_MSS_FILE_RUNS={61408: [0, 1], 61409: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19], 61410: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13], 61411: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39], 61412: [0, 2], 61413: [4, 6, 8, 10, 12, 14, 16, 18], 61416: [0, 2, 4, 6, 8, 10, 12, 14, 16, 18], 61417: [68, 70, 72, 74, 76, 78], 61421: [12, 16, 18], 61422: [21, 23], 61423: [20, 22], 61426: [33, 35, 37], 61395: [0, 1], 61396: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55], 61397: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47], 61398: [0, 1, 2, 3, 4, 5, 6], 61399: [0, 1, 2, 3], 61428: [9, 11, 13], 61402: [0, 1], 61405: [0, 1]}
=================================================
Launch Summary
-----------------------------------------------
RunPeriod: 2019-01
Launch type: recon
Version: 01-perl
batch: NERSC-multi
WORKFLOW: recon_2019-01_ver01-perl_batchNERSC-multi
Origin of run list: All runs in range 61322-61951
Number of runs: 630
Number of files: 12433 (maximum 1000 files/run)
Min. file no.: 0
Max. file no.: 999
QOS: regular
Time limit per job: 3:00:00
JANA config file: /launch/jana_recon_nersc.config
NERSC project: m3120
Shifter image: docker:jeffersonlab/gluex_prod:v1
halld_recon version: halld_recon/halld_recon-4.38.0 (from CVMFS)
launch directory: /global/cfs/cdirs/m3120/launch (at NERSC)
Output directory: /lustre/expphy/volatile/halld/offsite_prod/RunPeriod-2019-01/recon/ver01-perl
=================================================
WORKFLOW: recon_2019-01_ver01-perl_batchNERSC-multi
------------------------------------
Number of runs: 630 (only good runs)
12197/12433 total files submitted (236 files missing from mss)
606/606 total jobs submitted
12197 directories created for output