Mini Data Challenge Status, September 10, 2012

Started submitting jobs Friday afternoon, August 24.

Jobs

1. bggen
   • no input file
   • create 400 k events per job with bggen
   • run mcsmear on hdgeant output
   • write results to tape library
   • about 7 hours of CPU time
   • output file is about 14 GB (35 kB per event)
2. hd_root
   • get bggen data file from library
   • run reconstruction using hd_root
   • write resulting root file to tape library
   • several dying due to exceeding self-imposed 4 GB memory limit
   • several percent dying due to exception being thrown

Status

As of September 10.

bggen

mysql> select count(*), sum(submitted), sum(output), sum(jput_submitted), sum(silo) from bggen;
+----------+----------------+-------------+---------------------+-----------+
| count(*) | sum(submitted) | sum(output) | sum(jput_submitted) | sum(silo) |
+----------+----------------+-------------+---------------------+-----------+
|     1000 |           1000 |         895 |                1000 |      1000 |
+----------+----------------+-------------+---------------------+-----------+

hd_root

mysql> select count(*), sum(submitted), sum(output), sum(jput_submitted), sum(silo) from hd_root;
+----------+----------------+-------------+---------------------+-----------+
| count(*) | sum(submitted) | sum(output) | sum(jput_submitted) | sum(silo) |
+----------+----------------+-------------+---------------------+-----------+
|     1000 |           1000 |         992 |                 992 |       998 |
+----------+----------------+-------------+---------------------+-----------+

Farm Usage Snapshots

Farm Usage Chart

Volatile Disk Usage

To Do

1. put staging disk and library location into config file
2. finesse memory problem
3. do large scale job with REST output
4. analyze root files
5. do REST output validation
6. save farm output and error files
7. check for job success
   • error file check
   • size of output data
8. allow two output files
9. add configuration lines for all steps
10. add "output once existed" column