https://halldweb.jlab.org/wiki/api.php?action=feedcontributions&user=Kmoriya&feedformat=atomGlueXWiki - User contributions [en]2024-03-19T09:24:35ZUser contributionsMediaWiki 1.24.1https://halldweb.jlab.org/wiki/index.php?title=Data_Monitoring_Procedures&diff=71919Data Monitoring Procedures2015-12-15T06:37:16Z<p>Kmoriya: /* Offline Monitoring: Running Over Archived Data */</p>
<hr />
<div>__TOC__<br />
<br />
== Master List of File / Database / Webpage Locations ==<br />
=== Run Conditions ===<br />
* Online Run-by-run condition files (B-field, current, etc.): /work/halld/online_monitoring/conditions/<br />
* Offline monitoring run conditions (software versions, jana config): /group/halld/data_monitoring/run_conditions/<br />
*[http://www.jlab.org/Hall-D/test/RunInfo/ Run Info vers. 1]<br />
*[https://halldweb.jlab.org/cgi-bin/data_monitoring/run_conditions.pl Run Info vers. 2]<br />
*[https://halldweb.jlab.org/rcdb RCDB]<br />
<br />
=== Monitoring Output Files ===<br />
* Run Periods 201Y-MM is for example 2015-03, launch ver verVV is for example ver15<br />
* Online monitoring histograms: /work/halld/online_monitoring/root/<br />
* Offline monitoring histogram ROOT files (merged): /work/halld/data_monitoring/RunPeriod-201Y-MM/verVV/rootfiles<br />
* individual files for each job (ROOT, REST, log, etc.): /volatile/halld/offline_monitoring/RunPeriod-201Y-MM/verVV/<br />
<br />
=== Monitoring Database ===<br />
* Accessing monitoring database (on ifarm): mysql -u datmon -h hallddb.jlab.org data_monitoring<br />
<br />
=== Monitoring Webpages ===<br />
*[https://halldweb.jlab.org/cgi-bin/data_monitoring/monitoring/plotBrowser.py Plot Browser]<br />
*[https://halldweb.jlab.org/cgi-bin/data_monitoring/monitoring/runBrowser.py Run Browser]<br />
*[https://halldweb.jlab.org/cgi-bin/data_monitoring/monitoring/versionBrowser.py Version Browser]<br />
*[https://halldweb.jlab.org/cgi-bin/data_monitoring/monitoring/timeSeries.py Time Series]<br />
*[https://halldweb.jlab.org/data_monitoring/launch_analysis/ Launch Analysis]<br />
<br />
== SciComp Job Links ==<br />
=== Main ===<br />
* [https://scicomp.jlab.org/scicomp/ Scientific Computing Home Page]<br />
* [https://scicomp.jlab.org/scicomp/#/auger/jobs Auger Job Status Page]<br />
* [https://scicomp.jlab.org/scicomp/#/jasmine/jobs Jasmine Tape Job Status Page]<br />
<br />
=== Documentation ===<br />
* [https://scicomp.jlab.org/docs/batch Batch System]<br />
* [https://scicomp.jlab.org/docs/storage Mass Storage System]<br />
* [https://scicomp.jlab.org/docs/swif SWIF]<br />
* [https://scicomp.jlab.org/docs/swif-cli SWIF Command Line]<br />
<br />
=== Job Tracking ===<br />
* [http://scicomp.jlab.org/farm2/job.html Completed Job History]<br />
* [http://scicomp.jlab.org/farm2/project.html Job Stats By Project]<br />
* [http://scicomp.jlab.org/farm2/trackOrg.html Job Stats By Track]<br />
* [http://scicomp.jlab.org/farm2/report.html Cluster Report]<br />
* [http://scicomp.jlab.org/farm2/walltime.html Walltime Distribution]<br />
<br />
== Saving Online Monitoring Data ==<br />
<br />
The procedure for writing the data out is given in, e.g.,<br />
[https://halldweb.jlab.org/wiki/index.php/Raid-to-Silo_Transfer_Strategy Raid-to-Silo Transfer Strategy].<br />
<br />
Once the DAQ writes out the data to the raid disk, cron jobs will copy the file to tape,<br />
and within ~20 min., we will have access to the file on tape at<br />
/mss/halld/$RUN_PERIOD/rawdata/RunXXXXXX.<br />
<br />
All online monitoring plugins will be run as data is taken.<br />
They will be accessible within the counting house via RootSpy, and<br />
for each run and file, a ROOT file containing the histograms will be saved<br />
within a subdirectory for each run.<br />
<br />
For immediate access to these files, the raid disk files may be accessed directly<br />
from the counting house, or the tape files will be available within ~20 min. of the<br />
file being written out.<br />
<br />
== Offline Monitoring: Running Over Archived Data ==<br />
<br />
Once files are written to tape we run the online plugins on these files to confirm what we were seeing in the online monitoring, and also to update the results from the latest calibration and software. <br />
Manual scripts and cron jobs are set up to look for new data and run the plugins over a sample of files.<br />
<br />
Every other Friday (usually the Friday before the offline meetings) jobs will be started to run the newest software on all previous runs,and allows everybody to see improvements in each detector.<br />
For each launch, independent builds of hdds, sim-recon, the monitoring plugins, and an sqlite file will be generated.<br />
<br />
Below the procedures are described for<br />
# Preparing the software for the launch<br />
# Starting the launch (using hdswif)<br />
# Post-analysis of statistics of the launch<br />
<br />
Processing the results and making them available to the collaboration<br />
is handled in the section [[#Post-Processing_Procedures | Post-Processing Procedures]] below.<br />
<br />
<!--<br />
==== Using cron to run automatically ====<br />
Within /home/gluex/halld/monitoring/cron/ there is a file cron_plugins<br />
that can be executed via<br />
crontab cron_plugins<br />
This will set up a cron job to call the script scan_for_jobs.sh, which will<br />
check in the rawdata directory and call generatejobs_plugins_rawdata.sh for<br />
any run that is more than 5 min old. The cron job is set up to run every 10 min.<br />
--><br />
<br />
=== General Information on Procedures ===<br />
Since we may want to simultaneously run offline monitoring for different run periods that require<br />
different environment variables, the scripts are set up so that a generic user can download the<br />
scripts and run them from anywhere. Most output directories for offline monitoring are created<br />
with group read/write permissions so that any Hall D group user has access to the contents,<br />
but there are some cases where use of the account that created the launch is necessary.<br />
<br />
The accounts used for offline monitoring are the gxprojN accounts created and maintained by<br />
Mark Ito (see [https://halldweb.jlab.org/wiki/index.php/GlueX-related_shared_accounts_on_the_JLab_CUE here] for how each account is used).<br />
As of October 2015, the following are used:<br />
* gxproj1 for running over incoming experimental data (as it hits the tape)<br />
* gxproj5 for running over previous experimental data (biweekly launches)<br />
<br />
For offline monitoring, the hdswif system that Kei developed is used for launching the jobs, and a new cross analysis<br />
system based on MySQL and Python is maintained. <br />
<br />
The scripts for the monitoring are maintained in svn:<br />
https://halldsvn.jlab.org/repos/trunk/scripts/monitoring/<br />
<br />
=== Preparing the software for the launch ===<br />
<br />
1. Setup the environment: <pre>source ~/env_monitoring_launch</pre><br />
<br />
2. Updating & building hdds: <br />
<pre><br />
cd $HDDS_HOME<br />
git pull # Get latest software<br />
scons -c install # Clean out the old install: EXTREMELY IMPORTANT for cleaning out stale headers<br />
scons install -j4 # Rebuild and re-install with 4 threads<br />
</pre><br />
<br />
3. Updating & building sim-recon: <br />
<pre><br />
cd $HALLD_HOME/src<br />
git pull # Get latest software<br />
scons -c install # Clean out the old install: EXTREMELY IMPORTANT for cleaning out stale headers<br />
scons install -j4 # Rebuild and re-install with 4 threads<br />
</pre><br />
<br />
4. Create a new sqlite file containing the very latest calibration constants. Original documentation on creating sqlite files are [https://halldweb.jlab.org/wiki/index.php/SQLite-form_of_the_CCDB_database here]. <br />
<pre><br />
cd $GLUEX_MYTOP/../sqlite/<br />
$CCDB_HOME/scripts/mysql2sqlite/mysql2sqlite.sh -hhallddb.jlab.org -uccdb_user ccdb | sqlite3 ccdb.sqlite<br />
mv ccdb.sqlite ccdb_monitoring_launch.sqlite #replacing the old file<br />
</pre><br />
<br />
5. Note that the above steps must be done <b>BEFORE</b> launch project creation. This is because we will track the revisions of the libraries used, and this is done by extracting the svn information in each directory. Also, note that the system assumes that we have the topmost build directory (usually called GLUEX_TOP) to be $HOME/builds . Such an assumption is necessary to be able to extract information about the library locations automatically.<br />
<br />
Create the appropriate project(s) and submit the jobs using hdswif, as detailed in the section below.<br />
<br />
=== Starting the Launch and Submitting Jobs ===<br />
<br />
1. Download the hdswif directory from svn. For the gxprojN accounts, use the directory ~/monitoring/hdswif. <br />
<pre><br />
svn co https://halldsvn.jlab.org/repos/trunk/scripts/monitoring/hdswif<br />
cd hdswif<br />
</pre><br />
<br />
2. Edit the job config file, input.config, which is used to register jobs in hdswif. A typical config file will look this:<br />
PROJECT gluex<br />
TRACK reconstruction<br />
OS centos65<br />
NCORES 6<br />
DISK 40<br />
RAM 8<br />
TIMELIMIT 8<br />
JOBNAMEBASE offmon_<br />
RUNPERIOD 2015-03<br />
VERSION 15<br />
OUTPUT_TOPDIR /volatile/halld/offline_monitoring/RunPeriod-[RUNPERIOD]/ver[VERSION] # Example of other variables included in variable<br />
SCRIPTFILE /home/gxproj5/monitoring/hdswif/script.sh # Must specify full path<br />
ENVFILE /home/gxproj5/env_monitoring_launch # Must specify full path<br />
<br />
3. Creating the workflow: Within SWIF jobs are registered into workflows. For offline monitoring, the workflow names are of the form <b>offmon_20YY_MM_verVV</b> with suitable replacements for the run period year YY, month BB, and the version number VV (with leading zeroes). The command "swif list" will list all existing workflows. Also, for most simple SWIF commands hdswif also provides a wrapper. <br />
swif list<br />
<br />
For creation of workflows for offline monitoring the command:<br />
hdswif.py create [workflow] -c input.config<br />
should be used. When a config file (here: input.config) is passed in, hdswif will automatically create files that record the configuration of the current launch. These files are stored as for example:<br />
/group/halld/data_monitoring/run_conditions/RunPeriod-2015-03/jana_rawdata_comm_2015_03_ver15.conf<br />
/group/halld/data_monitoring/run_conditions/RunPeriod-2015-03/soft_comm_2015_03_ver15.xml<br />
<br />
The software packages stored in git (sim-recon and hdds) can have git tags applied to them, which makes it easier to find versions of the software than a SHA-1 hash. hdswif will ask if you would like to create a tag, and execute the following sequence: <pre>git tag -a offmon-201Y_MM-verVV -m "Used for offline monitoring 201Y-MM verVV started on 201y/mm/dd"</pre> <pre>git push offmon-201Y_MM-verVV</pre> This will only be invoked when the user name is gxprojN, and for the configuration files, the output directory will be /group/halld/data_monitoring/run_conditions/ for gxprojN accounts while it will be the current directory for other users.<br />
* To use the git-tagged software versions do for example <pre>cd $HALLD_HOME</pre> <pre>git checkout offmon-2015_03-ver15</pre><br />
<br />
4. Registering jobs in the workflow: To register jobs within the workflow, hdswif provides the use of config files. Jobs can be registered by specifying the workflow, config file (-c), run (-r) and file (-f) numbers if necessary. <br />
<span style="color:red">Note: Job configuration parameters can be set differently for jobs within the same workflow if necessary.</span><br />
Jobs can be added via <pre>hdswif.py add [workflow] -c input.config</pre><br />
By default, hdswif will add all files found within the directory /mss/halld/RunPeriod-201Y-MM/rawdata/ where 201Y-MM is specified by the RUNPERIOD parameter in the config file. If only some of the runs or files are needed, these can be specified for example with<br />
<pre>hdswif.py add [workflow] -c input.config -r 3180 -f '00[0-4]'</pre><br />
to specify to register running only over run 3180 files 000 - 004 (Unix-style brackets and wildcards can be used).<br />
<br />
5. Running the workflow: To run the workflow, simply use the hdswif wrapper:<br />
hdswif.py run [workflow]<br />
<br />
<b>It is recommended that some jobs be tested to make sure that everything is working rather than fail thousands of jobs.</b><br><b>Check the configuration setup when creating a workflow, and what is in script.sh. Also check the following:<br />
* Check stderr files. Are they small (<kB)?<br />
* Check stdout files. Are they very large (>MB)?<br />
* Check output ROOT files. Are they larger than several MB?<br />
* Check output REST files. Are they larger than several tens of MB?<br />
</b><br />
<br> For this purpose, hdswif will take an additional parameter to run which limits the number of jobs to submit: <pre>hdswif.py run [workflow] 10</pre><br />
in which case only 10 jobs will be submitted.<br />
<br />
To submit all jobs after checking the results, do <pre>hdswif.py run [workflow]</pre><br />
<br />
=== Checking the Status and Resubmitting ===<br />
1. The status of jobs can be checked on the terminal with <pre>jobstat -u gxprojN</pre> For the status of jobs on Auger see http://scicomp.jlab.org/scicomp/#/auger/jobs and for SWIF use <pre>swif list</pre> or for more information, <pre>swif status [workflow] -summary</pre> Note that "swif status" tends to be out of date sometimes, so don't panic if your workflow/jobs aren't showing up right away. Also see the Auger [http://scicomp.jlab.org/scicomp/#/auger/jobs job website].<br />
<br />
2. For failed jobs, SWIF can resubmit jobs based on the problem. For resubmission for failed jobs with the same resources, <pre>swif retry-jobs [workflow] -problems [problem name]</pre> can be used, and for jobs to be submitted with more resources, e.g., use <pre>swif modify-jobs [workflow] -ram add 2gb -problems AUGER-OVER_RLIMIT</pre> <br />
This only re-stages the jobs, be sure to resubmit them with:<br />
<pre>swif run -workflow [workflow] -errorlimit none</pre><br />
<br />
hdswif has a wrapper for both of these: <pre>hdswif.py resubmit [workflow] [problem]</pre> In this case [problem] can be one of <b>SYSTEM, TIMEOUT, RLIMIT</b>. If SYSTEM is specified, the jobs will be retried. For TIMEOUT and RLIMIT, the jobs will be modified by default with 2 additional hours or GB of RAM. If one more number is added as an option, then that many hours or GB of RAM will be added., e.g., <pre>hdswif.py resubmit [workflow] TIMEOUT 5</pre> will add 5 hours of processing time. You can wait until almost all jobs finish before resubmitting failed jobs since the number should be relatively small. <b>Even if jobs are resubmitted for one type of failure, jobs that later fail with that failure will not be automatically resubmitted.</b><br />
<br />
3. For information on swif, use the "swif help" commands and for hdswif see the attached documentaion in https://halldsvn.jlab.org/repos/trunk/scripts/monitoring/hdswif/manual_hdswif.pdf<br />
<br />
4. Below is a table describing the various errors that can occur.<br />
{| border="1" cellpadding="0" valign="left" style="text-align: left;"<br />
!width="150"| ERROR NAME<br />
!width="400"| Description<br />
!width="400"| Resolution<br />
!width="400"| hdswif command<br />
|-<br />
| AUGER-SUBMIT<br />
||<br />
SWIF’s attempt to submit jobs to Auger failed. Includes server-side problems as well as user failing to provide valid job parameters (e.g. incorrect project name, too many resources, etc.)<br />
||<br />
If requested resources are known to be correct resubmit. Otherwise modify job resources using swif directly.<br />
||<br />
<span style="color:purple"><b>hdswif.py resubmit [workflow] SYSTEM</b></span><br />
|-<br />
| AUGER-FAILED<br />
||<br />
Auger reports the job FAILED with no specific details.<br />
||<br />
Resubmit jobs. If problem persists, contact Chris Larrieu or SciComp.<br />
||<br />
<span style="color:purple"><b>hdswif.py resubmit [workflow] SYSTEM</b></span><br />
|-<br />
| AUGER-OUTPUT-FAIL<br />
||<br />
Failure to copy one or more output files.Can be due to permission problem, quota problem, system error, etc.<br />
||<br />
Check if output files will exist after job execution and that output directory exists, resubmit jobs. If problem persists, contact Chris Larrieu or SciComp.<br />
||<br />
<span style="color:purple"><b>hdswif.py resubmit [workflow] SYSTEM</b></span><br />
|-<br />
| AUGER-INPUT-FAIL<br />
||<br />
Auger failed to copy one or more of the requested input files, similar to output failures. Can also happen if tape file is unavailable (e.g. missing/damaged tape)<br />
||<br />
Check if input file exists, resubmit jobs. If problem persists, contact Chris Larrieu or SciComp.<br />
||<br />
<span style="color:purple"><b>hdswif.py resubmit [workflow] SYSTEM</b></span><br />
|-<br />
| AUGER-TIMEOUT<br />
||<br />
Job timed out.<br />
||<br />
If more time is needed for job add more resources.<br />
Default is to add 2 hrs of processing time. Also check whether code is hanging.<br />
||<br />
<span style="color:red"><b>hdswif.py resubmit [workflow] TIMEOUT</b></span><br><br />
Default is to add 2 hours. Optionally specify number of hours at end.<br />
|-<br />
| AUGER-OVER_RLIMIT<br />
||<br />
Not enough resources, RAM or disk space.<br />
||<br />
Add more resources for job.<br />
||<br />
<span style="color:blue"><b>hdswif.py resubmit [workflow] RLIMIT</b></span><br><br />
Default is to add 2 GB of RAM. Optionally specify GB at end. To add more disk space use SWIF directly.<br />
|-<br />
| SWIF-MISSING-OUTPUT<br />
||<br />
Output file specified by user was not found.<br />
||<br />
Check if output file exists at end of job.<br />
||<br />
<br />
|-<br />
| SWIF-USER-NON-ZERO <br />
||<br />
User script exited with non-zero status code.<br />
||<br />
Your script exited with non-zero status. Check the code you are running.<br />
||<br />
<br />
|-<br />
| SWIF-SYSTEM-ERROR <br />
||<br />
Job failed owing to a problem with swif (e.g. network connection timeout)<br />
||<br />
Resubmit jobs. If problem persists, contact Chris Larrieu or SciComp.<br />
||<br />
<span style="color:purple"><b>hdswif.py resubmit [workflow] SYSTEM</b></span><br />
|}<br />
<br style="clear:both;"/><br />
<br />
=== Post-analysis of statistics of the launch ===<br />
<br />
After jobs have been submitted, it will usually take a few days for all of the jobs to be processed.<br />
The next step is to check the resource usage for the current launch and publish the results online.<br />
<br />
# <b>Create summary XML, HTML files</b><br> The status and results of jobs are saved within the SWIF internal server, and are available via the command <pre>swif status [workflow] -summary -runs</pre> where the arguments -summary and -runs show summary statistics and statistics for individual jobs, respectively. hdswif has a command that takes this output in XML output and creates an HTML webpage showing results of the launch. To do this, do <pre>hdswif.py summary [workflow]</pre> This will create the XML output file from SWIF called swif_output_[workflow].xml and create a webpage containing png figure files. If the XML file already exists, hdswif will ask whether to overwrite the existing file.<br />
# <b>Publish output files online</b><br> At this stage the html output and figure files are created and ready to be put online. The html output capabilities of hdswif is useful for any user using SWIF, but since publication of the html output is specific to the offline monitoring, the script to do this is contained in the cross_analysis scripts directory at https://halldsvn.jlab.org/repos/trunk/scripts/monitoring/cross_analysis. For the gxprojN accounts this directory should exist as ~/halld/monitoring/cross_analysis. To publish the results online do for example <pre>python ~/halld/monitoring/cross_analysis/publish_offmon_results.py 2015_03 18</pre> The script copies the html output and corresponding figures to /group/halld/www/halldweb/html/data_monitoring/launch_analysis/ and also creates a link to it in the html page.<br />
# <b>Editing the summary HTML page</b><br> The top page for is offline monitoring https://halldweb.jlab.org/data_monitoring/launch_analysis/index.html and has links to the summary page for each run period. The summary files are <pre>/group/halld/www/halldweb/html/data_monitoring/launch_analysis/[run period]/[run period].html </pre> Edit the file to:<br />
## Add a new line to the first table which contains the version number, date, and comments for the current launch<br />
## Create a link to the webpage for the current launch. Simply copy and paste, modify the previous launch's link to have the correct launch ver.<br />
# <b>Freezing SWIF tables</b><br> Since we are now finished with the SWIF workflow that we used, the workflow should be "frozen" so that it cannot be mistakenly altered or modified. Do <pre>swif freeze [workflow]</pre><br />
# <b>Backing up SWIF output</b><br> With the workflow frozen we should be able to reproduce the XML output from SWIF, but we will backup the XML output just in case. Do <pre>cp ~/halld/hdswif/xml/swif_output_[workflow].xml /group/halld/data_monitoring/swif_xml_backup/ </pre><br />
<br />
=== Cross Analysis of Launches ===<br />
<br />
The purpose of the cross analysis is to correlate how resource usage changed for the same files across different launches.<br />
To do this it is useful to create MySQL tables that contain information on each launch, and do queries across tables.<br />
<br />
#The scripts to do this are maintained in the svn directory https://halldsvn.jlab.org/repos/trunk/scripts/monitoring/cross_analysis Do <pre> svn co https://halldsvn.jlab.org/repos/trunk/scripts/monitoring/cross_analysis </pre> For the gxprojN accounts used for offline monitoring the directory should be ~/halld/monitoring/cross_analysis<br />
<br />
# The main script is run_cross_analysis.sh, which can be run with <pre>./run_cross_analysis.sh [RUNPERIOD] [VERSION]</pre>, where e.g. [RUNPERIOD] = 2015_03 and [VERSION] = 22. However, <b>it is strongly recommended that the commands in this script be run by hand</b> to catch any errors.<br />
<br />
# Enter the python commands that are in run_cross_analysis.sh . Below are the steps and explanations:<br />
## Create a table for the current launch using <pre>./create_cross_analysis_table.sh [RUNPERIOD] [VERSION]</pre>. The table will be created from the file template_table_schema.sql and contain columns id, run, file, timeChange, cpu_sec, wall_sec, mem_kb, vmem_kb, nevents, input_copy_sec, plugin_sec, final_state, problems<br />
## Run <pre>python fill_cross_analysis_entries.py [RUNPERIOD] [VERSION]</pre> The script will gather all of the necessary information either from SWIF output or the stdout files for the jobs<br />
## Run <pre>python create_stats_table_row.py [RUNPERIOD] [VERSION]</pre> This will loop over the jobs in the current launch and create a row in an HTML table that summarizes the statistics for the final state and problems for the jobs. This table row is then inserted into the main HTML webpage for the run period.<br />
## Run <pre>python create_stats_for_each_file.py [RUNPERIOD] [MINVERSION] [VERSION]</pre> This creates a comparison table of the final state and problems for each file between launches [MINVERSION] and [VERSION]. In the run_cross_analysis.sh script, the default is to set MINVERSION to be 15 for run period 2015_03, but as long as SWIF was used for all previous launches, any number will work.<br />
## Run <pre>python create_resource_correlation_plots.py [RUNPERIOD] [CMPMINVERSION] [VERSION]</pre> This creates correlation plots of resource use between launches between CMPMINVERSION and VERSION. By default CMPMINVERSION is 5 launches earlier.<br />
<br />
=== Starting a new run period ===<br />
When a new run period is started, it is best to make sure all top-level directories are created with the right permissions. This can save headaches later on when a different gxprojN account is used for offline monitoring.<br />
# Create top-level directories<pre> /volatile/halld/offline_monitoring/RunPeriod-20YY-MM/</pre> <pre>/group/halld/data_monitoring/run_conditions/RunPeriod-20YY-MM/</pre><br />
#Make sure other gxprojN users can write in with chmod g+w [dir name]. Check that permissions match those from previous run periods.<br />
<br />
== Explanation of Scripts ==<br />
Below are explanations of each script used in the offline monitoring system and a breif explanation of how they work.<br />
<br />
=== hdswif scripts ===<br />
<b>Summary:</b> hdswif.py is the main script that calls the other utility scripts. For the utility scripts, they can be run standalone by giving the appropriate parameters. Visual graphics are made using the PyROOT extension of ROOT. To use this, the environment variable PYTHONPATH must include ROOTSYS. If ROOTSYS has been set, adding it to PYTHONPATH will be done by the script, but if ROOTSYS is not set, then the scripts will abort.<br />
<br />
{| border="1" cellpadding="0" valign="left" style="text-align: left;"<br />
!width="150"| file name<br />
!width="800"| Description<br />
|-<br />
| hdswif.py ||<br />
Main script to control the behavior of SWIF. Most commands follow the form hdswif.py [command] [workflow] (options)<br />
|-<br />
| parse_swif.py ||<br />
Called within hdswif.py to create html output from SWIF results.<br />
|-<br />
| createXMLfiles.py ||<br />
* Called by the "create" option of hdswif.py<br />
* Creates XML files for logging information about launch. Must specify config file with option -c.<br />
* Also adds tags to git repositories of sim-recon and hdds.<br />
* For XML file creation, the file will be written out to /group/halld/data_monitoring/run_conditions/ if the user is gxprojN (used for offline monitoring). If not, to avoid general users adding things to the above directory, the output files will be the current directory.<br />
* To write out the versions of each package, environment variables such as HDDS_HOME will need to be set.<br />
* Versions for each software package are extracted using the directory structure, so if these are changed the scripts must change accordingly.<br />
* For each launch, <b>the output soft_comm_[RUNPERIOD]_ver[VER].xml file should be checked that all version numbers have been extracted.</b><br />
|-<br />
| read_config.py ||<br />
* Called by the "add" option of hdswif.py<br />
* Takes in config file name, optionally set verbose<br />
* Return a dictionary between config parameter names and values (e.g., 'PROJECT' : 'gluex', 'NCORES' : 6, ...)<br />
* Prints the config parameters at the end. For parameters changed from default, a '*' will be printed<br />
|-<br />
| output_job_details.py ||<br />
* Called by the "details" option of hdswif.py<br />
* Takes in workflow name, run and file. Run and file must be numbers, no wildcards<br />
* Finds ids for jobs specified by the run and file number and returns info on each one<br />
* Job info is retrieved from pbs farm system and shows configuration parameters for that job<br />
|-<br />
| results_by_resources.py ||<br />
* Called within parse_swif.py<br />
* Takes in XML output from SWIF and creates 2 plots<br />
* Creates html table showing results of jobs by resources requested for that job<br />
* This table is shown under "Status by Resources" in the output html file of hdswif.py summary [workflow]<br />
|-<br />
| create_ordered_hists.py ||<br />
* Called within parse_swif.py<br />
* Takes in XML output from SWIF and creates 2 plots<br />
* Plots are dependency time and pending time of each job, ordered by submission.<br />
* Different colors represent jobs submitted at different times. For jobs submitted at the same time, jobs are ordered in increasing time.<br />
|-<br />
| create_stacked_times.py ||<br />
* Called within parse_swif.py<br />
* Takes in XML output from SWIF and creates 2 plots<br />
* Plots show total job time divided into colors for different stages<br />
* One shows all jobs in order of Auger ID (roughly submission order), the other one shows in order of total job time<br />
* The jobs show at a glance which stage contributes how much of the job's time<br />
|}<br />
<br />
=== Utility scripts ===<br />
{| border="1" cellpadding="0" valign="left" style="text-align: left;"<br />
!width="150"| file name<br />
!width="800"| Description<br />
|-<br />
| stderr_by_size.py ||<br />
* Independent of all other scripts in hdswif directory<br />
* When diagnosing problems it is useful to check the stderr/stdout files. Frequently, different problems are easier to find based on the size of stderr files<br />
* Takes in run period and version as arguments, creates a directory /volatile/halld/offline_monitoring/RunPeriod-[RUNPERIOD]/ver[VER]/log/bysize that contains soft links to all stdout and stderr files from the specified launch, <b>in separate directories given by stderr file size</b>.<br />
|}<br />
<br />
=== cross_analysis scripts ===<br />
<b>Summary:</b> The script run_cross_analysis.sh is the main script. In principle, running this with <pre>./run_cross_analysis.sh [RUNPERIOD] [VERSION]</pre> should work, but it is recommended that the commands within the script<br />
be run by hand to catch any errors.<br />
<br />
{| border="1" cellpadding="0" valign="left" style="text-align: left;"<br />
!width="150"| file name<br />
!width="800"| Description<br />
|-<br />
| run_cross_analysis.sh ||<br />
* Main script to call other Python scripts.<br />
* Takes in run period and version as arguments and runs cross analysis<br />
|-<br />
| create_cross_analysis_table.sh ||<br />
* Creates MySQL table for current launch that is used by other scripts<br />
* Takes run period and version as input, and creates table cross_analysis_table_[RUNPERIOD]_ver[VERSION]<br />
|-<br />
| create_stats_table_row.py ||<br />
* Create a row in the html table showing the overall statistics of final states and problems for a launch<br />
* The final states are "Success", and "Segfault". "Success" includes all jobs that had problems but still finished with Success.<br />
* The problems are "Over Limit", "Timeout", and "System". If any of these occurred for any attempt of the job they are counted.<br />
* The script creates a new html table row for the current launch. This html snippet is inserted into the web-accessible file showing the results <br />
|-<br />
| create_stats_for_each_file.py ||<br />
* Create html tables showing the status of each file against different launch versions.<br />
* Takes in run period, min version, max version and shows final result and problems for all versions in between<br />
* Different final results and problems are shown by combinations of ext content, olor coding of text, background color<br />
|-<br />
|-<br />
| create_resource_correlation_plots.py ||<br />
* Create plots showing correlation of resources between different launches for each file.<br />
* Takes in run period, min version, version of interest. Points are shown only for files included in version of interest<br />
* Creates plots for CPU time, Wall time, memory, virtual memory, #events, difference in #events, time to copy input evio file, time to run plugin<br />
|}<br />
<br />
== Running Over Data As It Comes In ==<br />
<br />
A special user gxproj1 will have a cron job set up to run the plugins as new data appears on /mss.<br />
During the week, gxproj1 will submit offline plugin jobs with the same setup as the weekly jobs<br />
run the previous Friday. The procedure for this is shown below.<br />
<br />
<!--<br />
=== Setting up the environment ===<br />
The file<br />
/home/gxproj1/setup_jlab.csh<br />
is sourced through .tcshrc.<br />
This file is the same as what is linked to by<br />
/home/gluex/setup_jlab_commissioning.csh,<br />
except HALLD_HOME, HDDS_HOME, and JANA_CALIB_URL are set separately so that this<br />
user can have a separate build.<br />
<br />
To obtain the builds from the previous Friday's runs,<br />
execute<br />
/home/gxproj1/halld/monitoring/newruns/setup_previous.sh [year] [month] [day]<br />
The build revisions from the previous Friday are archived in files<br />
/work/halld/data_monitoring/run_conditions/soft_comm_[year]_[month]_[day].xml<br />
and the script will build libraries based on those stored revision numbers.<br />
--><br />
<br />
=== Running the cron job ===<br />
<br />
'''IMPORTANT:''' The cron job should not be running while you are manually submitting jobs using the jproj.pl script for the same project, or else you will probably multiply-submit a job. <br />
<br />
* Go to the cron job directory: <br />
<pre><br />
cd /u/home/gxproj1/halld/monitoring/newruns<br />
</pre><br />
<br />
* The cron_plugins file is the cronjob that will be executed. During execution, it runs the exec.sh command in the same folder. This command takes two arguments: the project name, and the maximum file number for each run. These fields should be updated in the cron_plugins file before running. <br />
<br />
* The exec.sh command updates the job management database table with any data that has arrived on tape since it was last updated, ignoring file numbers greater than the maximum file number. It then submits jobs for these files. <br />
<br />
* To start the cron job, run:<br />
<pre><br />
crontab cron_plugins<br />
</pre><br />
<br />
* To check whether the cron job is running, do<br />
<pre><br />
crontab -l<br />
</pre><br />
<br />
* To remove the cron job do<br />
<pre><br />
crontab -r<br />
</pre><br />
<br />
<!--<br />
The cron job will run the script scan_for_jobs.sh,<br />
which runs generatejobs_plugins_rawdata.sh for any<br />
new runs that it had not seen before. All previous<br />
runs are recorded in the file filelists/files_current.txt<br />
so clear this to run over runs, or set the parameters<br />
MINRUN and MAXRUN which will set the range of runs submitted.<br />
--><br />
<br />
==Post-Processing Procedures==<br />
<br />
To visualize the monitoring data, we save images of selected histograms and store time series of selected quantities in a database, which are then displayed on a web page. This section describes how to generate the monitoring images and database information.<br />
<br />
The scripts used to generate this summary data are primarily run from /home/gxprojN/halld/monitoring/process . If you want a new copy of the scripts, e.g., for a new monitoring run, you should check the scripts out from SVN:<br />
<syntaxhighlight><br />
svn co https://halldsvn.jlab.org/repos/trunk/scripts/monitoring/process<br />
</syntaxhighlight><br />
Note that these scripts currently have some parameters which must be periodically set by hand.<br />
<br />
The default python version on most JLab machine does not have the modules to allow these scripts to connect to the MySQL database. To run these scripts, load the environment with the following command<br />
<syntaxhighlight><br />
source /home/gxproj1/halld/monitoring/process/monitoring_env.csh<br />
</syntaxhighlight><br />
<br />
===Online Monitoring===<br />
<br />
There are two primary scripts for running over the monitoring data generated by the online system and offline reconstruction. The online script can be run with either of the following commands:<br />
<syntaxhighlight><br />
/home/gluex/halld/monitoring/process/check_new_runs.py<br />
<br />
OR <br />
<br />
/home/gluex/halld/monitoring/process/check_new_runs.csh<br />
</syntaxhighlight><br />
The shell script sets up the environment properly to run the python script. To connect to the monitoring database on the JLab CUE, modules included in the local installation of python >= 2.7 are needed. The shell script is appropriate to use in a cron job. The cronjob is currently run under the "gluex" account.<br />
<br />
The online monitoring system copies a ROOT file containing the results of the online monitoring, and other configuration files into a directory accessible outside the counting house. This python script automatically checks for new ROOT files, which it will then automatically process. It contains several configuration variables that must be correctly set, which contains the location of input/output directories, etc...<br />
<br />
===Offline Monitoring===<br />
<br />
After the data is run over, the results should be processed, so that summary data is entered into the monitoring database and plots are made for the monitoring webpages. Currently, this processing is controlled by a cronjob that runs the following script:<br />
<syntaxhighlight><br />
/home/gxproj1/halld/monitoring/process/check_monitoring_data.csh <br />
</syntaxhighlight><br />
This script checks for new ROOT files, and only runs over those it hasn't processed yet. Since one monitoring ROOT file is produced for each EVIO file, whenever a new file is produced, the plots for the corresponding run are recreated and all the ROOT files for a run are combined into one file. Information is stored in the database on a per-file basis. <br />
<br />
Plots for the monitoring web page can be made from single histograms or multiple histograms using RootSpy macros. If you want to change the list of plots made, you must modify one of the following files:<br />
* histograms_to_monitor - specify either the name of the histogram or its the full ROOT path <br />
* macros_to_monitor - specify the full path to the RootSpy macro .C file<br />
<br />
When a new monitoring run is started, or the conditions are changed, the following steps should be taken to process the new files:<br />
# Add a new data version, as described below:<br />
# Change the following parameters in check_monitoring_data.csh:<br />
## JOBDATE should correspond to the ouptut date used by the job submission script<br />
## OUTPUTDIR should correspond to the directory corresponding to the run period and revision corresponding to the new version you just submitted. Presumably, this directory will be empty at the beginning.<br />
## Once you create a new data version as defined below, you should pass the needed information as a command line option. Currently this is done by the ARGS variable. For example, the argument "-v RunPeriod-2014-10,8" tells the monitoring scripts to look up the version corresponding to revision 8 of RunPeriod-2014-10 in the monitoring DB and to use to store the results.<br />
<br />
<syntaxhighlight><br />
Example configuration parameters:<br />
set JOBDATE=2015-01-09<br />
set INPUTDIR=/volatile/halld/RunPeriod-2014-10/offline_monitoring<br />
set OUTPUTDIR=/w/halld-scifs1a/data_monitoring/RunPeriod-2014-10/ver08<br />
set ARGS=" -v RunPeriod-2014-10,8 "<br />
</syntaxhighlight><br />
If you want to process the results manually, the data is processed using the following script:<br />
<syntaxhighlight><br />
./process_new_offline_data.py <input directory> <output directory><br />
<br />
EXAMPLE:<br />
<br />
./process_new_offline_data.py 2014-11-14 /volatile/halld/RunPeriod-2014-10/offline_monitoring/ /w/halld-scifs1a/data_monitoring/RunPeriod-2014-10/ver02<br />
</syntaxhighlight><br />
The python script takes several options to enable/disable various steps in the processing. Of interest is the "--force" option, which will run over all monitoring ROOT files, whether or not they've been previously identified.<br />
<br />
Every time a new reconstruction pass is performed, a new version number must be generated. To do this, prepare a version file as described below. Then run the register_new_version.py script to store the information in the database. The script will return a version number, which then should be set by hand in process_new_offline_data.py - future versions of the script will streamline this part of the procedure. An example of how to generate a new version is:<br />
<syntaxhighlight><br />
./register_new_version.py add /home/gxproj1/halld/monitoring/process/versions/vers_RunPeriod-2014-10_pass1.txt<br />
</syntaxhighlight><br />
<br />
<b>If you are running the offline monitoring by checking out the files in trunk/scripts/monitoring/jproj/projects/<br />
of the svn repository</b>, and created a project with<br />
create_project.sh [project name] hd_rawdata<br />
Then go to the directory [project name]/processing/<br />
and execute<br />
./run_processing.sh<br />
which will run register_new_version.py as well as check_monitoring_data.csh for that project.<br />
<br />
===Step-by-Step Instructions For Processing a New Monitoring Run===<br />
<br />
The monitoring runs are current run out of the gxproj1 and gxproj5 accounts. After an offline monitoring run has been successfully started on the batch farm, the following steps should be followed to setup the post-processing for these runs.<br />
<br />
# The post-processing scripts are stored in $HOME/halld/monitoring/process and are automatically run by cron.<br />
# Run "svn update" to bring any changes in. Be sure that the list of histograms and macros to plot are current.<br />
# Add a new data version<br />
# Edit check_monitoring_data.csh to point to the current revisions/directories<br />
#* VERSION<br />
#* ARGS<br />
#* Note that the environment depends on a standard script - $HOME/setup_jlab.csh<br />
# Update files in the web directory, so that the results are displayed on the web pages: /group/halld/www/halldweb/html/data_monitoring/textdata<br />
# Copy the REST files to more permanent locations:<br />
#* cp -a /volatile/halld/offline_monitoring/RunPeriod-YYYY-MM/verVV/REST /work/halld/data_monitoring/RunPeriod-YYYY-MM/REST/verVV<br />
#* cp -a /volatile/halld/offline_monitoring/RunPeriod-YYYY-MM/verVV/REST /cache/halld/RunPeriod-YYYY-MM/REST/verVV [under testing]<br />
<br />
Check log files in $HOME/halld/monitoring/process/log for more information on how each run went. If there are problems, check log files, and modify check_monitoring_data.csh to vary the verbosity of the output.<br />
<br />
==Data Versions==<br />
<br />
To document the conditions of the monitoring data that is created, for the sake of reproducability and further analysis we save several pieces of information. The format is intended to be comprehensive enough to document not just monitoring data, but versions of raw and reconstructed data, so that this database table can be used for the event database as well.<br />
<br />
We store one record per pass through one run period, with the following structure:<br />
<br />
{| class="wikitable"<br />
! Field !! Description<br />
|-<br />
| data_type || The level of data we are processing. For the purposes of monitoring, "rawdata" is the online monitoring, "recon" is the offline monitoring<br />
|- <br />
| run_period || The run period of the data<br />
|- <br />
| revision || An integer specifying which pass through the run period this data corresponds to<br />
|- <br />
| software_version || The name of the XML file that specifies the different software versions used<br />
|-<br />
| jana_config || The name of the text file that specifies which JANA options were passed to the reconstruction program<br />
|-<br />
| ccdb_context || The value of JANA_CALIB_CONTEXT, which specifies the version of calibration constants that were used<br />
|-<br />
| production_time || The data at which monitoring/reconstruction began<br />
|-<br />
| dataVersionString || A convenient string for identifying this version of the data<br />
|}<br />
<br />
<br />
An example file used as as input to ./register_new_version.py is:<br />
<syntaxhighlight><br />
data_type = recon<br />
run_period = RunPeriod-2014-10<br />
revision = 1<br />
software_version = soft_comm_2014_11_06.xml<br />
jana_config = jana_rawdata_comm_2014_11_06.conf<br />
ccdb_context = calibtime=2014-11-10<br />
production_time = 2014-11-10<br />
dataVersionString = recon_RunPeriod-2014-10_20141110_ver01<br />
</syntaxhighlight></div>Kmoriyahttps://halldweb.jlab.org/wiki/index.php?title=Data_Monitoring_Procedures&diff=71863Data Monitoring Procedures2015-12-11T21:39:07Z<p>Kmoriya: /* Cross Analysis of Launches */</p>
<hr />
<div>__TOC__<br />
<br />
== Master List of File / Database / Webpage Locations ==<br />
=== Run Conditions ===<br />
* Online Run-by-run condition files (B-field, current, etc.): /work/halld/online_monitoring/conditions/<br />
* Offline monitoring run conditions (software versions, jana config): /group/halld/data_monitoring/run_conditions/<br />
*[http://www.jlab.org/Hall-D/test/RunInfo/ Run Info vers. 1]<br />
*[https://halldweb.jlab.org/cgi-bin/data_monitoring/run_conditions.pl Run Info vers. 2]<br />
<br />
=== Monitoring Output Files ===<br />
* Run Periods 201Y-MM is for example 2015-03, launch ver verVV is for example ver15<br />
* Online monitoring histograms: /work/halld/online_monitoring/root/<br />
* Offline monitoring histogram ROOT files: /work/halld/data_monitoring/RunPeriod-201Y-MM/verVV/rootfiles<br />
* REST files (most recent launch only): /work/halld/data_monitoring/RunPeriod-201Y-MM/REST/verVV<br />
* individual files for each job (ROOT, log, etc.): /volatile/halld/offline_monitoring/RunPeriod-201Y-MM/verVV/<br />
<br />
=== Monitoring Database ===<br />
* Accessing monitoring database (on ifarm): mysql -u datmon -h hallddb.jlab.org data_monitoring<br />
<br />
=== Monitoring Webpages ===<br />
*[https://halldweb.jlab.org/cgi-bin/data_monitoring/monitoring/plotBrowser.py Plot Browser]<br />
*[https://halldweb.jlab.org/cgi-bin/data_monitoring/monitoring/runBrowser.py Run Browser]<br />
*[https://halldweb.jlab.org/cgi-bin/data_monitoring/monitoring/timeSeries.py Time Series]<br />
*[https://halldweb.jlab.org/data_monitoring/launch_analysis/ Launch Analysis]<br />
<br />
== Job Monitoring Links ==<br />
* [http://scicomp.jlab.org/farm2/job.html Completed Job History]<br />
* [http://scicomp.jlab.org/farm2/project.html Job Stats By Project]<br />
* [http://scicomp.jlab.org/farm2/trackOrg.html Job Stats By Track]<br />
* [http://scicomp.jlab.org/farm2/report.html Cluster Report]<br />
* [http://scicomp.jlab.org/farm2/walltime.html Walltime Distribution]<br />
<br />
== Saving Online Monitoring Data ==<br />
<br />
The procedure for writing the data out is given in, e.g.,<br />
[https://halldweb.jlab.org/wiki/index.php/Raid-to-Silo_Transfer_Strategy Raid-to-Silo Transfer Strategy].<br />
<br />
Once the DAQ writes out the data to the raid disk, cron jobs will copy the file to tape,<br />
and within ~20 min., we will have access to the file on tape at<br />
/mss/halld/$RUN_PERIOD/rawdata/RunXXXXXX.<br />
<br />
All online monitoring plugins will be run as data is taken.<br />
They will be accessible within the counting house via RootSpy, and<br />
for each run and file, a ROOT file containing the histograms will be saved<br />
within a subdirectory for each run.<br />
<br />
For immediate access to these files, the raid disk files may be accessed directly<br />
from the counting house, or the tape files will be available within ~20 min. of the<br />
file being written out.<br />
<br />
== Offline Monitoring: Running Over Archived Data ==<br />
<br />
Once files are written to tape we run the online plugins on these files to confirm what we were seeing in the online monitoring, and also to update the results from the latest calibration and software. <br />
Manual scripts and cron jobs are set up to look for new data and run the plugins over a sample of files.<br />
<br />
Every other Friday (usually the Friday before the offline meetings) jobs will be started to run the newest software on all previous runs,and allows everybody to see improvements in each detector.<br />
For each launch, independent builds of hdds, sim-recon, the monitoring plugins, and an sqlite file will be generated.<br />
<br />
Below the procedures are described for<br />
# Preparing the software for the launch<br />
# Starting the launch (using hdswif)<br />
# Post-analysis of statistics of the launch<br />
<br />
Processing the results and making them available to the collaboration<br />
is handled in the section [[#Post-Processing_Procedures | Post-Processing Procedures]] below.<br />
<br />
<!--<br />
==== Generating an offline plugin job ====<br />
<br />
The user gxproj1 should be used for official offline monitoring jobs.<br />
Within /home/gxproj1/halld/monitoring/batch/ there will be scripts to run the online monitoring plugins over tape files.<br />
The main script is generatejobs_plugins_rawdata.sh, which can be used as<br />
generatejobs_plugins_rawdata.sh [minrun] [maxrun] (minfile) (maxfile)<br />
where minrun, maxrun specify the range of the run #, and minfile, maxfile (optional) specify<br />
the file range for that run.<br />
<br />
This will generate a script run_rawdata_XXXXXX.sh, where the run # has now been formatted to be 6 digits.<br />
Executing this script will send the monitoring plugins job to the Auger batch system.<br />
<br />
There is also a script clean.sh which can be used as<br />
clean.sh XXX<br />
This will clean up all associated files created in association with run XXX.<br />
<br />
Internally, the xml file used to submit the job will be created, and the job to run<br />
will be given within script.sh. All run parameters should be specified in at the beginning<br />
of generatejobs_plugins_rawdata.sh<br />
Since we are running on tape, the tape file will first be copied over to the cache disk, and the job will run<br />
over this cached file.<br />
<br />
==== Using cron to run automatically ====<br />
Within /home/gluex/halld/monitoring/cron/ there is a file cron_plugins<br />
that can be executed via<br />
crontab cron_plugins<br />
This will set up a cron job to call the script scan_for_jobs.sh, which will<br />
check in the rawdata directory and call generatejobs_plugins_rawdata.sh for<br />
any run that is more than 5 min old. The cron job is set up to run every 10 min.<br />
<br />
==== Magnetic Field Settings ====<br />
For tracking, it is necessary to set the correct magnetic field settings.<br />
The below table is now obsolete. Please refer to the [https://halldweb.jlab.org/cgi-bin/data_monitoring/run_conditions.pl GlueX Run Conditions] webpage.<br />
<br />
The actual field settings for each run have not been documented well.<br />
Below are values based on going through entries in the [https://logbooks.jlab.org/book/halld halld log].<br />
<br />
{| border="1" cellpadding="1" style="text-align: center;"<br />
!width="100"| Run #s<br />
!width="150"| Solenoid Current (A)<br />
!width="300"| JANA option<br />
!width="300"| notes<br />
|-<br />
| 940 - 996 || 1000 || -PBFIELD_MAP=Magnets/Solenoid/solenoid_1000A_poisson_20141104 || At run 997, the solenoid started ramping down<br />
|-<br />
| 998 - 1448 || 0 || -PBFIELD_TYPE=NoField -PDEFTAG:DTrackCandidate=StraightLine || Solenoid started ramping up around run 1431. Runs 1432 - 1448 should not be used for tracking. See [https://logbooks.jlab.org/entry/3309039 this] and [https://logbooks.jlab.org/entry/3309092 this] entry.<br />
|-<br />
| 1449 - 1620 || 1200 || -PBFIELD_MAP=Magnets/Solenoid/solenoid_1200A_poisson_20140520 || <br />
|-<br />
|}<br />
<br />
'''Note:''' The following run ranges were taken with a 300A field, and could be analyzed with the magnetic field map Magnets/Solenoid/solenoid_300A_poisson_20140819 , more study is needed to see if the straight line track fitter is needed or not: 1036 - 1053, 1065 - 1121, 1212? - 1254, 1309 - 1318 [1307,1308,1319-1329 were taken while the magnet was ramping].<br />
--><br />
<br />
=== General Information on Procedures ===<br />
This section explains how the offline monitoring should be run.<br />
Since we may want to simultaneously run offline monitoring for different run periods that require<br />
different environment variables, the scripts are set up so that a generic user can download the<br />
scripts and run them from anywhere. Most output directories for offline monitoring are created<br />
with group read/write permissions so that any Hall D group user has access to the contents,<br />
but there are some cases where use of the account that created the launch is necessary.<br />
<br />
The accounts used for offline monitoring are the gxprojN accounts created and maintained by<br />
Mark Ito (see [https://halldweb.jlab.org/wiki/index.php/GlueX-related_shared_accounts_on_the_JLab_CUE here] for how each account is used).<br />
As of October 2015, the following are used:<br />
* gxproj1 for running over Fall 2014 data (deprecated since June 2015)<br />
* gxproj5 for running over Spring 2015 data<br />
<br />
Since the summer of 2015 we have transitioned from a system using Mark Ito's jproj scripts<br />
to integrating the swif system that Chris Larrieu (SciComp) has been developing.<br />
For offline monitoring, the hdswif system that Kei developed is used for launching the jobs, and a new cross analysis<br />
system based on MySQL and Python is maintained. The jproj system is now deprecated for offline monitoring.<br />
<br />
<br />
The scripts for hdswif, cross analysis and jproj are maintained in svn:<br />
* hdswif: https://halldsvn.jlab.org/repos/trunk/scripts/monitoring/hdswif<br />
* cross analysis: https://halldsvn.jlab.org/repos/trunk/scripts/monitoring/cross_analysis<br />
* jproj : https://halldsvn.jlab.org/repos/trunk/scripts/monitoring/jproj<br />
<br />
To run the offline monitoring each package should be checked out and all necessary<br />
scripts are included.<br />
<br />
=== Preparing the software for the launch ===<br />
<br />
1. Setup the environment: <pre>source ~/env_monitoring_launch</pre><br />
<br />
2. Updating & building hdds: <br />
<pre><br />
cd $HDDS_HOME<br />
git pull # Get latest software<br />
scons -c install # Clean out the old install: EXTREMELY IMPORTANT for cleaning out stale headers<br />
scons install -j4 # Rebuild and re-install with 4 threads<br />
</pre><br />
<br />
3. Updating & building sim-recon: <br />
<pre><br />
cd $HALLD_HOME/src<br />
git pull # Get latest software<br />
scons -c install # Clean out the old install: EXTREMELY IMPORTANT for cleaning out stale headers<br />
scons install -j4 # Rebuild and re-install with 4 threads<br />
</pre><br />
<br />
4. Create a new sqlite file containing the very latest calibration constants. Original documentation on creating sqlite files are [https://halldweb.jlab.org/wiki/index.php/SQLite-form_of_the_CCDB_database here]. <br />
<pre><br />
cd $GLUEX_MYTOP/../sqlite/<br />
$CCDB_HOME/scripts/mysql2sqlite/mysql2sqlite.sh -hhallddb.jlab.org -uccdb_user ccdb | sqlite3 ccdb.sqlite<br />
mv ccdb.sqlite ccdb_monitoring_launch.sqlite #replacing the old file<br />
</pre><br />
<br />
5. Note that the above steps must be done <b>BEFORE</b> launch project creation. This is because we will track the revisions of the libraries used, and this is done by extracting the svn information in each directory. Also, note that the system assumes that we have the topmost build directory (usually called GLUEX_TOP) to be $HOME/builds . Such an assumption is necessary to be able to extract information about the library locations automatically.<br />
<br />
Create the appropriate project(s) and submit the jobs using hdswif, as detailed in the section below.<br />
<br />
=== Starting the Launch and Submitting Jobs ===<br />
<br />
Until the summer of 2015 we relied solely on Mark Ito's jproj system for submitting and keeping track of jobs. We have since moved to the swif system and use the hdswif wrapper for this. Below are instructions for how to use these.<br />
<br />
1. Downloading hdswif: Download the hdswif directory from svn. For the gxprojN accounts, use the directory ~/halld/hdswif. <br />
<pre><br />
cd ~/halld<br />
svn co https://halldsvn.jlab.org/repos/trunk/scripts/monitoring/hdswif<br />
cd hdswif<br />
</pre><br />
<br />
2. Edit the job config file, input.config, which is used to register jobs in hdswif. A typical config file will look this:<br />
PROJECT gluex<br />
TRACK reconstruction<br />
OS centos65<br />
NCORES 6<br />
DISK 40<br />
RAM 8<br />
TIMELIMIT 8<br />
JOBNAMEBASE offmon_<br />
RUNPERIOD 2015-03<br />
VERSION 15<br />
OUTPUT_TOPDIR /volatile/halld/offline_monitoring/RunPeriod-[RUNPERIOD]/ver[VERSION] # Example of other variables included in variable<br />
SCRIPTFILE /home/gxproj5/halld/hdswif/script.sh # Must specify full path<br />
ENVFILE /home/gxproj5/halld/hdswif/setup_jlab-2015-03.csh # Must specify full path<br />
<br />
3. Creating the workflow: Within SWIF jobs are registered into workflows. For offline monitoring, the workflow names are of the form <b>offline_monitoring_RunPeriod201Y_MM_verVV_hd_rawdata</b> with suitable replacements for the run period and version number. The command "swif list" will list all existing workflows. Also, for most simple SWIF commands hdswif also provides a wrapper. <br />
swif list<br />
<br />
For creation of workflows for offline monitoring the command:<br />
hdswif.py create [workflow] -c input.config<br />
should be used. When a config file (here: input.config) is passed in, hdswif will automatically create files that record the configuration of the current launch. These files are stored as for example:<br />
/group/halld/data_monitoring/run_conditions/jana_rawdata_comm_2015_03_ver15.conf<br />
/group/halld/data_monitoring/run_conditions/soft_comm_2015_03_ver15.xml<br />
<br />
The software packages stored in git (sim-recon and hdds) can have git tags applied to them, which makes it easier to find versions of the software than a SHA-1 hash. hdswif will ask if you would like to create a tag, and execute the following sequence: <pre>git tag -a offmon-201Y_MM-verVV -m "Used for offline monitoring 201Y-MM verVV started on 201y/mm/dd"</pre> <pre>git push offmon-201Y_MM-verVV</pre> This will only be invoked when the user name is gxprojN, and for the configuration files, the output directory will be /group/halld/data_monitoring/run_conditions/ for gxprojN accounts while it will be the current directory for other users.<br />
* To use the git-tagged software versions do for example <pre>cd $HALLD_HOME</pre> <pre>git checkout offmon-2015_03-ver15</pre><br />
<br />
4. Registering jobs in the workflow: To register jobs within the workflow, hdswif provides the use of config files. Jobs can be registered by specifying the workflow, config file (-c), run (-r) and file (-f) numbers if necessary. <br />
<span style="color:red">Note: Job configuration parameters can be set differently for jobs within the same workflow if necessary.</span><br />
Jobs can be added via <pre>hdswif.py add [workflow] -c input.config</pre><br />
By default, hdswif will add all files found within the directory /mss/halld/RunPeriod-201Y-MM/rawdata/ where 201Y-MM is specified by the RUNPERIOD parameter in the config file. If only some of the runs or files are needed, these can be specified for example with<br />
<pre>hdswif.py add [workflow] -c input.config -r 3180 -f '00[0-4]'</pre><br />
to specify to register running only over run 3180 files 000 - 004 (Unix-style brackets and wildcards can be used).<br />
<br />
5. Running the workflow: To run the workflow, simply use the hdswif wrapper:<br />
hdswif.py run [workflow]<br />
<br />
<b>It is recommended that some jobs be tested to make sure that everything is working rather than fail thousands of jobs.</b><br><b>Check the configuration setup when creating a workflow, and what is in script.sh. Also check the following:<br />
* Check stderr files. Are they small (<kB)?<br />
* Check stdout files. Are they very large (>MB)?<br />
* Check output ROOT files. Are they larger than several MB?<br />
* Check output REST files. Are they larger than several tens of MB?<br />
</b><br />
<br> For this purpose, hdswif will take an additional parameter to run which limits the number of jobs to submit: <pre>hdswif.py run [workflow] 10</pre><br />
in which case only 10 jobs will be submitted.<br />
<br />
To submit all jobs after checking the results, do <pre>hdswif.py run [workflow]</pre><br />
<br />
=== Checking the Status and Resubmitting ===<br />
1. The status of jobs can be checked on the terminal with <pre>jobstat -u gxprojN</pre> For the status of jobs on Auger see http://scicomp.jlab.org/scicomp/#/auger/jobs and for SWIF use <pre>swif list</pre> or for more information, <pre>swif status [workflow] -summary</pre> Note that "swif status" tends to be out of date sometimes, so don't panic if your workflow/jobs aren't showing up right away. Also see the Auger [http://scicomp.jlab.org/scicomp/#/auger/jobs job website].<br />
<br />
2. For failed jobs, SWIF can resubmit jobs based on the problem. For resubmission for failed jobs with the same resources, <pre>swif retry-jobs [workflow] -problems [problem name]</pre> can be used, and for jobs to be submitted with more resources, e.g., use <pre>swif modify-jobs [workflow] -ram add 2gb -problems AUGER-OVER_RLIMIT</pre> <br />
This only re-stages the jobs, be sure to resubmit them with:<br />
<pre>swif run -workflow [workflow] -errorlimit none</pre><br />
<br />
hdswif has a wrapper for both of these: <pre>hdswif.py resubmit [workflow] [problem]</pre> In this case [problem] can be one of <b>SYSTEM, TIMEOUT, RLIMIT</b>. If SYSTEM is specified, the jobs will be retried. For TIMEOUT and RLIMIT, the jobs will be modified by default with 2 additional hours or GB of RAM. If one more number is added as an option, then that many hours or GB of RAM will be added., e.g., <pre>hdswif.py resubmit [workflow] TIMEOUT 5</pre> will add 5 hours of processing time. You can wait until almost all jobs finish before resubmitting failed jobs since the number should be relatively small. <b>Even if jobs are resubmitted for one type of failure, jobs that later fail with that failure will not be automatically resubmitted.</b><br />
<br />
3. For information on swif, use the "swif help" commands and for hdswif see the attached documentaion in https://halldsvn.jlab.org/repos/trunk/scripts/monitoring/hdswif/manual_hdswif.pdf<br />
<br />
4. Below is a table describing the various errors that can occur.<br />
{| border="1" cellpadding="0" valign="left" style="text-align: left;"<br />
!width="150"| ERROR NAME<br />
!width="400"| Description<br />
!width="400"| Resolution<br />
!width="400"| hdswif command<br />
|-<br />
| AUGER-SUBMIT<br />
||<br />
SWIF’s attempt to submit jobs to Auger failed. Includes server-side problems as well as user failing to provide valid job parameters (e.g. incorrect project name, too many resources, etc.)<br />
||<br />
If requested resources are known to be correct resubmit. Otherwise modify job resources using swif directly.<br />
||<br />
<span style="color:purple"><b>hdswif.py resubmit [workflow] SYSTEM</b></span><br />
|-<br />
| AUGER-FAILED<br />
||<br />
Auger reports the job FAILED with no specific details.<br />
||<br />
Resubmit jobs. If problem persists, contact Chris Larrieu or SciComp.<br />
||<br />
<span style="color:purple"><b>hdswif.py resubmit [workflow] SYSTEM</b></span><br />
|-<br />
| AUGER-OUTPUT-FAIL<br />
||<br />
Failure to copy one or more output files.Can be due to permission problem, quota problem, system error, etc.<br />
||<br />
Check if output files will exist after job execution and that output directory exists, resubmit jobs. If problem persists, contact Chris Larrieu or SciComp.<br />
||<br />
<span style="color:purple"><b>hdswif.py resubmit [workflow] SYSTEM</b></span><br />
|-<br />
| AUGER-INPUT-FAIL<br />
||<br />
Auger failed to copy one or more of the requested input files, similar to output failures. Can also happen if tape file is unavailable (e.g. missing/damaged tape)<br />
||<br />
Check if input file exists, resubmit jobs. If problem persists, contact Chris Larrieu or SciComp.<br />
||<br />
<span style="color:purple"><b>hdswif.py resubmit [workflow] SYSTEM</b></span><br />
|-<br />
| AUGER-TIMEOUT<br />
||<br />
Job timed out.<br />
||<br />
If more time is needed for job add more resources.<br />
Default is to add 2 hrs of processing time. Also check whether code is hanging.<br />
||<br />
<span style="color:red"><b>hdswif.py resubmit [workflow] TIMEOUT</b></span><br><br />
Default is to add 2 hours. Optionally specify number of hours at end.<br />
|-<br />
| AUGER-OVER_RLIMIT<br />
||<br />
Not enough resources, RAM or disk space.<br />
||<br />
Add more resources for job.<br />
||<br />
<span style="color:blue"><b>hdswif.py resubmit [workflow] RLIMIT</b></span><br><br />
Default is to add 2 GB of RAM. Optionally specify GB at end. To add more disk space use SWIF directly.<br />
|-<br />
| SWIF-MISSING-OUTPUT<br />
||<br />
Output file specified by user was not found.<br />
||<br />
Check if output file exists at end of job.<br />
||<br />
<br />
|-<br />
| SWIF-USER-NON-ZERO <br />
||<br />
User script exited with non-zero status code.<br />
||<br />
Your script exited with non-zero status. Check the code you are running.<br />
||<br />
<br />
|-<br />
| SWIF-SYSTEM-ERROR <br />
||<br />
Job failed owing to a problem with swif (e.g. network connection timeout)<br />
||<br />
Resubmit jobs. If problem persists, contact Chris Larrieu or SciComp.<br />
||<br />
<span style="color:purple"><b>hdswif.py resubmit [workflow] SYSTEM</b></span><br />
|}<br />
<br style="clear:both;"/><br />
<br />
=== Post-analysis of statistics of the launch ===<br />
<br />
After jobs have been submitted, it will usually take a few days for all of the jobs to be processed.<br />
The next step is to check the resource usage for the current launch and publish the results online.<br />
<br />
# <b>Create summary XML, HTML files</b><br> The status and results of jobs are saved within the SWIF internal server, and are available via the command <pre>swif status [workflow] -summary -runs</pre> where the arguments -summary and -runs show summary statistics and statistics for individual jobs, respectively. hdswif has a command that takes this output in XML output and creates an HTML webpage showing results of the launch. To do this, do <pre>hdswif.py summary [workflow]</pre> This will create the XML output file from SWIF called swif_output_[workflow].xml and create a webpage containing png figure files. If the XML file already exists, hdswif will ask whether to overwrite the existing file.<br />
# <b>Publish output files online</b><br> At this stage the html output and figure files are created and ready to be put online. The html output capabilities of hdswif is useful for any user using SWIF, but since publication of the html output is specific to the offline monitoring, the script to do this is contained in the cross_analysis scripts directory at https://halldsvn.jlab.org/repos/trunk/scripts/monitoring/cross_analysis. For the gxprojN accounts this directory should exist as ~/halld/monitoring/cross_analysis. To publish the results online do for example <pre>python ~/halld/monitoring/cross_analysis/publish_offmon_results.py 2015_03 18</pre> The script copies the html output and corresponding figures to /group/halld/www/halldweb/html/data_monitoring/launch_analysis/ and also creates a link to it in the html page.<br />
# <b>Editing the summary HTML page</b><br> The top page for is offline monitoring https://halldweb.jlab.org/data_monitoring/launch_analysis/index.html and has links to the summary page for each run period. The summary files are /group/halld/www/halldweb/html/data_monitoring/launch_analysis/[run period]/[run period].html Edit the file to<br />
## Add a new line to the first table which contains the version number, date, and comments for the current launch<br />
## Create a link to the webpage for the current launch. Simply copy and paste, modify the previous launch's link to have the correct launch ver.<br />
# <b>Freezing SWIF tables</b><br> Since we are now finished with the SWIF workflow that we used, the workflow should be "frozen" so that it cannot be mistakenly altered or modified. Do <pre>swif freeze [workflow]</pre><br />
# <b>Backing up SWIF output</b><br> With the workflow frozen we should be able to reproduce the XML output from SWIF, but we will backup the XML output just in case. Do <pre>cp ~/halld/hdswif/xml/swif_output_[workflow].xml /group/halld/data_monitoring/swif_xml_backup/ </pre><br />
<br />
=== Cross Analysis of Launches ===<br />
<br />
The purpose of the cross analysis is to correlate how resource usage changed for the same files across different launches.<br />
To do this it is useful to create MySQL tables that contain information on each launch, and do queries across tables.<br />
<br />
#The scripts to do this are maintained in the svn directory https://halldsvn.jlab.org/repos/trunk/scripts/monitoring/cross_analysis Do <pre> svn co https://halldsvn.jlab.org/repos/trunk/scripts/monitoring/cross_analysis </pre> For the gxprojN accounts used for offline monitoring the directory should be ~/halld/monitoring/cross_analysis<br />
# The main script is run_cross_analysis.sh, which can be run with <pre>./run_cross_analysis.sh [RUNPERIOD] [VERSION]</pre>, where e.g. [RUNPERIOD] = 2015_03 and [VERSION] = 22. However, <b>it is strongly recommended that the commands in this script be run by hand</b> to catch any errors.<br />
# Enter the python commands that are in run_cross_analysis.sh . Below are the steps and explanations:<br />
## Create a table for the current launch using <pre>./create_cross_analysis_table.sh [RUNPERIOD] [VERSION]</pre>. The table will be created from the file template_table_schema.sql and contain columns id, run, file, timeChange, cpu_sec, wall_sec, mem_kb, vmem_kb, nevents, input_copy_sec, plugin_sec, final_state, problems<br />
## Run <pre>python fill_cross_analysis_entries.py [RUNPERIOD] [VERSION]</pre> The script will gather all of the necessary information either from SWIF output or the stdout files for the jobs<br />
## Run <pre>python create_stats_table_row.py</pre> This will loop over the jobs in the current launch and create a row in an HTML table that summarizes the statistics for the final state and problems for the jobs. This table row is then inserted into the main HTML webpage for the run period.<br />
## Run <pre>python create_stats_for_each_file.py [RUNPERIOD] [MINVERSION] [VERSION]</pre> This creates a comparison table of the final state and problems for each file between launches [MINVERSION] and [VERSION]. In the run_cross_analysis.sh script, the default is to set MINVERSION to be 15 for run period 2015_03, but as long as SWIF was used for all previous launches, any number will work.<br />
## Run <pre>python create_resource_correlation_plots.py [RUNVERSION] [CMPMINVERSION] [VERSION]</pre> This creates correlation plots of resource use between launches between CMPMINVERSION and VERSION. By default CMPMINVERSION is 5 launches earlier.<br />
<br />
== Explanation of Scripts ==<br />
Below are explanations of each script used in the offline monitoring system and a breif explanation of how they work.<br />
<br />
=== hdswif scripts ===<br />
<b>Summary:</b> hdswif.py is the main script that calls the other utility scripts. For the utility scripts, they can be run standalone by giving the appropriate parameters. Visual graphics are made using the PyROOT extension of ROOT. To use this, the environment variable PYTHONPATH must include ROOTSYS. If ROOTSYS has been set, adding it to PYTHONPATH will be done by the script, but if ROOTSYS is not set, then the scripts will abort.<br />
<br />
{| border="1" cellpadding="0" valign="left" style="text-align: left;"<br />
!width="150"| file name<br />
!width="800"| Description<br />
|-<br />
| hdswif.py ||<br />
Main script to control the behavior of SWIF. Most commands follow the form hdswif.py [command] [workflow] (options)<br />
|-<br />
| parse_swif.py ||<br />
Called within hdswif.py to create html output from SWIF results.<br />
|-<br />
| createXMLfiles.py ||<br />
* Called by the "create" option of hdswif.py<br />
* Creates XML files for logging information about launch. Must specify config file with option -c.<br />
* Also adds tags to git repositories of sim-recon and hdds.<br />
* For XML file creation, the file will be written out to /group/halld/data_monitoring/run_conditions/ if the user is gxprojN (used for offline monitoring). If not, to avoid general users adding things to the above directory, the output files will be the current directory.<br />
* To write out the versions of each package, environment variables such as HDDS_HOME will need to be set.<br />
* Versions for each software package are extracted using the directory structure, so if these are changed the scripts must change accordingly.<br />
* For each launch, <b>the output soft_comm_[RUNPERIOD]_ver[VER].xml file should be checked that all version numbers have been extracted.</b><br />
|-<br />
| read_config.py ||<br />
* Called by the "add" option of hdswif.py<br />
* Takes in config file name, optionally set verbose<br />
* Return a dictionary between config parameter names and values (e.g., 'PROJECT' : 'gluex', 'NCORES' : 6, ...)<br />
* Prints the config parameters at the end. For parameters changed from default, a '*' will be printed<br />
|-<br />
| output_job_details.py ||<br />
* Called by the "details" option of hdswif.py<br />
* Takes in workflow name, run and file. Run and file must be numbers, no wildcards<br />
* Finds ids for jobs specified by the run and file number and returns info on each one<br />
* Job info is retrieved from pbs farm system and shows configuration parameters for that job<br />
|-<br />
| results_by_resources.py ||<br />
* Called within parse_swif.py<br />
* Takes in XML output from SWIF and creates 2 plots<br />
* Creates html table showing results of jobs by resources requested for that job<br />
* This table is shown under "Status by Resources" in the output html file of hdswif.py summary [workflow]<br />
|-<br />
| create_ordered_hists.py ||<br />
* Called within parse_swif.py<br />
* Takes in XML output from SWIF and creates 2 plots<br />
* Plots are dependency time and pending time of each job, ordered by submission.<br />
* Different colors represent jobs submitted at different times. For jobs submitted at the same time, jobs are ordered in increasing time.<br />
|-<br />
| create_stacked_times.py ||<br />
* Called within parse_swif.py<br />
* Takes in XML output from SWIF and creates 2 plots<br />
* Plots show total job time divided into colors for different stages<br />
* One shows all jobs in order of Auger ID (roughly submission order), the other one shows in order of total job time<br />
* The jobs show at a glance which stage contributes how much of the job's time<br />
|}<br />
<br />
=== Utility scripts ===<br />
{| border="1" cellpadding="0" valign="left" style="text-align: left;"<br />
!width="150"| file name<br />
!width="800"| Description<br />
|-<br />
| stderr_by_size.py ||<br />
* Independent of all other scripts in hdswif directory<br />
* When diagnosing problems it is useful to check the stderr/stdout files. Frequently, different problems are easier to find based on the size of stderr files<br />
* Takes in run period and version as arguments, creates a directory /volatile/halld/offline_monitoring/RunPeriod-[RUNPERIOD]/ver[VER]/log/bysize that contains soft links to all stdout and stderr files from the specified launch, <b>in separate directories given by stderr file size</b>.<br />
|}<br />
<br />
=== cross_analysis scripts ===<br />
<b>Summary:</b> The script run_cross_analysis.sh is the main script. In principle, running this with <pre>./run_cross_analysis.sh [RUNPERIOD] [VERSION]</pre> should work, but it is recommended that the commands within the script<br />
be run by hand to catch any errors.<br />
<br />
{| border="1" cellpadding="0" valign="left" style="text-align: left;"<br />
!width="150"| file name<br />
!width="800"| Description<br />
|-<br />
| run_cross_analysis.sh ||<br />
* Main script to call other Python scripts.<br />
* Takes in run period and version as arguments and runs cross analysis<br />
|-<br />
| create_cross_analysis_table.sh ||<br />
* Creates MySQL table for current launch that is used by other scripts<br />
* Takes run period and version as input, and creates table cross_analysis_table_[RUNPERIOD]_ver[VERSION]<br />
|-<br />
| create_stats_table_row.py ||<br />
* Create a row in the html table showing the overall statistics of final states and problems for a launch<br />
* The final states are "Success", and "Segfault". "Success" includes all jobs that had problems but still finished with Success.<br />
* The problems are "Over Limit", "Timeout", and "System". If any of these occurred for any attempt of the job they are counted.<br />
* The script creates a new html table row for the current launch. This html snippet is inserted into the web-accessible file showing the results <br />
|-<br />
| create_stats_for_each_file.py ||<br />
* Create html tables showing the status of each file against different launch versions.<br />
* Takes in run period, min version, max version and shows final result and problems for all versions in between<br />
* Different final results and problems are shown by combinations of ext content, olor coding of text, background color<br />
|-<br />
|-<br />
| create_resource_correlation_plots.py ||<br />
* Create plots showing correlation of resources between different launches for each file.<br />
* Takes in run period, min version, version of interest. Points are shown only for files included in version of interest<br />
* Creates plots for CPU time, Wall time, memory, virtual memory, #events, difference in #events, time to copy input evio file, time to run plugin<br />
|}<br />
<br />
== Hall D Job Management System ==<br />
<br />
This section details instructions on how to create and launch a set of jobs using the Hall-D Job Management System developed by Mark Ito. These instructions are generic: this system can be used for the weekly monitoring jobs, but can also be used for other sets of job launches as well. <br />
<br />
=== Database Table Overview ===<br />
<br />
* Job management database table (<project_name>): For each input file, keeps track of whether or not a job for it has been submitted, along with other optional fields. <br />
<br />
* Job status database table (<project_name>Job (no space)): For each job, keeps track of the job-id, the job status, memory used, cpu & wall time, time taken to complete various stages (e.g. pending, dependency, active), and others. <br />
<br />
* Job metrics database table (<project_name>_aux (no space)): For each job, keeps track of the job-id, how many events were processed, the time it took to copy the cache file, and the time it took to run the plugin. This information is culled from the log files of each job, and is done within the analysis directory of each launch.<br />
<br />
=== Initialize Project Management ===<br />
<br />
* Log into the ifarm machine with one of the gxproj accounts. For this example we will use gxproj1.<br />
<pre><br />
ssh gxproj1@ifarm -Y<br />
</pre><br />
<br />
* Go to a directory to do the launch. In principle, any directory will work, but for gxproj1 this is usually done in /home/gxproj1/halld/<br />
<br />
* Check out the necessary scripts <pre>svn co https://halldsvn.jlab.org/repos/trunk/scripts/monitoring/jproj</pre> This will get all necessary scripts for launching. Once checked out, <pre>cd projects</pre><br />
<br />
* The script create_project.sh can be used to create a new project. It will take a single argument, the project name. It is assumed that the project name is of the form offline_monitoring_RunPeriod20YY_MM_verVV_hd_rawdata where this string will be parsed to give the run period 20YY_MM and version number VV. One thing to do BEFORE creation of a new project is to editthe conditions of the launch (plugins to run over, memory requested, disk space requested) within templates/template.jsub . This<br />
information is saved automatically at project creation time into files in /group/halld/data_monitoring/run_conditions .<br />
To create a project do <pre>./create_project.sh offline_monitoring_RunPeriod20YY_MM_verVV_hd_rawdata</pre><br />
The name has been chosen to be as consistent as possible with other directory structures.<br />
However, mysql requires that "-" be escaped in table names, so unfortunately run periods will be<br />
given as 2YYY_MM instead of 2YYY-MM.<br />
<br />
* A new directory with the project name (offline_monitoring_RunPeriod20YY_MM_verVV_hd_rawdata) and files will be copied and modified from the template directory to reflect the run period, the user, the directory that it was created in, project name, etc.<br />
<br />
* For each project, cd into the new directory<br />
* The script clear.sh will remove any existing tables for the current project name, then recreate it. Do<br />
<pre>./clear.sh</pre><br />
* To use the jproj.pl script that was checked in, add the directory to your path with<br />
<pre>source ../../scripts/setup.csh</pre><br />
or always specify the full path<br />
<pre>../../scripts/jproj.pl</pre><br />
* Now update the table of runs with<br />
<pre>jproj.pl <project name> update</pre><br />
This will fill the table with all files within /mss that are of the same form as what is in <project name>.jproj . If you want to register only on a subset of all such files, you can edit this file directly.<br />
* Once you have registered all of the files you would like to run over, do<br />
<pre>jproj.pl <project name> submit [max # of jobs] [run number]</pre><br />
where the additional options specify how many jobs to submit and which run number to run on. Without these options all files that are registered and have not been submitted yet will be submitted.<br />
<br />
At this stage you are ready to submit all files. It is a good idea to submit a few test jobs<br />
at first to check that all scripts are working and that the plugins do not crash. Once you are<br />
sure that this does not happen, you can send all jobs in. The remaining jobs are then the monitoring<br />
which will (among other things) put the results on the online webpage for the collaboration to view, and<br />
the analysis of the launch.<br />
<br />
=== Project File Overview ===<br />
<br />
An overview of each project file:<br />
* '''clear.sh:''' For the current project, deletes the job status and management database tables (if any), and creates new, empty ones. <br />
* '''<project_name>.jproj:''' Contains the path and file name format for the input files for the jobs. <br />
* '''<project_name>.jsub:''' The xml job submission script. The run number and file number variables are set during job submission for each input file. <br />
* '''script.sh:''' The script that is executed during the job. If output job directories are not pre-created manually, they should be created in this script with the proper permissions:<br />
<pre><br />
mkdir -p -m 775 my_directory<br />
</pre><br />
* '''setup_jlab-[run period].csh:''' The environment that is sourced at the beginning of the job execution. <br />
* '''status.sh:''' Updates the job status database table, and prints some of its columns to screen.<br />
<br />
=== Project Management ===<br />
<br />
* Delete (if any) and create the database table(s) for the current set of job submissions:<br />
<pre><br />
./clear.sh<br />
</pre><br />
Also, if testing was done with jobs, it is best to delete the output directory and the configuration files:<br />
<pre><br />
rm -frv /volatile/halld/offline_monitoring/RunPeriod-20YY-MM/verVV /group/halld/data_monitoring/run_conditions/soft_comm_20YY_MM_verVV.xml /group/halld/data_monitoring/run_conditions/jana_rawdata_comm_20YY_MM_verVV.conf<br />
</pre><br />
<br />
* Search for input files matching the string in the .jproj file, and create a row for each in the job management database table (called <project_name>). You can test by adding an optional argument at the end, which only selects files with a specific file number:<br />
<pre><br />
jproj.pl <project_name> update <optional_file_number><br />
</pre><br />
<br />
* Confirm that the job management database is accurate by printing it's contents to screen:<br />
<pre><br />
mysql -hhallddb -ufarmer farming -e "select * from <project_name>"<br />
</pre><br />
<br />
* ONLY if a mistake was made, to delete the tables from the database and recreate new, empty ones, run: <br />
<pre><br />
./clear.sh<br />
</pre><br />
<br />
* To look at the status of the submitted jobs, first query auger and update the job status database:<br />
<pre><br />
fill_in_job_details.pl <project_name><br />
</pre><br />
<br />
* The job status can then be viewed by submitting a query to the job status database (called <project_name>Job (no space in between)):<br />
<pre><br />
mysql -hhallddb -ufarmer farming -e "select id,run,file,jobId,hostname,status,timeSubmitted,timeActive,walltime,cput,timeComplete,result,error from <project_name>Job"<br />
</pre><br />
<br />
* These last two commands can instead be executed simultaneously by running:<br />
<pre><br />
./status.sh<br />
</pre><br />
<br />
=== Handy mysql Instructions ===<br />
<br />
* Handy mysql instructions:<br />
<pre><br />
mysql -hhallddb -ufarmer farming # Enter the "farming" mysql database on "hallddb" as user "farmer"<br />
quit; # Exit mysql<br />
show tables; # Show a list of the tables in the current database<br />
show columns from <project_name>; # show all of the columns for the given table<br />
select * from <project_name>; # show the contents of all rows from the given table<br />
</pre><br />
<br />
=== Backing Up Offline Monitoring Tables ===<br />
Tables created for offline monitoring can be backed up using the script backup_tables.sh which<br />
can be checked out with the other files from https://halldsvn.jlab.org/repos/trunk/scripts/monitoring/jproj/projects<br />
<br />
The script uses the command mysqldump to print out a file that can be executed to recreate the tables.<br />
Since executing this output file will drop the table when it exists, caution is advised.<br />
Example usage to backup all three tables created for run period 2014_10 ver 17:<br />
<pre>backup_tables.sh 2014_10 17</pre><br />
<br />
== Running Over Data As It Comes In ==<br />
<br />
A special user gxproj1 will have a cron job set up to run the plugins as new data appears on /mss.<br />
During the week, gxproj1 will submit offline plugin jobs with the same setup as the weekly jobs<br />
run the previous Friday. The procedure for this is shown below.<br />
<br />
<!--<br />
=== Setting up the environment ===<br />
The file<br />
/home/gxproj1/setup_jlab.csh<br />
is sourced through .tcshrc.<br />
This file is the same as what is linked to by<br />
/home/gluex/setup_jlab_commissioning.csh,<br />
except HALLD_HOME, HDDS_HOME, and JANA_CALIB_URL are set separately so that this<br />
user can have a separate build.<br />
<br />
To obtain the builds from the previous Friday's runs,<br />
execute<br />
/home/gxproj1/halld/monitoring/newruns/setup_previous.sh [year] [month] [day]<br />
The build revisions from the previous Friday are archived in files<br />
/work/halld/data_monitoring/run_conditions/soft_comm_[year]_[month]_[day].xml<br />
and the script will build libraries based on those stored revision numbers.<br />
--><br />
<br />
=== Running the cron job ===<br />
<br />
'''IMPORTANT:''' The cron job should not be running while you are manually submitting jobs using the jproj.pl script for the same project, or else you will probably multiply-submit a job. <br />
<br />
* Go to the cron job directory: <br />
<pre><br />
cd /u/home/gxproj1/halld/monitoring/newruns<br />
</pre><br />
<br />
* The cron_plugins file is the cronjob that will be executed. During execution, it runs the exec.sh command in the same folder. This command takes two arguments: the project name, and the maximum file number for each run. These fields should be updated in the cron_plugins file before running. <br />
<br />
* The exec.sh command updates the job management database table with any data that has arrived on tape since it was last updated, ignoring file numbers greater than the maximum file number. It then submits jobs for these files. <br />
<br />
* To start the cron job, run:<br />
<pre><br />
crontab cron_plugins<br />
</pre><br />
<br />
* To check whether the cron job is running, do<br />
<pre><br />
crontab -l<br />
</pre><br />
<br />
* To remove the cron job do<br />
<pre><br />
crontab -r<br />
</pre><br />
<br />
<!--<br />
The cron job will run the script scan_for_jobs.sh,<br />
which runs generatejobs_plugins_rawdata.sh for any<br />
new runs that it had not seen before. All previous<br />
runs are recorded in the file filelists/files_current.txt<br />
so clear this to run over runs, or set the parameters<br />
MINRUN and MAXRUN which will set the range of runs submitted.<br />
--><br />
<br />
==Post-Processing Procedures==<br />
<br />
To visualize the monitoring data, we save images of selected histograms and store time series of selected quantities in a database, which are then displayed on a web page. This section describes how to generate the monitoring images and database information.<br />
<br />
The scripts used to generate this summary data are primarily run from /home/gxprojN/halld/monitoring/process . If you want a new copy of the scripts, e.g., for a new monitoring run, you should check the scripts out from SVN:<br />
<syntaxhighlight><br />
svn co https://halldsvn.jlab.org/repos/trunk/scripts/monitoring/process<br />
</syntaxhighlight><br />
Note that these scripts currently have some parameters which must be periodically set by hand.<br />
<br />
The default python version on most JLab machine does not have the modules to allow these scripts to connect to the MySQL database. To run these scripts, load the environment with the following command<br />
<syntaxhighlight><br />
source /home/gxproj1/halld/monitoring/process/monitoring_env.csh<br />
</syntaxhighlight><br />
<br />
===Online Monitoring===<br />
<br />
There are two primary scripts for running over the monitoring data generated by the online system and offline reconstruction. The online script can be run with either of the following commands:<br />
<syntaxhighlight><br />
/home/gluex/halld/monitoring/process/check_new_runs.py<br />
<br />
OR <br />
<br />
/home/gluex/halld/monitoring/process/check_new_runs.csh<br />
</syntaxhighlight><br />
The shell script sets up the environment properly to run the python script. To connect to the monitoring database on the JLab CUE, modules included in the local installation of python >= 2.7 are needed. The shell script is appropriate to use in a cron job. The cronjob is currently run under the "gluex" account.<br />
<br />
The online monitoring system copies a ROOT file containing the results of the online monitoring, and other configuration files into a directory accessible outside the counting house. This python script automatically checks for new ROOT files, which it will then automatically process. It contains several configuration variables that must be correctly set, which contains the location of input/output directories, etc...<br />
<br />
===Offline Monitoring===<br />
<br />
After the data is run over, the results should be processed, so that summary data is entered into the monitoring database and plots are made for the monitoring webpages. Currently, this processing is controlled by a cronjob that runs the following script:<br />
<syntaxhighlight><br />
/home/gxproj1/halld/monitoring/process/check_monitoring_data.csh <br />
</syntaxhighlight><br />
This script checks for new ROOT files, and only runs over those it hasn't processed yet. Since one monitoring ROOT file is produced for each EVIO file, whenever a new file is produced, the plots for the corresponding run are recreated and all the ROOT files for a run are combined into one file. Information is stored in the database on a per-file basis. <br />
<br />
Plots for the monitoring web page can be made from single histograms or multiple histograms using RootSpy macros. If you want to change the list of plots made, you must modify one of the following files:<br />
* histograms_to_monitor - specify either the name of the histogram or its the full ROOT path <br />
* macros_to_monitor - specify the full path to the RootSpy macro .C file<br />
<br />
When a new monitoring run is started, or the conditions are changed, the following steps should be taken to process the new files:<br />
# Add a new data version, as described below:<br />
# Change the following parameters in check_monitoring_data.csh:<br />
## JOBDATE should correspond to the ouptut date used by the job submission script<br />
## OUTPUTDIR should correspond to the directory corresponding to the run period and revision corresponding to the new version you just submitted. Presumably, this directory will be empty at the beginning.<br />
## Once you create a new data version as defined below, you should pass the needed information as a command line option. Currently this is done by the ARGS variable. For example, the argument "-v RunPeriod-2014-10,8" tells the monitoring scripts to look up the version corresponding to revision 8 of RunPeriod-2014-10 in the monitoring DB and to use to store the results.<br />
<br />
<syntaxhighlight><br />
Example configuration parameters:<br />
set JOBDATE=2015-01-09<br />
set INPUTDIR=/volatile/halld/RunPeriod-2014-10/offline_monitoring<br />
set OUTPUTDIR=/w/halld-scifs1a/data_monitoring/RunPeriod-2014-10/ver08<br />
set ARGS=" -v RunPeriod-2014-10,8 "<br />
</syntaxhighlight><br />
If you want to process the results manually, the data is processed using the following script:<br />
<syntaxhighlight><br />
./process_new_offline_data.py <input directory> <output directory><br />
<br />
EXAMPLE:<br />
<br />
./process_new_offline_data.py 2014-11-14 /volatile/halld/RunPeriod-2014-10/offline_monitoring/ /w/halld-scifs1a/data_monitoring/RunPeriod-2014-10/ver02<br />
</syntaxhighlight><br />
The python script takes several options to enable/disable various steps in the processing. Of interest is the "--force" option, which will run over all monitoring ROOT files, whether or not they've been previously identified.<br />
<br />
Every time a new reconstruction pass is performed, a new version number must be generated. To do this, prepare a version file as described below. Then run the register_new_version.py script to store the information in the database. The script will return a version number, which then should be set by hand in process_new_offline_data.py - future versions of the script will streamline this part of the procedure. An example of how to generate a new version is:<br />
<syntaxhighlight><br />
./register_new_version.py add /home/gxproj1/halld/monitoring/process/versions/vers_RunPeriod-2014-10_pass1.txt<br />
</syntaxhighlight><br />
<br />
<b>If you are running the offline monitoring by checking out the files in trunk/scripts/monitoring/jproj/projects/<br />
of the svn repository</b>, and created a project with<br />
create_project.sh [project name] hd_rawdata<br />
Then go to the directory [project name]/processing/<br />
and execute<br />
./run_processing.sh<br />
which will run register_new_version.py as well as check_monitoring_data.csh for that project.<br />
<br />
===Step-by-Step Instructions For Processing a New Monitoring Run===<br />
<br />
The monitoring runs are current run out of the gxproj1 and gxproj5 accounts. After an offline monitoring run has been successfully started on the batch farm, the following steps should be followed to setup the post-processing for these runs.<br />
<br />
# The post-processing scripts are stored in $HOME/halld/monitoring/process and are automatically run by cron.<br />
# Run "svn update" to bring any changes in. Be sure that the list of histograms and macros to plot are current.<br />
# Add a new data version<br />
# Edit check_monitoring_data.csh to point to the current revisions/directories<br />
#* VERSION<br />
#* ARGS<br />
#* Note that the environment depends on a standard script - $HOME/setup_jlab.csh<br />
# Update files in the web directory, so that the results are displayed on the web pages: /group/halld/www/halldweb/html/data_monitoring/textdata<br />
# Copy the REST files to more permanent locations:<br />
#* cp -a /volatile/halld/offline_monitoring/RunPeriod-YYYY-MM/verVV/REST /work/halld/data_monitoring/RunPeriod-YYYY-MM/REST/verVV<br />
#* cp -a /volatile/halld/offline_monitoring/RunPeriod-YYYY-MM/verVV/REST /cache/halld/RunPeriod-YYYY-MM/REST/verVV [under testing]<br />
<br />
Check log files in $HOME/halld/monitoring/process/log for more information on how each run went. If there are problems, check log files, and modify check_monitoring_data.csh to vary the verbosity of the output.<br />
<br />
==Data Versions==<br />
<br />
To document the conditions of the monitoring data that is created, for the sake of reproducability and further analysis we save several pieces of information. The format is intended to be comprehensive enough to document not just monitoring data, but versions of raw and reconstructed data, so that this database table can be used for the event database as well.<br />
<br />
We store one record per pass through one run period, with the following structure:<br />
<br />
{| class="wikitable"<br />
! Field !! Description<br />
|-<br />
| data_type || The level of data we are processing. For the purposes of monitoring, "rawdata" is the online monitoring, "recon" is the offline monitoring<br />
|- <br />
| run_period || The run period of the data<br />
|- <br />
| revision || An integer specifying which pass through the run period this data corresponds to<br />
|- <br />
| software_version || The name of the XML file that specifies the different software versions used<br />
|-<br />
| jana_config || The name of the text file that specifies which JANA options were passed to the reconstruction program<br />
|-<br />
| ccdb_context || The value of JANA_CALIB_CONTEXT, which specifies the version of calibration constants that were used<br />
|-<br />
| production_time || The data at which monitoring/reconstruction began<br />
|-<br />
| dataVersionString || A convenient string for identifying this version of the data<br />
|}<br />
<br />
<br />
An example file used as as input to ./register_new_version.py is:<br />
<syntaxhighlight><br />
data_type = recon<br />
run_period = RunPeriod-2014-10<br />
revision = 1<br />
software_version = soft_comm_2014_11_06.xml<br />
jana_config = jana_rawdata_comm_2014_11_06.conf<br />
ccdb_context = calibtime=2014-11-10<br />
production_time = 2014-11-10<br />
dataVersionString = recon_RunPeriod-2014-10_20141110_ver01<br />
</syntaxhighlight></div>Kmoriyahttps://halldweb.jlab.org/wiki/index.php?title=Data_Monitoring_Procedures&diff=71854Data Monitoring Procedures2015-12-11T15:38:22Z<p>Kmoriya: /* Post-analysis of statistics of the launch */</p>
<hr />
<div>__TOC__<br />
<br />
== Master List of File / Database / Webpage Locations ==<br />
=== Run Conditions ===<br />
* Online Run-by-run condition files (B-field, current, etc.): /work/halld/online_monitoring/conditions/<br />
* Offline monitoring run conditions (software versions, jana config): /group/halld/data_monitoring/run_conditions/<br />
*[http://www.jlab.org/Hall-D/test/RunInfo/ Run Info vers. 1]<br />
*[https://halldweb.jlab.org/cgi-bin/data_monitoring/run_conditions.pl Run Info vers. 2]<br />
<br />
=== Monitoring Output Files ===<br />
* Run Periods 201Y-MM is for example 2015-03, launch ver verVV is for example ver15<br />
* Online monitoring histograms: /work/halld/online_monitoring/root/<br />
* Offline monitoring histogram ROOT files: /work/halld/data_monitoring/RunPeriod-201Y-MM/verVV/rootfiles<br />
* REST files (most recent launch only): /work/halld/data_monitoring/RunPeriod-201Y-MM/REST/verVV<br />
* individual files for each job (ROOT, log, etc.): /volatile/halld/offline_monitoring/RunPeriod-201Y-MM/verVV/<br />
<br />
=== Monitoring Database ===<br />
* Accessing monitoring database (on ifarm): mysql -u datmon -h hallddb.jlab.org data_monitoring<br />
<br />
=== Monitoring Webpages ===<br />
*[https://halldweb.jlab.org/cgi-bin/data_monitoring/monitoring/plotBrowser.py Plot Browser]<br />
*[https://halldweb.jlab.org/cgi-bin/data_monitoring/monitoring/runBrowser.py Run Browser]<br />
*[https://halldweb.jlab.org/cgi-bin/data_monitoring/monitoring/timeSeries.py Time Series]<br />
*[https://halldweb.jlab.org/data_monitoring/launch_analysis/ Launch Analysis]<br />
<br />
== Job Monitoring Links ==<br />
* [http://scicomp.jlab.org/farm2/job.html Completed Job History]<br />
* [http://scicomp.jlab.org/farm2/project.html Job Stats By Project]<br />
* [http://scicomp.jlab.org/farm2/trackOrg.html Job Stats By Track]<br />
* [http://scicomp.jlab.org/farm2/report.html Cluster Report]<br />
* [http://scicomp.jlab.org/farm2/walltime.html Walltime Distribution]<br />
<br />
== Saving Online Monitoring Data ==<br />
<br />
The procedure for writing the data out is given in, e.g.,<br />
[https://halldweb.jlab.org/wiki/index.php/Raid-to-Silo_Transfer_Strategy Raid-to-Silo Transfer Strategy].<br />
<br />
Once the DAQ writes out the data to the raid disk, cron jobs will copy the file to tape,<br />
and within ~20 min., we will have access to the file on tape at<br />
/mss/halld/$RUN_PERIOD/rawdata/RunXXXXXX.<br />
<br />
All online monitoring plugins will be run as data is taken.<br />
They will be accessible within the counting house via RootSpy, and<br />
for each run and file, a ROOT file containing the histograms will be saved<br />
within a subdirectory for each run.<br />
<br />
For immediate access to these files, the raid disk files may be accessed directly<br />
from the counting house, or the tape files will be available within ~20 min. of the<br />
file being written out.<br />
<br />
== Offline Monitoring: Running Over Archived Data ==<br />
<br />
Once files are written to tape we run the online plugins on these files to confirm what we were seeing in the online monitoring, and also to update the results from the latest calibration and software. <br />
Manual scripts and cron jobs are set up to look for new data and run the plugins over a sample of files.<br />
<br />
Every other Friday (usually the Friday before the offline meetings) jobs will be started to run the newest software on all previous runs,and allows everybody to see improvements in each detector.<br />
For each launch, independent builds of hdds, sim-recon, the monitoring plugins, and an sqlite file will be generated.<br />
<br />
Below the procedures are described for<br />
# Preparing the software for the launch<br />
# Starting the launch (using hdswif)<br />
# Post-analysis of statistics of the launch<br />
<br />
Processing the results and making them available to the collaboration<br />
is handled in the section [[#Post-Processing_Procedures | Post-Processing Procedures]] below.<br />
<br />
<!--<br />
==== Generating an offline plugin job ====<br />
<br />
The user gxproj1 should be used for official offline monitoring jobs.<br />
Within /home/gxproj1/halld/monitoring/batch/ there will be scripts to run the online monitoring plugins over tape files.<br />
The main script is generatejobs_plugins_rawdata.sh, which can be used as<br />
generatejobs_plugins_rawdata.sh [minrun] [maxrun] (minfile) (maxfile)<br />
where minrun, maxrun specify the range of the run #, and minfile, maxfile (optional) specify<br />
the file range for that run.<br />
<br />
This will generate a script run_rawdata_XXXXXX.sh, where the run # has now been formatted to be 6 digits.<br />
Executing this script will send the monitoring plugins job to the Auger batch system.<br />
<br />
There is also a script clean.sh which can be used as<br />
clean.sh XXX<br />
This will clean up all associated files created in association with run XXX.<br />
<br />
Internally, the xml file used to submit the job will be created, and the job to run<br />
will be given within script.sh. All run parameters should be specified in at the beginning<br />
of generatejobs_plugins_rawdata.sh<br />
Since we are running on tape, the tape file will first be copied over to the cache disk, and the job will run<br />
over this cached file.<br />
<br />
==== Using cron to run automatically ====<br />
Within /home/gluex/halld/monitoring/cron/ there is a file cron_plugins<br />
that can be executed via<br />
crontab cron_plugins<br />
This will set up a cron job to call the script scan_for_jobs.sh, which will<br />
check in the rawdata directory and call generatejobs_plugins_rawdata.sh for<br />
any run that is more than 5 min old. The cron job is set up to run every 10 min.<br />
<br />
==== Magnetic Field Settings ====<br />
For tracking, it is necessary to set the correct magnetic field settings.<br />
The below table is now obsolete. Please refer to the [https://halldweb.jlab.org/cgi-bin/data_monitoring/run_conditions.pl GlueX Run Conditions] webpage.<br />
<br />
The actual field settings for each run have not been documented well.<br />
Below are values based on going through entries in the [https://logbooks.jlab.org/book/halld halld log].<br />
<br />
{| border="1" cellpadding="1" style="text-align: center;"<br />
!width="100"| Run #s<br />
!width="150"| Solenoid Current (A)<br />
!width="300"| JANA option<br />
!width="300"| notes<br />
|-<br />
| 940 - 996 || 1000 || -PBFIELD_MAP=Magnets/Solenoid/solenoid_1000A_poisson_20141104 || At run 997, the solenoid started ramping down<br />
|-<br />
| 998 - 1448 || 0 || -PBFIELD_TYPE=NoField -PDEFTAG:DTrackCandidate=StraightLine || Solenoid started ramping up around run 1431. Runs 1432 - 1448 should not be used for tracking. See [https://logbooks.jlab.org/entry/3309039 this] and [https://logbooks.jlab.org/entry/3309092 this] entry.<br />
|-<br />
| 1449 - 1620 || 1200 || -PBFIELD_MAP=Magnets/Solenoid/solenoid_1200A_poisson_20140520 || <br />
|-<br />
|}<br />
<br />
'''Note:''' The following run ranges were taken with a 300A field, and could be analyzed with the magnetic field map Magnets/Solenoid/solenoid_300A_poisson_20140819 , more study is needed to see if the straight line track fitter is needed or not: 1036 - 1053, 1065 - 1121, 1212? - 1254, 1309 - 1318 [1307,1308,1319-1329 were taken while the magnet was ramping].<br />
--><br />
<br />
=== General Information on Procedures ===<br />
This section explains how the offline monitoring should be run.<br />
Since we may want to simultaneously run offline monitoring for different run periods that require<br />
different environment variables, the scripts are set up so that a generic user can download the<br />
scripts and run them from anywhere. Most output directories for offline monitoring are created<br />
with group read/write permissions so that any Hall D group user has access to the contents,<br />
but there are some cases where use of the account that created the launch is necessary.<br />
<br />
The accounts used for offline monitoring are the gxprojN accounts created and maintained by<br />
Mark Ito (see [https://halldweb.jlab.org/wiki/index.php/GlueX-related_shared_accounts_on_the_JLab_CUE here] for how each account is used).<br />
As of October 2015, the following are used:<br />
* gxproj1 for running over Fall 2014 data (deprecated since June 2015)<br />
* gxproj5 for running over Spring 2015 data<br />
<br />
Since the summer of 2015 we have transitioned from a system using Mark Ito's jproj scripts<br />
to integrating the swif system that Chris Larrieu (SciComp) has been developing.<br />
For offline monitoring, the hdswif system that Kei developed is used for launching the jobs, and a new cross analysis<br />
system based on MySQL and Python is maintained. The jproj system is now deprecated for offline monitoring.<br />
<br />
<br />
The scripts for hdswif, cross analysis and jproj are maintained in svn:<br />
* hdswif: https://halldsvn.jlab.org/repos/trunk/scripts/monitoring/hdswif<br />
* cross analysis: https://halldsvn.jlab.org/repos/trunk/scripts/monitoring/cross_analysis<br />
* jproj : https://halldsvn.jlab.org/repos/trunk/scripts/monitoring/jproj<br />
<br />
To run the offline monitoring each package should be checked out and all necessary<br />
scripts are included.<br />
<br />
=== Preparing the software for the launch ===<br />
<br />
1. Setup the environment: <pre>source ~/env_monitoring_launch</pre><br />
<br />
2. Updating & building hdds: <br />
<pre><br />
cd $HDDS_HOME<br />
git pull # Get latest software<br />
scons -c install # Clean out the old install: EXTREMELY IMPORTANT for cleaning out stale headers<br />
scons install -j4 # Rebuild and re-install with 4 threads<br />
</pre><br />
<br />
3. Updating & building sim-recon: <br />
<pre><br />
cd $HALLD_HOME/src<br />
git pull # Get latest software<br />
scons -c install # Clean out the old install: EXTREMELY IMPORTANT for cleaning out stale headers<br />
scons install -j4 # Rebuild and re-install with 4 threads<br />
</pre><br />
<br />
4. Create a new sqlite file containing the very latest calibration constants. Original documentation on creating sqlite files are [https://halldweb.jlab.org/wiki/index.php/SQLite-form_of_the_CCDB_database here]. <br />
<pre><br />
cd $GLUEX_MYTOP/../sqlite/<br />
$CCDB_HOME/scripts/mysql2sqlite/mysql2sqlite.sh -hhallddb.jlab.org -uccdb_user ccdb | sqlite3 ccdb.sqlite<br />
mv ccdb.sqlite ccdb_monitoring_launch.sqlite #replacing the old file<br />
</pre><br />
<br />
5. Note that the above steps must be done <b>BEFORE</b> launch project creation. This is because we will track the revisions of the libraries used, and this is done by extracting the svn information in each directory. Also, note that the system assumes that we have the topmost build directory (usually called GLUEX_TOP) to be $HOME/builds . Such an assumption is necessary to be able to extract information about the library locations automatically.<br />
<br />
Create the appropriate project(s) and submit the jobs using hdswif, as detailed in the section below.<br />
<br />
=== Starting the Launch and Submitting Jobs ===<br />
<br />
Until the summer of 2015 we relied solely on Mark Ito's jproj system for submitting and keeping track of jobs. We have since moved to the swif system and use the hdswif wrapper for this. Below are instructions for how to use these.<br />
<br />
1. Downloading hdswif: Download the hdswif directory from svn. For the gxprojN accounts, use the directory ~/halld/hdswif. <br />
<pre><br />
cd ~/halld<br />
svn co https://halldsvn.jlab.org/repos/trunk/scripts/monitoring/hdswif<br />
cd hdswif<br />
</pre><br />
<br />
2. Edit the job config file, input.config, which is used to register jobs in hdswif. A typical config file will look this:<br />
PROJECT gluex<br />
TRACK reconstruction<br />
OS centos65<br />
NCORES 6<br />
DISK 40<br />
RAM 8<br />
TIMELIMIT 8<br />
JOBNAMEBASE offmon_<br />
RUNPERIOD 2015-03<br />
VERSION 15<br />
OUTPUT_TOPDIR /volatile/halld/offline_monitoring/RunPeriod-[RUNPERIOD]/ver[VERSION] # Example of other variables included in variable<br />
SCRIPTFILE /home/gxproj5/halld/hdswif/script.sh # Must specify full path<br />
ENVFILE /home/gxproj5/halld/hdswif/setup_jlab-2015-03.csh # Must specify full path<br />
<br />
3. Creating the workflow: Within SWIF jobs are registered into workflows. For offline monitoring, the workflow names are of the form <b>offline_monitoring_RunPeriod201Y_MM_verVV_hd_rawdata</b> with suitable replacements for the run period and version number. The command "swif list" will list all existing workflows. Also, for most simple SWIF commands hdswif also provides a wrapper. <br />
swif list<br />
<br />
For creation of workflows for offline monitoring the command:<br />
hdswif.py create [workflow] -c input.config<br />
should be used. When a config file (here: input.config) is passed in, hdswif will automatically create files that record the configuration of the current launch. These files are stored as for example:<br />
/group/halld/data_monitoring/run_conditions/jana_rawdata_comm_2015_03_ver15.conf<br />
/group/halld/data_monitoring/run_conditions/soft_comm_2015_03_ver15.xml<br />
<br />
The software packages stored in git (sim-recon and hdds) can have git tags applied to them, which makes it easier to find versions of the software than a SHA-1 hash. hdswif will ask if you would like to create a tag, and execute the following sequence: <pre>git tag -a offmon-201Y_MM-verVV -m "Used for offline monitoring 201Y-MM verVV started on 201y/mm/dd"</pre> <pre>git push offmon-201Y_MM-verVV</pre> This will only be invoked when the user name is gxprojN, and for the configuration files, the output directory will be /group/halld/data_monitoring/run_conditions/ for gxprojN accounts while it will be the current directory for other users.<br />
* To use the git-tagged software versions do for example <pre>cd $HALLD_HOME</pre> <pre>git checkout offmon-2015_03-ver15</pre><br />
<br />
4. Registering jobs in the workflow: To register jobs within the workflow, hdswif provides the use of config files. Jobs can be registered by specifying the workflow, config file (-c), run (-r) and file (-f) numbers if necessary. <br />
<span style="color:red">Note: Job configuration parameters can be set differently for jobs within the same workflow if necessary.</span><br />
Jobs can be added via <pre>hdswif.py add [workflow] -c input.config</pre><br />
By default, hdswif will add all files found within the directory /mss/halld/RunPeriod-201Y-MM/rawdata/ where 201Y-MM is specified by the RUNPERIOD parameter in the config file. If only some of the runs or files are needed, these can be specified for example with<br />
<pre>hdswif.py add [workflow] -c input.config -r 3180 -f '00[0-4]'</pre><br />
to specify to register running only over run 3180 files 000 - 004 (Unix-style brackets and wildcards can be used).<br />
<br />
5. Running the workflow: To run the workflow, simply use the hdswif wrapper:<br />
hdswif.py run [workflow]<br />
<br />
<b>It is recommended that some jobs be tested to make sure that everything is working rather than fail thousands of jobs.</b><br><b>Check the configuration setup when creating a workflow, and what is in script.sh. Also check the following:<br />
* Check stderr files. Are they small (<kB)?<br />
* Check stdout files. Are they very large (>MB)?<br />
* Check output ROOT files. Are they larger than several MB?<br />
* Check output REST files. Are they larger than several tens of MB?<br />
</b><br />
<br> For this purpose, hdswif will take an additional parameter to run which limits the number of jobs to submit: <pre>hdswif.py run [workflow] 10</pre><br />
in which case only 10 jobs will be submitted.<br />
<br />
To submit all jobs after checking the results, do <pre>hdswif.py run [workflow]</pre><br />
<br />
=== Checking the Status and Resubmitting ===<br />
1. The status of jobs can be checked on the terminal with <pre>jobstat -u gxprojN</pre> For the status of jobs on Auger see http://scicomp.jlab.org/scicomp/#/auger/jobs and for SWIF use <pre>swif list</pre> or for more information, <pre>swif status [workflow] -summary</pre> Note that "swif status" tends to be out of date sometimes, so don't panic if your workflow/jobs aren't showing up right away. Also see the Auger [http://scicomp.jlab.org/scicomp/#/auger/jobs job website].<br />
<br />
2. For failed jobs, SWIF can resubmit jobs based on the problem. For resubmission for failed jobs with the same resources, <pre>swif retry-jobs [workflow] -problems [problem name]</pre> can be used, and for jobs to be submitted with more resources, e.g., use <pre>swif modify-jobs [workflow] -ram add 2gb -problems AUGER-OVER_RLIMIT</pre> <br />
This only re-stages the jobs, be sure to resubmit them with:<br />
<pre>swif run -workflow [workflow] -errorlimit none</pre><br />
<br />
hdswif has a wrapper for both of these: <pre>hdswif.py resubmit [workflow] [problem]</pre> In this case [problem] can be one of <b>SYSTEM, TIMEOUT, RLIMIT</b>. If SYSTEM is specified, the jobs will be retried. For TIMEOUT and RLIMIT, the jobs will be modified by default with 2 additional hours or GB of RAM. If one more number is added as an option, then that many hours or GB of RAM will be added., e.g., <pre>hdswif.py resubmit [workflow] TIMEOUT 5</pre> will add 5 hours of processing time. You can wait until almost all jobs finish before resubmitting failed jobs since the number should be relatively small. <b>Even if jobs are resubmitted for one type of failure, jobs that later fail with that failure will not be automatically resubmitted.</b><br />
<br />
3. For information on swif, use the "swif help" commands and for hdswif see the attached documentaion in https://halldsvn.jlab.org/repos/trunk/scripts/monitoring/hdswif/manual_hdswif.pdf<br />
<br />
4. Below is a table describing the various errors that can occur.<br />
{| border="1" cellpadding="0" valign="left" style="text-align: left;"<br />
!width="150"| ERROR NAME<br />
!width="400"| Description<br />
!width="400"| Resolution<br />
!width="400"| hdswif command<br />
|-<br />
| AUGER-SUBMIT<br />
||<br />
SWIF’s attempt to submit jobs to Auger failed. Includes server-side problems as well as user failing to provide valid job parameters (e.g. incorrect project name, too many resources, etc.)<br />
||<br />
If requested resources are known to be correct resubmit. Otherwise modify job resources using swif directly.<br />
||<br />
<span style="color:purple"><b>hdswif.py resubmit [workflow] SYSTEM</b></span><br />
|-<br />
| AUGER-FAILED<br />
||<br />
Auger reports the job FAILED with no specific details.<br />
||<br />
Resubmit jobs. If problem persists, contact Chris Larrieu or SciComp.<br />
||<br />
<span style="color:purple"><b>hdswif.py resubmit [workflow] SYSTEM</b></span><br />
|-<br />
| AUGER-OUTPUT-FAIL<br />
||<br />
Failure to copy one or more output files.Can be due to permission problem, quota problem, system error, etc.<br />
||<br />
Check if output files will exist after job execution and that output directory exists, resubmit jobs. If problem persists, contact Chris Larrieu or SciComp.<br />
||<br />
<span style="color:purple"><b>hdswif.py resubmit [workflow] SYSTEM</b></span><br />
|-<br />
| AUGER-INPUT-FAIL<br />
||<br />
Auger failed to copy one or more of the requested input files, similar to output failures. Can also happen if tape file is unavailable (e.g. missing/damaged tape)<br />
||<br />
Check if input file exists, resubmit jobs. If problem persists, contact Chris Larrieu or SciComp.<br />
||<br />
<span style="color:purple"><b>hdswif.py resubmit [workflow] SYSTEM</b></span><br />
|-<br />
| AUGER-TIMEOUT<br />
||<br />
Job timed out.<br />
||<br />
If more time is needed for job add more resources.<br />
Default is to add 2 hrs of processing time. Also check whether code is hanging.<br />
||<br />
<span style="color:red"><b>hdswif.py resubmit [workflow] TIMEOUT</b></span><br><br />
Default is to add 2 hours. Optionally specify number of hours at end.<br />
|-<br />
| AUGER-OVER_RLIMIT<br />
||<br />
Not enough resources, RAM or disk space.<br />
||<br />
Add more resources for job.<br />
||<br />
<span style="color:blue"><b>hdswif.py resubmit [workflow] RLIMIT</b></span><br><br />
Default is to add 2 GB of RAM. Optionally specify GB at end. To add more disk space use SWIF directly.<br />
|-<br />
| SWIF-MISSING-OUTPUT<br />
||<br />
Output file specified by user was not found.<br />
||<br />
Check if output file exists at end of job.<br />
||<br />
<br />
|-<br />
| SWIF-USER-NON-ZERO <br />
||<br />
User script exited with non-zero status code.<br />
||<br />
Your script exited with non-zero status. Check the code you are running.<br />
||<br />
<br />
|-<br />
| SWIF-SYSTEM-ERROR <br />
||<br />
Job failed owing to a problem with swif (e.g. network connection timeout)<br />
||<br />
Resubmit jobs. If problem persists, contact Chris Larrieu or SciComp.<br />
||<br />
<span style="color:purple"><b>hdswif.py resubmit [workflow] SYSTEM</b></span><br />
|}<br />
<br style="clear:both;"/><br />
<br />
=== Post-analysis of statistics of the launch ===<br />
<br />
After jobs have been submitted, it will usually take a few days for all of the jobs to be processed.<br />
The next step is to check the resource usage for the current launch and publish the results online.<br />
<br />
# <b>Create summary XML, HTML files</b><br> The status and results of jobs are saved within the SWIF internal server, and are available via the command <pre>swif status [workflow] -summary -runs</pre> where the arguments -summary and -runs show summary statistics and statistics for individual jobs, respectively. hdswif has a command that takes this output in XML output and creates an HTML webpage showing results of the launch. To do this, do <pre>hdswif.py summary [workflow]</pre> This will create the XML output file from SWIF called swif_output_[workflow].xml and create a webpage containing png figure files. If the XML file already exists, hdswif will ask whether to overwrite the existing file.<br />
# <b>Publish output files online</b><br> At this stage the html output and figure files are created and ready to be put online. The html output capabilities of hdswif is useful for any user using SWIF, but since publication of the html output is specific to the offline monitoring, the script to do this is contained in the cross_analysis scripts directory at https://halldsvn.jlab.org/repos/trunk/scripts/monitoring/cross_analysis. For the gxprojN accounts this directory should exist as ~/halld/monitoring/cross_analysis. To publish the results online do for example <pre>python ~/halld/monitoring/cross_analysis/publish_offmon_results.py 2015_03 18</pre> The script copies the html output and corresponding figures to /group/halld/www/halldweb/html/data_monitoring/launch_analysis/ and also creates a link to it in the html page.<br />
# <b>Editing the summary HTML page</b><br> The top page for is offline monitoring https://halldweb.jlab.org/data_monitoring/launch_analysis/index.html and has links to the summary page for each run period. The summary files are /group/halld/www/halldweb/html/data_monitoring/launch_analysis/[run period].html Edit the file to<br />
## Add a new line to the first table which contains the version number, date, and comments for the current launch<br />
## Create a link to the webpage for the current launch. Simply copy and paste, modify the previous launch's link to have the correct launch ver.<br />
# <b>Freezing SWIF tables</b><br> Since we are now finished with the SWIF workflow that we used, the workflow should be "frozen" so that it cannot be mistakenly altered or modified. Do <pre>swif freeze [workflow]</pre><br />
# <b>Backing up SWIF output</b><br> With the workflow frozen we should be able to reproduce the XML output from SWIF, but we will backup the XML output just in case. Do <pre>cp ~/halld/hdswif/swif_output_[workflow].xml /group/halld/data_monitoring/swif_xml_backup/ </pre><br />
<br />
=== Cross Analysis of Launches ===<br />
<br />
The purpose of the cross analysis is to correlate how resource usage changed for the same files across different launches.<br />
To do this it is useful to create MySQL tables that contain information on each launch, and do queries across tables.<br />
<br />
#The scripts to do this are maintained in the svn directory https://halldsvn.jlab.org/repos/trunk/scripts/monitoring/cross_analysis Do <pre> svn co https://halldsvn.jlab.org/repos/trunk/scripts/monitoring/cross_analysis </pre> For the gxprojN accounts used for offline monitoring the directory should be ~/halld/monitoring/cross_analysis<br />
# The main script is run_cross_analysis.sh, which can be run with <pre>./run_cross_analysis.sh [RUNPERIOD] [VERSION]</pre>. However, <b>it is strongly recommended that the commands in this script be run by hand</b> to catch any errors.<br />
# Enter the python commands that are in run_cross_analysis.sh . Below are the steps and explanations:<br />
## Create a table for the current launch using create_cross_analysis_table.sh. The table will be created from the file template_table_schema.sql and contain columns id, run, file, timeChange, cpu_sec, wall_sec, mem_kb, vmem_kb, nevents, input_copy_sec, plugin_sec, final_state, problems<br />
## Run <pre>python fill_cross_analysis_entries.py [RUNPERIOD] [VERSION]</pre> The script will gather all of the necessary information either from SWIF output or the stdout files for the jobs<br />
## Run <pre>python create_stats_table_row.py</pre> This will loop over the jobs in the current launch and create a row in an HTML table that summarizes the statistics for the final state and problems for the jobs. This table row is then inserted into the main HTML webpage for the run period.<br />
## Run <pre>python create_stats_for_each_file.py [RUNPERIOD] [MINVERSION] [VERSION]</pre> This creates a comparison table of the final state and problems for each file between launches [MINVERSION] and [VERSION]. In the run_cross_analysis.sh script, the default is to set MINVERSION to be 15 for run period 2015_03, but as long as SWIF was used for all previous launches, any number will work.<br />
## Run <pre>python create_resource_correlation_plots.py [RUNVERSION] [CMPMINVERSION] [VERSION]</pre> This creates correlation plots of resource use between launches between CMPMINVERSION and VERSION. By default CMPMINVERSION is 5 launches earlier.<br />
<br />
== Explanation of Scripts ==<br />
Below are explanations of each script used in the offline monitoring system and a breif explanation of how they work.<br />
<br />
=== hdswif scripts ===<br />
<b>Summary:</b> hdswif.py is the main script that calls the other utility scripts. For the utility scripts, they can be run standalone by giving the appropriate parameters. Visual graphics are made using the PyROOT extension of ROOT. To use this, the environment variable PYTHONPATH must include ROOTSYS. If ROOTSYS has been set, adding it to PYTHONPATH will be done by the script, but if ROOTSYS is not set, then the scripts will abort.<br />
<br />
{| border="1" cellpadding="0" valign="left" style="text-align: left;"<br />
!width="150"| file name<br />
!width="800"| Description<br />
|-<br />
| hdswif.py ||<br />
Main script to control the behavior of SWIF. Most commands follow the form hdswif.py [command] [workflow] (options)<br />
|-<br />
| parse_swif.py ||<br />
Called within hdswif.py to create html output from SWIF results.<br />
|-<br />
| createXMLfiles.py ||<br />
* Called by the "create" option of hdswif.py<br />
* Creates XML files for logging information about launch. Must specify config file with option -c.<br />
* Also adds tags to git repositories of sim-recon and hdds.<br />
* For XML file creation, the file will be written out to /group/halld/data_monitoring/run_conditions/ if the user is gxprojN (used for offline monitoring). If not, to avoid general users adding things to the above directory, the output files will be the current directory.<br />
* To write out the versions of each package, environment variables such as HDDS_HOME will need to be set.<br />
* Versions for each software package are extracted using the directory structure, so if these are changed the scripts must change accordingly.<br />
* For each launch, <b>the output soft_comm_[RUNPERIOD]_ver[VER].xml file should be checked that all version numbers have been extracted.</b><br />
|-<br />
| read_config.py ||<br />
* Called by the "add" option of hdswif.py<br />
* Takes in config file name, optionally set verbose<br />
* Return a dictionary between config parameter names and values (e.g., 'PROJECT' : 'gluex', 'NCORES' : 6, ...)<br />
* Prints the config parameters at the end. For parameters changed from default, a '*' will be printed<br />
|-<br />
| output_job_details.py ||<br />
* Called by the "details" option of hdswif.py<br />
* Takes in workflow name, run and file. Run and file must be numbers, no wildcards<br />
* Finds ids for jobs specified by the run and file number and returns info on each one<br />
* Job info is retrieved from pbs farm system and shows configuration parameters for that job<br />
|-<br />
| results_by_resources.py ||<br />
* Called within parse_swif.py<br />
* Takes in XML output from SWIF and creates 2 plots<br />
* Creates html table showing results of jobs by resources requested for that job<br />
* This table is shown under "Status by Resources" in the output html file of hdswif.py summary [workflow]<br />
|-<br />
| create_ordered_hists.py ||<br />
* Called within parse_swif.py<br />
* Takes in XML output from SWIF and creates 2 plots<br />
* Plots are dependency time and pending time of each job, ordered by submission.<br />
* Different colors represent jobs submitted at different times. For jobs submitted at the same time, jobs are ordered in increasing time.<br />
|-<br />
| create_stacked_times.py ||<br />
* Called within parse_swif.py<br />
* Takes in XML output from SWIF and creates 2 plots<br />
* Plots show total job time divided into colors for different stages<br />
* One shows all jobs in order of Auger ID (roughly submission order), the other one shows in order of total job time<br />
* The jobs show at a glance which stage contributes how much of the job's time<br />
|}<br />
<br />
=== Utility scripts ===<br />
{| border="1" cellpadding="0" valign="left" style="text-align: left;"<br />
!width="150"| file name<br />
!width="800"| Description<br />
|-<br />
| stderr_by_size.py ||<br />
* Independent of all other scripts in hdswif directory<br />
* When diagnosing problems it is useful to check the stderr/stdout files. Frequently, different problems are easier to find based on the size of stderr files<br />
* Takes in run period and version as arguments, creates a directory /volatile/halld/offline_monitoring/RunPeriod-[RUNPERIOD]/ver[VER]/log/bysize that contains soft links to all stdout and stderr files from the specified launch, <b>in separate directories given by stderr file size</b>.<br />
|}<br />
<br />
=== cross_analysis scripts ===<br />
<b>Summary:</b> The script run_cross_analysis.sh is the main script. In principle, running this with <pre>./run_cross_analysis.sh [RUNPERIOD] [VERSION]</pre> should work, but it is recommended that the commands within the script<br />
be run by hand to catch any errors.<br />
<br />
{| border="1" cellpadding="0" valign="left" style="text-align: left;"<br />
!width="150"| file name<br />
!width="800"| Description<br />
|-<br />
| run_cross_analysis.sh ||<br />
* Main script to call other Python scripts.<br />
* Takes in run period and version as arguments and runs cross analysis<br />
|-<br />
| create_cross_analysis_table.sh ||<br />
* Creates MySQL table for current launch that is used by other scripts<br />
* Takes run period and version as input, and creates table cross_analysis_table_[RUNPERIOD]_ver[VERSION]<br />
|-<br />
| create_stats_table_row.py ||<br />
* Create a row in the html table showing the overall statistics of final states and problems for a launch<br />
* The final states are "Success", and "Segfault". "Success" includes all jobs that had problems but still finished with Success.<br />
* The problems are "Over Limit", "Timeout", and "System". If any of these occurred for any attempt of the job they are counted.<br />
* The script creates a new html table row for the current launch. This html snippet is inserted into the web-accessible file showing the results <br />
|-<br />
| create_stats_for_each_file.py ||<br />
* Create html tables showing the status of each file against different launch versions.<br />
* Takes in run period, min version, max version and shows final result and problems for all versions in between<br />
* Different final results and problems are shown by combinations of ext content, olor coding of text, background color<br />
|-<br />
|-<br />
| create_resource_correlation_plots.py ||<br />
* Create plots showing correlation of resources between different launches for each file.<br />
* Takes in run period, min version, version of interest. Points are shown only for files included in version of interest<br />
* Creates plots for CPU time, Wall time, memory, virtual memory, #events, difference in #events, time to copy input evio file, time to run plugin<br />
|}<br />
<br />
== Hall D Job Management System ==<br />
<br />
This section details instructions on how to create and launch a set of jobs using the Hall-D Job Management System developed by Mark Ito. These instructions are generic: this system can be used for the weekly monitoring jobs, but can also be used for other sets of job launches as well. <br />
<br />
=== Database Table Overview ===<br />
<br />
* Job management database table (<project_name>): For each input file, keeps track of whether or not a job for it has been submitted, along with other optional fields. <br />
<br />
* Job status database table (<project_name>Job (no space)): For each job, keeps track of the job-id, the job status, memory used, cpu & wall time, time taken to complete various stages (e.g. pending, dependency, active), and others. <br />
<br />
* Job metrics database table (<project_name>_aux (no space)): For each job, keeps track of the job-id, how many events were processed, the time it took to copy the cache file, and the time it took to run the plugin. This information is culled from the log files of each job, and is done within the analysis directory of each launch.<br />
<br />
=== Initialize Project Management ===<br />
<br />
* Log into the ifarm machine with one of the gxproj accounts. For this example we will use gxproj1.<br />
<pre><br />
ssh gxproj1@ifarm -Y<br />
</pre><br />
<br />
* Go to a directory to do the launch. In principle, any directory will work, but for gxproj1 this is usually done in /home/gxproj1/halld/<br />
<br />
* Check out the necessary scripts <pre>svn co https://halldsvn.jlab.org/repos/trunk/scripts/monitoring/jproj</pre> This will get all necessary scripts for launching. Once checked out, <pre>cd projects</pre><br />
<br />
* The script create_project.sh can be used to create a new project. It will take a single argument, the project name. It is assumed that the project name is of the form offline_monitoring_RunPeriod20YY_MM_verVV_hd_rawdata where this string will be parsed to give the run period 20YY_MM and version number VV. One thing to do BEFORE creation of a new project is to editthe conditions of the launch (plugins to run over, memory requested, disk space requested) within templates/template.jsub . This<br />
information is saved automatically at project creation time into files in /group/halld/data_monitoring/run_conditions .<br />
To create a project do <pre>./create_project.sh offline_monitoring_RunPeriod20YY_MM_verVV_hd_rawdata</pre><br />
The name has been chosen to be as consistent as possible with other directory structures.<br />
However, mysql requires that "-" be escaped in table names, so unfortunately run periods will be<br />
given as 2YYY_MM instead of 2YYY-MM.<br />
<br />
* A new directory with the project name (offline_monitoring_RunPeriod20YY_MM_verVV_hd_rawdata) and files will be copied and modified from the template directory to reflect the run period, the user, the directory that it was created in, project name, etc.<br />
<br />
* For each project, cd into the new directory<br />
* The script clear.sh will remove any existing tables for the current project name, then recreate it. Do<br />
<pre>./clear.sh</pre><br />
* To use the jproj.pl script that was checked in, add the directory to your path with<br />
<pre>source ../../scripts/setup.csh</pre><br />
or always specify the full path<br />
<pre>../../scripts/jproj.pl</pre><br />
* Now update the table of runs with<br />
<pre>jproj.pl <project name> update</pre><br />
This will fill the table with all files within /mss that are of the same form as what is in <project name>.jproj . If you want to register only on a subset of all such files, you can edit this file directly.<br />
* Once you have registered all of the files you would like to run over, do<br />
<pre>jproj.pl <project name> submit [max # of jobs] [run number]</pre><br />
where the additional options specify how many jobs to submit and which run number to run on. Without these options all files that are registered and have not been submitted yet will be submitted.<br />
<br />
At this stage you are ready to submit all files. It is a good idea to submit a few test jobs<br />
at first to check that all scripts are working and that the plugins do not crash. Once you are<br />
sure that this does not happen, you can send all jobs in. The remaining jobs are then the monitoring<br />
which will (among other things) put the results on the online webpage for the collaboration to view, and<br />
the analysis of the launch.<br />
<br />
=== Project File Overview ===<br />
<br />
An overview of each project file:<br />
* '''clear.sh:''' For the current project, deletes the job status and management database tables (if any), and creates new, empty ones. <br />
* '''<project_name>.jproj:''' Contains the path and file name format for the input files for the jobs. <br />
* '''<project_name>.jsub:''' The xml job submission script. The run number and file number variables are set during job submission for each input file. <br />
* '''script.sh:''' The script that is executed during the job. If output job directories are not pre-created manually, they should be created in this script with the proper permissions:<br />
<pre><br />
mkdir -p -m 775 my_directory<br />
</pre><br />
* '''setup_jlab-[run period].csh:''' The environment that is sourced at the beginning of the job execution. <br />
* '''status.sh:''' Updates the job status database table, and prints some of its columns to screen.<br />
<br />
=== Project Management ===<br />
<br />
* Delete (if any) and create the database table(s) for the current set of job submissions:<br />
<pre><br />
./clear.sh<br />
</pre><br />
Also, if testing was done with jobs, it is best to delete the output directory and the configuration files:<br />
<pre><br />
rm -frv /volatile/halld/offline_monitoring/RunPeriod-20YY-MM/verVV /group/halld/data_monitoring/run_conditions/soft_comm_20YY_MM_verVV.xml /group/halld/data_monitoring/run_conditions/jana_rawdata_comm_20YY_MM_verVV.conf<br />
</pre><br />
<br />
* Search for input files matching the string in the .jproj file, and create a row for each in the job management database table (called <project_name>). You can test by adding an optional argument at the end, which only selects files with a specific file number:<br />
<pre><br />
jproj.pl <project_name> update <optional_file_number><br />
</pre><br />
<br />
* Confirm that the job management database is accurate by printing it's contents to screen:<br />
<pre><br />
mysql -hhallddb -ufarmer farming -e "select * from <project_name>"<br />
</pre><br />
<br />
* ONLY if a mistake was made, to delete the tables from the database and recreate new, empty ones, run: <br />
<pre><br />
./clear.sh<br />
</pre><br />
<br />
* To look at the status of the submitted jobs, first query auger and update the job status database:<br />
<pre><br />
fill_in_job_details.pl <project_name><br />
</pre><br />
<br />
* The job status can then be viewed by submitting a query to the job status database (called <project_name>Job (no space in between)):<br />
<pre><br />
mysql -hhallddb -ufarmer farming -e "select id,run,file,jobId,hostname,status,timeSubmitted,timeActive,walltime,cput,timeComplete,result,error from <project_name>Job"<br />
</pre><br />
<br />
* These last two commands can instead be executed simultaneously by running:<br />
<pre><br />
./status.sh<br />
</pre><br />
<br />
=== Handy mysql Instructions ===<br />
<br />
* Handy mysql instructions:<br />
<pre><br />
mysql -hhallddb -ufarmer farming # Enter the "farming" mysql database on "hallddb" as user "farmer"<br />
quit; # Exit mysql<br />
show tables; # Show a list of the tables in the current database<br />
show columns from <project_name>; # show all of the columns for the given table<br />
select * from <project_name>; # show the contents of all rows from the given table<br />
</pre><br />
<br />
=== Backing Up Offline Monitoring Tables ===<br />
Tables created for offline monitoring can be backed up using the script backup_tables.sh which<br />
can be checked out with the other files from https://halldsvn.jlab.org/repos/trunk/scripts/monitoring/jproj/projects<br />
<br />
The script uses the command mysqldump to print out a file that can be executed to recreate the tables.<br />
Since executing this output file will drop the table when it exists, caution is advised.<br />
Example usage to backup all three tables created for run period 2014_10 ver 17:<br />
<pre>backup_tables.sh 2014_10 17</pre><br />
<br />
== Running Over Data As It Comes In ==<br />
<br />
A special user gxproj1 will have a cron job set up to run the plugins as new data appears on /mss.<br />
During the week, gxproj1 will submit offline plugin jobs with the same setup as the weekly jobs<br />
run the previous Friday. The procedure for this is shown below.<br />
<br />
<!--<br />
=== Setting up the environment ===<br />
The file<br />
/home/gxproj1/setup_jlab.csh<br />
is sourced through .tcshrc.<br />
This file is the same as what is linked to by<br />
/home/gluex/setup_jlab_commissioning.csh,<br />
except HALLD_HOME, HDDS_HOME, and JANA_CALIB_URL are set separately so that this<br />
user can have a separate build.<br />
<br />
To obtain the builds from the previous Friday's runs,<br />
execute<br />
/home/gxproj1/halld/monitoring/newruns/setup_previous.sh [year] [month] [day]<br />
The build revisions from the previous Friday are archived in files<br />
/work/halld/data_monitoring/run_conditions/soft_comm_[year]_[month]_[day].xml<br />
and the script will build libraries based on those stored revision numbers.<br />
--><br />
<br />
=== Running the cron job ===<br />
<br />
'''IMPORTANT:''' The cron job should not be running while you are manually submitting jobs using the jproj.pl script for the same project, or else you will probably multiply-submit a job. <br />
<br />
* Go to the cron job directory: <br />
<pre><br />
cd /u/home/gxproj1/halld/monitoring/newruns<br />
</pre><br />
<br />
* The cron_plugins file is the cronjob that will be executed. During execution, it runs the exec.sh command in the same folder. This command takes two arguments: the project name, and the maximum file number for each run. These fields should be updated in the cron_plugins file before running. <br />
<br />
* The exec.sh command updates the job management database table with any data that has arrived on tape since it was last updated, ignoring file numbers greater than the maximum file number. It then submits jobs for these files. <br />
<br />
* To start the cron job, run:<br />
<pre><br />
crontab cron_plugins<br />
</pre><br />
<br />
* To check whether the cron job is running, do<br />
<pre><br />
crontab -l<br />
</pre><br />
<br />
* To remove the cron job do<br />
<pre><br />
crontab -r<br />
</pre><br />
<br />
<!--<br />
The cron job will run the script scan_for_jobs.sh,<br />
which runs generatejobs_plugins_rawdata.sh for any<br />
new runs that it had not seen before. All previous<br />
runs are recorded in the file filelists/files_current.txt<br />
so clear this to run over runs, or set the parameters<br />
MINRUN and MAXRUN which will set the range of runs submitted.<br />
--><br />
<br />
==Post-Processing Procedures==<br />
<br />
To visualize the monitoring data, we save images of selected histograms and store time series of selected quantities in a database, which are then displayed on a web page. This section describes how to generate the monitoring images and database information.<br />
<br />
The scripts used to generate this summary data are primarily run from /home/gxprojN/halld/monitoring/process . If you want a new copy of the scripts, e.g., for a new monitoring run, you should check the scripts out from SVN:<br />
<syntaxhighlight><br />
svn co https://halldsvn.jlab.org/repos/trunk/scripts/monitoring/process<br />
</syntaxhighlight><br />
Note that these scripts currently have some parameters which must be periodically set by hand.<br />
<br />
The default python version on most JLab machine does not have the modules to allow these scripts to connect to the MySQL database. To run these scripts, load the environment with the following command<br />
<syntaxhighlight><br />
source /home/gxproj1/halld/monitoring/process/monitoring_env.csh<br />
</syntaxhighlight><br />
<br />
===Online Monitoring===<br />
<br />
There are two primary scripts for running over the monitoring data generated by the online system and offline reconstruction. The online script can be run with either of the following commands:<br />
<syntaxhighlight><br />
/home/gluex/halld/monitoring/process/check_new_runs.py<br />
<br />
OR <br />
<br />
/home/gluex/halld/monitoring/process/check_new_runs.csh<br />
</syntaxhighlight><br />
The shell script sets up the environment properly to run the python script. To connect to the monitoring database on the JLab CUE, modules included in the local installation of python >= 2.7 are needed. The shell script is appropriate to use in a cron job. The cronjob is currently run under the "gluex" account.<br />
<br />
The online monitoring system copies a ROOT file containing the results of the online monitoring, and other configuration files into a directory accessible outside the counting house. This python script automatically checks for new ROOT files, which it will then automatically process. It contains several configuration variables that must be correctly set, which contains the location of input/output directories, etc...<br />
<br />
===Offline Monitoring===<br />
<br />
After the data is run over, the results should be processed, so that summary data is entered into the monitoring database and plots are made for the monitoring webpages. Currently, this processing is controlled by a cronjob that runs the following script:<br />
<syntaxhighlight><br />
/home/gxproj1/halld/monitoring/process/check_monitoring_data.csh <br />
</syntaxhighlight><br />
This script checks for new ROOT files, and only runs over those it hasn't processed yet. Since one monitoring ROOT file is produced for each EVIO file, whenever a new file is produced, the plots for the corresponding run are recreated and all the ROOT files for a run are combined into one file. Information is stored in the database on a per-file basis. <br />
<br />
Plots for the monitoring web page can be made from single histograms or multiple histograms using RootSpy macros. If you want to change the list of plots made, you must modify one of the following files:<br />
* histograms_to_monitor - specify either the name of the histogram or its the full ROOT path <br />
* macros_to_monitor - specify the full path to the RootSpy macro .C file<br />
<br />
When a new monitoring run is started, or the conditions are changed, the following steps should be taken to process the new files:<br />
# Add a new data version, as described below:<br />
# Change the following parameters in check_monitoring_data.csh:<br />
## JOBDATE should correspond to the ouptut date used by the job submission script<br />
## OUTPUTDIR should correspond to the directory corresponding to the run period and revision corresponding to the new version you just submitted. Presumably, this directory will be empty at the beginning.<br />
## Once you create a new data version as defined below, you should pass the needed information as a command line option. Currently this is done by the ARGS variable. For example, the argument "-v RunPeriod-2014-10,8" tells the monitoring scripts to look up the version corresponding to revision 8 of RunPeriod-2014-10 in the monitoring DB and to use to store the results.<br />
<br />
<syntaxhighlight><br />
Example configuration parameters:<br />
set JOBDATE=2015-01-09<br />
set INPUTDIR=/volatile/halld/RunPeriod-2014-10/offline_monitoring<br />
set OUTPUTDIR=/w/halld-scifs1a/data_monitoring/RunPeriod-2014-10/ver08<br />
set ARGS=" -v RunPeriod-2014-10,8 "<br />
</syntaxhighlight><br />
If you want to process the results manually, the data is processed using the following script:<br />
<syntaxhighlight><br />
./process_new_offline_data.py <input directory> <output directory><br />
<br />
EXAMPLE:<br />
<br />
./process_new_offline_data.py 2014-11-14 /volatile/halld/RunPeriod-2014-10/offline_monitoring/ /w/halld-scifs1a/data_monitoring/RunPeriod-2014-10/ver02<br />
</syntaxhighlight><br />
The python script takes several options to enable/disable various steps in the processing. Of interest is the "--force" option, which will run over all monitoring ROOT files, whether or not they've been previously identified.<br />
<br />
Every time a new reconstruction pass is performed, a new version number must be generated. To do this, prepare a version file as described below. Then run the register_new_version.py script to store the information in the database. The script will return a version number, which then should be set by hand in process_new_offline_data.py - future versions of the script will streamline this part of the procedure. An example of how to generate a new version is:<br />
<syntaxhighlight><br />
./register_new_version.py add /home/gxproj1/halld/monitoring/process/versions/vers_RunPeriod-2014-10_pass1.txt<br />
</syntaxhighlight><br />
<br />
<b>If you are running the offline monitoring by checking out the files in trunk/scripts/monitoring/jproj/projects/<br />
of the svn repository</b>, and created a project with<br />
create_project.sh [project name] hd_rawdata<br />
Then go to the directory [project name]/processing/<br />
and execute<br />
./run_processing.sh<br />
which will run register_new_version.py as well as check_monitoring_data.csh for that project.<br />
<br />
===Step-by-Step Instructions For Processing a New Monitoring Run===<br />
<br />
The monitoring runs are current run out of the gxproj1 and gxproj5 accounts. After an offline monitoring run has been successfully started on the batch farm, the following steps should be followed to setup the post-processing for these runs.<br />
<br />
# The post-processing scripts are stored in $HOME/halld/monitoring/process and are automatically run by cron.<br />
# Run "svn update" to bring any changes in. Be sure that the list of histograms and macros to plot are current.<br />
# Add a new data version<br />
# Edit check_monitoring_data.csh to point to the current revisions/directories<br />
#* VERSION<br />
#* ARGS<br />
#* Note that the environment depends on a standard script - $HOME/setup_jlab.csh<br />
# Update files in the web directory, so that the results are displayed on the web pages: /group/halld/www/halldweb/html/data_monitoring/textdata<br />
# Copy the REST files to more permanent locations:<br />
#* cp -a /volatile/halld/offline_monitoring/RunPeriod-YYYY-MM/verVV/REST /work/halld/data_monitoring/RunPeriod-YYYY-MM/REST/verVV<br />
#* cp -a /volatile/halld/offline_monitoring/RunPeriod-YYYY-MM/verVV/REST /cache/halld/RunPeriod-YYYY-MM/REST/verVV [under testing]<br />
<br />
Check log files in $HOME/halld/monitoring/process/log for more information on how each run went. If there are problems, check log files, and modify check_monitoring_data.csh to vary the verbosity of the output.<br />
<br />
==Data Versions==<br />
<br />
To document the conditions of the monitoring data that is created, for the sake of reproducability and further analysis we save several pieces of information. The format is intended to be comprehensive enough to document not just monitoring data, but versions of raw and reconstructed data, so that this database table can be used for the event database as well.<br />
<br />
We store one record per pass through one run period, with the following structure:<br />
<br />
{| class="wikitable"<br />
! Field !! Description<br />
|-<br />
| data_type || The level of data we are processing. For the purposes of monitoring, "rawdata" is the online monitoring, "recon" is the offline monitoring<br />
|- <br />
| run_period || The run period of the data<br />
|- <br />
| revision || An integer specifying which pass through the run period this data corresponds to<br />
|- <br />
| software_version || The name of the XML file that specifies the different software versions used<br />
|-<br />
| jana_config || The name of the text file that specifies which JANA options were passed to the reconstruction program<br />
|-<br />
| ccdb_context || The value of JANA_CALIB_CONTEXT, which specifies the version of calibration constants that were used<br />
|-<br />
| production_time || The data at which monitoring/reconstruction began<br />
|-<br />
| dataVersionString || A convenient string for identifying this version of the data<br />
|}<br />
<br />
<br />
An example file used as as input to ./register_new_version.py is:<br />
<syntaxhighlight><br />
data_type = recon<br />
run_period = RunPeriod-2014-10<br />
revision = 1<br />
software_version = soft_comm_2014_11_06.xml<br />
jana_config = jana_rawdata_comm_2014_11_06.conf<br />
ccdb_context = calibtime=2014-11-10<br />
production_time = 2014-11-10<br />
dataVersionString = recon_RunPeriod-2014-10_20141110_ver01<br />
</syntaxhighlight></div>Kmoriyahttps://halldweb.jlab.org/wiki/index.php?title=Data_Monitoring_Procedures&diff=71853Data Monitoring Procedures2015-12-11T15:37:34Z<p>Kmoriya: /* General Information on Procedures */</p>
<hr />
<div>__TOC__<br />
<br />
== Master List of File / Database / Webpage Locations ==<br />
=== Run Conditions ===<br />
* Online Run-by-run condition files (B-field, current, etc.): /work/halld/online_monitoring/conditions/<br />
* Offline monitoring run conditions (software versions, jana config): /group/halld/data_monitoring/run_conditions/<br />
*[http://www.jlab.org/Hall-D/test/RunInfo/ Run Info vers. 1]<br />
*[https://halldweb.jlab.org/cgi-bin/data_monitoring/run_conditions.pl Run Info vers. 2]<br />
<br />
=== Monitoring Output Files ===<br />
* Run Periods 201Y-MM is for example 2015-03, launch ver verVV is for example ver15<br />
* Online monitoring histograms: /work/halld/online_monitoring/root/<br />
* Offline monitoring histogram ROOT files: /work/halld/data_monitoring/RunPeriod-201Y-MM/verVV/rootfiles<br />
* REST files (most recent launch only): /work/halld/data_monitoring/RunPeriod-201Y-MM/REST/verVV<br />
* individual files for each job (ROOT, log, etc.): /volatile/halld/offline_monitoring/RunPeriod-201Y-MM/verVV/<br />
<br />
=== Monitoring Database ===<br />
* Accessing monitoring database (on ifarm): mysql -u datmon -h hallddb.jlab.org data_monitoring<br />
<br />
=== Monitoring Webpages ===<br />
*[https://halldweb.jlab.org/cgi-bin/data_monitoring/monitoring/plotBrowser.py Plot Browser]<br />
*[https://halldweb.jlab.org/cgi-bin/data_monitoring/monitoring/runBrowser.py Run Browser]<br />
*[https://halldweb.jlab.org/cgi-bin/data_monitoring/monitoring/timeSeries.py Time Series]<br />
*[https://halldweb.jlab.org/data_monitoring/launch_analysis/ Launch Analysis]<br />
<br />
== Job Monitoring Links ==<br />
* [http://scicomp.jlab.org/farm2/job.html Completed Job History]<br />
* [http://scicomp.jlab.org/farm2/project.html Job Stats By Project]<br />
* [http://scicomp.jlab.org/farm2/trackOrg.html Job Stats By Track]<br />
* [http://scicomp.jlab.org/farm2/report.html Cluster Report]<br />
* [http://scicomp.jlab.org/farm2/walltime.html Walltime Distribution]<br />
<br />
== Saving Online Monitoring Data ==<br />
<br />
The procedure for writing the data out is given in, e.g.,<br />
[https://halldweb.jlab.org/wiki/index.php/Raid-to-Silo_Transfer_Strategy Raid-to-Silo Transfer Strategy].<br />
<br />
Once the DAQ writes out the data to the raid disk, cron jobs will copy the file to tape,<br />
and within ~20 min., we will have access to the file on tape at<br />
/mss/halld/$RUN_PERIOD/rawdata/RunXXXXXX.<br />
<br />
All online monitoring plugins will be run as data is taken.<br />
They will be accessible within the counting house via RootSpy, and<br />
for each run and file, a ROOT file containing the histograms will be saved<br />
within a subdirectory for each run.<br />
<br />
For immediate access to these files, the raid disk files may be accessed directly<br />
from the counting house, or the tape files will be available within ~20 min. of the<br />
file being written out.<br />
<br />
== Offline Monitoring: Running Over Archived Data ==<br />
<br />
Once files are written to tape we run the online plugins on these files to confirm what we were seeing in the online monitoring, and also to update the results from the latest calibration and software. <br />
Manual scripts and cron jobs are set up to look for new data and run the plugins over a sample of files.<br />
<br />
Every other Friday (usually the Friday before the offline meetings) jobs will be started to run the newest software on all previous runs,and allows everybody to see improvements in each detector.<br />
For each launch, independent builds of hdds, sim-recon, the monitoring plugins, and an sqlite file will be generated.<br />
<br />
Below the procedures are described for<br />
# Preparing the software for the launch<br />
# Starting the launch (using hdswif)<br />
# Post-analysis of statistics of the launch<br />
<br />
Processing the results and making them available to the collaboration<br />
is handled in the section [[#Post-Processing_Procedures | Post-Processing Procedures]] below.<br />
<br />
<!--<br />
==== Generating an offline plugin job ====<br />
<br />
The user gxproj1 should be used for official offline monitoring jobs.<br />
Within /home/gxproj1/halld/monitoring/batch/ there will be scripts to run the online monitoring plugins over tape files.<br />
The main script is generatejobs_plugins_rawdata.sh, which can be used as<br />
generatejobs_plugins_rawdata.sh [minrun] [maxrun] (minfile) (maxfile)<br />
where minrun, maxrun specify the range of the run #, and minfile, maxfile (optional) specify<br />
the file range for that run.<br />
<br />
This will generate a script run_rawdata_XXXXXX.sh, where the run # has now been formatted to be 6 digits.<br />
Executing this script will send the monitoring plugins job to the Auger batch system.<br />
<br />
There is also a script clean.sh which can be used as<br />
clean.sh XXX<br />
This will clean up all associated files created in association with run XXX.<br />
<br />
Internally, the xml file used to submit the job will be created, and the job to run<br />
will be given within script.sh. All run parameters should be specified in at the beginning<br />
of generatejobs_plugins_rawdata.sh<br />
Since we are running on tape, the tape file will first be copied over to the cache disk, and the job will run<br />
over this cached file.<br />
<br />
==== Using cron to run automatically ====<br />
Within /home/gluex/halld/monitoring/cron/ there is a file cron_plugins<br />
that can be executed via<br />
crontab cron_plugins<br />
This will set up a cron job to call the script scan_for_jobs.sh, which will<br />
check in the rawdata directory and call generatejobs_plugins_rawdata.sh for<br />
any run that is more than 5 min old. The cron job is set up to run every 10 min.<br />
<br />
==== Magnetic Field Settings ====<br />
For tracking, it is necessary to set the correct magnetic field settings.<br />
The below table is now obsolete. Please refer to the [https://halldweb.jlab.org/cgi-bin/data_monitoring/run_conditions.pl GlueX Run Conditions] webpage.<br />
<br />
The actual field settings for each run have not been documented well.<br />
Below are values based on going through entries in the [https://logbooks.jlab.org/book/halld halld log].<br />
<br />
{| border="1" cellpadding="1" style="text-align: center;"<br />
!width="100"| Run #s<br />
!width="150"| Solenoid Current (A)<br />
!width="300"| JANA option<br />
!width="300"| notes<br />
|-<br />
| 940 - 996 || 1000 || -PBFIELD_MAP=Magnets/Solenoid/solenoid_1000A_poisson_20141104 || At run 997, the solenoid started ramping down<br />
|-<br />
| 998 - 1448 || 0 || -PBFIELD_TYPE=NoField -PDEFTAG:DTrackCandidate=StraightLine || Solenoid started ramping up around run 1431. Runs 1432 - 1448 should not be used for tracking. See [https://logbooks.jlab.org/entry/3309039 this] and [https://logbooks.jlab.org/entry/3309092 this] entry.<br />
|-<br />
| 1449 - 1620 || 1200 || -PBFIELD_MAP=Magnets/Solenoid/solenoid_1200A_poisson_20140520 || <br />
|-<br />
|}<br />
<br />
'''Note:''' The following run ranges were taken with a 300A field, and could be analyzed with the magnetic field map Magnets/Solenoid/solenoid_300A_poisson_20140819 , more study is needed to see if the straight line track fitter is needed or not: 1036 - 1053, 1065 - 1121, 1212? - 1254, 1309 - 1318 [1307,1308,1319-1329 were taken while the magnet was ramping].<br />
--><br />
<br />
=== General Information on Procedures ===<br />
This section explains how the offline monitoring should be run.<br />
Since we may want to simultaneously run offline monitoring for different run periods that require<br />
different environment variables, the scripts are set up so that a generic user can download the<br />
scripts and run them from anywhere. Most output directories for offline monitoring are created<br />
with group read/write permissions so that any Hall D group user has access to the contents,<br />
but there are some cases where use of the account that created the launch is necessary.<br />
<br />
The accounts used for offline monitoring are the gxprojN accounts created and maintained by<br />
Mark Ito (see [https://halldweb.jlab.org/wiki/index.php/GlueX-related_shared_accounts_on_the_JLab_CUE here] for how each account is used).<br />
As of October 2015, the following are used:<br />
* gxproj1 for running over Fall 2014 data (deprecated since June 2015)<br />
* gxproj5 for running over Spring 2015 data<br />
<br />
Since the summer of 2015 we have transitioned from a system using Mark Ito's jproj scripts<br />
to integrating the swif system that Chris Larrieu (SciComp) has been developing.<br />
For offline monitoring, the hdswif system that Kei developed is used for launching the jobs, and a new cross analysis<br />
system based on MySQL and Python is maintained. The jproj system is now deprecated for offline monitoring.<br />
<br />
<br />
The scripts for hdswif, cross analysis and jproj are maintained in svn:<br />
* hdswif: https://halldsvn.jlab.org/repos/trunk/scripts/monitoring/hdswif<br />
* cross analysis: https://halldsvn.jlab.org/repos/trunk/scripts/monitoring/cross_analysis<br />
* jproj : https://halldsvn.jlab.org/repos/trunk/scripts/monitoring/jproj<br />
<br />
To run the offline monitoring each package should be checked out and all necessary<br />
scripts are included.<br />
<br />
=== Preparing the software for the launch ===<br />
<br />
1. Setup the environment: <pre>source ~/env_monitoring_launch</pre><br />
<br />
2. Updating & building hdds: <br />
<pre><br />
cd $HDDS_HOME<br />
git pull # Get latest software<br />
scons -c install # Clean out the old install: EXTREMELY IMPORTANT for cleaning out stale headers<br />
scons install -j4 # Rebuild and re-install with 4 threads<br />
</pre><br />
<br />
3. Updating & building sim-recon: <br />
<pre><br />
cd $HALLD_HOME/src<br />
git pull # Get latest software<br />
scons -c install # Clean out the old install: EXTREMELY IMPORTANT for cleaning out stale headers<br />
scons install -j4 # Rebuild and re-install with 4 threads<br />
</pre><br />
<br />
4. Create a new sqlite file containing the very latest calibration constants. Original documentation on creating sqlite files are [https://halldweb.jlab.org/wiki/index.php/SQLite-form_of_the_CCDB_database here]. <br />
<pre><br />
cd $GLUEX_MYTOP/../sqlite/<br />
$CCDB_HOME/scripts/mysql2sqlite/mysql2sqlite.sh -hhallddb.jlab.org -uccdb_user ccdb | sqlite3 ccdb.sqlite<br />
mv ccdb.sqlite ccdb_monitoring_launch.sqlite #replacing the old file<br />
</pre><br />
<br />
5. Note that the above steps must be done <b>BEFORE</b> launch project creation. This is because we will track the revisions of the libraries used, and this is done by extracting the svn information in each directory. Also, note that the system assumes that we have the topmost build directory (usually called GLUEX_TOP) to be $HOME/builds . Such an assumption is necessary to be able to extract information about the library locations automatically.<br />
<br />
Create the appropriate project(s) and submit the jobs using hdswif, as detailed in the section below.<br />
<br />
=== Starting the Launch and Submitting Jobs ===<br />
<br />
Until the summer of 2015 we relied solely on Mark Ito's jproj system for submitting and keeping track of jobs. We have since moved to the swif system and use the hdswif wrapper for this. Below are instructions for how to use these.<br />
<br />
1. Downloading hdswif: Download the hdswif directory from svn. For the gxprojN accounts, use the directory ~/halld/hdswif. <br />
<pre><br />
cd ~/halld<br />
svn co https://halldsvn.jlab.org/repos/trunk/scripts/monitoring/hdswif<br />
cd hdswif<br />
</pre><br />
<br />
2. Edit the job config file, input.config, which is used to register jobs in hdswif. A typical config file will look this:<br />
PROJECT gluex<br />
TRACK reconstruction<br />
OS centos65<br />
NCORES 6<br />
DISK 40<br />
RAM 8<br />
TIMELIMIT 8<br />
JOBNAMEBASE offmon_<br />
RUNPERIOD 2015-03<br />
VERSION 15<br />
OUTPUT_TOPDIR /volatile/halld/offline_monitoring/RunPeriod-[RUNPERIOD]/ver[VERSION] # Example of other variables included in variable<br />
SCRIPTFILE /home/gxproj5/halld/hdswif/script.sh # Must specify full path<br />
ENVFILE /home/gxproj5/halld/hdswif/setup_jlab-2015-03.csh # Must specify full path<br />
<br />
3. Creating the workflow: Within SWIF jobs are registered into workflows. For offline monitoring, the workflow names are of the form <b>offline_monitoring_RunPeriod201Y_MM_verVV_hd_rawdata</b> with suitable replacements for the run period and version number. The command "swif list" will list all existing workflows. Also, for most simple SWIF commands hdswif also provides a wrapper. <br />
swif list<br />
<br />
For creation of workflows for offline monitoring the command:<br />
hdswif.py create [workflow] -c input.config<br />
should be used. When a config file (here: input.config) is passed in, hdswif will automatically create files that record the configuration of the current launch. These files are stored as for example:<br />
/group/halld/data_monitoring/run_conditions/jana_rawdata_comm_2015_03_ver15.conf<br />
/group/halld/data_monitoring/run_conditions/soft_comm_2015_03_ver15.xml<br />
<br />
The software packages stored in git (sim-recon and hdds) can have git tags applied to them, which makes it easier to find versions of the software than a SHA-1 hash. hdswif will ask if you would like to create a tag, and execute the following sequence: <pre>git tag -a offmon-201Y_MM-verVV -m "Used for offline monitoring 201Y-MM verVV started on 201y/mm/dd"</pre> <pre>git push offmon-201Y_MM-verVV</pre> This will only be invoked when the user name is gxprojN, and for the configuration files, the output directory will be /group/halld/data_monitoring/run_conditions/ for gxprojN accounts while it will be the current directory for other users.<br />
* To use the git-tagged software versions do for example <pre>cd $HALLD_HOME</pre> <pre>git checkout offmon-2015_03-ver15</pre><br />
<br />
4. Registering jobs in the workflow: To register jobs within the workflow, hdswif provides the use of config files. Jobs can be registered by specifying the workflow, config file (-c), run (-r) and file (-f) numbers if necessary. <br />
<span style="color:red">Note: Job configuration parameters can be set differently for jobs within the same workflow if necessary.</span><br />
Jobs can be added via <pre>hdswif.py add [workflow] -c input.config</pre><br />
By default, hdswif will add all files found within the directory /mss/halld/RunPeriod-201Y-MM/rawdata/ where 201Y-MM is specified by the RUNPERIOD parameter in the config file. If only some of the runs or files are needed, these can be specified for example with<br />
<pre>hdswif.py add [workflow] -c input.config -r 3180 -f '00[0-4]'</pre><br />
to specify to register running only over run 3180 files 000 - 004 (Unix-style brackets and wildcards can be used).<br />
<br />
5. Running the workflow: To run the workflow, simply use the hdswif wrapper:<br />
hdswif.py run [workflow]<br />
<br />
<b>It is recommended that some jobs be tested to make sure that everything is working rather than fail thousands of jobs.</b><br><b>Check the configuration setup when creating a workflow, and what is in script.sh. Also check the following:<br />
* Check stderr files. Are they small (<kB)?<br />
* Check stdout files. Are they very large (>MB)?<br />
* Check output ROOT files. Are they larger than several MB?<br />
* Check output REST files. Are they larger than several tens of MB?<br />
</b><br />
<br> For this purpose, hdswif will take an additional parameter to run which limits the number of jobs to submit: <pre>hdswif.py run [workflow] 10</pre><br />
in which case only 10 jobs will be submitted.<br />
<br />
To submit all jobs after checking the results, do <pre>hdswif.py run [workflow]</pre><br />
<br />
=== Checking the Status and Resubmitting ===<br />
1. The status of jobs can be checked on the terminal with <pre>jobstat -u gxprojN</pre> For the status of jobs on Auger see http://scicomp.jlab.org/scicomp/#/auger/jobs and for SWIF use <pre>swif list</pre> or for more information, <pre>swif status [workflow] -summary</pre> Note that "swif status" tends to be out of date sometimes, so don't panic if your workflow/jobs aren't showing up right away. Also see the Auger [http://scicomp.jlab.org/scicomp/#/auger/jobs job website].<br />
<br />
2. For failed jobs, SWIF can resubmit jobs based on the problem. For resubmission for failed jobs with the same resources, <pre>swif retry-jobs [workflow] -problems [problem name]</pre> can be used, and for jobs to be submitted with more resources, e.g., use <pre>swif modify-jobs [workflow] -ram add 2gb -problems AUGER-OVER_RLIMIT</pre> <br />
This only re-stages the jobs, be sure to resubmit them with:<br />
<pre>swif run -workflow [workflow] -errorlimit none</pre><br />
<br />
hdswif has a wrapper for both of these: <pre>hdswif.py resubmit [workflow] [problem]</pre> In this case [problem] can be one of <b>SYSTEM, TIMEOUT, RLIMIT</b>. If SYSTEM is specified, the jobs will be retried. For TIMEOUT and RLIMIT, the jobs will be modified by default with 2 additional hours or GB of RAM. If one more number is added as an option, then that many hours or GB of RAM will be added., e.g., <pre>hdswif.py resubmit [workflow] TIMEOUT 5</pre> will add 5 hours of processing time. You can wait until almost all jobs finish before resubmitting failed jobs since the number should be relatively small. <b>Even if jobs are resubmitted for one type of failure, jobs that later fail with that failure will not be automatically resubmitted.</b><br />
<br />
3. For information on swif, use the "swif help" commands and for hdswif see the attached documentaion in https://halldsvn.jlab.org/repos/trunk/scripts/monitoring/hdswif/manual_hdswif.pdf<br />
<br />
4. Below is a table describing the various errors that can occur.<br />
{| border="1" cellpadding="0" valign="left" style="text-align: left;"<br />
!width="150"| ERROR NAME<br />
!width="400"| Description<br />
!width="400"| Resolution<br />
!width="400"| hdswif command<br />
|-<br />
| AUGER-SUBMIT<br />
||<br />
SWIF’s attempt to submit jobs to Auger failed. Includes server-side problems as well as user failing to provide valid job parameters (e.g. incorrect project name, too many resources, etc.)<br />
||<br />
If requested resources are known to be correct resubmit. Otherwise modify job resources using swif directly.<br />
||<br />
<span style="color:purple"><b>hdswif.py resubmit [workflow] SYSTEM</b></span><br />
|-<br />
| AUGER-FAILED<br />
||<br />
Auger reports the job FAILED with no specific details.<br />
||<br />
Resubmit jobs. If problem persists, contact Chris Larrieu or SciComp.<br />
||<br />
<span style="color:purple"><b>hdswif.py resubmit [workflow] SYSTEM</b></span><br />
|-<br />
| AUGER-OUTPUT-FAIL<br />
||<br />
Failure to copy one or more output files.Can be due to permission problem, quota problem, system error, etc.<br />
||<br />
Check if output files will exist after job execution and that output directory exists, resubmit jobs. If problem persists, contact Chris Larrieu or SciComp.<br />
||<br />
<span style="color:purple"><b>hdswif.py resubmit [workflow] SYSTEM</b></span><br />
|-<br />
| AUGER-INPUT-FAIL<br />
||<br />
Auger failed to copy one or more of the requested input files, similar to output failures. Can also happen if tape file is unavailable (e.g. missing/damaged tape)<br />
||<br />
Check if input file exists, resubmit jobs. If problem persists, contact Chris Larrieu or SciComp.<br />
||<br />
<span style="color:purple"><b>hdswif.py resubmit [workflow] SYSTEM</b></span><br />
|-<br />
| AUGER-TIMEOUT<br />
||<br />
Job timed out.<br />
||<br />
If more time is needed for job add more resources.<br />
Default is to add 2 hrs of processing time. Also check whether code is hanging.<br />
||<br />
<span style="color:red"><b>hdswif.py resubmit [workflow] TIMEOUT</b></span><br><br />
Default is to add 2 hours. Optionally specify number of hours at end.<br />
|-<br />
| AUGER-OVER_RLIMIT<br />
||<br />
Not enough resources, RAM or disk space.<br />
||<br />
Add more resources for job.<br />
||<br />
<span style="color:blue"><b>hdswif.py resubmit [workflow] RLIMIT</b></span><br><br />
Default is to add 2 GB of RAM. Optionally specify GB at end. To add more disk space use SWIF directly.<br />
|-<br />
| SWIF-MISSING-OUTPUT<br />
||<br />
Output file specified by user was not found.<br />
||<br />
Check if output file exists at end of job.<br />
||<br />
<br />
|-<br />
| SWIF-USER-NON-ZERO <br />
||<br />
User script exited with non-zero status code.<br />
||<br />
Your script exited with non-zero status. Check the code you are running.<br />
||<br />
<br />
|-<br />
| SWIF-SYSTEM-ERROR <br />
||<br />
Job failed owing to a problem with swif (e.g. network connection timeout)<br />
||<br />
Resubmit jobs. If problem persists, contact Chris Larrieu or SciComp.<br />
||<br />
<span style="color:purple"><b>hdswif.py resubmit [workflow] SYSTEM</b></span><br />
|}<br />
<br style="clear:both;"/><br />
<br />
=== Post-analysis of statistics of the launch ===<br />
<br />
After jobs have been submitted, it will usually take a few days for all of the jobs to be processed.<br />
The next step is to check the resource usage for the current launch and publish the results online.<br />
<br />
# <b>Create summary XML, HTML files</b><br> The status and results of jobs are saved within the SWIF internal server, and are available via the command <pre>swif status [workflow] -summary -runs</pre> where the arguments -summary and -runs show summary statistics and statistics for individual jobs, respectively. hdswif has a command that takes this output in XML output and creates an HTML webpage showing results of the launch. To do this, do <pre>hdswif.py summary [workflow]</pre> This will create the XML output file from SWIF called swif_output_[workflow].xml and create a webpage containing png figure files. If the XML file already exists, hdswif will ask whether to overwrite the existing file.<br />
# <b>Publish output files online</b><br> At this stage the html output and figure files are created and ready to be put online. The html output capabilities of hdswif is useful for any user using SWIF, but since publication of the html output is specific to the offline monitoring, the script to do this is contained in the jproj scripts directory at https://halldsvn.jlab.org/repos/trunk/scripts/monitoring/cross_analysis. For the gxprojN accounts this directory should exist as ~/halld/monitoring/cross_analysis. To publish the results online do for example <pre>python ~/halld/monitoring/cross_analysis/publish_offmon_results.py 2015_03 18</pre> The script copies the html output and corresponding figures to /group/halld/www/halldweb/html/data_monitoring/launch_analysis/ and also creates a link to it in the html page.<br />
# <b>Editing the summary HTML page</b><br> The top page for is offline monitoring https://halldweb.jlab.org/data_monitoring/launch_analysis/index.html and has links to the summary page for each run period. The summary files are /group/halld/www/halldweb/html/data_monitoring/launch_analysis/[run period].html Edit the file to<br />
## Add a new line to the first table which contains the version number, date, and comments for the current launch<br />
## Create a link to the webpage for the current launch. Simply copy and paste, modify the previous launch's link to have the correct launch ver.<br />
# <b>Freezing SWIF tables</b><br> Since we are now finished with the SWIF workflow that we used, the workflow should be "frozen" so that it cannot be mistakenly altered or modified. Do <pre>swif freeze [workflow]</pre><br />
# <b>Backing up SWIF output</b><br> With the workflow frozen we should be able to reproduce the XML output from SWIF, but we will backup the XML output just in case. Do <pre>cp ~/halld/hdswif/swif_output_[workflow].xml /group/halld/data_monitoring/swif_xml_backup/ </pre><br />
<br />
=== Cross Analysis of Launches ===<br />
<br />
The purpose of the cross analysis is to correlate how resource usage changed for the same files across different launches.<br />
To do this it is useful to create MySQL tables that contain information on each launch, and do queries across tables.<br />
<br />
#The scripts to do this are maintained in the svn directory https://halldsvn.jlab.org/repos/trunk/scripts/monitoring/cross_analysis Do <pre> svn co https://halldsvn.jlab.org/repos/trunk/scripts/monitoring/cross_analysis </pre> For the gxprojN accounts used for offline monitoring the directory should be ~/halld/monitoring/cross_analysis<br />
# The main script is run_cross_analysis.sh, which can be run with <pre>./run_cross_analysis.sh [RUNPERIOD] [VERSION]</pre>. However, <b>it is strongly recommended that the commands in this script be run by hand</b> to catch any errors.<br />
# Enter the python commands that are in run_cross_analysis.sh . Below are the steps and explanations:<br />
## Create a table for the current launch using create_cross_analysis_table.sh. The table will be created from the file template_table_schema.sql and contain columns id, run, file, timeChange, cpu_sec, wall_sec, mem_kb, vmem_kb, nevents, input_copy_sec, plugin_sec, final_state, problems<br />
## Run <pre>python fill_cross_analysis_entries.py [RUNPERIOD] [VERSION]</pre> The script will gather all of the necessary information either from SWIF output or the stdout files for the jobs<br />
## Run <pre>python create_stats_table_row.py</pre> This will loop over the jobs in the current launch and create a row in an HTML table that summarizes the statistics for the final state and problems for the jobs. This table row is then inserted into the main HTML webpage for the run period.<br />
## Run <pre>python create_stats_for_each_file.py [RUNPERIOD] [MINVERSION] [VERSION]</pre> This creates a comparison table of the final state and problems for each file between launches [MINVERSION] and [VERSION]. In the run_cross_analysis.sh script, the default is to set MINVERSION to be 15 for run period 2015_03, but as long as SWIF was used for all previous launches, any number will work.<br />
## Run <pre>python create_resource_correlation_plots.py [RUNVERSION] [CMPMINVERSION] [VERSION]</pre> This creates correlation plots of resource use between launches between CMPMINVERSION and VERSION. By default CMPMINVERSION is 5 launches earlier.<br />
<br />
== Explanation of Scripts ==<br />
Below are explanations of each script used in the offline monitoring system and a breif explanation of how they work.<br />
<br />
=== hdswif scripts ===<br />
<b>Summary:</b> hdswif.py is the main script that calls the other utility scripts. For the utility scripts, they can be run standalone by giving the appropriate parameters. Visual graphics are made using the PyROOT extension of ROOT. To use this, the environment variable PYTHONPATH must include ROOTSYS. If ROOTSYS has been set, adding it to PYTHONPATH will be done by the script, but if ROOTSYS is not set, then the scripts will abort.<br />
<br />
{| border="1" cellpadding="0" valign="left" style="text-align: left;"<br />
!width="150"| file name<br />
!width="800"| Description<br />
|-<br />
| hdswif.py ||<br />
Main script to control the behavior of SWIF. Most commands follow the form hdswif.py [command] [workflow] (options)<br />
|-<br />
| parse_swif.py ||<br />
Called within hdswif.py to create html output from SWIF results.<br />
|-<br />
| createXMLfiles.py ||<br />
* Called by the "create" option of hdswif.py<br />
* Creates XML files for logging information about launch. Must specify config file with option -c.<br />
* Also adds tags to git repositories of sim-recon and hdds.<br />
* For XML file creation, the file will be written out to /group/halld/data_monitoring/run_conditions/ if the user is gxprojN (used for offline monitoring). If not, to avoid general users adding things to the above directory, the output files will be the current directory.<br />
* To write out the versions of each package, environment variables such as HDDS_HOME will need to be set.<br />
* Versions for each software package are extracted using the directory structure, so if these are changed the scripts must change accordingly.<br />
* For each launch, <b>the output soft_comm_[RUNPERIOD]_ver[VER].xml file should be checked that all version numbers have been extracted.</b><br />
|-<br />
| read_config.py ||<br />
* Called by the "add" option of hdswif.py<br />
* Takes in config file name, optionally set verbose<br />
* Return a dictionary between config parameter names and values (e.g., 'PROJECT' : 'gluex', 'NCORES' : 6, ...)<br />
* Prints the config parameters at the end. For parameters changed from default, a '*' will be printed<br />
|-<br />
| output_job_details.py ||<br />
* Called by the "details" option of hdswif.py<br />
* Takes in workflow name, run and file. Run and file must be numbers, no wildcards<br />
* Finds ids for jobs specified by the run and file number and returns info on each one<br />
* Job info is retrieved from pbs farm system and shows configuration parameters for that job<br />
|-<br />
| results_by_resources.py ||<br />
* Called within parse_swif.py<br />
* Takes in XML output from SWIF and creates 2 plots<br />
* Creates html table showing results of jobs by resources requested for that job<br />
* This table is shown under "Status by Resources" in the output html file of hdswif.py summary [workflow]<br />
|-<br />
| create_ordered_hists.py ||<br />
* Called within parse_swif.py<br />
* Takes in XML output from SWIF and creates 2 plots<br />
* Plots are dependency time and pending time of each job, ordered by submission.<br />
* Different colors represent jobs submitted at different times. For jobs submitted at the same time, jobs are ordered in increasing time.<br />
|-<br />
| create_stacked_times.py ||<br />
* Called within parse_swif.py<br />
* Takes in XML output from SWIF and creates 2 plots<br />
* Plots show total job time divided into colors for different stages<br />
* One shows all jobs in order of Auger ID (roughly submission order), the other one shows in order of total job time<br />
* The jobs show at a glance which stage contributes how much of the job's time<br />
|}<br />
<br />
=== Utility scripts ===<br />
{| border="1" cellpadding="0" valign="left" style="text-align: left;"<br />
!width="150"| file name<br />
!width="800"| Description<br />
|-<br />
| stderr_by_size.py ||<br />
* Independent of all other scripts in hdswif directory<br />
* When diagnosing problems it is useful to check the stderr/stdout files. Frequently, different problems are easier to find based on the size of stderr files<br />
* Takes in run period and version as arguments, creates a directory /volatile/halld/offline_monitoring/RunPeriod-[RUNPERIOD]/ver[VER]/log/bysize that contains soft links to all stdout and stderr files from the specified launch, <b>in separate directories given by stderr file size</b>.<br />
|}<br />
<br />
=== cross_analysis scripts ===<br />
<b>Summary:</b> The script run_cross_analysis.sh is the main script. In principle, running this with <pre>./run_cross_analysis.sh [RUNPERIOD] [VERSION]</pre> should work, but it is recommended that the commands within the script<br />
be run by hand to catch any errors.<br />
<br />
{| border="1" cellpadding="0" valign="left" style="text-align: left;"<br />
!width="150"| file name<br />
!width="800"| Description<br />
|-<br />
| run_cross_analysis.sh ||<br />
* Main script to call other Python scripts.<br />
* Takes in run period and version as arguments and runs cross analysis<br />
|-<br />
| create_cross_analysis_table.sh ||<br />
* Creates MySQL table for current launch that is used by other scripts<br />
* Takes run period and version as input, and creates table cross_analysis_table_[RUNPERIOD]_ver[VERSION]<br />
|-<br />
| create_stats_table_row.py ||<br />
* Create a row in the html table showing the overall statistics of final states and problems for a launch<br />
* The final states are "Success", and "Segfault". "Success" includes all jobs that had problems but still finished with Success.<br />
* The problems are "Over Limit", "Timeout", and "System". If any of these occurred for any attempt of the job they are counted.<br />
* The script creates a new html table row for the current launch. This html snippet is inserted into the web-accessible file showing the results <br />
|-<br />
| create_stats_for_each_file.py ||<br />
* Create html tables showing the status of each file against different launch versions.<br />
* Takes in run period, min version, max version and shows final result and problems for all versions in between<br />
* Different final results and problems are shown by combinations of ext content, olor coding of text, background color<br />
|-<br />
|-<br />
| create_resource_correlation_plots.py ||<br />
* Create plots showing correlation of resources between different launches for each file.<br />
* Takes in run period, min version, version of interest. Points are shown only for files included in version of interest<br />
* Creates plots for CPU time, Wall time, memory, virtual memory, #events, difference in #events, time to copy input evio file, time to run plugin<br />
|}<br />
<br />
== Hall D Job Management System ==<br />
<br />
This section details instructions on how to create and launch a set of jobs using the Hall-D Job Management System developed by Mark Ito. These instructions are generic: this system can be used for the weekly monitoring jobs, but can also be used for other sets of job launches as well. <br />
<br />
=== Database Table Overview ===<br />
<br />
* Job management database table (<project_name>): For each input file, keeps track of whether or not a job for it has been submitted, along with other optional fields. <br />
<br />
* Job status database table (<project_name>Job (no space)): For each job, keeps track of the job-id, the job status, memory used, cpu & wall time, time taken to complete various stages (e.g. pending, dependency, active), and others. <br />
<br />
* Job metrics database table (<project_name>_aux (no space)): For each job, keeps track of the job-id, how many events were processed, the time it took to copy the cache file, and the time it took to run the plugin. This information is culled from the log files of each job, and is done within the analysis directory of each launch.<br />
<br />
=== Initialize Project Management ===<br />
<br />
* Log into the ifarm machine with one of the gxproj accounts. For this example we will use gxproj1.<br />
<pre><br />
ssh gxproj1@ifarm -Y<br />
</pre><br />
<br />
* Go to a directory to do the launch. In principle, any directory will work, but for gxproj1 this is usually done in /home/gxproj1/halld/<br />
<br />
* Check out the necessary scripts <pre>svn co https://halldsvn.jlab.org/repos/trunk/scripts/monitoring/jproj</pre> This will get all necessary scripts for launching. Once checked out, <pre>cd projects</pre><br />
<br />
* The script create_project.sh can be used to create a new project. It will take a single argument, the project name. It is assumed that the project name is of the form offline_monitoring_RunPeriod20YY_MM_verVV_hd_rawdata where this string will be parsed to give the run period 20YY_MM and version number VV. One thing to do BEFORE creation of a new project is to editthe conditions of the launch (plugins to run over, memory requested, disk space requested) within templates/template.jsub . This<br />
information is saved automatically at project creation time into files in /group/halld/data_monitoring/run_conditions .<br />
To create a project do <pre>./create_project.sh offline_monitoring_RunPeriod20YY_MM_verVV_hd_rawdata</pre><br />
The name has been chosen to be as consistent as possible with other directory structures.<br />
However, mysql requires that "-" be escaped in table names, so unfortunately run periods will be<br />
given as 2YYY_MM instead of 2YYY-MM.<br />
<br />
* A new directory with the project name (offline_monitoring_RunPeriod20YY_MM_verVV_hd_rawdata) and files will be copied and modified from the template directory to reflect the run period, the user, the directory that it was created in, project name, etc.<br />
<br />
* For each project, cd into the new directory<br />
* The script clear.sh will remove any existing tables for the current project name, then recreate it. Do<br />
<pre>./clear.sh</pre><br />
* To use the jproj.pl script that was checked in, add the directory to your path with<br />
<pre>source ../../scripts/setup.csh</pre><br />
or always specify the full path<br />
<pre>../../scripts/jproj.pl</pre><br />
* Now update the table of runs with<br />
<pre>jproj.pl <project name> update</pre><br />
This will fill the table with all files within /mss that are of the same form as what is in <project name>.jproj . If you want to register only on a subset of all such files, you can edit this file directly.<br />
* Once you have registered all of the files you would like to run over, do<br />
<pre>jproj.pl <project name> submit [max # of jobs] [run number]</pre><br />
where the additional options specify how many jobs to submit and which run number to run on. Without these options all files that are registered and have not been submitted yet will be submitted.<br />
<br />
At this stage you are ready to submit all files. It is a good idea to submit a few test jobs<br />
at first to check that all scripts are working and that the plugins do not crash. Once you are<br />
sure that this does not happen, you can send all jobs in. The remaining jobs are then the monitoring<br />
which will (among other things) put the results on the online webpage for the collaboration to view, and<br />
the analysis of the launch.<br />
<br />
=== Project File Overview ===<br />
<br />
An overview of each project file:<br />
* '''clear.sh:''' For the current project, deletes the job status and management database tables (if any), and creates new, empty ones. <br />
* '''<project_name>.jproj:''' Contains the path and file name format for the input files for the jobs. <br />
* '''<project_name>.jsub:''' The xml job submission script. The run number and file number variables are set during job submission for each input file. <br />
* '''script.sh:''' The script that is executed during the job. If output job directories are not pre-created manually, they should be created in this script with the proper permissions:<br />
<pre><br />
mkdir -p -m 775 my_directory<br />
</pre><br />
* '''setup_jlab-[run period].csh:''' The environment that is sourced at the beginning of the job execution. <br />
* '''status.sh:''' Updates the job status database table, and prints some of its columns to screen.<br />
<br />
=== Project Management ===<br />
<br />
* Delete (if any) and create the database table(s) for the current set of job submissions:<br />
<pre><br />
./clear.sh<br />
</pre><br />
Also, if testing was done with jobs, it is best to delete the output directory and the configuration files:<br />
<pre><br />
rm -frv /volatile/halld/offline_monitoring/RunPeriod-20YY-MM/verVV /group/halld/data_monitoring/run_conditions/soft_comm_20YY_MM_verVV.xml /group/halld/data_monitoring/run_conditions/jana_rawdata_comm_20YY_MM_verVV.conf<br />
</pre><br />
<br />
* Search for input files matching the string in the .jproj file, and create a row for each in the job management database table (called <project_name>). You can test by adding an optional argument at the end, which only selects files with a specific file number:<br />
<pre><br />
jproj.pl <project_name> update <optional_file_number><br />
</pre><br />
<br />
* Confirm that the job management database is accurate by printing it's contents to screen:<br />
<pre><br />
mysql -hhallddb -ufarmer farming -e "select * from <project_name>"<br />
</pre><br />
<br />
* ONLY if a mistake was made, to delete the tables from the database and recreate new, empty ones, run: <br />
<pre><br />
./clear.sh<br />
</pre><br />
<br />
* To look at the status of the submitted jobs, first query auger and update the job status database:<br />
<pre><br />
fill_in_job_details.pl <project_name><br />
</pre><br />
<br />
* The job status can then be viewed by submitting a query to the job status database (called <project_name>Job (no space in between)):<br />
<pre><br />
mysql -hhallddb -ufarmer farming -e "select id,run,file,jobId,hostname,status,timeSubmitted,timeActive,walltime,cput,timeComplete,result,error from <project_name>Job"<br />
</pre><br />
<br />
* These last two commands can instead be executed simultaneously by running:<br />
<pre><br />
./status.sh<br />
</pre><br />
<br />
=== Handy mysql Instructions ===<br />
<br />
* Handy mysql instructions:<br />
<pre><br />
mysql -hhallddb -ufarmer farming # Enter the "farming" mysql database on "hallddb" as user "farmer"<br />
quit; # Exit mysql<br />
show tables; # Show a list of the tables in the current database<br />
show columns from <project_name>; # show all of the columns for the given table<br />
select * from <project_name>; # show the contents of all rows from the given table<br />
</pre><br />
<br />
=== Backing Up Offline Monitoring Tables ===<br />
Tables created for offline monitoring can be backed up using the script backup_tables.sh which<br />
can be checked out with the other files from https://halldsvn.jlab.org/repos/trunk/scripts/monitoring/jproj/projects<br />
<br />
The script uses the command mysqldump to print out a file that can be executed to recreate the tables.<br />
Since executing this output file will drop the table when it exists, caution is advised.<br />
Example usage to backup all three tables created for run period 2014_10 ver 17:<br />
<pre>backup_tables.sh 2014_10 17</pre><br />
<br />
== Running Over Data As It Comes In ==<br />
<br />
A special user gxproj1 will have a cron job set up to run the plugins as new data appears on /mss.<br />
During the week, gxproj1 will submit offline plugin jobs with the same setup as the weekly jobs<br />
run the previous Friday. The procedure for this is shown below.<br />
<br />
<!--<br />
=== Setting up the environment ===<br />
The file<br />
/home/gxproj1/setup_jlab.csh<br />
is sourced through .tcshrc.<br />
This file is the same as what is linked to by<br />
/home/gluex/setup_jlab_commissioning.csh,<br />
except HALLD_HOME, HDDS_HOME, and JANA_CALIB_URL are set separately so that this<br />
user can have a separate build.<br />
<br />
To obtain the builds from the previous Friday's runs,<br />
execute<br />
/home/gxproj1/halld/monitoring/newruns/setup_previous.sh [year] [month] [day]<br />
The build revisions from the previous Friday are archived in files<br />
/work/halld/data_monitoring/run_conditions/soft_comm_[year]_[month]_[day].xml<br />
and the script will build libraries based on those stored revision numbers.<br />
--><br />
<br />
=== Running the cron job ===<br />
<br />
'''IMPORTANT:''' The cron job should not be running while you are manually submitting jobs using the jproj.pl script for the same project, or else you will probably multiply-submit a job. <br />
<br />
* Go to the cron job directory: <br />
<pre><br />
cd /u/home/gxproj1/halld/monitoring/newruns<br />
</pre><br />
<br />
* The cron_plugins file is the cronjob that will be executed. During execution, it runs the exec.sh command in the same folder. This command takes two arguments: the project name, and the maximum file number for each run. These fields should be updated in the cron_plugins file before running. <br />
<br />
* The exec.sh command updates the job management database table with any data that has arrived on tape since it was last updated, ignoring file numbers greater than the maximum file number. It then submits jobs for these files. <br />
<br />
* To start the cron job, run:<br />
<pre><br />
crontab cron_plugins<br />
</pre><br />
<br />
* To check whether the cron job is running, do<br />
<pre><br />
crontab -l<br />
</pre><br />
<br />
* To remove the cron job do<br />
<pre><br />
crontab -r<br />
</pre><br />
<br />
<!--<br />
The cron job will run the script scan_for_jobs.sh,<br />
which runs generatejobs_plugins_rawdata.sh for any<br />
new runs that it had not seen before. All previous<br />
runs are recorded in the file filelists/files_current.txt<br />
so clear this to run over runs, or set the parameters<br />
MINRUN and MAXRUN which will set the range of runs submitted.<br />
--><br />
<br />
==Post-Processing Procedures==<br />
<br />
To visualize the monitoring data, we save images of selected histograms and store time series of selected quantities in a database, which are then displayed on a web page. This section describes how to generate the monitoring images and database information.<br />
<br />
The scripts used to generate this summary data are primarily run from /home/gxprojN/halld/monitoring/process . If you want a new copy of the scripts, e.g., for a new monitoring run, you should check the scripts out from SVN:<br />
<syntaxhighlight><br />
svn co https://halldsvn.jlab.org/repos/trunk/scripts/monitoring/process<br />
</syntaxhighlight><br />
Note that these scripts currently have some parameters which must be periodically set by hand.<br />
<br />
The default python version on most JLab machine does not have the modules to allow these scripts to connect to the MySQL database. To run these scripts, load the environment with the following command<br />
<syntaxhighlight><br />
source /home/gxproj1/halld/monitoring/process/monitoring_env.csh<br />
</syntaxhighlight><br />
<br />
===Online Monitoring===<br />
<br />
There are two primary scripts for running over the monitoring data generated by the online system and offline reconstruction. The online script can be run with either of the following commands:<br />
<syntaxhighlight><br />
/home/gluex/halld/monitoring/process/check_new_runs.py<br />
<br />
OR <br />
<br />
/home/gluex/halld/monitoring/process/check_new_runs.csh<br />
</syntaxhighlight><br />
The shell script sets up the environment properly to run the python script. To connect to the monitoring database on the JLab CUE, modules included in the local installation of python >= 2.7 are needed. The shell script is appropriate to use in a cron job. The cronjob is currently run under the "gluex" account.<br />
<br />
The online monitoring system copies a ROOT file containing the results of the online monitoring, and other configuration files into a directory accessible outside the counting house. This python script automatically checks for new ROOT files, which it will then automatically process. It contains several configuration variables that must be correctly set, which contains the location of input/output directories, etc...<br />
<br />
===Offline Monitoring===<br />
<br />
After the data is run over, the results should be processed, so that summary data is entered into the monitoring database and plots are made for the monitoring webpages. Currently, this processing is controlled by a cronjob that runs the following script:<br />
<syntaxhighlight><br />
/home/gxproj1/halld/monitoring/process/check_monitoring_data.csh <br />
</syntaxhighlight><br />
This script checks for new ROOT files, and only runs over those it hasn't processed yet. Since one monitoring ROOT file is produced for each EVIO file, whenever a new file is produced, the plots for the corresponding run are recreated and all the ROOT files for a run are combined into one file. Information is stored in the database on a per-file basis. <br />
<br />
Plots for the monitoring web page can be made from single histograms or multiple histograms using RootSpy macros. If you want to change the list of plots made, you must modify one of the following files:<br />
* histograms_to_monitor - specify either the name of the histogram or its the full ROOT path <br />
* macros_to_monitor - specify the full path to the RootSpy macro .C file<br />
<br />
When a new monitoring run is started, or the conditions are changed, the following steps should be taken to process the new files:<br />
# Add a new data version, as described below:<br />
# Change the following parameters in check_monitoring_data.csh:<br />
## JOBDATE should correspond to the ouptut date used by the job submission script<br />
## OUTPUTDIR should correspond to the directory corresponding to the run period and revision corresponding to the new version you just submitted. Presumably, this directory will be empty at the beginning.<br />
## Once you create a new data version as defined below, you should pass the needed information as a command line option. Currently this is done by the ARGS variable. For example, the argument "-v RunPeriod-2014-10,8" tells the monitoring scripts to look up the version corresponding to revision 8 of RunPeriod-2014-10 in the monitoring DB and to use to store the results.<br />
<br />
<syntaxhighlight><br />
Example configuration parameters:<br />
set JOBDATE=2015-01-09<br />
set INPUTDIR=/volatile/halld/RunPeriod-2014-10/offline_monitoring<br />
set OUTPUTDIR=/w/halld-scifs1a/data_monitoring/RunPeriod-2014-10/ver08<br />
set ARGS=" -v RunPeriod-2014-10,8 "<br />
</syntaxhighlight><br />
If you want to process the results manually, the data is processed using the following script:<br />
<syntaxhighlight><br />
./process_new_offline_data.py <input directory> <output directory><br />
<br />
EXAMPLE:<br />
<br />
./process_new_offline_data.py 2014-11-14 /volatile/halld/RunPeriod-2014-10/offline_monitoring/ /w/halld-scifs1a/data_monitoring/RunPeriod-2014-10/ver02<br />
</syntaxhighlight><br />
The python script takes several options to enable/disable various steps in the processing. Of interest is the "--force" option, which will run over all monitoring ROOT files, whether or not they've been previously identified.<br />
<br />
Every time a new reconstruction pass is performed, a new version number must be generated. To do this, prepare a version file as described below. Then run the register_new_version.py script to store the information in the database. The script will return a version number, which then should be set by hand in process_new_offline_data.py - future versions of the script will streamline this part of the procedure. An example of how to generate a new version is:<br />
<syntaxhighlight><br />
./register_new_version.py add /home/gxproj1/halld/monitoring/process/versions/vers_RunPeriod-2014-10_pass1.txt<br />
</syntaxhighlight><br />
<br />
<b>If you are running the offline monitoring by checking out the files in trunk/scripts/monitoring/jproj/projects/<br />
of the svn repository</b>, and created a project with<br />
create_project.sh [project name] hd_rawdata<br />
Then go to the directory [project name]/processing/<br />
and execute<br />
./run_processing.sh<br />
which will run register_new_version.py as well as check_monitoring_data.csh for that project.<br />
<br />
===Step-by-Step Instructions For Processing a New Monitoring Run===<br />
<br />
The monitoring runs are current run out of the gxproj1 and gxproj5 accounts. After an offline monitoring run has been successfully started on the batch farm, the following steps should be followed to setup the post-processing for these runs.<br />
<br />
# The post-processing scripts are stored in $HOME/halld/monitoring/process and are automatically run by cron.<br />
# Run "svn update" to bring any changes in. Be sure that the list of histograms and macros to plot are current.<br />
# Add a new data version<br />
# Edit check_monitoring_data.csh to point to the current revisions/directories<br />
#* VERSION<br />
#* ARGS<br />
#* Note that the environment depends on a standard script - $HOME/setup_jlab.csh<br />
# Update files in the web directory, so that the results are displayed on the web pages: /group/halld/www/halldweb/html/data_monitoring/textdata<br />
# Copy the REST files to more permanent locations:<br />
#* cp -a /volatile/halld/offline_monitoring/RunPeriod-YYYY-MM/verVV/REST /work/halld/data_monitoring/RunPeriod-YYYY-MM/REST/verVV<br />
#* cp -a /volatile/halld/offline_monitoring/RunPeriod-YYYY-MM/verVV/REST /cache/halld/RunPeriod-YYYY-MM/REST/verVV [under testing]<br />
<br />
Check log files in $HOME/halld/monitoring/process/log for more information on how each run went. If there are problems, check log files, and modify check_monitoring_data.csh to vary the verbosity of the output.<br />
<br />
==Data Versions==<br />
<br />
To document the conditions of the monitoring data that is created, for the sake of reproducability and further analysis we save several pieces of information. The format is intended to be comprehensive enough to document not just monitoring data, but versions of raw and reconstructed data, so that this database table can be used for the event database as well.<br />
<br />
We store one record per pass through one run period, with the following structure:<br />
<br />
{| class="wikitable"<br />
! Field !! Description<br />
|-<br />
| data_type || The level of data we are processing. For the purposes of monitoring, "rawdata" is the online monitoring, "recon" is the offline monitoring<br />
|- <br />
| run_period || The run period of the data<br />
|- <br />
| revision || An integer specifying which pass through the run period this data corresponds to<br />
|- <br />
| software_version || The name of the XML file that specifies the different software versions used<br />
|-<br />
| jana_config || The name of the text file that specifies which JANA options were passed to the reconstruction program<br />
|-<br />
| ccdb_context || The value of JANA_CALIB_CONTEXT, which specifies the version of calibration constants that were used<br />
|-<br />
| production_time || The data at which monitoring/reconstruction began<br />
|-<br />
| dataVersionString || A convenient string for identifying this version of the data<br />
|}<br />
<br />
<br />
An example file used as as input to ./register_new_version.py is:<br />
<syntaxhighlight><br />
data_type = recon<br />
run_period = RunPeriod-2014-10<br />
revision = 1<br />
software_version = soft_comm_2014_11_06.xml<br />
jana_config = jana_rawdata_comm_2014_11_06.conf<br />
ccdb_context = calibtime=2014-11-10<br />
production_time = 2014-11-10<br />
dataVersionString = recon_RunPeriod-2014-10_20141110_ver01<br />
</syntaxhighlight></div>Kmoriyahttps://halldweb.jlab.org/wiki/index.php?title=Data_Monitoring_Procedures&diff=71851Data Monitoring Procedures2015-12-11T04:36:53Z<p>Kmoriya: /* Cross Analysis of Launches */</p>
<hr />
<div>__TOC__<br />
<br />
== Master List of File / Database / Webpage Locations ==<br />
=== Run Conditions ===<br />
* Online Run-by-run condition files (B-field, current, etc.): /work/halld/online_monitoring/conditions/<br />
* Offline monitoring run conditions (software versions, jana config): /group/halld/data_monitoring/run_conditions/<br />
*[http://www.jlab.org/Hall-D/test/RunInfo/ Run Info vers. 1]<br />
*[https://halldweb.jlab.org/cgi-bin/data_monitoring/run_conditions.pl Run Info vers. 2]<br />
<br />
=== Monitoring Output Files ===<br />
* Run Periods 201Y-MM is for example 2015-03, launch ver verVV is for example ver15<br />
* Online monitoring histograms: /work/halld/online_monitoring/root/<br />
* Offline monitoring histogram ROOT files: /work/halld/data_monitoring/RunPeriod-201Y-MM/verVV/rootfiles<br />
* REST files (most recent launch only): /work/halld/data_monitoring/RunPeriod-201Y-MM/REST/verVV<br />
* individual files for each job (ROOT, log, etc.): /volatile/halld/offline_monitoring/RunPeriod-201Y-MM/verVV/<br />
<br />
=== Monitoring Database ===<br />
* Accessing monitoring database (on ifarm): mysql -u datmon -h hallddb.jlab.org data_monitoring<br />
<br />
=== Monitoring Webpages ===<br />
*[https://halldweb.jlab.org/cgi-bin/data_monitoring/monitoring/plotBrowser.py Plot Browser]<br />
*[https://halldweb.jlab.org/cgi-bin/data_monitoring/monitoring/runBrowser.py Run Browser]<br />
*[https://halldweb.jlab.org/cgi-bin/data_monitoring/monitoring/timeSeries.py Time Series]<br />
*[https://halldweb.jlab.org/data_monitoring/launch_analysis/ Launch Analysis]<br />
<br />
== Job Monitoring Links ==<br />
* [http://scicomp.jlab.org/farm2/job.html Completed Job History]<br />
* [http://scicomp.jlab.org/farm2/project.html Job Stats By Project]<br />
* [http://scicomp.jlab.org/farm2/trackOrg.html Job Stats By Track]<br />
* [http://scicomp.jlab.org/farm2/report.html Cluster Report]<br />
* [http://scicomp.jlab.org/farm2/walltime.html Walltime Distribution]<br />
<br />
== Saving Online Monitoring Data ==<br />
<br />
The procedure for writing the data out is given in, e.g.,<br />
[https://halldweb.jlab.org/wiki/index.php/Raid-to-Silo_Transfer_Strategy Raid-to-Silo Transfer Strategy].<br />
<br />
Once the DAQ writes out the data to the raid disk, cron jobs will copy the file to tape,<br />
and within ~20 min., we will have access to the file on tape at<br />
/mss/halld/$RUN_PERIOD/rawdata/RunXXXXXX.<br />
<br />
All online monitoring plugins will be run as data is taken.<br />
They will be accessible within the counting house via RootSpy, and<br />
for each run and file, a ROOT file containing the histograms will be saved<br />
within a subdirectory for each run.<br />
<br />
For immediate access to these files, the raid disk files may be accessed directly<br />
from the counting house, or the tape files will be available within ~20 min. of the<br />
file being written out.<br />
<br />
== Offline Monitoring: Running Over Archived Data ==<br />
<br />
Once files are written to tape we run the online plugins on these files to confirm what we were seeing in the online monitoring, and also to update the results from the latest calibration and software. <br />
Manual scripts and cron jobs are set up to look for new data and run the plugins over a sample of files.<br />
<br />
Every other Friday (usually the Friday before the offline meetings) jobs will be started to run the newest software on all previous runs,and allows everybody to see improvements in each detector.<br />
For each launch, independent builds of hdds, sim-recon, the monitoring plugins, and an sqlite file will be generated.<br />
<br />
Below the procedures are described for<br />
# Preparing the software for the launch<br />
# Starting the launch (using hdswif)<br />
# Post-analysis of statistics of the launch<br />
<br />
Processing the results and making them available to the collaboration<br />
is handled in the section [[#Post-Processing_Procedures | Post-Processing Procedures]] below.<br />
<br />
<!--<br />
==== Generating an offline plugin job ====<br />
<br />
The user gxproj1 should be used for official offline monitoring jobs.<br />
Within /home/gxproj1/halld/monitoring/batch/ there will be scripts to run the online monitoring plugins over tape files.<br />
The main script is generatejobs_plugins_rawdata.sh, which can be used as<br />
generatejobs_plugins_rawdata.sh [minrun] [maxrun] (minfile) (maxfile)<br />
where minrun, maxrun specify the range of the run #, and minfile, maxfile (optional) specify<br />
the file range for that run.<br />
<br />
This will generate a script run_rawdata_XXXXXX.sh, where the run # has now been formatted to be 6 digits.<br />
Executing this script will send the monitoring plugins job to the Auger batch system.<br />
<br />
There is also a script clean.sh which can be used as<br />
clean.sh XXX<br />
This will clean up all associated files created in association with run XXX.<br />
<br />
Internally, the xml file used to submit the job will be created, and the job to run<br />
will be given within script.sh. All run parameters should be specified in at the beginning<br />
of generatejobs_plugins_rawdata.sh<br />
Since we are running on tape, the tape file will first be copied over to the cache disk, and the job will run<br />
over this cached file.<br />
<br />
==== Using cron to run automatically ====<br />
Within /home/gluex/halld/monitoring/cron/ there is a file cron_plugins<br />
that can be executed via<br />
crontab cron_plugins<br />
This will set up a cron job to call the script scan_for_jobs.sh, which will<br />
check in the rawdata directory and call generatejobs_plugins_rawdata.sh for<br />
any run that is more than 5 min old. The cron job is set up to run every 10 min.<br />
<br />
==== Magnetic Field Settings ====<br />
For tracking, it is necessary to set the correct magnetic field settings.<br />
The below table is now obsolete. Please refer to the [https://halldweb.jlab.org/cgi-bin/data_monitoring/run_conditions.pl GlueX Run Conditions] webpage.<br />
<br />
The actual field settings for each run have not been documented well.<br />
Below are values based on going through entries in the [https://logbooks.jlab.org/book/halld halld log].<br />
<br />
{| border="1" cellpadding="1" style="text-align: center;"<br />
!width="100"| Run #s<br />
!width="150"| Solenoid Current (A)<br />
!width="300"| JANA option<br />
!width="300"| notes<br />
|-<br />
| 940 - 996 || 1000 || -PBFIELD_MAP=Magnets/Solenoid/solenoid_1000A_poisson_20141104 || At run 997, the solenoid started ramping down<br />
|-<br />
| 998 - 1448 || 0 || -PBFIELD_TYPE=NoField -PDEFTAG:DTrackCandidate=StraightLine || Solenoid started ramping up around run 1431. Runs 1432 - 1448 should not be used for tracking. See [https://logbooks.jlab.org/entry/3309039 this] and [https://logbooks.jlab.org/entry/3309092 this] entry.<br />
|-<br />
| 1449 - 1620 || 1200 || -PBFIELD_MAP=Magnets/Solenoid/solenoid_1200A_poisson_20140520 || <br />
|-<br />
|}<br />
<br />
'''Note:''' The following run ranges were taken with a 300A field, and could be analyzed with the magnetic field map Magnets/Solenoid/solenoid_300A_poisson_20140819 , more study is needed to see if the straight line track fitter is needed or not: 1036 - 1053, 1065 - 1121, 1212? - 1254, 1309 - 1318 [1307,1308,1319-1329 were taken while the magnet was ramping].<br />
--><br />
<br />
=== General Information on Procedures ===<br />
This section explains how the offline monitoring should be run.<br />
Since we may want to simultaneously run offline monitoring for different run periods that require<br />
different environment variables, the scripts are set up so that a generic user can download the<br />
scripts and run them from anywhere. Most output directories for offline monitoring are created<br />
with group read/write permissions so that any Hall D group user has access to the contents,<br />
but there are some cases where use of the account that created the launch is necessary.<br />
<br />
The accounts used for offline monitoring are the gxprojN accounts created and maintained by<br />
Mark Ito (see [https://halldweb.jlab.org/wiki/index.php/GlueX-related_shared_accounts_on_the_JLab_CUE here] for how each account is used).<br />
As of October 2015, the following are used:<br />
* gxproj1 for running over Fall 2014 data (deprecated since June 2015)<br />
* gxproj5 for running over Spring 2015 data<br />
<br />
Since the summer of 2015 we have transitioned from a system using Mark Ito's jproj scripts<br />
to integrating the swif system that Chris Larrieu (SciComp) has been developing. For offline monitoring,<br />
the hdswif system that Kei developed is used for launching the jobs, and the jproj system is used<br />
for meta-analysis of launch statistics.<br />
<br />
Both hdswif and jproj are maintained in svn:<br />
* hdswif: https://halldsvn.jlab.org/repos/trunk/scripts/monitoring/hdswif<br />
* jproj : https://halldsvn.jlab.org/repos/trunk/scripts/monitoring/jproj<br />
<br />
To run the offline monitoring each package should be checked out and all necessary<br />
scripts are included.<br />
<br />
=== Preparing the software for the launch ===<br />
<br />
1. Setup the environment: <pre>source ~/env_monitoring_launch</pre><br />
<br />
2. Updating & building hdds: <br />
<pre><br />
cd $HDDS_HOME<br />
git pull # Get latest software<br />
scons -c install # Clean out the old install: EXTREMELY IMPORTANT for cleaning out stale headers<br />
scons install -j4 # Rebuild and re-install with 4 threads<br />
</pre><br />
<br />
3. Updating & building sim-recon: <br />
<pre><br />
cd $HALLD_HOME/src<br />
git pull # Get latest software<br />
scons -c install # Clean out the old install: EXTREMELY IMPORTANT for cleaning out stale headers<br />
scons install -j4 # Rebuild and re-install with 4 threads<br />
</pre><br />
<br />
4. Create a new sqlite file containing the very latest calibration constants. Original documentation on creating sqlite files are [https://halldweb.jlab.org/wiki/index.php/SQLite-form_of_the_CCDB_database here]. <br />
<pre><br />
cd $GLUEX_MYTOP/../sqlite/<br />
$CCDB_HOME/scripts/mysql2sqlite/mysql2sqlite.sh -hhallddb.jlab.org -uccdb_user ccdb | sqlite3 ccdb.sqlite<br />
mv ccdb.sqlite ccdb_monitoring_launch.sqlite #replacing the old file<br />
</pre><br />
<br />
5. Note that the above steps must be done <b>BEFORE</b> launch project creation. This is because we will track the revisions of the libraries used, and this is done by extracting the svn information in each directory. Also, note that the system assumes that we have the topmost build directory (usually called GLUEX_TOP) to be $HOME/builds . Such an assumption is necessary to be able to extract information about the library locations automatically.<br />
<br />
Create the appropriate project(s) and submit the jobs using hdswif, as detailed in the section below.<br />
<br />
=== Starting the Launch and Submitting Jobs ===<br />
<br />
Until the summer of 2015 we relied solely on Mark Ito's jproj system for submitting and keeping track of jobs. We have since moved to the swif system and use the hdswif wrapper for this. Below are instructions for how to use these.<br />
<br />
1. Downloading hdswif: Download the hdswif directory from svn. For the gxprojN accounts, use the directory ~/halld/hdswif. <br />
<pre><br />
cd ~/halld<br />
svn co https://halldsvn.jlab.org/repos/trunk/scripts/monitoring/hdswif<br />
cd hdswif<br />
</pre><br />
<br />
2. Edit the job config file, input.config, which is used to register jobs in hdswif. A typical config file will look this:<br />
PROJECT gluex<br />
TRACK reconstruction<br />
OS centos65<br />
NCORES 6<br />
DISK 40<br />
RAM 8<br />
TIMELIMIT 8<br />
JOBNAMEBASE offmon_<br />
RUNPERIOD 2015-03<br />
VERSION 15<br />
OUTPUT_TOPDIR /volatile/halld/offline_monitoring/RunPeriod-[RUNPERIOD]/ver[VERSION] # Example of other variables included in variable<br />
SCRIPTFILE /home/gxproj5/halld/hdswif/script.sh # Must specify full path<br />
ENVFILE /home/gxproj5/halld/hdswif/setup_jlab-2015-03.csh # Must specify full path<br />
<br />
3. Creating the workflow: Within SWIF jobs are registered into workflows. For offline monitoring, the workflow names are of the form <b>offline_monitoring_RunPeriod201Y_MM_verVV_hd_rawdata</b> with suitable replacements for the run period and version number. The command "swif list" will list all existing workflows. Also, for most simple SWIF commands hdswif also provides a wrapper. <br />
swif list<br />
<br />
For creation of workflows for offline monitoring the command:<br />
hdswif.py create [workflow] -c input.config<br />
should be used. When a config file (here: input.config) is passed in, hdswif will automatically create files that record the configuration of the current launch. These files are stored as for example:<br />
/group/halld/data_monitoring/run_conditions/jana_rawdata_comm_2015_03_ver15.conf<br />
/group/halld/data_monitoring/run_conditions/soft_comm_2015_03_ver15.xml<br />
<br />
The software packages stored in git (sim-recon and hdds) can have git tags applied to them, which makes it easier to find versions of the software than a SHA-1 hash. hdswif will ask if you would like to create a tag, and execute the following sequence: <pre>git tag -a offmon-201Y_MM-verVV -m "Used for offline monitoring 201Y-MM verVV started on 201y/mm/dd"</pre> <pre>git push offmon-201Y_MM-verVV</pre> This will only be invoked when the user name is gxprojN, and for the configuration files, the output directory will be /group/halld/data_monitoring/run_conditions/ for gxprojN accounts while it will be the current directory for other users.<br />
* To use the git-tagged software versions do for example <pre>cd $HALLD_HOME</pre> <pre>git checkout offmon-2015_03-ver15</pre><br />
<br />
4. Registering jobs in the workflow: To register jobs within the workflow, hdswif provides the use of config files. Jobs can be registered by specifying the workflow, config file (-c), run (-r) and file (-f) numbers if necessary. <br />
<span style="color:red">Note: Job configuration parameters can be set differently for jobs within the same workflow if necessary.</span><br />
Jobs can be added via <pre>hdswif.py add [workflow] -c input.config</pre><br />
By default, hdswif will add all files found within the directory /mss/halld/RunPeriod-201Y-MM/rawdata/ where 201Y-MM is specified by the RUNPERIOD parameter in the config file. If only some of the runs or files are needed, these can be specified for example with<br />
<pre>hdswif.py add [workflow] -c input.config -r 3180 -f '00[0-4]'</pre><br />
to specify to register running only over run 3180 files 000 - 004 (Unix-style brackets and wildcards can be used).<br />
<br />
5. Running the workflow: To run the workflow, simply use the hdswif wrapper:<br />
hdswif.py run [workflow]<br />
<br />
<b>It is recommended that some jobs be tested to make sure that everything is working rather than fail thousands of jobs.</b><br><b>Check the configuration setup when creating a workflow, and what is in script.sh. Also check the following:<br />
* Check stderr files. Are they small (<kB)?<br />
* Check stdout files. Are they very large (>MB)?<br />
* Check output ROOT files. Are they larger than several MB?<br />
* Check output REST files. Are they larger than several tens of MB?<br />
</b><br />
<br> For this purpose, hdswif will take an additional parameter to run which limits the number of jobs to submit: <pre>hdswif.py run [workflow] 10</pre><br />
in which case only 10 jobs will be submitted.<br />
<br />
To submit all jobs after checking the results, do <pre>hdswif.py run [workflow]</pre><br />
<br />
=== Checking the Status and Resubmitting ===<br />
1. The status of jobs can be checked on the terminal with <pre>jobstat -u gxprojN</pre> For the status of jobs on Auger see http://scicomp.jlab.org/scicomp/#/auger/jobs and for SWIF use <pre>swif list</pre> or for more information, <pre>swif status [workflow] -summary</pre> Note that "swif status" tends to be out of date sometimes, so don't panic if your workflow/jobs aren't showing up right away. Also see the Auger [http://scicomp.jlab.org/scicomp/#/auger/jobs job website].<br />
<br />
2. For failed jobs, SWIF can resubmit jobs based on the problem. For resubmission for failed jobs with the same resources, <pre>swif retry-jobs [workflow] -problems [problem name]</pre> can be used, and for jobs to be submitted with more resources, e.g., use <pre>swif modify-jobs [workflow] -ram add 2gb -problems AUGER-OVER_RLIMIT</pre> <br />
This only re-stages the jobs, be sure to resubmit them with:<br />
<pre>swif run -workflow [workflow] -errorlimit none</pre><br />
<br />
hdswif has a wrapper for both of these: <pre>hdswif.py resubmit [workflow] [problem]</pre> In this case [problem] can be one of <b>SYSTEM, TIMEOUT, RLIMIT</b>. If SYSTEM is specified, the jobs will be retried. For TIMEOUT and RLIMIT, the jobs will be modified by default with 2 additional hours or GB of RAM. If one more number is added as an option, then that many hours or GB of RAM will be added., e.g., <pre>hdswif.py resubmit [workflow] TIMEOUT 5</pre> will add 5 hours of processing time. You can wait until almost all jobs finish before resubmitting failed jobs since the number should be relatively small. <b>Even if jobs are resubmitted for one type of failure, jobs that later fail with that failure will not be automatically resubmitted.</b><br />
<br />
3. For information on swif, use the "swif help" commands and for hdswif see the attached documentaion in https://halldsvn.jlab.org/repos/trunk/scripts/monitoring/hdswif/manual_hdswif.pdf<br />
<br />
4. Below is a table describing the various errors that can occur.<br />
{| border="1" cellpadding="0" valign="left" style="text-align: left;"<br />
!width="150"| ERROR NAME<br />
!width="400"| Description<br />
!width="400"| Resolution<br />
!width="400"| hdswif command<br />
|-<br />
| AUGER-SUBMIT<br />
||<br />
SWIF’s attempt to submit jobs to Auger failed. Includes server-side problems as well as user failing to provide valid job parameters (e.g. incorrect project name, too many resources, etc.)<br />
||<br />
If requested resources are known to be correct resubmit. Otherwise modify job resources using swif directly.<br />
||<br />
<span style="color:purple"><b>hdswif.py resubmit [workflow] SYSTEM</b></span><br />
|-<br />
| AUGER-FAILED<br />
||<br />
Auger reports the job FAILED with no specific details.<br />
||<br />
Resubmit jobs. If problem persists, contact Chris Larrieu or SciComp.<br />
||<br />
<span style="color:purple"><b>hdswif.py resubmit [workflow] SYSTEM</b></span><br />
|-<br />
| AUGER-OUTPUT-FAIL<br />
||<br />
Failure to copy one or more output files.Can be due to permission problem, quota problem, system error, etc.<br />
||<br />
Check if output files will exist after job execution and that output directory exists, resubmit jobs. If problem persists, contact Chris Larrieu or SciComp.<br />
||<br />
<span style="color:purple"><b>hdswif.py resubmit [workflow] SYSTEM</b></span><br />
|-<br />
| AUGER-INPUT-FAIL<br />
||<br />
Auger failed to copy one or more of the requested input files, similar to output failures. Can also happen if tape file is unavailable (e.g. missing/damaged tape)<br />
||<br />
Check if input file exists, resubmit jobs. If problem persists, contact Chris Larrieu or SciComp.<br />
||<br />
<span style="color:purple"><b>hdswif.py resubmit [workflow] SYSTEM</b></span><br />
|-<br />
| AUGER-TIMEOUT<br />
||<br />
Job timed out.<br />
||<br />
If more time is needed for job add more resources.<br />
Default is to add 2 hrs of processing time. Also check whether code is hanging.<br />
||<br />
<span style="color:red"><b>hdswif.py resubmit [workflow] TIMEOUT</b></span><br><br />
Default is to add 2 hours. Optionally specify number of hours at end.<br />
|-<br />
| AUGER-OVER_RLIMIT<br />
||<br />
Not enough resources, RAM or disk space.<br />
||<br />
Add more resources for job.<br />
||<br />
<span style="color:blue"><b>hdswif.py resubmit [workflow] RLIMIT</b></span><br><br />
Default is to add 2 GB of RAM. Optionally specify GB at end. To add more disk space use SWIF directly.<br />
|-<br />
| SWIF-MISSING-OUTPUT<br />
||<br />
Output file specified by user was not found.<br />
||<br />
Check if output file exists at end of job.<br />
||<br />
<br />
|-<br />
| SWIF-USER-NON-ZERO <br />
||<br />
User script exited with non-zero status code.<br />
||<br />
Your script exited with non-zero status. Check the code you are running.<br />
||<br />
<br />
|-<br />
| SWIF-SYSTEM-ERROR <br />
||<br />
Job failed owing to a problem with swif (e.g. network connection timeout)<br />
||<br />
Resubmit jobs. If problem persists, contact Chris Larrieu or SciComp.<br />
||<br />
<span style="color:purple"><b>hdswif.py resubmit [workflow] SYSTEM</b></span><br />
|}<br />
<br style="clear:both;"/><br />
<br />
=== Post-analysis of statistics of the launch ===<br />
<br />
After jobs have been submitted, it will usually take a few days for all of the jobs to be processed.<br />
The next step is to check the resource usage for the current launch and publish the results online.<br />
<br />
# <b>Create summary XML, HTML files</b><br> The status and results of jobs are saved within the SWIF internal server, and are available via the command <pre>swif status [workflow] -summary -runs</pre> where the arguments -summary and -runs show summary statistics and statistics for individual jobs, respectively. hdswif has a command that takes this output in XML output and creates an HTML webpage showing results of the launch. To do this, do <pre>hdswif.py summary [workflow]</pre> This will create the XML output file from SWIF called swif_output_[workflow].xml and create a webpage containing png figure files. If the XML file already exists, hdswif will ask whether to overwrite the existing file.<br />
# <b>Publish output files online</b><br> At this stage the html output and figure files are created and ready to be put online. The html output capabilities of hdswif is useful for any user using SWIF, but since publication of the html output is specific to the offline monitoring, the script to do this is contained in the jproj scripts directory at https://halldsvn.jlab.org/repos/trunk/scripts/monitoring/cross_analysis. For the gxprojN accounts this directory should exist as ~/halld/monitoring/cross_analysis. To publish the results online do for example <pre>python ~/halld/monitoring/cross_analysis/publish_offmon_results.py 2015_03 18</pre> The script copies the html output and corresponding figures to /group/halld/www/halldweb/html/data_monitoring/launch_analysis/ and also creates a link to it in the html page.<br />
# <b>Editing the summary HTML page</b><br> The top page for is offline monitoring https://halldweb.jlab.org/data_monitoring/launch_analysis/index.html and has links to the summary page for each run period. The summary files are /group/halld/www/halldweb/html/data_monitoring/launch_analysis/[run period].html Edit the file to<br />
## Add a new line to the first table which contains the version number, date, and comments for the current launch<br />
## Create a link to the webpage for the current launch. Simply copy and paste, modify the previous launch's link to have the correct launch ver.<br />
# <b>Freezing SWIF tables</b><br> Since we are now finished with the SWIF workflow that we used, the workflow should be "frozen" so that it cannot be mistakenly altered or modified. Do <pre>swif freeze [workflow]</pre><br />
# <b>Backing up SWIF output</b><br> With the workflow frozen we should be able to reproduce the XML output from SWIF, but we will backup the XML output just in case. Do <pre>cp ~/halld/hdswif/swif_output_[workflow].xml /group/halld/data_monitoring/swif_xml_backup/ </pre><br />
<br />
=== Cross Analysis of Launches ===<br />
<br />
The purpose of the cross analysis is to correlate how resource usage changed for the same files across different launches.<br />
To do this it is useful to create MySQL tables that contain information on each launch, and do queries across tables.<br />
<br />
#The scripts to do this are maintained in the svn directory https://halldsvn.jlab.org/repos/trunk/scripts/monitoring/cross_analysis Do <pre> svn co https://halldsvn.jlab.org/repos/trunk/scripts/monitoring/cross_analysis </pre> For the gxprojN accounts used for offline monitoring the directory should be ~/halld/monitoring/cross_analysis<br />
# The main script is run_cross_analysis.sh, which can be run with <pre>./run_cross_analysis.sh [RUNPERIOD] [VERSION]</pre>. However, <b>it is strongly recommended that the commands in this script be run by hand</b> to catch any errors.<br />
# Enter the python commands that are in run_cross_analysis.sh . Below are the steps and explanations:<br />
## Create a table for the current launch using create_cross_analysis_table.sh. The table will be created from the file template_table_schema.sql and contain columns id, run, file, timeChange, cpu_sec, wall_sec, mem_kb, vmem_kb, nevents, input_copy_sec, plugin_sec, final_state, problems<br />
## Run <pre>python fill_cross_analysis_entries.py [RUNPERIOD] [VERSION]</pre> The script will gather all of the necessary information either from SWIF output or the stdout files for the jobs<br />
## Run <pre>python create_stats_table_row.py</pre> This will loop over the jobs in the current launch and create a row in an HTML table that summarizes the statistics for the final state and problems for the jobs. This table row is then inserted into the main HTML webpage for the run period.<br />
## Run <pre>python create_stats_for_each_file.py [RUNPERIOD] [MINVERSION] [VERSION]</pre> This creates a comparison table of the final state and problems for each file between launches [MINVERSION] and [VERSION]. In the run_cross_analysis.sh script, the default is to set MINVERSION to be 15 for run period 2015_03, but as long as SWIF was used for all previous launches, any number will work.<br />
## Run <pre>python create_resource_correlation_plots.py [RUNVERSION] [CMPMINVERSION] [VERSION]</pre> This creates correlation plots of resource use between launches between CMPMINVERSION and VERSION. By default CMPMINVERSION is 5 launches earlier.<br />
<br />
== Explanation of Scripts ==<br />
Below are explanations of each script used in the offline monitoring system and a breif explanation of how they work.<br />
<br />
=== hdswif scripts ===<br />
<b>Summary:</b> hdswif.py is the main script that calls the other utility scripts. For the utility scripts, they can be run standalone by giving the appropriate parameters. Visual graphics are made using the PyROOT extension of ROOT. To use this, the environment variable PYTHONPATH must include ROOTSYS. If ROOTSYS has been set, adding it to PYTHONPATH will be done by the script, but if ROOTSYS is not set, then the scripts will abort.<br />
<br />
{| border="1" cellpadding="0" valign="left" style="text-align: left;"<br />
!width="150"| file name<br />
!width="800"| Description<br />
|-<br />
| hdswif.py ||<br />
Main script to control the behavior of SWIF. Most commands follow the form hdswif.py [command] [workflow] (options)<br />
|-<br />
| parse_swif.py ||<br />
Called within hdswif.py to create html output from SWIF results.<br />
|-<br />
| createXMLfiles.py ||<br />
* Called by the "create" option of hdswif.py<br />
* Creates XML files for logging information about launch. Must specify config file with option -c.<br />
* Also adds tags to git repositories of sim-recon and hdds.<br />
* For XML file creation, the file will be written out to /group/halld/data_monitoring/run_conditions/ if the user is gxprojN (used for offline monitoring). If not, to avoid general users adding things to the above directory, the output files will be the current directory.<br />
* To write out the versions of each package, environment variables such as HDDS_HOME will need to be set.<br />
* Versions for each software package are extracted using the directory structure, so if these are changed the scripts must change accordingly.<br />
* For each launch, <b>the output soft_comm_[RUNPERIOD]_ver[VER].xml file should be checked that all version numbers have been extracted.</b><br />
|-<br />
| read_config.py ||<br />
* Called by the "add" option of hdswif.py<br />
* Takes in config file name, optionally set verbose<br />
* Return a dictionary between config parameter names and values (e.g., 'PROJECT' : 'gluex', 'NCORES' : 6, ...)<br />
* Prints the config parameters at the end. For parameters changed from default, a '*' will be printed<br />
|-<br />
| output_job_details.py ||<br />
* Called by the "details" option of hdswif.py<br />
* Takes in workflow name, run and file. Run and file must be numbers, no wildcards<br />
* Finds ids for jobs specified by the run and file number and returns info on each one<br />
* Job info is retrieved from pbs farm system and shows configuration parameters for that job<br />
|-<br />
| results_by_resources.py ||<br />
* Called within parse_swif.py<br />
* Takes in XML output from SWIF and creates 2 plots<br />
* Creates html table showing results of jobs by resources requested for that job<br />
* This table is shown under "Status by Resources" in the output html file of hdswif.py summary [workflow]<br />
|-<br />
| create_ordered_hists.py ||<br />
* Called within parse_swif.py<br />
* Takes in XML output from SWIF and creates 2 plots<br />
* Plots are dependency time and pending time of each job, ordered by submission.<br />
* Different colors represent jobs submitted at different times. For jobs submitted at the same time, jobs are ordered in increasing time.<br />
|-<br />
| create_stacked_times.py ||<br />
* Called within parse_swif.py<br />
* Takes in XML output from SWIF and creates 2 plots<br />
* Plots show total job time divided into colors for different stages<br />
* One shows all jobs in order of Auger ID (roughly submission order), the other one shows in order of total job time<br />
* The jobs show at a glance which stage contributes how much of the job's time<br />
|}<br />
<br />
=== Utility scripts ===<br />
{| border="1" cellpadding="0" valign="left" style="text-align: left;"<br />
!width="150"| file name<br />
!width="800"| Description<br />
|-<br />
| stderr_by_size.py ||<br />
* Independent of all other scripts in hdswif directory<br />
* When diagnosing problems it is useful to check the stderr/stdout files. Frequently, different problems are easier to find based on the size of stderr files<br />
* Takes in run period and version as arguments, creates a directory /volatile/halld/offline_monitoring/RunPeriod-[RUNPERIOD]/ver[VER]/log/bysize that contains soft links to all stdout and stderr files from the specified launch, <b>in separate directories given by stderr file size</b>.<br />
|}<br />
<br />
=== cross_analysis scripts ===<br />
<b>Summary:</b> The script run_cross_analysis.sh is the main script. In principle, running this with <pre>./run_cross_analysis.sh [RUNPERIOD] [VERSION]</pre> should work, but it is recommended that the commands within the script<br />
be run by hand to catch any errors.<br />
<br />
{| border="1" cellpadding="0" valign="left" style="text-align: left;"<br />
!width="150"| file name<br />
!width="800"| Description<br />
|-<br />
| run_cross_analysis.sh ||<br />
* Main script to call other Python scripts.<br />
* Takes in run period and version as arguments and runs cross analysis<br />
|-<br />
| create_cross_analysis_table.sh ||<br />
* Creates MySQL table for current launch that is used by other scripts<br />
* Takes run period and version as input, and creates table cross_analysis_table_[RUNPERIOD]_ver[VERSION]<br />
|-<br />
| create_stats_table_row.py ||<br />
* Create a row in the html table showing the overall statistics of final states and problems for a launch<br />
* The final states are "Success", and "Segfault". "Success" includes all jobs that had problems but still finished with Success.<br />
* The problems are "Over Limit", "Timeout", and "System". If any of these occurred for any attempt of the job they are counted.<br />
* The script creates a new html table row for the current launch. This html snippet is inserted into the web-accessible file showing the results <br />
|-<br />
| create_stats_for_each_file.py ||<br />
* Create html tables showing the status of each file against different launch versions.<br />
* Takes in run period, min version, max version and shows final result and problems for all versions in between<br />
* Different final results and problems are shown by combinations of ext content, olor coding of text, background color<br />
|-<br />
|-<br />
| create_resource_correlation_plots.py ||<br />
* Create plots showing correlation of resources between different launches for each file.<br />
* Takes in run period, min version, version of interest. Points are shown only for files included in version of interest<br />
* Creates plots for CPU time, Wall time, memory, virtual memory, #events, difference in #events, time to copy input evio file, time to run plugin<br />
|}<br />
<br />
== Hall D Job Management System ==<br />
<br />
This section details instructions on how to create and launch a set of jobs using the Hall-D Job Management System developed by Mark Ito. These instructions are generic: this system can be used for the weekly monitoring jobs, but can also be used for other sets of job launches as well. <br />
<br />
=== Database Table Overview ===<br />
<br />
* Job management database table (<project_name>): For each input file, keeps track of whether or not a job for it has been submitted, along with other optional fields. <br />
<br />
* Job status database table (<project_name>Job (no space)): For each job, keeps track of the job-id, the job status, memory used, cpu & wall time, time taken to complete various stages (e.g. pending, dependency, active), and others. <br />
<br />
* Job metrics database table (<project_name>_aux (no space)): For each job, keeps track of the job-id, how many events were processed, the time it took to copy the cache file, and the time it took to run the plugin. This information is culled from the log files of each job, and is done within the analysis directory of each launch.<br />
<br />
=== Initialize Project Management ===<br />
<br />
* Log into the ifarm machine with one of the gxproj accounts. For this example we will use gxproj1.<br />
<pre><br />
ssh gxproj1@ifarm -Y<br />
</pre><br />
<br />
* Go to a directory to do the launch. In principle, any directory will work, but for gxproj1 this is usually done in /home/gxproj1/halld/<br />
<br />
* Check out the necessary scripts <pre>svn co https://halldsvn.jlab.org/repos/trunk/scripts/monitoring/jproj</pre> This will get all necessary scripts for launching. Once checked out, <pre>cd projects</pre><br />
<br />
* The script create_project.sh can be used to create a new project. It will take a single argument, the project name. It is assumed that the project name is of the form offline_monitoring_RunPeriod20YY_MM_verVV_hd_rawdata where this string will be parsed to give the run period 20YY_MM and version number VV. One thing to do BEFORE creation of a new project is to editthe conditions of the launch (plugins to run over, memory requested, disk space requested) within templates/template.jsub . This<br />
information is saved automatically at project creation time into files in /group/halld/data_monitoring/run_conditions .<br />
To create a project do <pre>./create_project.sh offline_monitoring_RunPeriod20YY_MM_verVV_hd_rawdata</pre><br />
The name has been chosen to be as consistent as possible with other directory structures.<br />
However, mysql requires that "-" be escaped in table names, so unfortunately run periods will be<br />
given as 2YYY_MM instead of 2YYY-MM.<br />
<br />
* A new directory with the project name (offline_monitoring_RunPeriod20YY_MM_verVV_hd_rawdata) and files will be copied and modified from the template directory to reflect the run period, the user, the directory that it was created in, project name, etc.<br />
<br />
* For each project, cd into the new directory<br />
* The script clear.sh will remove any existing tables for the current project name, then recreate it. Do<br />
<pre>./clear.sh</pre><br />
* To use the jproj.pl script that was checked in, add the directory to your path with<br />
<pre>source ../../scripts/setup.csh</pre><br />
or always specify the full path<br />
<pre>../../scripts/jproj.pl</pre><br />
* Now update the table of runs with<br />
<pre>jproj.pl <project name> update</pre><br />
This will fill the table with all files within /mss that are of the same form as what is in <project name>.jproj . If you want to register only on a subset of all such files, you can edit this file directly.<br />
* Once you have registered all of the files you would like to run over, do<br />
<pre>jproj.pl <project name> submit [max # of jobs] [run number]</pre><br />
where the additional options specify how many jobs to submit and which run number to run on. Without these options all files that are registered and have not been submitted yet will be submitted.<br />
<br />
At this stage you are ready to submit all files. It is a good idea to submit a few test jobs<br />
at first to check that all scripts are working and that the plugins do not crash. Once you are<br />
sure that this does not happen, you can send all jobs in. The remaining jobs are then the monitoring<br />
which will (among other things) put the results on the online webpage for the collaboration to view, and<br />
the analysis of the launch.<br />
<br />
=== Project File Overview ===<br />
<br />
An overview of each project file:<br />
* '''clear.sh:''' For the current project, deletes the job status and management database tables (if any), and creates new, empty ones. <br />
* '''<project_name>.jproj:''' Contains the path and file name format for the input files for the jobs. <br />
* '''<project_name>.jsub:''' The xml job submission script. The run number and file number variables are set during job submission for each input file. <br />
* '''script.sh:''' The script that is executed during the job. If output job directories are not pre-created manually, they should be created in this script with the proper permissions:<br />
<pre><br />
mkdir -p -m 775 my_directory<br />
</pre><br />
* '''setup_jlab-[run period].csh:''' The environment that is sourced at the beginning of the job execution. <br />
* '''status.sh:''' Updates the job status database table, and prints some of its columns to screen.<br />
<br />
=== Project Management ===<br />
<br />
* Delete (if any) and create the database table(s) for the current set of job submissions:<br />
<pre><br />
./clear.sh<br />
</pre><br />
Also, if testing was done with jobs, it is best to delete the output directory and the configuration files:<br />
<pre><br />
rm -frv /volatile/halld/offline_monitoring/RunPeriod-20YY-MM/verVV /group/halld/data_monitoring/run_conditions/soft_comm_20YY_MM_verVV.xml /group/halld/data_monitoring/run_conditions/jana_rawdata_comm_20YY_MM_verVV.conf<br />
</pre><br />
<br />
* Search for input files matching the string in the .jproj file, and create a row for each in the job management database table (called <project_name>). You can test by adding an optional argument at the end, which only selects files with a specific file number:<br />
<pre><br />
jproj.pl <project_name> update <optional_file_number><br />
</pre><br />
<br />
* Confirm that the job management database is accurate by printing it's contents to screen:<br />
<pre><br />
mysql -hhallddb -ufarmer farming -e "select * from <project_name>"<br />
</pre><br />
<br />
* ONLY if a mistake was made, to delete the tables from the database and recreate new, empty ones, run: <br />
<pre><br />
./clear.sh<br />
</pre><br />
<br />
* To look at the status of the submitted jobs, first query auger and update the job status database:<br />
<pre><br />
fill_in_job_details.pl <project_name><br />
</pre><br />
<br />
* The job status can then be viewed by submitting a query to the job status database (called <project_name>Job (no space in between)):<br />
<pre><br />
mysql -hhallddb -ufarmer farming -e "select id,run,file,jobId,hostname,status,timeSubmitted,timeActive,walltime,cput,timeComplete,result,error from <project_name>Job"<br />
</pre><br />
<br />
* These last two commands can instead be executed simultaneously by running:<br />
<pre><br />
./status.sh<br />
</pre><br />
<br />
=== Handy mysql Instructions ===<br />
<br />
* Handy mysql instructions:<br />
<pre><br />
mysql -hhallddb -ufarmer farming # Enter the "farming" mysql database on "hallddb" as user "farmer"<br />
quit; # Exit mysql<br />
show tables; # Show a list of the tables in the current database<br />
show columns from <project_name>; # show all of the columns for the given table<br />
select * from <project_name>; # show the contents of all rows from the given table<br />
</pre><br />
<br />
=== Backing Up Offline Monitoring Tables ===<br />
Tables created for offline monitoring can be backed up using the script backup_tables.sh which<br />
can be checked out with the other files from https://halldsvn.jlab.org/repos/trunk/scripts/monitoring/jproj/projects<br />
<br />
The script uses the command mysqldump to print out a file that can be executed to recreate the tables.<br />
Since executing this output file will drop the table when it exists, caution is advised.<br />
Example usage to backup all three tables created for run period 2014_10 ver 17:<br />
<pre>backup_tables.sh 2014_10 17</pre><br />
<br />
== Running Over Data As It Comes In ==<br />
<br />
A special user gxproj1 will have a cron job set up to run the plugins as new data appears on /mss.<br />
During the week, gxproj1 will submit offline plugin jobs with the same setup as the weekly jobs<br />
run the previous Friday. The procedure for this is shown below.<br />
<br />
<!--<br />
=== Setting up the environment ===<br />
The file<br />
/home/gxproj1/setup_jlab.csh<br />
is sourced through .tcshrc.<br />
This file is the same as what is linked to by<br />
/home/gluex/setup_jlab_commissioning.csh,<br />
except HALLD_HOME, HDDS_HOME, and JANA_CALIB_URL are set separately so that this<br />
user can have a separate build.<br />
<br />
To obtain the builds from the previous Friday's runs,<br />
execute<br />
/home/gxproj1/halld/monitoring/newruns/setup_previous.sh [year] [month] [day]<br />
The build revisions from the previous Friday are archived in files<br />
/work/halld/data_monitoring/run_conditions/soft_comm_[year]_[month]_[day].xml<br />
and the script will build libraries based on those stored revision numbers.<br />
--><br />
<br />
=== Running the cron job ===<br />
<br />
'''IMPORTANT:''' The cron job should not be running while you are manually submitting jobs using the jproj.pl script for the same project, or else you will probably multiply-submit a job. <br />
<br />
* Go to the cron job directory: <br />
<pre><br />
cd /u/home/gxproj1/halld/monitoring/newruns<br />
</pre><br />
<br />
* The cron_plugins file is the cronjob that will be executed. During execution, it runs the exec.sh command in the same folder. This command takes two arguments: the project name, and the maximum file number for each run. These fields should be updated in the cron_plugins file before running. <br />
<br />
* The exec.sh command updates the job management database table with any data that has arrived on tape since it was last updated, ignoring file numbers greater than the maximum file number. It then submits jobs for these files. <br />
<br />
* To start the cron job, run:<br />
<pre><br />
crontab cron_plugins<br />
</pre><br />
<br />
* To check whether the cron job is running, do<br />
<pre><br />
crontab -l<br />
</pre><br />
<br />
* To remove the cron job do<br />
<pre><br />
crontab -r<br />
</pre><br />
<br />
<!--<br />
The cron job will run the script scan_for_jobs.sh,<br />
which runs generatejobs_plugins_rawdata.sh for any<br />
new runs that it had not seen before. All previous<br />
runs are recorded in the file filelists/files_current.txt<br />
so clear this to run over runs, or set the parameters<br />
MINRUN and MAXRUN which will set the range of runs submitted.<br />
--><br />
<br />
==Post-Processing Procedures==<br />
<br />
To visualize the monitoring data, we save images of selected histograms and store time series of selected quantities in a database, which are then displayed on a web page. This section describes how to generate the monitoring images and database information.<br />
<br />
The scripts used to generate this summary data are primarily run from /home/gxprojN/halld/monitoring/process . If you want a new copy of the scripts, e.g., for a new monitoring run, you should check the scripts out from SVN:<br />
<syntaxhighlight><br />
svn co https://halldsvn.jlab.org/repos/trunk/scripts/monitoring/process<br />
</syntaxhighlight><br />
Note that these scripts currently have some parameters which must be periodically set by hand.<br />
<br />
The default python version on most JLab machine does not have the modules to allow these scripts to connect to the MySQL database. To run these scripts, load the environment with the following command<br />
<syntaxhighlight><br />
source /home/gxproj1/halld/monitoring/process/monitoring_env.csh<br />
</syntaxhighlight><br />
<br />
===Online Monitoring===<br />
<br />
There are two primary scripts for running over the monitoring data generated by the online system and offline reconstruction. The online script can be run with either of the following commands:<br />
<syntaxhighlight><br />
/home/gluex/halld/monitoring/process/check_new_runs.py<br />
<br />
OR <br />
<br />
/home/gluex/halld/monitoring/process/check_new_runs.csh<br />
</syntaxhighlight><br />
The shell script sets up the environment properly to run the python script. To connect to the monitoring database on the JLab CUE, modules included in the local installation of python >= 2.7 are needed. The shell script is appropriate to use in a cron job. The cronjob is currently run under the "gluex" account.<br />
<br />
The online monitoring system copies a ROOT file containing the results of the online monitoring, and other configuration files into a directory accessible outside the counting house. This python script automatically checks for new ROOT files, which it will then automatically process. It contains several configuration variables that must be correctly set, which contains the location of input/output directories, etc...<br />
<br />
===Offline Monitoring===<br />
<br />
After the data is run over, the results should be processed, so that summary data is entered into the monitoring database and plots are made for the monitoring webpages. Currently, this processing is controlled by a cronjob that runs the following script:<br />
<syntaxhighlight><br />
/home/gxproj1/halld/monitoring/process/check_monitoring_data.csh <br />
</syntaxhighlight><br />
This script checks for new ROOT files, and only runs over those it hasn't processed yet. Since one monitoring ROOT file is produced for each EVIO file, whenever a new file is produced, the plots for the corresponding run are recreated and all the ROOT files for a run are combined into one file. Information is stored in the database on a per-file basis. <br />
<br />
Plots for the monitoring web page can be made from single histograms or multiple histograms using RootSpy macros. If you want to change the list of plots made, you must modify one of the following files:<br />
* histograms_to_monitor - specify either the name of the histogram or its the full ROOT path <br />
* macros_to_monitor - specify the full path to the RootSpy macro .C file<br />
<br />
When a new monitoring run is started, or the conditions are changed, the following steps should be taken to process the new files:<br />
# Add a new data version, as described below:<br />
# Change the following parameters in check_monitoring_data.csh:<br />
## JOBDATE should correspond to the ouptut date used by the job submission script<br />
## OUTPUTDIR should correspond to the directory corresponding to the run period and revision corresponding to the new version you just submitted. Presumably, this directory will be empty at the beginning.<br />
## Once you create a new data version as defined below, you should pass the needed information as a command line option. Currently this is done by the ARGS variable. For example, the argument "-v RunPeriod-2014-10,8" tells the monitoring scripts to look up the version corresponding to revision 8 of RunPeriod-2014-10 in the monitoring DB and to use to store the results.<br />
<br />
<syntaxhighlight><br />
Example configuration parameters:<br />
set JOBDATE=2015-01-09<br />
set INPUTDIR=/volatile/halld/RunPeriod-2014-10/offline_monitoring<br />
set OUTPUTDIR=/w/halld-scifs1a/data_monitoring/RunPeriod-2014-10/ver08<br />
set ARGS=" -v RunPeriod-2014-10,8 "<br />
</syntaxhighlight><br />
If you want to process the results manually, the data is processed using the following script:<br />
<syntaxhighlight><br />
./process_new_offline_data.py <input directory> <output directory><br />
<br />
EXAMPLE:<br />
<br />
./process_new_offline_data.py 2014-11-14 /volatile/halld/RunPeriod-2014-10/offline_monitoring/ /w/halld-scifs1a/data_monitoring/RunPeriod-2014-10/ver02<br />
</syntaxhighlight><br />
The python script takes several options to enable/disable various steps in the processing. Of interest is the "--force" option, which will run over all monitoring ROOT files, whether or not they've been previously identified.<br />
<br />
Every time a new reconstruction pass is performed, a new version number must be generated. To do this, prepare a version file as described below. Then run the register_new_version.py script to store the information in the database. The script will return a version number, which then should be set by hand in process_new_offline_data.py - future versions of the script will streamline this part of the procedure. An example of how to generate a new version is:<br />
<syntaxhighlight><br />
./register_new_version.py add /home/gxproj1/halld/monitoring/process/versions/vers_RunPeriod-2014-10_pass1.txt<br />
</syntaxhighlight><br />
<br />
<b>If you are running the offline monitoring by checking out the files in trunk/scripts/monitoring/jproj/projects/<br />
of the svn repository</b>, and created a project with<br />
create_project.sh [project name] hd_rawdata<br />
Then go to the directory [project name]/processing/<br />
and execute<br />
./run_processing.sh<br />
which will run register_new_version.py as well as check_monitoring_data.csh for that project.<br />
<br />
===Step-by-Step Instructions For Processing a New Monitoring Run===<br />
<br />
The monitoring runs are current run out of the gxproj1 and gxproj5 accounts. After an offline monitoring run has been successfully started on the batch farm, the following steps should be followed to setup the post-processing for these runs.<br />
<br />
# The post-processing scripts are stored in $HOME/halld/monitoring/process and are automatically run by cron.<br />
# Run "svn update" to bring any changes in. Be sure that the list of histograms and macros to plot are current.<br />
# Add a new data version<br />
# Edit check_monitoring_data.csh to point to the current revisions/directories<br />
#* VERSION<br />
#* ARGS<br />
#* Note that the environment depends on a standard script - $HOME/setup_jlab.csh<br />
# Update files in the web directory, so that the results are displayed on the web pages: /group/halld/www/halldweb/html/data_monitoring/textdata<br />
# Copy the REST files to more permanent locations:<br />
#* cp -a /volatile/halld/offline_monitoring/RunPeriod-YYYY-MM/verVV/REST /work/halld/data_monitoring/RunPeriod-YYYY-MM/REST/verVV<br />
#* cp -a /volatile/halld/offline_monitoring/RunPeriod-YYYY-MM/verVV/REST /cache/halld/RunPeriod-YYYY-MM/REST/verVV [under testing]<br />
<br />
Check log files in $HOME/halld/monitoring/process/log for more information on how each run went. If there are problems, check log files, and modify check_monitoring_data.csh to vary the verbosity of the output.<br />
<br />
==Data Versions==<br />
<br />
To document the conditions of the monitoring data that is created, for the sake of reproducability and further analysis we save several pieces of information. The format is intended to be comprehensive enough to document not just monitoring data, but versions of raw and reconstructed data, so that this database table can be used for the event database as well.<br />
<br />
We store one record per pass through one run period, with the following structure:<br />
<br />
{| class="wikitable"<br />
! Field !! Description<br />
|-<br />
| data_type || The level of data we are processing. For the purposes of monitoring, "rawdata" is the online monitoring, "recon" is the offline monitoring<br />
|- <br />
| run_period || The run period of the data<br />
|- <br />
| revision || An integer specifying which pass through the run period this data corresponds to<br />
|- <br />
| software_version || The name of the XML file that specifies the different software versions used<br />
|-<br />
| jana_config || The name of the text file that specifies which JANA options were passed to the reconstruction program<br />
|-<br />
| ccdb_context || The value of JANA_CALIB_CONTEXT, which specifies the version of calibration constants that were used<br />
|-<br />
| production_time || The data at which monitoring/reconstruction began<br />
|-<br />
| dataVersionString || A convenient string for identifying this version of the data<br />
|}<br />
<br />
<br />
An example file used as as input to ./register_new_version.py is:<br />
<syntaxhighlight><br />
data_type = recon<br />
run_period = RunPeriod-2014-10<br />
revision = 1<br />
software_version = soft_comm_2014_11_06.xml<br />
jana_config = jana_rawdata_comm_2014_11_06.conf<br />
ccdb_context = calibtime=2014-11-10<br />
production_time = 2014-11-10<br />
dataVersionString = recon_RunPeriod-2014-10_20141110_ver01<br />
</syntaxhighlight></div>Kmoriyahttps://halldweb.jlab.org/wiki/index.php?title=Data_Monitoring_Procedures&diff=71850Data Monitoring Procedures2015-12-11T04:19:45Z<p>Kmoriya: /* Post-analysis of statistics of the launch */</p>
<hr />
<div>__TOC__<br />
<br />
== Master List of File / Database / Webpage Locations ==<br />
=== Run Conditions ===<br />
* Online Run-by-run condition files (B-field, current, etc.): /work/halld/online_monitoring/conditions/<br />
* Offline monitoring run conditions (software versions, jana config): /group/halld/data_monitoring/run_conditions/<br />
*[http://www.jlab.org/Hall-D/test/RunInfo/ Run Info vers. 1]<br />
*[https://halldweb.jlab.org/cgi-bin/data_monitoring/run_conditions.pl Run Info vers. 2]<br />
<br />
=== Monitoring Output Files ===<br />
* Run Periods 201Y-MM is for example 2015-03, launch ver verVV is for example ver15<br />
* Online monitoring histograms: /work/halld/online_monitoring/root/<br />
* Offline monitoring histogram ROOT files: /work/halld/data_monitoring/RunPeriod-201Y-MM/verVV/rootfiles<br />
* REST files (most recent launch only): /work/halld/data_monitoring/RunPeriod-201Y-MM/REST/verVV<br />
* individual files for each job (ROOT, log, etc.): /volatile/halld/offline_monitoring/RunPeriod-201Y-MM/verVV/<br />
<br />
=== Monitoring Database ===<br />
* Accessing monitoring database (on ifarm): mysql -u datmon -h hallddb.jlab.org data_monitoring<br />
<br />
=== Monitoring Webpages ===<br />
*[https://halldweb.jlab.org/cgi-bin/data_monitoring/monitoring/plotBrowser.py Plot Browser]<br />
*[https://halldweb.jlab.org/cgi-bin/data_monitoring/monitoring/runBrowser.py Run Browser]<br />
*[https://halldweb.jlab.org/cgi-bin/data_monitoring/monitoring/timeSeries.py Time Series]<br />
*[https://halldweb.jlab.org/data_monitoring/launch_analysis/ Launch Analysis]<br />
<br />
== Job Monitoring Links ==<br />
* [http://scicomp.jlab.org/farm2/job.html Completed Job History]<br />
* [http://scicomp.jlab.org/farm2/project.html Job Stats By Project]<br />
* [http://scicomp.jlab.org/farm2/trackOrg.html Job Stats By Track]<br />
* [http://scicomp.jlab.org/farm2/report.html Cluster Report]<br />
* [http://scicomp.jlab.org/farm2/walltime.html Walltime Distribution]<br />
<br />
== Saving Online Monitoring Data ==<br />
<br />
The procedure for writing the data out is given in, e.g.,<br />
[https://halldweb.jlab.org/wiki/index.php/Raid-to-Silo_Transfer_Strategy Raid-to-Silo Transfer Strategy].<br />
<br />
Once the DAQ writes out the data to the raid disk, cron jobs will copy the file to tape,<br />
and within ~20 min., we will have access to the file on tape at<br />
/mss/halld/$RUN_PERIOD/rawdata/RunXXXXXX.<br />
<br />
All online monitoring plugins will be run as data is taken.<br />
They will be accessible within the counting house via RootSpy, and<br />
for each run and file, a ROOT file containing the histograms will be saved<br />
within a subdirectory for each run.<br />
<br />
For immediate access to these files, the raid disk files may be accessed directly<br />
from the counting house, or the tape files will be available within ~20 min. of the<br />
file being written out.<br />
<br />
== Offline Monitoring: Running Over Archived Data ==<br />
<br />
Once files are written to tape we run the online plugins on these files to confirm what we were seeing in the online monitoring, and also to update the results from the latest calibration and software. <br />
Manual scripts and cron jobs are set up to look for new data and run the plugins over a sample of files.<br />
<br />
Every other Friday (usually the Friday before the offline meetings) jobs will be started to run the newest software on all previous runs,and allows everybody to see improvements in each detector.<br />
For each launch, independent builds of hdds, sim-recon, the monitoring plugins, and an sqlite file will be generated.<br />
<br />
Below the procedures are described for<br />
# Preparing the software for the launch<br />
# Starting the launch (using hdswif)<br />
# Post-analysis of statistics of the launch<br />
<br />
Processing the results and making them available to the collaboration<br />
is handled in the section [[#Post-Processing_Procedures | Post-Processing Procedures]] below.<br />
<br />
<!--<br />
==== Generating an offline plugin job ====<br />
<br />
The user gxproj1 should be used for official offline monitoring jobs.<br />
Within /home/gxproj1/halld/monitoring/batch/ there will be scripts to run the online monitoring plugins over tape files.<br />
The main script is generatejobs_plugins_rawdata.sh, which can be used as<br />
generatejobs_plugins_rawdata.sh [minrun] [maxrun] (minfile) (maxfile)<br />
where minrun, maxrun specify the range of the run #, and minfile, maxfile (optional) specify<br />
the file range for that run.<br />
<br />
This will generate a script run_rawdata_XXXXXX.sh, where the run # has now been formatted to be 6 digits.<br />
Executing this script will send the monitoring plugins job to the Auger batch system.<br />
<br />
There is also a script clean.sh which can be used as<br />
clean.sh XXX<br />
This will clean up all associated files created in association with run XXX.<br />
<br />
Internally, the xml file used to submit the job will be created, and the job to run<br />
will be given within script.sh. All run parameters should be specified in at the beginning<br />
of generatejobs_plugins_rawdata.sh<br />
Since we are running on tape, the tape file will first be copied over to the cache disk, and the job will run<br />
over this cached file.<br />
<br />
==== Using cron to run automatically ====<br />
Within /home/gluex/halld/monitoring/cron/ there is a file cron_plugins<br />
that can be executed via<br />
crontab cron_plugins<br />
This will set up a cron job to call the script scan_for_jobs.sh, which will<br />
check in the rawdata directory and call generatejobs_plugins_rawdata.sh for<br />
any run that is more than 5 min old. The cron job is set up to run every 10 min.<br />
<br />
==== Magnetic Field Settings ====<br />
For tracking, it is necessary to set the correct magnetic field settings.<br />
The below table is now obsolete. Please refer to the [https://halldweb.jlab.org/cgi-bin/data_monitoring/run_conditions.pl GlueX Run Conditions] webpage.<br />
<br />
The actual field settings for each run have not been documented well.<br />
Below are values based on going through entries in the [https://logbooks.jlab.org/book/halld halld log].<br />
<br />
{| border="1" cellpadding="1" style="text-align: center;"<br />
!width="100"| Run #s<br />
!width="150"| Solenoid Current (A)<br />
!width="300"| JANA option<br />
!width="300"| notes<br />
|-<br />
| 940 - 996 || 1000 || -PBFIELD_MAP=Magnets/Solenoid/solenoid_1000A_poisson_20141104 || At run 997, the solenoid started ramping down<br />
|-<br />
| 998 - 1448 || 0 || -PBFIELD_TYPE=NoField -PDEFTAG:DTrackCandidate=StraightLine || Solenoid started ramping up around run 1431. Runs 1432 - 1448 should not be used for tracking. See [https://logbooks.jlab.org/entry/3309039 this] and [https://logbooks.jlab.org/entry/3309092 this] entry.<br />
|-<br />
| 1449 - 1620 || 1200 || -PBFIELD_MAP=Magnets/Solenoid/solenoid_1200A_poisson_20140520 || <br />
|-<br />
|}<br />
<br />
'''Note:''' The following run ranges were taken with a 300A field, and could be analyzed with the magnetic field map Magnets/Solenoid/solenoid_300A_poisson_20140819 , more study is needed to see if the straight line track fitter is needed or not: 1036 - 1053, 1065 - 1121, 1212? - 1254, 1309 - 1318 [1307,1308,1319-1329 were taken while the magnet was ramping].<br />
--><br />
<br />
=== General Information on Procedures ===<br />
This section explains how the offline monitoring should be run.<br />
Since we may want to simultaneously run offline monitoring for different run periods that require<br />
different environment variables, the scripts are set up so that a generic user can download the<br />
scripts and run them from anywhere. Most output directories for offline monitoring are created<br />
with group read/write permissions so that any Hall D group user has access to the contents,<br />
but there are some cases where use of the account that created the launch is necessary.<br />
<br />
The accounts used for offline monitoring are the gxprojN accounts created and maintained by<br />
Mark Ito (see [https://halldweb.jlab.org/wiki/index.php/GlueX-related_shared_accounts_on_the_JLab_CUE here] for how each account is used).<br />
As of October 2015, the following are used:<br />
* gxproj1 for running over Fall 2014 data (deprecated since June 2015)<br />
* gxproj5 for running over Spring 2015 data<br />
<br />
Since the summer of 2015 we have transitioned from a system using Mark Ito's jproj scripts<br />
to integrating the swif system that Chris Larrieu (SciComp) has been developing. For offline monitoring,<br />
the hdswif system that Kei developed is used for launching the jobs, and the jproj system is used<br />
for meta-analysis of launch statistics.<br />
<br />
Both hdswif and jproj are maintained in svn:<br />
* hdswif: https://halldsvn.jlab.org/repos/trunk/scripts/monitoring/hdswif<br />
* jproj : https://halldsvn.jlab.org/repos/trunk/scripts/monitoring/jproj<br />
<br />
To run the offline monitoring each package should be checked out and all necessary<br />
scripts are included.<br />
<br />
=== Preparing the software for the launch ===<br />
<br />
1. Setup the environment: <pre>source ~/env_monitoring_launch</pre><br />
<br />
2. Updating & building hdds: <br />
<pre><br />
cd $HDDS_HOME<br />
git pull # Get latest software<br />
scons -c install # Clean out the old install: EXTREMELY IMPORTANT for cleaning out stale headers<br />
scons install -j4 # Rebuild and re-install with 4 threads<br />
</pre><br />
<br />
3. Updating & building sim-recon: <br />
<pre><br />
cd $HALLD_HOME/src<br />
git pull # Get latest software<br />
scons -c install # Clean out the old install: EXTREMELY IMPORTANT for cleaning out stale headers<br />
scons install -j4 # Rebuild and re-install with 4 threads<br />
</pre><br />
<br />
4. Create a new sqlite file containing the very latest calibration constants. Original documentation on creating sqlite files are [https://halldweb.jlab.org/wiki/index.php/SQLite-form_of_the_CCDB_database here]. <br />
<pre><br />
cd $GLUEX_MYTOP/../sqlite/<br />
$CCDB_HOME/scripts/mysql2sqlite/mysql2sqlite.sh -hhallddb.jlab.org -uccdb_user ccdb | sqlite3 ccdb.sqlite<br />
mv ccdb.sqlite ccdb_monitoring_launch.sqlite #replacing the old file<br />
</pre><br />
<br />
5. Note that the above steps must be done <b>BEFORE</b> launch project creation. This is because we will track the revisions of the libraries used, and this is done by extracting the svn information in each directory. Also, note that the system assumes that we have the topmost build directory (usually called GLUEX_TOP) to be $HOME/builds . Such an assumption is necessary to be able to extract information about the library locations automatically.<br />
<br />
Create the appropriate project(s) and submit the jobs using hdswif, as detailed in the section below.<br />
<br />
=== Starting the Launch and Submitting Jobs ===<br />
<br />
Until the summer of 2015 we relied solely on Mark Ito's jproj system for submitting and keeping track of jobs. We have since moved to the swif system and use the hdswif wrapper for this. Below are instructions for how to use these.<br />
<br />
1. Downloading hdswif: Download the hdswif directory from svn. For the gxprojN accounts, use the directory ~/halld/hdswif. <br />
<pre><br />
cd ~/halld<br />
svn co https://halldsvn.jlab.org/repos/trunk/scripts/monitoring/hdswif<br />
cd hdswif<br />
</pre><br />
<br />
2. Edit the job config file, input.config, which is used to register jobs in hdswif. A typical config file will look this:<br />
PROJECT gluex<br />
TRACK reconstruction<br />
OS centos65<br />
NCORES 6<br />
DISK 40<br />
RAM 8<br />
TIMELIMIT 8<br />
JOBNAMEBASE offmon_<br />
RUNPERIOD 2015-03<br />
VERSION 15<br />
OUTPUT_TOPDIR /volatile/halld/offline_monitoring/RunPeriod-[RUNPERIOD]/ver[VERSION] # Example of other variables included in variable<br />
SCRIPTFILE /home/gxproj5/halld/hdswif/script.sh # Must specify full path<br />
ENVFILE /home/gxproj5/halld/hdswif/setup_jlab-2015-03.csh # Must specify full path<br />
<br />
3. Creating the workflow: Within SWIF jobs are registered into workflows. For offline monitoring, the workflow names are of the form <b>offline_monitoring_RunPeriod201Y_MM_verVV_hd_rawdata</b> with suitable replacements for the run period and version number. The command "swif list" will list all existing workflows. Also, for most simple SWIF commands hdswif also provides a wrapper. <br />
swif list<br />
<br />
For creation of workflows for offline monitoring the command:<br />
hdswif.py create [workflow] -c input.config<br />
should be used. When a config file (here: input.config) is passed in, hdswif will automatically create files that record the configuration of the current launch. These files are stored as for example:<br />
/group/halld/data_monitoring/run_conditions/jana_rawdata_comm_2015_03_ver15.conf<br />
/group/halld/data_monitoring/run_conditions/soft_comm_2015_03_ver15.xml<br />
<br />
The software packages stored in git (sim-recon and hdds) can have git tags applied to them, which makes it easier to find versions of the software than a SHA-1 hash. hdswif will ask if you would like to create a tag, and execute the following sequence: <pre>git tag -a offmon-201Y_MM-verVV -m "Used for offline monitoring 201Y-MM verVV started on 201y/mm/dd"</pre> <pre>git push offmon-201Y_MM-verVV</pre> This will only be invoked when the user name is gxprojN, and for the configuration files, the output directory will be /group/halld/data_monitoring/run_conditions/ for gxprojN accounts while it will be the current directory for other users.<br />
* To use the git-tagged software versions do for example <pre>cd $HALLD_HOME</pre> <pre>git checkout offmon-2015_03-ver15</pre><br />
<br />
4. Registering jobs in the workflow: To register jobs within the workflow, hdswif provides the use of config files. Jobs can be registered by specifying the workflow, config file (-c), run (-r) and file (-f) numbers if necessary. <br />
<span style="color:red">Note: Job configuration parameters can be set differently for jobs within the same workflow if necessary.</span><br />
Jobs can be added via <pre>hdswif.py add [workflow] -c input.config</pre><br />
By default, hdswif will add all files found within the directory /mss/halld/RunPeriod-201Y-MM/rawdata/ where 201Y-MM is specified by the RUNPERIOD parameter in the config file. If only some of the runs or files are needed, these can be specified for example with<br />
<pre>hdswif.py add [workflow] -c input.config -r 3180 -f '00[0-4]'</pre><br />
to specify to register running only over run 3180 files 000 - 004 (Unix-style brackets and wildcards can be used).<br />
<br />
5. Running the workflow: To run the workflow, simply use the hdswif wrapper:<br />
hdswif.py run [workflow]<br />
<br />
<b>It is recommended that some jobs be tested to make sure that everything is working rather than fail thousands of jobs.</b><br><b>Check the configuration setup when creating a workflow, and what is in script.sh. Also check the following:<br />
* Check stderr files. Are they small (<kB)?<br />
* Check stdout files. Are they very large (>MB)?<br />
* Check output ROOT files. Are they larger than several MB?<br />
* Check output REST files. Are they larger than several tens of MB?<br />
</b><br />
<br> For this purpose, hdswif will take an additional parameter to run which limits the number of jobs to submit: <pre>hdswif.py run [workflow] 10</pre><br />
in which case only 10 jobs will be submitted.<br />
<br />
To submit all jobs after checking the results, do <pre>hdswif.py run [workflow]</pre><br />
<br />
=== Checking the Status and Resubmitting ===<br />
1. The status of jobs can be checked on the terminal with <pre>jobstat -u gxprojN</pre> For the status of jobs on Auger see http://scicomp.jlab.org/scicomp/#/auger/jobs and for SWIF use <pre>swif list</pre> or for more information, <pre>swif status [workflow] -summary</pre> Note that "swif status" tends to be out of date sometimes, so don't panic if your workflow/jobs aren't showing up right away. Also see the Auger [http://scicomp.jlab.org/scicomp/#/auger/jobs job website].<br />
<br />
2. For failed jobs, SWIF can resubmit jobs based on the problem. For resubmission for failed jobs with the same resources, <pre>swif retry-jobs [workflow] -problems [problem name]</pre> can be used, and for jobs to be submitted with more resources, e.g., use <pre>swif modify-jobs [workflow] -ram add 2gb -problems AUGER-OVER_RLIMIT</pre> <br />
This only re-stages the jobs, be sure to resubmit them with:<br />
<pre>swif run -workflow [workflow] -errorlimit none</pre><br />
<br />
hdswif has a wrapper for both of these: <pre>hdswif.py resubmit [workflow] [problem]</pre> In this case [problem] can be one of <b>SYSTEM, TIMEOUT, RLIMIT</b>. If SYSTEM is specified, the jobs will be retried. For TIMEOUT and RLIMIT, the jobs will be modified by default with 2 additional hours or GB of RAM. If one more number is added as an option, then that many hours or GB of RAM will be added., e.g., <pre>hdswif.py resubmit [workflow] TIMEOUT 5</pre> will add 5 hours of processing time. You can wait until almost all jobs finish before resubmitting failed jobs since the number should be relatively small. <b>Even if jobs are resubmitted for one type of failure, jobs that later fail with that failure will not be automatically resubmitted.</b><br />
<br />
3. For information on swif, use the "swif help" commands and for hdswif see the attached documentaion in https://halldsvn.jlab.org/repos/trunk/scripts/monitoring/hdswif/manual_hdswif.pdf<br />
<br />
4. Below is a table describing the various errors that can occur.<br />
{| border="1" cellpadding="0" valign="left" style="text-align: left;"<br />
!width="150"| ERROR NAME<br />
!width="400"| Description<br />
!width="400"| Resolution<br />
!width="400"| hdswif command<br />
|-<br />
| AUGER-SUBMIT<br />
||<br />
SWIF’s attempt to submit jobs to Auger failed. Includes server-side problems as well as user failing to provide valid job parameters (e.g. incorrect project name, too many resources, etc.)<br />
||<br />
If requested resources are known to be correct resubmit. Otherwise modify job resources using swif directly.<br />
||<br />
<span style="color:purple"><b>hdswif.py resubmit [workflow] SYSTEM</b></span><br />
|-<br />
| AUGER-FAILED<br />
||<br />
Auger reports the job FAILED with no specific details.<br />
||<br />
Resubmit jobs. If problem persists, contact Chris Larrieu or SciComp.<br />
||<br />
<span style="color:purple"><b>hdswif.py resubmit [workflow] SYSTEM</b></span><br />
|-<br />
| AUGER-OUTPUT-FAIL<br />
||<br />
Failure to copy one or more output files.Can be due to permission problem, quota problem, system error, etc.<br />
||<br />
Check if output files will exist after job execution and that output directory exists, resubmit jobs. If problem persists, contact Chris Larrieu or SciComp.<br />
||<br />
<span style="color:purple"><b>hdswif.py resubmit [workflow] SYSTEM</b></span><br />
|-<br />
| AUGER-INPUT-FAIL<br />
||<br />
Auger failed to copy one or more of the requested input files, similar to output failures. Can also happen if tape file is unavailable (e.g. missing/damaged tape)<br />
||<br />
Check if input file exists, resubmit jobs. If problem persists, contact Chris Larrieu or SciComp.<br />
||<br />
<span style="color:purple"><b>hdswif.py resubmit [workflow] SYSTEM</b></span><br />
|-<br />
| AUGER-TIMEOUT<br />
||<br />
Job timed out.<br />
||<br />
If more time is needed for job add more resources.<br />
Default is to add 2 hrs of processing time. Also check whether code is hanging.<br />
||<br />
<span style="color:red"><b>hdswif.py resubmit [workflow] TIMEOUT</b></span><br><br />
Default is to add 2 hours. Optionally specify number of hours at end.<br />
|-<br />
| AUGER-OVER_RLIMIT<br />
||<br />
Not enough resources, RAM or disk space.<br />
||<br />
Add more resources for job.<br />
||<br />
<span style="color:blue"><b>hdswif.py resubmit [workflow] RLIMIT</b></span><br><br />
Default is to add 2 GB of RAM. Optionally specify GB at end. To add more disk space use SWIF directly.<br />
|-<br />
| SWIF-MISSING-OUTPUT<br />
||<br />
Output file specified by user was not found.<br />
||<br />
Check if output file exists at end of job.<br />
||<br />
<br />
|-<br />
| SWIF-USER-NON-ZERO <br />
||<br />
User script exited with non-zero status code.<br />
||<br />
Your script exited with non-zero status. Check the code you are running.<br />
||<br />
<br />
|-<br />
| SWIF-SYSTEM-ERROR <br />
||<br />
Job failed owing to a problem with swif (e.g. network connection timeout)<br />
||<br />
Resubmit jobs. If problem persists, contact Chris Larrieu or SciComp.<br />
||<br />
<span style="color:purple"><b>hdswif.py resubmit [workflow] SYSTEM</b></span><br />
|}<br />
<br style="clear:both;"/><br />
<br />
=== Post-analysis of statistics of the launch ===<br />
<br />
After jobs have been submitted, it will usually take a few days for all of the jobs to be processed.<br />
The next step is to check the resource usage for the current launch and publish the results online.<br />
<br />
# <b>Create summary XML, HTML files</b><br> The status and results of jobs are saved within the SWIF internal server, and are available via the command <pre>swif status [workflow] -summary -runs</pre> where the arguments -summary and -runs show summary statistics and statistics for individual jobs, respectively. hdswif has a command that takes this output in XML output and creates an HTML webpage showing results of the launch. To do this, do <pre>hdswif.py summary [workflow]</pre> This will create the XML output file from SWIF called swif_output_[workflow].xml and create a webpage containing png figure files. If the XML file already exists, hdswif will ask whether to overwrite the existing file.<br />
# <b>Publish output files online</b><br> At this stage the html output and figure files are created and ready to be put online. The html output capabilities of hdswif is useful for any user using SWIF, but since publication of the html output is specific to the offline monitoring, the script to do this is contained in the jproj scripts directory at https://halldsvn.jlab.org/repos/trunk/scripts/monitoring/cross_analysis. For the gxprojN accounts this directory should exist as ~/halld/monitoring/cross_analysis. To publish the results online do for example <pre>python ~/halld/monitoring/cross_analysis/publish_offmon_results.py 2015_03 18</pre> The script copies the html output and corresponding figures to /group/halld/www/halldweb/html/data_monitoring/launch_analysis/ and also creates a link to it in the html page.<br />
# <b>Editing the summary HTML page</b><br> The top page for is offline monitoring https://halldweb.jlab.org/data_monitoring/launch_analysis/index.html and has links to the summary page for each run period. The summary files are /group/halld/www/halldweb/html/data_monitoring/launch_analysis/[run period].html Edit the file to<br />
## Add a new line to the first table which contains the version number, date, and comments for the current launch<br />
## Create a link to the webpage for the current launch. Simply copy and paste, modify the previous launch's link to have the correct launch ver.<br />
# <b>Freezing SWIF tables</b><br> Since we are now finished with the SWIF workflow that we used, the workflow should be "frozen" so that it cannot be mistakenly altered or modified. Do <pre>swif freeze [workflow]</pre><br />
# <b>Backing up SWIF output</b><br> With the workflow frozen we should be able to reproduce the XML output from SWIF, but we will backup the XML output just in case. Do <pre>cp ~/halld/hdswif/swif_output_[workflow].xml /group/halld/data_monitoring/swif_xml_backup/ </pre><br />
<br />
=== Cross Analysis of Launches ===<br />
<br />
The purpose of the cross analysis is to correlate how resource usage changed for the same files across different launches.<br />
To do this it is useful to create MySQL tables that contain information on each launch, and do queries across tables.<br />
<br />
#The scripts to do this are maintained in the svn directory https://halldsvn.jlab.org/repos/trunk/scripts/monitoring/jproj Do <pre> svn co https://halldsvn.jlab.org/repos/trunk/scripts/monitoring/jproj </pre> For the gxprojN accounts used for offline monitoring the directory should be ~/halld/jproj<br />
# The jproj directory contains two subdirectories, scripts and projects. The scripts directory contains useful scripts for processing the jobs registered in the jproj system, and each of the offline monitoring launches will be handled in the projects directory.<br />
# Go to the projects directory<pre>cd ~/halld/jproj/projects</pre> and use the script create_project.sh to create a new directory that contains the processing scripts for the current launch <pre>./create_project.sh [workflow]</pre> This should create a directory such as offline_monitoring_RunPeriod2015_03_ver15_hd_rawdata (same as the workflow name). The script uses the template files in the directory templates and by substitution creates script files for the current launch. Now go to the newly created analysis directory: <pre>cd [workflow]/analysis</pre><br />
which for the gxprojN accounts should have the full path /home/gxprojN/halld/jproj/projects/[workflow]/analysis.<br><br><br />
<span style="color:red">All of the analysis commands including arguments are contained in run_analysis.sh, but it is <b>strongly</b> recommended that all commands are run manually to check for errors.</span><br><br><br />
# Next, we need to create a few MySQL tables for the current launch. The MySQL tables are useful for comparing run/file combinations for different launches. For the SWIF launches, there are two tables needed, and will be named<br />
#* [workflow]Job<br />
#* [workflow]_aux<br />
# This naming scheme of tables and their roles are the same as from the jproj only launches. The [workflow]Job table will contain information gathered from SWIF about each job (which node it went to, start time of each stage, memory usage, etc.). The [workflow]_aux table will contain information gathered from the output stdout files from each job. First, create the Job table using <pre>python create_jproj_job_table.py [run period] [version]</pre> Note that this script uses the XML output from hdswif summary and inserts the contents into the MySQL table, so the XML output file must exist.<br />
# To check the contents of this MySQL table, do <pre>mysql -hhalldb -ufarmer farming</pre> <pre>mysql> describe offline_monitoring_RunPeriod2015_03_ver11_hd_rawdataJob;</pre><br />
# <b>Backing up tables</b><br> Tables created in MySQL should be backed up.<br />
<br />
== Explanation of Scripts ==<br />
Below are explanations of each script used in the offline monitoring system and a breif explanation of how they work.<br />
<br />
=== hdswif scripts ===<br />
<b>Summary:</b> hdswif.py is the main script that calls the other utility scripts. For the utility scripts, they can be run standalone by giving the appropriate parameters. Visual graphics are made using the PyROOT extension of ROOT. To use this, the environment variable PYTHONPATH must include ROOTSYS. If ROOTSYS has been set, adding it to PYTHONPATH will be done by the script, but if ROOTSYS is not set, then the scripts will abort.<br />
<br />
{| border="1" cellpadding="0" valign="left" style="text-align: left;"<br />
!width="150"| file name<br />
!width="800"| Description<br />
|-<br />
| hdswif.py ||<br />
Main script to control the behavior of SWIF. Most commands follow the form hdswif.py [command] [workflow] (options)<br />
|-<br />
| parse_swif.py ||<br />
Called within hdswif.py to create html output from SWIF results.<br />
|-<br />
| createXMLfiles.py ||<br />
* Called by the "create" option of hdswif.py<br />
* Creates XML files for logging information about launch. Must specify config file with option -c.<br />
* Also adds tags to git repositories of sim-recon and hdds.<br />
* For XML file creation, the file will be written out to /group/halld/data_monitoring/run_conditions/ if the user is gxprojN (used for offline monitoring). If not, to avoid general users adding things to the above directory, the output files will be the current directory.<br />
* To write out the versions of each package, environment variables such as HDDS_HOME will need to be set.<br />
* Versions for each software package are extracted using the directory structure, so if these are changed the scripts must change accordingly.<br />
* For each launch, <b>the output soft_comm_[RUNPERIOD]_ver[VER].xml file should be checked that all version numbers have been extracted.</b><br />
|-<br />
| read_config.py ||<br />
* Called by the "add" option of hdswif.py<br />
* Takes in config file name, optionally set verbose<br />
* Return a dictionary between config parameter names and values (e.g., 'PROJECT' : 'gluex', 'NCORES' : 6, ...)<br />
* Prints the config parameters at the end. For parameters changed from default, a '*' will be printed<br />
|-<br />
| output_job_details.py ||<br />
* Called by the "details" option of hdswif.py<br />
* Takes in workflow name, run and file. Run and file must be numbers, no wildcards<br />
* Finds ids for jobs specified by the run and file number and returns info on each one<br />
* Job info is retrieved from pbs farm system and shows configuration parameters for that job<br />
|-<br />
| results_by_resources.py ||<br />
* Called within parse_swif.py<br />
* Takes in XML output from SWIF and creates 2 plots<br />
* Creates html table showing results of jobs by resources requested for that job<br />
* This table is shown under "Status by Resources" in the output html file of hdswif.py summary [workflow]<br />
|-<br />
| create_ordered_hists.py ||<br />
* Called within parse_swif.py<br />
* Takes in XML output from SWIF and creates 2 plots<br />
* Plots are dependency time and pending time of each job, ordered by submission.<br />
* Different colors represent jobs submitted at different times. For jobs submitted at the same time, jobs are ordered in increasing time.<br />
|-<br />
| create_stacked_times.py ||<br />
* Called within parse_swif.py<br />
* Takes in XML output from SWIF and creates 2 plots<br />
* Plots show total job time divided into colors for different stages<br />
* One shows all jobs in order of Auger ID (roughly submission order), the other one shows in order of total job time<br />
* The jobs show at a glance which stage contributes how much of the job's time<br />
|}<br />
<br />
=== Utility scripts ===<br />
{| border="1" cellpadding="0" valign="left" style="text-align: left;"<br />
!width="150"| file name<br />
!width="800"| Description<br />
|-<br />
| stderr_by_size.py ||<br />
* Independent of all other scripts in hdswif directory<br />
* When diagnosing problems it is useful to check the stderr/stdout files. Frequently, different problems are easier to find based on the size of stderr files<br />
* Takes in run period and version as arguments, creates a directory /volatile/halld/offline_monitoring/RunPeriod-[RUNPERIOD]/ver[VER]/log/bysize that contains soft links to all stdout and stderr files from the specified launch, <b>in separate directories given by stderr file size</b>.<br />
|}<br />
<br />
=== cross_analysis scripts ===<br />
<b>Summary:</b> The script run_cross_analysis.sh is the main script. In principle, running this with <pre>./run_cross_analysis.sh [RUNPERIOD] [VERSION]</pre> should work, but it is recommended that the commands within the script<br />
be run by hand to catch any errors.<br />
<br />
{| border="1" cellpadding="0" valign="left" style="text-align: left;"<br />
!width="150"| file name<br />
!width="800"| Description<br />
|-<br />
| run_cross_analysis.sh ||<br />
* Main script to call other Python scripts.<br />
* Takes in run period and version as arguments and runs cross analysis<br />
|-<br />
| create_cross_analysis_table.sh ||<br />
* Creates MySQL table for current launch that is used by other scripts<br />
* Takes run period and version as input, and creates table cross_analysis_table_[RUNPERIOD]_ver[VERSION]<br />
|-<br />
| create_stats_table_row.py ||<br />
* Create a row in the html table showing the overall statistics of final states and problems for a launch<br />
* The final states are "Success", and "Segfault". "Success" includes all jobs that had problems but still finished with Success.<br />
* The problems are "Over Limit", "Timeout", and "System". If any of these occurred for any attempt of the job they are counted.<br />
* The script creates a new html table row for the current launch. This html snippet is inserted into the web-accessible file showing the results <br />
|-<br />
| create_stats_for_each_file.py ||<br />
* Create html tables showing the status of each file against different launch versions.<br />
* Takes in run period, min version, max version and shows final result and problems for all versions in between<br />
* Different final results and problems are shown by combinations of ext content, olor coding of text, background color<br />
|-<br />
|-<br />
| create_resource_correlation_plots.py ||<br />
* Create plots showing correlation of resources between different launches for each file.<br />
* Takes in run period, min version, version of interest. Points are shown only for files included in version of interest<br />
* Creates plots for CPU time, Wall time, memory, virtual memory, #events, difference in #events, time to copy input evio file, time to run plugin<br />
|}<br />
<br />
== Hall D Job Management System ==<br />
<br />
This section details instructions on how to create and launch a set of jobs using the Hall-D Job Management System developed by Mark Ito. These instructions are generic: this system can be used for the weekly monitoring jobs, but can also be used for other sets of job launches as well. <br />
<br />
=== Database Table Overview ===<br />
<br />
* Job management database table (<project_name>): For each input file, keeps track of whether or not a job for it has been submitted, along with other optional fields. <br />
<br />
* Job status database table (<project_name>Job (no space)): For each job, keeps track of the job-id, the job status, memory used, cpu & wall time, time taken to complete various stages (e.g. pending, dependency, active), and others. <br />
<br />
* Job metrics database table (<project_name>_aux (no space)): For each job, keeps track of the job-id, how many events were processed, the time it took to copy the cache file, and the time it took to run the plugin. This information is culled from the log files of each job, and is done within the analysis directory of each launch.<br />
<br />
=== Initialize Project Management ===<br />
<br />
* Log into the ifarm machine with one of the gxproj accounts. For this example we will use gxproj1.<br />
<pre><br />
ssh gxproj1@ifarm -Y<br />
</pre><br />
<br />
* Go to a directory to do the launch. In principle, any directory will work, but for gxproj1 this is usually done in /home/gxproj1/halld/<br />
<br />
* Check out the necessary scripts <pre>svn co https://halldsvn.jlab.org/repos/trunk/scripts/monitoring/jproj</pre> This will get all necessary scripts for launching. Once checked out, <pre>cd projects</pre><br />
<br />
* The script create_project.sh can be used to create a new project. It will take a single argument, the project name. It is assumed that the project name is of the form offline_monitoring_RunPeriod20YY_MM_verVV_hd_rawdata where this string will be parsed to give the run period 20YY_MM and version number VV. One thing to do BEFORE creation of a new project is to editthe conditions of the launch (plugins to run over, memory requested, disk space requested) within templates/template.jsub . This<br />
information is saved automatically at project creation time into files in /group/halld/data_monitoring/run_conditions .<br />
To create a project do <pre>./create_project.sh offline_monitoring_RunPeriod20YY_MM_verVV_hd_rawdata</pre><br />
The name has been chosen to be as consistent as possible with other directory structures.<br />
However, mysql requires that "-" be escaped in table names, so unfortunately run periods will be<br />
given as 2YYY_MM instead of 2YYY-MM.<br />
<br />
* A new directory with the project name (offline_monitoring_RunPeriod20YY_MM_verVV_hd_rawdata) and files will be copied and modified from the template directory to reflect the run period, the user, the directory that it was created in, project name, etc.<br />
<br />
* For each project, cd into the new directory<br />
* The script clear.sh will remove any existing tables for the current project name, then recreate it. Do<br />
<pre>./clear.sh</pre><br />
* To use the jproj.pl script that was checked in, add the directory to your path with<br />
<pre>source ../../scripts/setup.csh</pre><br />
or always specify the full path<br />
<pre>../../scripts/jproj.pl</pre><br />
* Now update the table of runs with<br />
<pre>jproj.pl <project name> update</pre><br />
This will fill the table with all files within /mss that are of the same form as what is in <project name>.jproj . If you want to register only on a subset of all such files, you can edit this file directly.<br />
* Once you have registered all of the files you would like to run over, do<br />
<pre>jproj.pl <project name> submit [max # of jobs] [run number]</pre><br />
where the additional options specify how many jobs to submit and which run number to run on. Without these options all files that are registered and have not been submitted yet will be submitted.<br />
<br />
At this stage you are ready to submit all files. It is a good idea to submit a few test jobs<br />
at first to check that all scripts are working and that the plugins do not crash. Once you are<br />
sure that this does not happen, you can send all jobs in. The remaining jobs are then the monitoring<br />
which will (among other things) put the results on the online webpage for the collaboration to view, and<br />
the analysis of the launch.<br />
<br />
=== Project File Overview ===<br />
<br />
An overview of each project file:<br />
* '''clear.sh:''' For the current project, deletes the job status and management database tables (if any), and creates new, empty ones. <br />
* '''<project_name>.jproj:''' Contains the path and file name format for the input files for the jobs. <br />
* '''<project_name>.jsub:''' The xml job submission script. The run number and file number variables are set during job submission for each input file. <br />
* '''script.sh:''' The script that is executed during the job. If output job directories are not pre-created manually, they should be created in this script with the proper permissions:<br />
<pre><br />
mkdir -p -m 775 my_directory<br />
</pre><br />
* '''setup_jlab-[run period].csh:''' The environment that is sourced at the beginning of the job execution. <br />
* '''status.sh:''' Updates the job status database table, and prints some of its columns to screen.<br />
<br />
=== Project Management ===<br />
<br />
* Delete (if any) and create the database table(s) for the current set of job submissions:<br />
<pre><br />
./clear.sh<br />
</pre><br />
Also, if testing was done with jobs, it is best to delete the output directory and the configuration files:<br />
<pre><br />
rm -frv /volatile/halld/offline_monitoring/RunPeriod-20YY-MM/verVV /group/halld/data_monitoring/run_conditions/soft_comm_20YY_MM_verVV.xml /group/halld/data_monitoring/run_conditions/jana_rawdata_comm_20YY_MM_verVV.conf<br />
</pre><br />
<br />
* Search for input files matching the string in the .jproj file, and create a row for each in the job management database table (called <project_name>). You can test by adding an optional argument at the end, which only selects files with a specific file number:<br />
<pre><br />
jproj.pl <project_name> update <optional_file_number><br />
</pre><br />
<br />
* Confirm that the job management database is accurate by printing it's contents to screen:<br />
<pre><br />
mysql -hhallddb -ufarmer farming -e "select * from <project_name>"<br />
</pre><br />
<br />
* ONLY if a mistake was made, to delete the tables from the database and recreate new, empty ones, run: <br />
<pre><br />
./clear.sh<br />
</pre><br />
<br />
* To look at the status of the submitted jobs, first query auger and update the job status database:<br />
<pre><br />
fill_in_job_details.pl <project_name><br />
</pre><br />
<br />
* The job status can then be viewed by submitting a query to the job status database (called <project_name>Job (no space in between)):<br />
<pre><br />
mysql -hhallddb -ufarmer farming -e "select id,run,file,jobId,hostname,status,timeSubmitted,timeActive,walltime,cput,timeComplete,result,error from <project_name>Job"<br />
</pre><br />
<br />
* These last two commands can instead be executed simultaneously by running:<br />
<pre><br />
./status.sh<br />
</pre><br />
<br />
=== Handy mysql Instructions ===<br />
<br />
* Handy mysql instructions:<br />
<pre><br />
mysql -hhallddb -ufarmer farming # Enter the "farming" mysql database on "hallddb" as user "farmer"<br />
quit; # Exit mysql<br />
show tables; # Show a list of the tables in the current database<br />
show columns from <project_name>; # show all of the columns for the given table<br />
select * from <project_name>; # show the contents of all rows from the given table<br />
</pre><br />
<br />
=== Backing Up Offline Monitoring Tables ===<br />
Tables created for offline monitoring can be backed up using the script backup_tables.sh which<br />
can be checked out with the other files from https://halldsvn.jlab.org/repos/trunk/scripts/monitoring/jproj/projects<br />
<br />
The script uses the command mysqldump to print out a file that can be executed to recreate the tables.<br />
Since executing this output file will drop the table when it exists, caution is advised.<br />
Example usage to backup all three tables created for run period 2014_10 ver 17:<br />
<pre>backup_tables.sh 2014_10 17</pre><br />
<br />
== Running Over Data As It Comes In ==<br />
<br />
A special user gxproj1 will have a cron job set up to run the plugins as new data appears on /mss.<br />
During the week, gxproj1 will submit offline plugin jobs with the same setup as the weekly jobs<br />
run the previous Friday. The procedure for this is shown below.<br />
<br />
<!--<br />
=== Setting up the environment ===<br />
The file<br />
/home/gxproj1/setup_jlab.csh<br />
is sourced through .tcshrc.<br />
This file is the same as what is linked to by<br />
/home/gluex/setup_jlab_commissioning.csh,<br />
except HALLD_HOME, HDDS_HOME, and JANA_CALIB_URL are set separately so that this<br />
user can have a separate build.<br />
<br />
To obtain the builds from the previous Friday's runs,<br />
execute<br />
/home/gxproj1/halld/monitoring/newruns/setup_previous.sh [year] [month] [day]<br />
The build revisions from the previous Friday are archived in files<br />
/work/halld/data_monitoring/run_conditions/soft_comm_[year]_[month]_[day].xml<br />
and the script will build libraries based on those stored revision numbers.<br />
--><br />
<br />
=== Running the cron job ===<br />
<br />
'''IMPORTANT:''' The cron job should not be running while you are manually submitting jobs using the jproj.pl script for the same project, or else you will probably multiply-submit a job. <br />
<br />
* Go to the cron job directory: <br />
<pre><br />
cd /u/home/gxproj1/halld/monitoring/newruns<br />
</pre><br />
<br />
* The cron_plugins file is the cronjob that will be executed. During execution, it runs the exec.sh command in the same folder. This command takes two arguments: the project name, and the maximum file number for each run. These fields should be updated in the cron_plugins file before running. <br />
<br />
* The exec.sh command updates the job management database table with any data that has arrived on tape since it was last updated, ignoring file numbers greater than the maximum file number. It then submits jobs for these files. <br />
<br />
* To start the cron job, run:<br />
<pre><br />
crontab cron_plugins<br />
</pre><br />
<br />
* To check whether the cron job is running, do<br />
<pre><br />
crontab -l<br />
</pre><br />
<br />
* To remove the cron job do<br />
<pre><br />
crontab -r<br />
</pre><br />
<br />
<!--<br />
The cron job will run the script scan_for_jobs.sh,<br />
which runs generatejobs_plugins_rawdata.sh for any<br />
new runs that it had not seen before. All previous<br />
runs are recorded in the file filelists/files_current.txt<br />
so clear this to run over runs, or set the parameters<br />
MINRUN and MAXRUN which will set the range of runs submitted.<br />
--><br />
<br />
==Post-Processing Procedures==<br />
<br />
To visualize the monitoring data, we save images of selected histograms and store time series of selected quantities in a database, which are then displayed on a web page. This section describes how to generate the monitoring images and database information.<br />
<br />
The scripts used to generate this summary data are primarily run from /home/gxprojN/halld/monitoring/process . If you want a new copy of the scripts, e.g., for a new monitoring run, you should check the scripts out from SVN:<br />
<syntaxhighlight><br />
svn co https://halldsvn.jlab.org/repos/trunk/scripts/monitoring/process<br />
</syntaxhighlight><br />
Note that these scripts currently have some parameters which must be periodically set by hand.<br />
<br />
The default python version on most JLab machine does not have the modules to allow these scripts to connect to the MySQL database. To run these scripts, load the environment with the following command<br />
<syntaxhighlight><br />
source /home/gxproj1/halld/monitoring/process/monitoring_env.csh<br />
</syntaxhighlight><br />
<br />
===Online Monitoring===<br />
<br />
There are two primary scripts for running over the monitoring data generated by the online system and offline reconstruction. The online script can be run with either of the following commands:<br />
<syntaxhighlight><br />
/home/gluex/halld/monitoring/process/check_new_runs.py<br />
<br />
OR <br />
<br />
/home/gluex/halld/monitoring/process/check_new_runs.csh<br />
</syntaxhighlight><br />
The shell script sets up the environment properly to run the python script. To connect to the monitoring database on the JLab CUE, modules included in the local installation of python >= 2.7 are needed. The shell script is appropriate to use in a cron job. The cronjob is currently run under the "gluex" account.<br />
<br />
The online monitoring system copies a ROOT file containing the results of the online monitoring, and other configuration files into a directory accessible outside the counting house. This python script automatically checks for new ROOT files, which it will then automatically process. It contains several configuration variables that must be correctly set, which contains the location of input/output directories, etc...<br />
<br />
===Offline Monitoring===<br />
<br />
After the data is run over, the results should be processed, so that summary data is entered into the monitoring database and plots are made for the monitoring webpages. Currently, this processing is controlled by a cronjob that runs the following script:<br />
<syntaxhighlight><br />
/home/gxproj1/halld/monitoring/process/check_monitoring_data.csh <br />
</syntaxhighlight><br />
This script checks for new ROOT files, and only runs over those it hasn't processed yet. Since one monitoring ROOT file is produced for each EVIO file, whenever a new file is produced, the plots for the corresponding run are recreated and all the ROOT files for a run are combined into one file. Information is stored in the database on a per-file basis. <br />
<br />
Plots for the monitoring web page can be made from single histograms or multiple histograms using RootSpy macros. If you want to change the list of plots made, you must modify one of the following files:<br />
* histograms_to_monitor - specify either the name of the histogram or its the full ROOT path <br />
* macros_to_monitor - specify the full path to the RootSpy macro .C file<br />
<br />
When a new monitoring run is started, or the conditions are changed, the following steps should be taken to process the new files:<br />
# Add a new data version, as described below:<br />
# Change the following parameters in check_monitoring_data.csh:<br />
## JOBDATE should correspond to the ouptut date used by the job submission script<br />
## OUTPUTDIR should correspond to the directory corresponding to the run period and revision corresponding to the new version you just submitted. Presumably, this directory will be empty at the beginning.<br />
## Once you create a new data version as defined below, you should pass the needed information as a command line option. Currently this is done by the ARGS variable. For example, the argument "-v RunPeriod-2014-10,8" tells the monitoring scripts to look up the version corresponding to revision 8 of RunPeriod-2014-10 in the monitoring DB and to use to store the results.<br />
<br />
<syntaxhighlight><br />
Example configuration parameters:<br />
set JOBDATE=2015-01-09<br />
set INPUTDIR=/volatile/halld/RunPeriod-2014-10/offline_monitoring<br />
set OUTPUTDIR=/w/halld-scifs1a/data_monitoring/RunPeriod-2014-10/ver08<br />
set ARGS=" -v RunPeriod-2014-10,8 "<br />
</syntaxhighlight><br />
If you want to process the results manually, the data is processed using the following script:<br />
<syntaxhighlight><br />
./process_new_offline_data.py <input directory> <output directory><br />
<br />
EXAMPLE:<br />
<br />
./process_new_offline_data.py 2014-11-14 /volatile/halld/RunPeriod-2014-10/offline_monitoring/ /w/halld-scifs1a/data_monitoring/RunPeriod-2014-10/ver02<br />
</syntaxhighlight><br />
The python script takes several options to enable/disable various steps in the processing. Of interest is the "--force" option, which will run over all monitoring ROOT files, whether or not they've been previously identified.<br />
<br />
Every time a new reconstruction pass is performed, a new version number must be generated. To do this, prepare a version file as described below. Then run the register_new_version.py script to store the information in the database. The script will return a version number, which then should be set by hand in process_new_offline_data.py - future versions of the script will streamline this part of the procedure. An example of how to generate a new version is:<br />
<syntaxhighlight><br />
./register_new_version.py add /home/gxproj1/halld/monitoring/process/versions/vers_RunPeriod-2014-10_pass1.txt<br />
</syntaxhighlight><br />
<br />
<b>If you are running the offline monitoring by checking out the files in trunk/scripts/monitoring/jproj/projects/<br />
of the svn repository</b>, and created a project with<br />
create_project.sh [project name] hd_rawdata<br />
Then go to the directory [project name]/processing/<br />
and execute<br />
./run_processing.sh<br />
which will run register_new_version.py as well as check_monitoring_data.csh for that project.<br />
<br />
===Step-by-Step Instructions For Processing a New Monitoring Run===<br />
<br />
The monitoring runs are current run out of the gxproj1 and gxproj5 accounts. After an offline monitoring run has been successfully started on the batch farm, the following steps should be followed to setup the post-processing for these runs.<br />
<br />
# The post-processing scripts are stored in $HOME/halld/monitoring/process and are automatically run by cron.<br />
# Run "svn update" to bring any changes in. Be sure that the list of histograms and macros to plot are current.<br />
# Add a new data version<br />
# Edit check_monitoring_data.csh to point to the current revisions/directories<br />
#* VERSION<br />
#* ARGS<br />
#* Note that the environment depends on a standard script - $HOME/setup_jlab.csh<br />
# Update files in the web directory, so that the results are displayed on the web pages: /group/halld/www/halldweb/html/data_monitoring/textdata<br />
# Copy the REST files to more permanent locations:<br />
#* cp -a /volatile/halld/offline_monitoring/RunPeriod-YYYY-MM/verVV/REST /work/halld/data_monitoring/RunPeriod-YYYY-MM/REST/verVV<br />
#* cp -a /volatile/halld/offline_monitoring/RunPeriod-YYYY-MM/verVV/REST /cache/halld/RunPeriod-YYYY-MM/REST/verVV [under testing]<br />
<br />
Check log files in $HOME/halld/monitoring/process/log for more information on how each run went. If there are problems, check log files, and modify check_monitoring_data.csh to vary the verbosity of the output.<br />
<br />
==Data Versions==<br />
<br />
To document the conditions of the monitoring data that is created, for the sake of reproducability and further analysis we save several pieces of information. The format is intended to be comprehensive enough to document not just monitoring data, but versions of raw and reconstructed data, so that this database table can be used for the event database as well.<br />
<br />
We store one record per pass through one run period, with the following structure:<br />
<br />
{| class="wikitable"<br />
! Field !! Description<br />
|-<br />
| data_type || The level of data we are processing. For the purposes of monitoring, "rawdata" is the online monitoring, "recon" is the offline monitoring<br />
|- <br />
| run_period || The run period of the data<br />
|- <br />
| revision || An integer specifying which pass through the run period this data corresponds to<br />
|- <br />
| software_version || The name of the XML file that specifies the different software versions used<br />
|-<br />
| jana_config || The name of the text file that specifies which JANA options were passed to the reconstruction program<br />
|-<br />
| ccdb_context || The value of JANA_CALIB_CONTEXT, which specifies the version of calibration constants that were used<br />
|-<br />
| production_time || The data at which monitoring/reconstruction began<br />
|-<br />
| dataVersionString || A convenient string for identifying this version of the data<br />
|}<br />
<br />
<br />
An example file used as as input to ./register_new_version.py is:<br />
<syntaxhighlight><br />
data_type = recon<br />
run_period = RunPeriod-2014-10<br />
revision = 1<br />
software_version = soft_comm_2014_11_06.xml<br />
jana_config = jana_rawdata_comm_2014_11_06.conf<br />
ccdb_context = calibtime=2014-11-10<br />
production_time = 2014-11-10<br />
dataVersionString = recon_RunPeriod-2014-10_20141110_ver01<br />
</syntaxhighlight></div>Kmoriyahttps://halldweb.jlab.org/wiki/index.php?title=GlueX_Offline_Meeting,_December_9,_2015&diff=71766GlueX Offline Meeting, December 9, 20152015-12-09T18:27:06Z<p>Kmoriya: /* Agenda */</p>
<hr />
<div>GlueX Offline Software Meeting<br><br />
Wednesday, December 9, 2015<br><br />
1:30 pm EST<br><br />
JLab: CEBAF Center F326/327<br />
<br />
==Agenda==<br />
<br />
# Announcements<br />
## [https://mailman.jlab.org/pipermail/halld-offline/2015-November/002175.html New sim-recon release: 1.7.0]<br />
## [https://mailman.jlab.org/pipermail/halld-offline/2015-December/002177.html CCDB release v1.06.00] (Dmitry)<br />
## [https://mailman.jlab.org/pipermail/halld-online/2015-December/000692.html JANA 0.7.4] (David)<br />
## We should expect a Software Review in Summer 2016.<br />
# Review of [[GlueX Offline Meeting, November 11, 2015#Minutes|minutes from November 11]] (all)<br />
# [https://halldweb1.jlab.org/wiki/images/0/0c/2015-12-09-offline_monitoring.pdf Offline Monitoring] (Kei)<br />
# Geant4 Update (Richard, David)<br />
# [https://mailman.jlab.org/pipermail/halld-tagger/2015-November/000565.html Update to CCDB for microscope energy] (Richard, Alex B.)<br />
# BCAL timing calibration and pion ID in simulation (Paul M., Sean, Tegan)<br />
# [https://mailman.jlab.org/pipermail/halld-offline/2015-December/002179.html C++ code analyzer + clang 3.7.0] (David)<br />
# [https://mailman.jlab.org/pipermail/halld-offline/2015-November/002172.html JANA status bits] (David)<br />
# Run number assignments for simulation for future run periods (Justin)<br />
# Review of [https://github.com/JeffersonLab/sim-recon/pulls?q=is%3Aopen+is%3Apr recent pull requests]<br />
# Deleting old pull-request builds (Sean)<br />
# Running tests on pull-request builds (Nathan)<br />
# [[Data Challenge 3]] update (Mark)<br />
# [[Sim1 Conditions|Future Commissioning Simulations]] (all)<br />
# Action Item Review<br />
<br />
==Communication Information==<br />
<br />
===Remote Connection===<br />
<br />
* The BlueJeans meeting number is 968 592 007 .<br />
* [http://bluejeans.com/968592007 Join the Meeting] via BlueJeans<br />
<br />
===Slides===<br />
<br />
Talks can be deposited in the directory <code>/group/halld/www/halldweb/html/talks/2015</code> on the JLab CUE. This directory is accessible from the web at https://halldweb.jlab.org/talks/2015/ .</div>Kmoriyahttps://halldweb.jlab.org/wiki/index.php?title=File:2015-12-09-offline_monitoring.pdf&diff=71765File:2015-12-09-offline monitoring.pdf2015-12-09T18:26:43Z<p>Kmoriya: Talk by Kei on offline monitoring for offline meeting 2015/12/09</p>
<hr />
<div>Talk by Kei on offline monitoring for offline meeting 2015/12/09</div>Kmoriyahttps://halldweb.jlab.org/wiki/index.php?title=Data_Monitoring_Procedures&diff=71725Data Monitoring Procedures2015-12-05T03:04:12Z<p>Kmoriya: /* Explanation of Scripts */</p>
<hr />
<div>__TOC__<br />
<br />
== Master List of File / Database / Webpage Locations ==<br />
=== Run Conditions ===<br />
* Online Run-by-run condition files (B-field, current, etc.): /work/halld/online_monitoring/conditions/<br />
* Offline monitoring run conditions (software versions, jana config): /group/halld/data_monitoring/run_conditions/<br />
*[http://www.jlab.org/Hall-D/test/RunInfo/ Run Info vers. 1]<br />
*[https://halldweb.jlab.org/cgi-bin/data_monitoring/run_conditions.pl Run Info vers. 2]<br />
<br />
=== Monitoring Output Files ===<br />
* Run Periods 201Y-MM is for example 2015-03, launch ver verVV is for example ver15<br />
* Online monitoring histograms: /work/halld/online_monitoring/root/<br />
* Offline monitoring histogram ROOT files: /work/halld/data_monitoring/RunPeriod-201Y-MM/verVV/rootfiles<br />
* REST files (most recent launch only): /work/halld/data_monitoring/RunPeriod-201Y-MM/REST/verVV<br />
* individual files for each job (ROOT, log, etc.): /volatile/halld/offline_monitoring/RunPeriod-201Y-MM/verVV/<br />
<br />
=== Monitoring Database ===<br />
* Accessing monitoring database (on ifarm): mysql -u datmon -h hallddb.jlab.org data_monitoring<br />
<br />
=== Monitoring Webpages ===<br />
*[https://halldweb.jlab.org/cgi-bin/data_monitoring/monitoring/plotBrowser.py Plot Browser]<br />
*[https://halldweb.jlab.org/cgi-bin/data_monitoring/monitoring/runBrowser.py Run Browser]<br />
*[https://halldweb.jlab.org/cgi-bin/data_monitoring/monitoring/timeSeries.py Time Series]<br />
*[https://halldweb.jlab.org/data_monitoring/launch_analysis/ Launch Analysis]<br />
<br />
== Job Monitoring Links ==<br />
* [http://scicomp.jlab.org/farm2/job.html Completed Job History]<br />
* [http://scicomp.jlab.org/farm2/project.html Job Stats By Project]<br />
* [http://scicomp.jlab.org/farm2/trackOrg.html Job Stats By Track]<br />
* [http://scicomp.jlab.org/farm2/report.html Cluster Report]<br />
* [http://scicomp.jlab.org/farm2/walltime.html Walltime Distribution]<br />
<br />
== Saving Online Monitoring Data ==<br />
<br />
The procedure for writing the data out is given in, e.g.,<br />
[https://halldweb.jlab.org/wiki/index.php/Raid-to-Silo_Transfer_Strategy Raid-to-Silo Transfer Strategy].<br />
<br />
Once the DAQ writes out the data to the raid disk, cron jobs will copy the file to tape,<br />
and within ~20 min., we will have access to the file on tape at<br />
/mss/halld/$RUN_PERIOD/rawdata/RunXXXXXX.<br />
<br />
All online monitoring plugins will be run as data is taken.<br />
They will be accessible within the counting house via RootSpy, and<br />
for each run and file, a ROOT file containing the histograms will be saved<br />
within a subdirectory for each run.<br />
<br />
For immediate access to these files, the raid disk files may be accessed directly<br />
from the counting house, or the tape files will be available within ~20 min. of the<br />
file being written out.<br />
<br />
== Offline Monitoring: Running Over Archived Data ==<br />
<br />
Once files are written to tape we run the online plugins on these files to confirm what we were seeing in the online monitoring, and also to update the results from the latest calibration and software. <br />
Manual scripts and cron jobs are set up to look for new data and run the plugins over a sample of files.<br />
<br />
Every other Friday (usually the Friday before the offline meetings) jobs will be started to run the newest software on all previous runs,and allows everybody to see improvements in each detector.<br />
For each launch, independent builds of hdds, sim-recon, the monitoring plugins, and an sqlite file will be generated.<br />
<br />
Below the procedures are described for<br />
# Preparing the software for the launch<br />
# Starting the launch (using hdswif)<br />
# Post-analysis of statistics of the launch<br />
<br />
Processing the results and making them available to the collaboration<br />
is handled in the section [[#Post-Processing_Procedures | Post-Processing Procedures]] below.<br />
<br />
<!--<br />
==== Generating an offline plugin job ====<br />
<br />
The user gxproj1 should be used for official offline monitoring jobs.<br />
Within /home/gxproj1/halld/monitoring/batch/ there will be scripts to run the online monitoring plugins over tape files.<br />
The main script is generatejobs_plugins_rawdata.sh, which can be used as<br />
generatejobs_plugins_rawdata.sh [minrun] [maxrun] (minfile) (maxfile)<br />
where minrun, maxrun specify the range of the run #, and minfile, maxfile (optional) specify<br />
the file range for that run.<br />
<br />
This will generate a script run_rawdata_XXXXXX.sh, where the run # has now been formatted to be 6 digits.<br />
Executing this script will send the monitoring plugins job to the Auger batch system.<br />
<br />
There is also a script clean.sh which can be used as<br />
clean.sh XXX<br />
This will clean up all associated files created in association with run XXX.<br />
<br />
Internally, the xml file used to submit the job will be created, and the job to run<br />
will be given within script.sh. All run parameters should be specified in at the beginning<br />
of generatejobs_plugins_rawdata.sh<br />
Since we are running on tape, the tape file will first be copied over to the cache disk, and the job will run<br />
over this cached file.<br />
<br />
==== Using cron to run automatically ====<br />
Within /home/gluex/halld/monitoring/cron/ there is a file cron_plugins<br />
that can be executed via<br />
crontab cron_plugins<br />
This will set up a cron job to call the script scan_for_jobs.sh, which will<br />
check in the rawdata directory and call generatejobs_plugins_rawdata.sh for<br />
any run that is more than 5 min old. The cron job is set up to run every 10 min.<br />
<br />
==== Magnetic Field Settings ====<br />
For tracking, it is necessary to set the correct magnetic field settings.<br />
The below table is now obsolete. Please refer to the [https://halldweb.jlab.org/cgi-bin/data_monitoring/run_conditions.pl GlueX Run Conditions] webpage.<br />
<br />
The actual field settings for each run have not been documented well.<br />
Below are values based on going through entries in the [https://logbooks.jlab.org/book/halld halld log].<br />
<br />
{| border="1" cellpadding="1" style="text-align: center;"<br />
!width="100"| Run #s<br />
!width="150"| Solenoid Current (A)<br />
!width="300"| JANA option<br />
!width="300"| notes<br />
|-<br />
| 940 - 996 || 1000 || -PBFIELD_MAP=Magnets/Solenoid/solenoid_1000A_poisson_20141104 || At run 997, the solenoid started ramping down<br />
|-<br />
| 998 - 1448 || 0 || -PBFIELD_TYPE=NoField -PDEFTAG:DTrackCandidate=StraightLine || Solenoid started ramping up around run 1431. Runs 1432 - 1448 should not be used for tracking. See [https://logbooks.jlab.org/entry/3309039 this] and [https://logbooks.jlab.org/entry/3309092 this] entry.<br />
|-<br />
| 1449 - 1620 || 1200 || -PBFIELD_MAP=Magnets/Solenoid/solenoid_1200A_poisson_20140520 || <br />
|-<br />
|}<br />
<br />
'''Note:''' The following run ranges were taken with a 300A field, and could be analyzed with the magnetic field map Magnets/Solenoid/solenoid_300A_poisson_20140819 , more study is needed to see if the straight line track fitter is needed or not: 1036 - 1053, 1065 - 1121, 1212? - 1254, 1309 - 1318 [1307,1308,1319-1329 were taken while the magnet was ramping].<br />
--><br />
<br />
=== General Information on Procedures ===<br />
This section explains how the offline monitoring should be run.<br />
Since we may want to simultaneously run offline monitoring for different run periods that require<br />
different environment variables, the scripts are set up so that a generic user can download the<br />
scripts and run them from anywhere. Most output directories for offline monitoring are created<br />
with group read/write permissions so that any Hall D group user has access to the contents,<br />
but there are some cases where use of the account that created the launch is necessary.<br />
<br />
The accounts used for offline monitoring are the gxprojN accounts created and maintained by<br />
Mark Ito (see [https://halldweb.jlab.org/wiki/index.php/GlueX-related_shared_accounts_on_the_JLab_CUE here] for how each account is used).<br />
As of October 2015, the following are used:<br />
* gxproj1 for running over Fall 2014 data (deprecated since June 2015)<br />
* gxproj5 for running over Spring 2015 data<br />
<br />
Since the summer of 2015 we have transitioned from a system using Mark Ito's jproj scripts<br />
to integrating the swif system that Chris Larrieu (SciComp) has been developing. For offline monitoring,<br />
the hdswif system that Kei developed is used for launching the jobs, and the jproj system is used<br />
for meta-analysis of launch statistics.<br />
<br />
Both hdswif and jproj are maintained in svn:<br />
* hdswif: https://halldsvn.jlab.org/repos/trunk/scripts/monitoring/hdswif<br />
* jproj : https://halldsvn.jlab.org/repos/trunk/scripts/monitoring/jproj<br />
<br />
To run the offline monitoring each package should be checked out and all necessary<br />
scripts are included.<br />
<br />
=== Preparing the software for the launch ===<br />
<br />
1. Setup the environment: <pre>source ~/env_monitoring_launch</pre><br />
<br />
2. Updating & building hdds: <br />
<pre><br />
cd $HDDS_HOME<br />
git pull # Get latest software<br />
scons -c install # Clean out the old install: EXTREMELY IMPORTANT for cleaning out stale headers<br />
scons install -j4 # Rebuild and re-install with 4 threads<br />
</pre><br />
<br />
3. Updating & building sim-recon: <br />
<pre><br />
cd $HALLD_HOME/src<br />
git pull # Get latest software<br />
scons -c install # Clean out the old install: EXTREMELY IMPORTANT for cleaning out stale headers<br />
scons install -j4 # Rebuild and re-install with 4 threads<br />
</pre><br />
<br />
4. Create a new sqlite file containing the very latest calibration constants. Original documentation on creating sqlite files are [https://halldweb.jlab.org/wiki/index.php/SQLite-form_of_the_CCDB_database here]. <br />
<pre><br />
cd $GLUEX_MYTOP/../sqlite/<br />
$CCDB_HOME/scripts/mysql2sqlite/mysql2sqlite.sh -hhallddb.jlab.org -uccdb_user ccdb | sqlite3 ccdb.sqlite<br />
mv ccdb.sqlite ccdb_monitoring_launch.sqlite #replacing the old file<br />
</pre><br />
<br />
5. Note that the above steps must be done <b>BEFORE</b> launch project creation. This is because we will track the revisions of the libraries used, and this is done by extracting the svn information in each directory. Also, note that the system assumes that we have the topmost build directory (usually called GLUEX_TOP) to be $HOME/builds . Such an assumption is necessary to be able to extract information about the library locations automatically.<br />
<br />
Create the appropriate project(s) and submit the jobs using hdswif, as detailed in the section below.<br />
<br />
=== Starting the Launch and Submitting Jobs ===<br />
<br />
Until the summer of 2015 we relied solely on Mark Ito's jproj system for submitting and keeping track of jobs. We have since moved to the swif system and use the hdswif wrapper for this. Below are instructions for how to use these.<br />
<br />
1. Downloading hdswif: Download the hdswif directory from svn. For the gxprojN accounts, use the directory ~/halld/hdswif. <br />
<pre><br />
cd ~/halld<br />
svn co https://halldsvn.jlab.org/repos/trunk/scripts/monitoring/hdswif<br />
cd hdswif<br />
</pre><br />
<br />
2. Edit the job config file, input.config, which is used to register jobs in hdswif. A typical config file will look this:<br />
PROJECT gluex<br />
TRACK reconstruction<br />
OS centos65<br />
NCORES 6<br />
DISK 40<br />
RAM 8<br />
TIMELIMIT 8<br />
JOBNAMEBASE offmon_<br />
RUNPERIOD 2015-03<br />
VERSION 15<br />
OUTPUT_TOPDIR /volatile/halld/offline_monitoring/RunPeriod-[RUNPERIOD]/ver[VERSION] # Example of other variables included in variable<br />
SCRIPTFILE /home/gxproj5/halld/hdswif/script.sh # Must specify full path<br />
ENVFILE /home/gxproj5/halld/hdswif/setup_jlab-2015-03.csh # Must specify full path<br />
<br />
3. Creating the workflow: Within SWIF jobs are registered into workflows. For offline monitoring, the workflow names are of the form <b>offline_monitoring_RunPeriod201Y_MM_verVV_hd_rawdata</b> with suitable replacements for the run period and version number. The command "swif list" will list all existing workflows. Also, for most simple SWIF commands hdswif also provides a wrapper. <br />
swif list<br />
<br />
For creation of workflows for offline monitoring the command:<br />
hdswif.py create [workflow] -c input.config<br />
should be used. When a config file (here: input.config) is passed in, hdswif will automatically create files that record the configuration of the current launch. These files are stored as for example:<br />
/group/halld/data_monitoring/run_conditions/jana_rawdata_comm_2015_03_ver15.conf<br />
/group/halld/data_monitoring/run_conditions/soft_comm_2015_03_ver15.xml<br />
<br />
The software packages stored in git (sim-recon and hdds) can have git tags applied to them, which makes it easier to find versions of the software than a SHA-1 hash. hdswif will ask if you would like to create a tag, and execute the following sequence: <pre>git tag -a offmon-201Y_MM-verVV -m "Used for offline monitoring 201Y-MM verVV started on 201y/mm/dd"</pre> <pre>git push offmon-201Y_MM-verVV</pre> This will only be invoked when the user name is gxprojN, and for the configuration files, the output directory will be /group/halld/data_monitoring/run_conditions/ for gxprojN accounts while it will be the current directory for other users.<br />
* To use the git-tagged software versions do for example <pre>cd $HALLD_HOME</pre> <pre>git checkout offmon-2015_03-ver15</pre><br />
<br />
4. Registering jobs in the workflow: To register jobs within the workflow, hdswif provides the use of config files. Jobs can be registered by specifying the workflow, config file (-c), run (-r) and file (-f) numbers if necessary. <br />
<span style="color:red">Note: Job configuration parameters can be set differently for jobs within the same workflow if necessary.</span><br />
Jobs can be added via <pre>hdswif.py add [workflow] -c input.config</pre><br />
By default, hdswif will add all files found within the directory /mss/halld/RunPeriod-201Y-MM/rawdata/ where 201Y-MM is specified by the RUNPERIOD parameter in the config file. If only some of the runs or files are needed, these can be specified for example with<br />
<pre>hdswif.py add [workflow] -c input.config -r 3180 -f '00[0-4]'</pre><br />
to specify to register running only over run 3180 files 000 - 004 (Unix-style brackets and wildcards can be used).<br />
<br />
5. Running the workflow: To run the workflow, simply use the hdswif wrapper:<br />
hdswif.py run [workflow]<br />
<br />
<b>It is recommended that some jobs be tested to make sure that everything is working rather than fail thousands of jobs.</b><br><b>Check the configuration setup when creating a workflow, and what is in script.sh. Also check the following:<br />
* Check stderr files. Are they small (<kB)?<br />
* Check stdout files. Are they very large (>MB)?<br />
* Check output ROOT files. Are they larger than several MB?<br />
* Check output REST files. Are they larger than several tens of MB?<br />
</b><br />
<br> For this purpose, hdswif will take an additional parameter to run which limits the number of jobs to submit: <pre>hdswif.py run [workflow] 10</pre><br />
in which case only 10 jobs will be submitted.<br />
<br />
To submit all jobs after checking the results, do <pre>hdswif.py run [workflow]</pre><br />
<br />
=== Checking the Status and Resubmitting ===<br />
1. The status of jobs can be checked on the terminal with <pre>jobstat -u gxprojN</pre> For the status of jobs on Auger see http://scicomp.jlab.org/scicomp/#/auger/jobs and for SWIF use <pre>swif list</pre> or for more information, <pre>swif status [workflow] -summary</pre> Note that "swif status" tends to be out of date sometimes, so don't panic if your workflow/jobs aren't showing up right away. Also see the Auger [http://scicomp.jlab.org/scicomp/#/auger/jobs job website].<br />
<br />
2. For failed jobs, SWIF can resubmit jobs based on the problem. For resubmission for failed jobs with the same resources, <pre>swif retry-jobs [workflow] -problems [problem name]</pre> can be used, and for jobs to be submitted with more resources, e.g., use <pre>swif modify-jobs [workflow] -ram add 2gb -problems AUGER-OVER_RLIMIT</pre> <br />
This only re-stages the jobs, be sure to resubmit them with:<br />
<pre>swif run -workflow [workflow] -errorlimit none</pre><br />
<br />
hdswif has a wrapper for both of these: <pre>hdswif.py resubmit [workflow] [problem]</pre> In this case [problem] can be one of <b>SYSTEM, TIMEOUT, RLIMIT</b>. If SYSTEM is specified, the jobs will be retried. For TIMEOUT and RLIMIT, the jobs will be modified by default with 2 additional hours or GB of RAM. If one more number is added as an option, then that many hours or GB of RAM will be added., e.g., <pre>hdswif.py resubmit [workflow] TIMEOUT 5</pre> will add 5 hours of processing time. You can wait until almost all jobs finish before resubmitting failed jobs since the number should be relatively small. <b>Even if jobs are resubmitted for one type of failure, jobs that later fail with that failure will not be automatically resubmitted.</b><br />
<br />
3. For information on swif, use the "swif help" commands and for hdswif see the attached documentaion in https://halldsvn.jlab.org/repos/trunk/scripts/monitoring/hdswif/manual_hdswif.pdf<br />
<br />
4. Below is a table describing the various errors that can occur.<br />
{| border="1" cellpadding="0" valign="left" style="text-align: left;"<br />
!width="150"| ERROR NAME<br />
!width="400"| Description<br />
!width="400"| Resolution<br />
!width="400"| hdswif command<br />
|-<br />
| AUGER-SUBMIT<br />
||<br />
SWIF’s attempt to submit jobs to Auger failed. Includes server-side problems as well as user failing to provide valid job parameters (e.g. incorrect project name, too many resources, etc.)<br />
||<br />
If requested resources are known to be correct resubmit. Otherwise modify job resources using swif directly.<br />
||<br />
<span style="color:purple"><b>hdswif.py resubmit [workflow] SYSTEM</b></span><br />
|-<br />
| AUGER-FAILED<br />
||<br />
Auger reports the job FAILED with no specific details.<br />
||<br />
Resubmit jobs. If problem persists, contact Chris Larrieu or SciComp.<br />
||<br />
<span style="color:purple"><b>hdswif.py resubmit [workflow] SYSTEM</b></span><br />
|-<br />
| AUGER-OUTPUT-FAIL<br />
||<br />
Failure to copy one or more output files.Can be due to permission problem, quota problem, system error, etc.<br />
||<br />
Check if output files will exist after job execution and that output directory exists, resubmit jobs. If problem persists, contact Chris Larrieu or SciComp.<br />
||<br />
<span style="color:purple"><b>hdswif.py resubmit [workflow] SYSTEM</b></span><br />
|-<br />
| AUGER-INPUT-FAIL<br />
||<br />
Auger failed to copy one or more of the requested input files, similar to output failures. Can also happen if tape file is unavailable (e.g. missing/damaged tape)<br />
||<br />
Check if input file exists, resubmit jobs. If problem persists, contact Chris Larrieu or SciComp.<br />
||<br />
<span style="color:purple"><b>hdswif.py resubmit [workflow] SYSTEM</b></span><br />
|-<br />
| AUGER-TIMEOUT<br />
||<br />
Job timed out.<br />
||<br />
If more time is needed for job add more resources.<br />
Default is to add 2 hrs of processing time. Also check whether code is hanging.<br />
||<br />
<span style="color:red"><b>hdswif.py resubmit [workflow] TIMEOUT</b></span><br><br />
Default is to add 2 hours. Optionally specify number of hours at end.<br />
|-<br />
| AUGER-OVER_RLIMIT<br />
||<br />
Not enough resources, RAM or disk space.<br />
||<br />
Add more resources for job.<br />
||<br />
<span style="color:blue"><b>hdswif.py resubmit [workflow] RLIMIT</b></span><br><br />
Default is to add 2 GB of RAM. Optionally specify GB at end. To add more disk space use SWIF directly.<br />
|-<br />
| SWIF-MISSING-OUTPUT<br />
||<br />
Output file specified by user was not found.<br />
||<br />
Check if output file exists at end of job.<br />
||<br />
<br />
|-<br />
| SWIF-USER-NON-ZERO <br />
||<br />
User script exited with non-zero status code.<br />
||<br />
Your script exited with non-zero status. Check the code you are running.<br />
||<br />
<br />
|-<br />
| SWIF-SYSTEM-ERROR <br />
||<br />
Job failed owing to a problem with swif (e.g. network connection timeout)<br />
||<br />
Resubmit jobs. If problem persists, contact Chris Larrieu or SciComp.<br />
||<br />
<span style="color:purple"><b>hdswif.py resubmit [workflow] SYSTEM</b></span><br />
|}<br />
<br style="clear:both;"/><br />
<br />
=== Post-analysis of statistics of the launch ===<br />
<br />
After jobs have been submitted, it will usually take a few days for all of the jobs to be processed.<br />
The next step is to check the resource usage for the current launch and publish the results online.<br />
<br />
# <b>Create summary XML, HTML files</b><br> The status and results of jobs are saved within the SWIF internal server, and are available via the command <pre>swif status [workflow] -summary -runs</pre> where the arguments -summary and -runs show summary statistics and statistics for individual jobs, respectively. hdswif has a command that takes this output in XML output and creates an HTML webpage showing results of the launch. To do this, do <pre>hdswif.py summary [workflow]</pre> This will create the XML output file from SWIF called swif_output_[workflow].xml and create a webpage containing png figure files. If the XML file already exists, hdswif will ask whether to overwrite the existing file.<br />
# <b>Publish output files online</b><br> At this stage the html output and figure files are created and ready to be put online. The html output capabilities of hdswif is useful for any user using SWIF, but since publication of the html output is specific to the offline monitoring, the script to do this is contained in the jproj scripts directory at https://halldsvn.jlab.org/repos/trunk/scripts/monitoring/jproj. For the gxprojN accounts this directory should exist as ~/halld/jproj. To publish the results online do for example <pre>python ~/halld/jproj/projects/templates/publish_offmon_results.py 2015_03 18</pre> The script simply copies the html output and corresponding figures to /group/halld/www/halldweb/html/data_monitoring/launch_analysis/<br />
# <b>Editing the summary HTML page</b><br> The top page for is offline monitoring https://halldweb.jlab.org/data_monitoring/launch_analysis/index.html and has links to the summary page for each run period. The summary files are /group/halld/www/halldweb/html/data_monitoring/launch_analysis/[run period].html Edit the file to<br />
## Add a new line to the first table which contains the version number, date, and comments for the current launch<br />
## Create a link to the webpage for the current launch. Simply copy and paste, modify the previous launch's link to have the correct launch ver.<br />
# <b>Freezing SWIF tables</b><br> Since we are now finished with the SWIF workflow that we used, the workflow should be "frozen" so that it cannot be mistakenly altered or modified. Do <pre>swif freeze [workflow]</pre><br />
# <b>Backing up SWIF output</b><br> With the workflow frozen we should be able to reproduce the XML output from SWIF, but we will backup the XML output just in case. Do <pre>cp ~/halld/hdswif/swif_output_[workflow].xml /group/halld/data_monitoring/swif_xml_backup/ </pre><br />
<br />
=== Cross Analysis of Launches ===<br />
<br />
The purpose of the cross analysis is to correlate how resource usage changed for the same files across different launches.<br />
To do this it is useful to create MySQL tables that contain information on each launch, and do queries across tables.<br />
<br />
#The scripts to do this are maintained in the svn directory https://halldsvn.jlab.org/repos/trunk/scripts/monitoring/jproj Do <pre> svn co https://halldsvn.jlab.org/repos/trunk/scripts/monitoring/jproj </pre> For the gxprojN accounts used for offline monitoring the directory should be ~/halld/jproj<br />
# The jproj directory contains two subdirectories, scripts and projects. The scripts directory contains useful scripts for processing the jobs registered in the jproj system, and each of the offline monitoring launches will be handled in the projects directory.<br />
# Go to the projects directory<pre>cd ~/halld/jproj/projects</pre> and use the script create_project.sh to create a new directory that contains the processing scripts for the current launch <pre>./create_project.sh [workflow]</pre> This should create a directory such as offline_monitoring_RunPeriod2015_03_ver15_hd_rawdata (same as the workflow name). The script uses the template files in the directory templates and by substitution creates script files for the current launch. Now go to the newly created analysis directory: <pre>cd [workflow]/analysis</pre><br />
which for the gxprojN accounts should have the full path /home/gxprojN/halld/jproj/projects/[workflow]/analysis.<br><br><br />
<span style="color:red">All of the analysis commands including arguments are contained in run_analysis.sh, but it is <b>strongly</b> recommended that all commands are run manually to check for errors.</span><br><br><br />
# Next, we need to create a few MySQL tables for the current launch. The MySQL tables are useful for comparing run/file combinations for different launches. For the SWIF launches, there are two tables needed, and will be named<br />
#* [workflow]Job<br />
#* [workflow]_aux<br />
# This naming scheme of tables and their roles are the same as from the jproj only launches. The [workflow]Job table will contain information gathered from SWIF about each job (which node it went to, start time of each stage, memory usage, etc.). The [workflow]_aux table will contain information gathered from the output stdout files from each job. First, create the Job table using <pre>python create_jproj_job_table.py [run period] [version]</pre> Note that this script uses the XML output from hdswif summary and inserts the contents into the MySQL table, so the XML output file must exist.<br />
# To check the contents of this MySQL table, do <pre>mysql -hhalldb -ufarmer farming</pre> <pre>mysql> describe offline_monitoring_RunPeriod2015_03_ver11_hd_rawdataJob;</pre><br />
# <b>Backing up tables</b><br> Tables created in MySQL should be backed up.<br />
<br />
== Explanation of Scripts ==<br />
Below are explanations of each script used in the offline monitoring system and a breif explanation of how they work.<br />
<br />
=== hdswif scripts ===<br />
<b>Summary:</b> hdswif.py is the main script that calls the other utility scripts. For the utility scripts, they can be run standalone by giving the appropriate parameters. Visual graphics are made using the PyROOT extension of ROOT. To use this, the environment variable PYTHONPATH must include ROOTSYS. If ROOTSYS has been set, adding it to PYTHONPATH will be done by the script, but if ROOTSYS is not set, then the scripts will abort.<br />
<br />
{| border="1" cellpadding="0" valign="left" style="text-align: left;"<br />
!width="150"| file name<br />
!width="800"| Description<br />
|-<br />
| hdswif.py ||<br />
Main script to control the behavior of SWIF. Most commands follow the form hdswif.py [command] [workflow] (options)<br />
|-<br />
| parse_swif.py ||<br />
Called within hdswif.py to create html output from SWIF results.<br />
|-<br />
| createXMLfiles.py ||<br />
* Called by the "create" option of hdswif.py<br />
* Creates XML files for logging information about launch. Must specify config file with option -c.<br />
* Also adds tags to git repositories of sim-recon and hdds.<br />
* For XML file creation, the file will be written out to /group/halld/data_monitoring/run_conditions/ if the user is gxprojN (used for offline monitoring). If not, to avoid general users adding things to the above directory, the output files will be the current directory.<br />
* To write out the versions of each package, environment variables such as HDDS_HOME will need to be set.<br />
* Versions for each software package are extracted using the directory structure, so if these are changed the scripts must change accordingly.<br />
* For each launch, <b>the output soft_comm_[RUNPERIOD]_ver[VER].xml file should be checked that all version numbers have been extracted.</b><br />
|-<br />
| read_config.py ||<br />
* Called by the "add" option of hdswif.py<br />
* Takes in config file name, optionally set verbose<br />
* Return a dictionary between config parameter names and values (e.g., 'PROJECT' : 'gluex', 'NCORES' : 6, ...)<br />
* Prints the config parameters at the end. For parameters changed from default, a '*' will be printed<br />
|-<br />
| output_job_details.py ||<br />
* Called by the "details" option of hdswif.py<br />
* Takes in workflow name, run and file. Run and file must be numbers, no wildcards<br />
* Finds ids for jobs specified by the run and file number and returns info on each one<br />
* Job info is retrieved from pbs farm system and shows configuration parameters for that job<br />
|-<br />
| results_by_resources.py ||<br />
* Called within parse_swif.py<br />
* Takes in XML output from SWIF and creates 2 plots<br />
* Creates html table showing results of jobs by resources requested for that job<br />
* This table is shown under "Status by Resources" in the output html file of hdswif.py summary [workflow]<br />
|-<br />
| create_ordered_hists.py ||<br />
* Called within parse_swif.py<br />
* Takes in XML output from SWIF and creates 2 plots<br />
* Plots are dependency time and pending time of each job, ordered by submission.<br />
* Different colors represent jobs submitted at different times. For jobs submitted at the same time, jobs are ordered in increasing time.<br />
|-<br />
| create_stacked_times.py ||<br />
* Called within parse_swif.py<br />
* Takes in XML output from SWIF and creates 2 plots<br />
* Plots show total job time divided into colors for different stages<br />
* One shows all jobs in order of Auger ID (roughly submission order), the other one shows in order of total job time<br />
* The jobs show at a glance which stage contributes how much of the job's time<br />
|}<br />
<br />
=== Utility scripts ===<br />
{| border="1" cellpadding="0" valign="left" style="text-align: left;"<br />
!width="150"| file name<br />
!width="800"| Description<br />
|-<br />
| stderr_by_size.py ||<br />
* Independent of all other scripts in hdswif directory<br />
* When diagnosing problems it is useful to check the stderr/stdout files. Frequently, different problems are easier to find based on the size of stderr files<br />
* Takes in run period and version as arguments, creates a directory /volatile/halld/offline_monitoring/RunPeriod-[RUNPERIOD]/ver[VER]/log/bysize that contains soft links to all stdout and stderr files from the specified launch, <b>in separate directories given by stderr file size</b>.<br />
|}<br />
<br />
=== cross_analysis scripts ===<br />
<b>Summary:</b> The script run_cross_analysis.sh is the main script. In principle, running this with <pre>./run_cross_analysis.sh [RUNPERIOD] [VERSION]</pre> should work, but it is recommended that the commands within the script<br />
be run by hand to catch any errors.<br />
<br />
{| border="1" cellpadding="0" valign="left" style="text-align: left;"<br />
!width="150"| file name<br />
!width="800"| Description<br />
|-<br />
| run_cross_analysis.sh ||<br />
* Main script to call other Python scripts.<br />
* Takes in run period and version as arguments and runs cross analysis<br />
|-<br />
| create_cross_analysis_table.sh ||<br />
* Creates MySQL table for current launch that is used by other scripts<br />
* Takes run period and version as input, and creates table cross_analysis_table_[RUNPERIOD]_ver[VERSION]<br />
|-<br />
| create_stats_table_row.py ||<br />
* Create a row in the html table showing the overall statistics of final states and problems for a launch<br />
* The final states are "Success", and "Segfault". "Success" includes all jobs that had problems but still finished with Success.<br />
* The problems are "Over Limit", "Timeout", and "System". If any of these occurred for any attempt of the job they are counted.<br />
* The script creates a new html table row for the current launch. This html snippet is inserted into the web-accessible file showing the results <br />
|-<br />
| create_stats_for_each_file.py ||<br />
* Create html tables showing the status of each file against different launch versions.<br />
* Takes in run period, min version, max version and shows final result and problems for all versions in between<br />
* Different final results and problems are shown by combinations of ext content, olor coding of text, background color<br />
|-<br />
|-<br />
| create_resource_correlation_plots.py ||<br />
* Create plots showing correlation of resources between different launches for each file.<br />
* Takes in run period, min version, version of interest. Points are shown only for files included in version of interest<br />
* Creates plots for CPU time, Wall time, memory, virtual memory, #events, difference in #events, time to copy input evio file, time to run plugin<br />
|}<br />
<br />
== Hall D Job Management System ==<br />
<br />
This section details instructions on how to create and launch a set of jobs using the Hall-D Job Management System developed by Mark Ito. These instructions are generic: this system can be used for the weekly monitoring jobs, but can also be used for other sets of job launches as well. <br />
<br />
=== Database Table Overview ===<br />
<br />
* Job management database table (<project_name>): For each input file, keeps track of whether or not a job for it has been submitted, along with other optional fields. <br />
<br />
* Job status database table (<project_name>Job (no space)): For each job, keeps track of the job-id, the job status, memory used, cpu & wall time, time taken to complete various stages (e.g. pending, dependency, active), and others. <br />
<br />
* Job metrics database table (<project_name>_aux (no space)): For each job, keeps track of the job-id, how many events were processed, the time it took to copy the cache file, and the time it took to run the plugin. This information is culled from the log files of each job, and is done within the analysis directory of each launch.<br />
<br />
=== Initialize Project Management ===<br />
<br />
* Log into the ifarm machine with one of the gxproj accounts. For this example we will use gxproj1.<br />
<pre><br />
ssh gxproj1@ifarm -Y<br />
</pre><br />
<br />
* Go to a directory to do the launch. In principle, any directory will work, but for gxproj1 this is usually done in /home/gxproj1/halld/<br />
<br />
* Check out the necessary scripts <pre>svn co https://halldsvn.jlab.org/repos/trunk/scripts/monitoring/jproj</pre> This will get all necessary scripts for launching. Once checked out, <pre>cd projects</pre><br />
<br />
* The script create_project.sh can be used to create a new project. It will take a single argument, the project name. It is assumed that the project name is of the form offline_monitoring_RunPeriod20YY_MM_verVV_hd_rawdata where this string will be parsed to give the run period 20YY_MM and version number VV. One thing to do BEFORE creation of a new project is to editthe conditions of the launch (plugins to run over, memory requested, disk space requested) within templates/template.jsub . This<br />
information is saved automatically at project creation time into files in /group/halld/data_monitoring/run_conditions .<br />
To create a project do <pre>./create_project.sh offline_monitoring_RunPeriod20YY_MM_verVV_hd_rawdata</pre><br />
The name has been chosen to be as consistent as possible with other directory structures.<br />
However, mysql requires that "-" be escaped in table names, so unfortunately run periods will be<br />
given as 2YYY_MM instead of 2YYY-MM.<br />
<br />
* A new directory with the project name (offline_monitoring_RunPeriod20YY_MM_verVV_hd_rawdata) and files will be copied and modified from the template directory to reflect the run period, the user, the directory that it was created in, project name, etc.<br />
<br />
* For each project, cd into the new directory<br />
* The script clear.sh will remove any existing tables for the current project name, then recreate it. Do<br />
<pre>./clear.sh</pre><br />
* To use the jproj.pl script that was checked in, add the directory to your path with<br />
<pre>source ../../scripts/setup.csh</pre><br />
or always specify the full path<br />
<pre>../../scripts/jproj.pl</pre><br />
* Now update the table of runs with<br />
<pre>jproj.pl <project name> update</pre><br />
This will fill the table with all files within /mss that are of the same form as what is in <project name>.jproj . If you want to register only on a subset of all such files, you can edit this file directly.<br />
* Once you have registered all of the files you would like to run over, do<br />
<pre>jproj.pl <project name> submit [max # of jobs] [run number]</pre><br />
where the additional options specify how many jobs to submit and which run number to run on. Without these options all files that are registered and have not been submitted yet will be submitted.<br />
<br />
At this stage you are ready to submit all files. It is a good idea to submit a few test jobs<br />
at first to check that all scripts are working and that the plugins do not crash. Once you are<br />
sure that this does not happen, you can send all jobs in. The remaining jobs are then the monitoring<br />
which will (among other things) put the results on the online webpage for the collaboration to view, and<br />
the analysis of the launch.<br />
<br />
=== Project File Overview ===<br />
<br />
An overview of each project file:<br />
* '''clear.sh:''' For the current project, deletes the job status and management database tables (if any), and creates new, empty ones. <br />
* '''<project_name>.jproj:''' Contains the path and file name format for the input files for the jobs. <br />
* '''<project_name>.jsub:''' The xml job submission script. The run number and file number variables are set during job submission for each input file. <br />
* '''script.sh:''' The script that is executed during the job. If output job directories are not pre-created manually, they should be created in this script with the proper permissions:<br />
<pre><br />
mkdir -p -m 775 my_directory<br />
</pre><br />
* '''setup_jlab-[run period].csh:''' The environment that is sourced at the beginning of the job execution. <br />
* '''status.sh:''' Updates the job status database table, and prints some of its columns to screen.<br />
<br />
=== Project Management ===<br />
<br />
* Delete (if any) and create the database table(s) for the current set of job submissions:<br />
<pre><br />
./clear.sh<br />
</pre><br />
Also, if testing was done with jobs, it is best to delete the output directory and the configuration files:<br />
<pre><br />
rm -frv /volatile/halld/offline_monitoring/RunPeriod-20YY-MM/verVV /group/halld/data_monitoring/run_conditions/soft_comm_20YY_MM_verVV.xml /group/halld/data_monitoring/run_conditions/jana_rawdata_comm_20YY_MM_verVV.conf<br />
</pre><br />
<br />
* Search for input files matching the string in the .jproj file, and create a row for each in the job management database table (called <project_name>). You can test by adding an optional argument at the end, which only selects files with a specific file number:<br />
<pre><br />
jproj.pl <project_name> update <optional_file_number><br />
</pre><br />
<br />
* Confirm that the job management database is accurate by printing it's contents to screen:<br />
<pre><br />
mysql -hhallddb -ufarmer farming -e "select * from <project_name>"<br />
</pre><br />
<br />
* ONLY if a mistake was made, to delete the tables from the database and recreate new, empty ones, run: <br />
<pre><br />
./clear.sh<br />
</pre><br />
<br />
* To look at the status of the submitted jobs, first query auger and update the job status database:<br />
<pre><br />
fill_in_job_details.pl <project_name><br />
</pre><br />
<br />
* The job status can then be viewed by submitting a query to the job status database (called <project_name>Job (no space in between)):<br />
<pre><br />
mysql -hhallddb -ufarmer farming -e "select id,run,file,jobId,hostname,status,timeSubmitted,timeActive,walltime,cput,timeComplete,result,error from <project_name>Job"<br />
</pre><br />
<br />
* These last two commands can instead be executed simultaneously by running:<br />
<pre><br />
./status.sh<br />
</pre><br />
<br />
=== Handy mysql Instructions ===<br />
<br />
* Handy mysql instructions:<br />
<pre><br />
mysql -hhallddb -ufarmer farming # Enter the "farming" mysql database on "hallddb" as user "farmer"<br />
quit; # Exit mysql<br />
show tables; # Show a list of the tables in the current database<br />
show columns from <project_name>; # show all of the columns for the given table<br />
select * from <project_name>; # show the contents of all rows from the given table<br />
</pre><br />
<br />
=== Backing Up Offline Monitoring Tables ===<br />
Tables created for offline monitoring can be backed up using the script backup_tables.sh which<br />
can be checked out with the other files from https://halldsvn.jlab.org/repos/trunk/scripts/monitoring/jproj/projects<br />
<br />
The script uses the command mysqldump to print out a file that can be executed to recreate the tables.<br />
Since executing this output file will drop the table when it exists, caution is advised.<br />
Example usage to backup all three tables created for run period 2014_10 ver 17:<br />
<pre>backup_tables.sh 2014_10 17</pre><br />
<br />
== Running Over Data As It Comes In ==<br />
<br />
A special user gxproj1 will have a cron job set up to run the plugins as new data appears on /mss.<br />
During the week, gxproj1 will submit offline plugin jobs with the same setup as the weekly jobs<br />
run the previous Friday. The procedure for this is shown below.<br />
<br />
<!--<br />
=== Setting up the environment ===<br />
The file<br />
/home/gxproj1/setup_jlab.csh<br />
is sourced through .tcshrc.<br />
This file is the same as what is linked to by<br />
/home/gluex/setup_jlab_commissioning.csh,<br />
except HALLD_HOME, HDDS_HOME, and JANA_CALIB_URL are set separately so that this<br />
user can have a separate build.<br />
<br />
To obtain the builds from the previous Friday's runs,<br />
execute<br />
/home/gxproj1/halld/monitoring/newruns/setup_previous.sh [year] [month] [day]<br />
The build revisions from the previous Friday are archived in files<br />
/work/halld/data_monitoring/run_conditions/soft_comm_[year]_[month]_[day].xml<br />
and the script will build libraries based on those stored revision numbers.<br />
--><br />
<br />
=== Running the cron job ===<br />
<br />
'''IMPORTANT:''' The cron job should not be running while you are manually submitting jobs using the jproj.pl script for the same project, or else you will probably multiply-submit a job. <br />
<br />
* Go to the cron job directory: <br />
<pre><br />
cd /u/home/gxproj1/halld/monitoring/newruns<br />
</pre><br />
<br />
* The cron_plugins file is the cronjob that will be executed. During execution, it runs the exec.sh command in the same folder. This command takes two arguments: the project name, and the maximum file number for each run. These fields should be updated in the cron_plugins file before running. <br />
<br />
* The exec.sh command updates the job management database table with any data that has arrived on tape since it was last updated, ignoring file numbers greater than the maximum file number. It then submits jobs for these files. <br />
<br />
* To start the cron job, run:<br />
<pre><br />
crontab cron_plugins<br />
</pre><br />
<br />
* To check whether the cron job is running, do<br />
<pre><br />
crontab -l<br />
</pre><br />
<br />
* To remove the cron job do<br />
<pre><br />
crontab -r<br />
</pre><br />
<br />
<!--<br />
The cron job will run the script scan_for_jobs.sh,<br />
which runs generatejobs_plugins_rawdata.sh for any<br />
new runs that it had not seen before. All previous<br />
runs are recorded in the file filelists/files_current.txt<br />
so clear this to run over runs, or set the parameters<br />
MINRUN and MAXRUN which will set the range of runs submitted.<br />
--><br />
<br />
==Post-Processing Procedures==<br />
<br />
To visualize the monitoring data, we save images of selected histograms and store time series of selected quantities in a database, which are then displayed on a web page. This section describes how to generate the monitoring images and database information.<br />
<br />
The scripts used to generate this summary data are primarily run from /home/gxprojN/halld/monitoring/process . If you want a new copy of the scripts, e.g., for a new monitoring run, you should check the scripts out from SVN:<br />
<syntaxhighlight><br />
svn co https://halldsvn.jlab.org/repos/trunk/scripts/monitoring/process<br />
</syntaxhighlight><br />
Note that these scripts currently have some parameters which must be periodically set by hand.<br />
<br />
The default python version on most JLab machine does not have the modules to allow these scripts to connect to the MySQL database. To run these scripts, load the environment with the following command<br />
<syntaxhighlight><br />
source /home/gxproj1/halld/monitoring/process/monitoring_env.csh<br />
</syntaxhighlight><br />
<br />
===Online Monitoring===<br />
<br />
There are two primary scripts for running over the monitoring data generated by the online system and offline reconstruction. The online script can be run with either of the following commands:<br />
<syntaxhighlight><br />
/home/gluex/halld/monitoring/process/check_new_runs.py<br />
<br />
OR <br />
<br />
/home/gluex/halld/monitoring/process/check_new_runs.csh<br />
</syntaxhighlight><br />
The shell script sets up the environment properly to run the python script. To connect to the monitoring database on the JLab CUE, modules included in the local installation of python >= 2.7 are needed. The shell script is appropriate to use in a cron job. The cronjob is currently run under the "gluex" account.<br />
<br />
The online monitoring system copies a ROOT file containing the results of the online monitoring, and other configuration files into a directory accessible outside the counting house. This python script automatically checks for new ROOT files, which it will then automatically process. It contains several configuration variables that must be correctly set, which contains the location of input/output directories, etc...<br />
<br />
===Offline Monitoring===<br />
<br />
After the data is run over, the results should be processed, so that summary data is entered into the monitoring database and plots are made for the monitoring webpages. Currently, this processing is controlled by a cronjob that runs the following script:<br />
<syntaxhighlight><br />
/home/gxproj1/halld/monitoring/process/check_monitoring_data.csh <br />
</syntaxhighlight><br />
This script checks for new ROOT files, and only runs over those it hasn't processed yet. Since one monitoring ROOT file is produced for each EVIO file, whenever a new file is produced, the plots for the corresponding run are recreated and all the ROOT files for a run are combined into one file. Information is stored in the database on a per-file basis. <br />
<br />
Plots for the monitoring web page can be made from single histograms or multiple histograms using RootSpy macros. If you want to change the list of plots made, you must modify one of the following files:<br />
* histograms_to_monitor - specify either the name of the histogram or its the full ROOT path <br />
* macros_to_monitor - specify the full path to the RootSpy macro .C file<br />
<br />
When a new monitoring run is started, or the conditions are changed, the following steps should be taken to process the new files:<br />
# Add a new data version, as described below:<br />
# Change the following parameters in check_monitoring_data.csh:<br />
## JOBDATE should correspond to the ouptut date used by the job submission script<br />
## OUTPUTDIR should correspond to the directory corresponding to the run period and revision corresponding to the new version you just submitted. Presumably, this directory will be empty at the beginning.<br />
## Once you create a new data version as defined below, you should pass the needed information as a command line option. Currently this is done by the ARGS variable. For example, the argument "-v RunPeriod-2014-10,8" tells the monitoring scripts to look up the version corresponding to revision 8 of RunPeriod-2014-10 in the monitoring DB and to use to store the results.<br />
<br />
<syntaxhighlight><br />
Example configuration parameters:<br />
set JOBDATE=2015-01-09<br />
set INPUTDIR=/volatile/halld/RunPeriod-2014-10/offline_monitoring<br />
set OUTPUTDIR=/w/halld-scifs1a/data_monitoring/RunPeriod-2014-10/ver08<br />
set ARGS=" -v RunPeriod-2014-10,8 "<br />
</syntaxhighlight><br />
If you want to process the results manually, the data is processed using the following script:<br />
<syntaxhighlight><br />
./process_new_offline_data.py <input directory> <output directory><br />
<br />
EXAMPLE:<br />
<br />
./process_new_offline_data.py 2014-11-14 /volatile/halld/RunPeriod-2014-10/offline_monitoring/ /w/halld-scifs1a/data_monitoring/RunPeriod-2014-10/ver02<br />
</syntaxhighlight><br />
The python script takes several options to enable/disable various steps in the processing. Of interest is the "--force" option, which will run over all monitoring ROOT files, whether or not they've been previously identified.<br />
<br />
Every time a new reconstruction pass is performed, a new version number must be generated. To do this, prepare a version file as described below. Then run the register_new_version.py script to store the information in the database. The script will return a version number, which then should be set by hand in process_new_offline_data.py - future versions of the script will streamline this part of the procedure. An example of how to generate a new version is:<br />
<syntaxhighlight><br />
./register_new_version.py add /home/gxproj1/halld/monitoring/process/versions/vers_RunPeriod-2014-10_pass1.txt<br />
</syntaxhighlight><br />
<br />
<b>If you are running the offline monitoring by checking out the files in trunk/scripts/monitoring/jproj/projects/<br />
of the svn repository</b>, and created a project with<br />
create_project.sh [project name] hd_rawdata<br />
Then go to the directory [project name]/processing/<br />
and execute<br />
./run_processing.sh<br />
which will run register_new_version.py as well as check_monitoring_data.csh for that project.<br />
<br />
===Step-by-Step Instructions For Processing a New Monitoring Run===<br />
<br />
The monitoring runs are current run out of the gxproj1 and gxproj5 accounts. After an offline monitoring run has been successfully started on the batch farm, the following steps should be followed to setup the post-processing for these runs.<br />
<br />
# The post-processing scripts are stored in $HOME/halld/monitoring/process and are automatically run by cron.<br />
# Run "svn update" to bring any changes in. Be sure that the list of histograms and macros to plot are current.<br />
# Add a new data version<br />
# Edit check_monitoring_data.csh to point to the current revisions/directories<br />
#* VERSION<br />
#* ARGS<br />
#* Note that the environment depends on a standard script - $HOME/setup_jlab.csh<br />
# Update files in the web directory, so that the results are displayed on the web pages: /group/halld/www/halldweb/html/data_monitoring/textdata<br />
# Copy the REST files to more permanent locations:<br />
#* cp -a /volatile/halld/offline_monitoring/RunPeriod-YYYY-MM/verVV/REST /work/halld/data_monitoring/RunPeriod-YYYY-MM/REST/verVV<br />
#* cp -a /volatile/halld/offline_monitoring/RunPeriod-YYYY-MM/verVV/REST /cache/halld/RunPeriod-YYYY-MM/REST/verVV [under testing]<br />
<br />
Check log files in $HOME/halld/monitoring/process/log for more information on how each run went. If there are problems, check log files, and modify check_monitoring_data.csh to vary the verbosity of the output.<br />
<br />
==Data Versions==<br />
<br />
To document the conditions of the monitoring data that is created, for the sake of reproducability and further analysis we save several pieces of information. The format is intended to be comprehensive enough to document not just monitoring data, but versions of raw and reconstructed data, so that this database table can be used for the event database as well.<br />
<br />
We store one record per pass through one run period, with the following structure:<br />
<br />
{| class="wikitable"<br />
! Field !! Description<br />
|-<br />
| data_type || The level of data we are processing. For the purposes of monitoring, "rawdata" is the online monitoring, "recon" is the offline monitoring<br />
|- <br />
| run_period || The run period of the data<br />
|- <br />
| revision || An integer specifying which pass through the run period this data corresponds to<br />
|- <br />
| software_version || The name of the XML file that specifies the different software versions used<br />
|-<br />
| jana_config || The name of the text file that specifies which JANA options were passed to the reconstruction program<br />
|-<br />
| ccdb_context || The value of JANA_CALIB_CONTEXT, which specifies the version of calibration constants that were used<br />
|-<br />
| production_time || The data at which monitoring/reconstruction began<br />
|-<br />
| dataVersionString || A convenient string for identifying this version of the data<br />
|}<br />
<br />
<br />
An example file used as as input to ./register_new_version.py is:<br />
<syntaxhighlight><br />
data_type = recon<br />
run_period = RunPeriod-2014-10<br />
revision = 1<br />
software_version = soft_comm_2014_11_06.xml<br />
jana_config = jana_rawdata_comm_2014_11_06.conf<br />
ccdb_context = calibtime=2014-11-10<br />
production_time = 2014-11-10<br />
dataVersionString = recon_RunPeriod-2014-10_20141110_ver01<br />
</syntaxhighlight></div>Kmoriyahttps://halldweb.jlab.org/wiki/index.php?title=Data_Monitoring_Procedures&diff=71724Data Monitoring Procedures2015-12-05T02:42:21Z<p>Kmoriya: /* Explanation of Scripts */</p>
<hr />
<div>__TOC__<br />
<br />
== Master List of File / Database / Webpage Locations ==<br />
=== Run Conditions ===<br />
* Online Run-by-run condition files (B-field, current, etc.): /work/halld/online_monitoring/conditions/<br />
* Offline monitoring run conditions (software versions, jana config): /group/halld/data_monitoring/run_conditions/<br />
*[http://www.jlab.org/Hall-D/test/RunInfo/ Run Info vers. 1]<br />
*[https://halldweb.jlab.org/cgi-bin/data_monitoring/run_conditions.pl Run Info vers. 2]<br />
<br />
=== Monitoring Output Files ===<br />
* Run Periods 201Y-MM is for example 2015-03, launch ver verVV is for example ver15<br />
* Online monitoring histograms: /work/halld/online_monitoring/root/<br />
* Offline monitoring histogram ROOT files: /work/halld/data_monitoring/RunPeriod-201Y-MM/verVV/rootfiles<br />
* REST files (most recent launch only): /work/halld/data_monitoring/RunPeriod-201Y-MM/REST/verVV<br />
* individual files for each job (ROOT, log, etc.): /volatile/halld/offline_monitoring/RunPeriod-201Y-MM/verVV/<br />
<br />
=== Monitoring Database ===<br />
* Accessing monitoring database (on ifarm): mysql -u datmon -h hallddb.jlab.org data_monitoring<br />
<br />
=== Monitoring Webpages ===<br />
*[https://halldweb.jlab.org/cgi-bin/data_monitoring/monitoring/plotBrowser.py Plot Browser]<br />
*[https://halldweb.jlab.org/cgi-bin/data_monitoring/monitoring/runBrowser.py Run Browser]<br />
*[https://halldweb.jlab.org/cgi-bin/data_monitoring/monitoring/timeSeries.py Time Series]<br />
*[https://halldweb.jlab.org/data_monitoring/launch_analysis/ Launch Analysis]<br />
<br />
== Job Monitoring Links ==<br />
* [http://scicomp.jlab.org/farm2/job.html Completed Job History]<br />
* [http://scicomp.jlab.org/farm2/project.html Job Stats By Project]<br />
* [http://scicomp.jlab.org/farm2/trackOrg.html Job Stats By Track]<br />
* [http://scicomp.jlab.org/farm2/report.html Cluster Report]<br />
* [http://scicomp.jlab.org/farm2/walltime.html Walltime Distribution]<br />
<br />
== Saving Online Monitoring Data ==<br />
<br />
The procedure for writing the data out is given in, e.g.,<br />
[https://halldweb.jlab.org/wiki/index.php/Raid-to-Silo_Transfer_Strategy Raid-to-Silo Transfer Strategy].<br />
<br />
Once the DAQ writes out the data to the raid disk, cron jobs will copy the file to tape,<br />
and within ~20 min., we will have access to the file on tape at<br />
/mss/halld/$RUN_PERIOD/rawdata/RunXXXXXX.<br />
<br />
All online monitoring plugins will be run as data is taken.<br />
They will be accessible within the counting house via RootSpy, and<br />
for each run and file, a ROOT file containing the histograms will be saved<br />
within a subdirectory for each run.<br />
<br />
For immediate access to these files, the raid disk files may be accessed directly<br />
from the counting house, or the tape files will be available within ~20 min. of the<br />
file being written out.<br />
<br />
== Offline Monitoring: Running Over Archived Data ==<br />
<br />
Once files are written to tape we run the online plugins on these files to confirm what we were seeing in the online monitoring, and also to update the results from the latest calibration and software. <br />
Manual scripts and cron jobs are set up to look for new data and run the plugins over a sample of files.<br />
<br />
Every other Friday (usually the Friday before the offline meetings) jobs will be started to run the newest software on all previous runs,and allows everybody to see improvements in each detector.<br />
For each launch, independent builds of hdds, sim-recon, the monitoring plugins, and an sqlite file will be generated.<br />
<br />
Below the procedures are described for<br />
# Preparing the software for the launch<br />
# Starting the launch (using hdswif)<br />
# Post-analysis of statistics of the launch<br />
<br />
Processing the results and making them available to the collaboration<br />
is handled in the section [[#Post-Processing_Procedures | Post-Processing Procedures]] below.<br />
<br />
<!--<br />
==== Generating an offline plugin job ====<br />
<br />
The user gxproj1 should be used for official offline monitoring jobs.<br />
Within /home/gxproj1/halld/monitoring/batch/ there will be scripts to run the online monitoring plugins over tape files.<br />
The main script is generatejobs_plugins_rawdata.sh, which can be used as<br />
generatejobs_plugins_rawdata.sh [minrun] [maxrun] (minfile) (maxfile)<br />
where minrun, maxrun specify the range of the run #, and minfile, maxfile (optional) specify<br />
the file range for that run.<br />
<br />
This will generate a script run_rawdata_XXXXXX.sh, where the run # has now been formatted to be 6 digits.<br />
Executing this script will send the monitoring plugins job to the Auger batch system.<br />
<br />
There is also a script clean.sh which can be used as<br />
clean.sh XXX<br />
This will clean up all associated files created in association with run XXX.<br />
<br />
Internally, the xml file used to submit the job will be created, and the job to run<br />
will be given within script.sh. All run parameters should be specified in at the beginning<br />
of generatejobs_plugins_rawdata.sh<br />
Since we are running on tape, the tape file will first be copied over to the cache disk, and the job will run<br />
over this cached file.<br />
<br />
==== Using cron to run automatically ====<br />
Within /home/gluex/halld/monitoring/cron/ there is a file cron_plugins<br />
that can be executed via<br />
crontab cron_plugins<br />
This will set up a cron job to call the script scan_for_jobs.sh, which will<br />
check in the rawdata directory and call generatejobs_plugins_rawdata.sh for<br />
any run that is more than 5 min old. The cron job is set up to run every 10 min.<br />
<br />
==== Magnetic Field Settings ====<br />
For tracking, it is necessary to set the correct magnetic field settings.<br />
The below table is now obsolete. Please refer to the [https://halldweb.jlab.org/cgi-bin/data_monitoring/run_conditions.pl GlueX Run Conditions] webpage.<br />
<br />
The actual field settings for each run have not been documented well.<br />
Below are values based on going through entries in the [https://logbooks.jlab.org/book/halld halld log].<br />
<br />
{| border="1" cellpadding="1" style="text-align: center;"<br />
!width="100"| Run #s<br />
!width="150"| Solenoid Current (A)<br />
!width="300"| JANA option<br />
!width="300"| notes<br />
|-<br />
| 940 - 996 || 1000 || -PBFIELD_MAP=Magnets/Solenoid/solenoid_1000A_poisson_20141104 || At run 997, the solenoid started ramping down<br />
|-<br />
| 998 - 1448 || 0 || -PBFIELD_TYPE=NoField -PDEFTAG:DTrackCandidate=StraightLine || Solenoid started ramping up around run 1431. Runs 1432 - 1448 should not be used for tracking. See [https://logbooks.jlab.org/entry/3309039 this] and [https://logbooks.jlab.org/entry/3309092 this] entry.<br />
|-<br />
| 1449 - 1620 || 1200 || -PBFIELD_MAP=Magnets/Solenoid/solenoid_1200A_poisson_20140520 || <br />
|-<br />
|}<br />
<br />
'''Note:''' The following run ranges were taken with a 300A field, and could be analyzed with the magnetic field map Magnets/Solenoid/solenoid_300A_poisson_20140819 , more study is needed to see if the straight line track fitter is needed or not: 1036 - 1053, 1065 - 1121, 1212? - 1254, 1309 - 1318 [1307,1308,1319-1329 were taken while the magnet was ramping].<br />
--><br />
<br />
=== General Information on Procedures ===<br />
This section explains how the offline monitoring should be run.<br />
Since we may want to simultaneously run offline monitoring for different run periods that require<br />
different environment variables, the scripts are set up so that a generic user can download the<br />
scripts and run them from anywhere. Most output directories for offline monitoring are created<br />
with group read/write permissions so that any Hall D group user has access to the contents,<br />
but there are some cases where use of the account that created the launch is necessary.<br />
<br />
The accounts used for offline monitoring are the gxprojN accounts created and maintained by<br />
Mark Ito (see [https://halldweb.jlab.org/wiki/index.php/GlueX-related_shared_accounts_on_the_JLab_CUE here] for how each account is used).<br />
As of October 2015, the following are used:<br />
* gxproj1 for running over Fall 2014 data (deprecated since June 2015)<br />
* gxproj5 for running over Spring 2015 data<br />
<br />
Since the summer of 2015 we have transitioned from a system using Mark Ito's jproj scripts<br />
to integrating the swif system that Chris Larrieu (SciComp) has been developing. For offline monitoring,<br />
the hdswif system that Kei developed is used for launching the jobs, and the jproj system is used<br />
for meta-analysis of launch statistics.<br />
<br />
Both hdswif and jproj are maintained in svn:<br />
* hdswif: https://halldsvn.jlab.org/repos/trunk/scripts/monitoring/hdswif<br />
* jproj : https://halldsvn.jlab.org/repos/trunk/scripts/monitoring/jproj<br />
<br />
To run the offline monitoring each package should be checked out and all necessary<br />
scripts are included.<br />
<br />
=== Preparing the software for the launch ===<br />
<br />
1. Setup the environment: <pre>source ~/env_monitoring_launch</pre><br />
<br />
2. Updating & building hdds: <br />
<pre><br />
cd $HDDS_HOME<br />
git pull # Get latest software<br />
scons -c install # Clean out the old install: EXTREMELY IMPORTANT for cleaning out stale headers<br />
scons install -j4 # Rebuild and re-install with 4 threads<br />
</pre><br />
<br />
3. Updating & building sim-recon: <br />
<pre><br />
cd $HALLD_HOME/src<br />
git pull # Get latest software<br />
scons -c install # Clean out the old install: EXTREMELY IMPORTANT for cleaning out stale headers<br />
scons install -j4 # Rebuild and re-install with 4 threads<br />
</pre><br />
<br />
4. Create a new sqlite file containing the very latest calibration constants. Original documentation on creating sqlite files are [https://halldweb.jlab.org/wiki/index.php/SQLite-form_of_the_CCDB_database here]. <br />
<pre><br />
cd $GLUEX_MYTOP/../sqlite/<br />
$CCDB_HOME/scripts/mysql2sqlite/mysql2sqlite.sh -hhallddb.jlab.org -uccdb_user ccdb | sqlite3 ccdb.sqlite<br />
mv ccdb.sqlite ccdb_monitoring_launch.sqlite #replacing the old file<br />
</pre><br />
<br />
5. Note that the above steps must be done <b>BEFORE</b> launch project creation. This is because we will track the revisions of the libraries used, and this is done by extracting the svn information in each directory. Also, note that the system assumes that we have the topmost build directory (usually called GLUEX_TOP) to be $HOME/builds . Such an assumption is necessary to be able to extract information about the library locations automatically.<br />
<br />
Create the appropriate project(s) and submit the jobs using hdswif, as detailed in the section below.<br />
<br />
=== Starting the Launch and Submitting Jobs ===<br />
<br />
Until the summer of 2015 we relied solely on Mark Ito's jproj system for submitting and keeping track of jobs. We have since moved to the swif system and use the hdswif wrapper for this. Below are instructions for how to use these.<br />
<br />
1. Downloading hdswif: Download the hdswif directory from svn. For the gxprojN accounts, use the directory ~/halld/hdswif. <br />
<pre><br />
cd ~/halld<br />
svn co https://halldsvn.jlab.org/repos/trunk/scripts/monitoring/hdswif<br />
cd hdswif<br />
</pre><br />
<br />
2. Edit the job config file, input.config, which is used to register jobs in hdswif. A typical config file will look this:<br />
PROJECT gluex<br />
TRACK reconstruction<br />
OS centos65<br />
NCORES 6<br />
DISK 40<br />
RAM 8<br />
TIMELIMIT 8<br />
JOBNAMEBASE offmon_<br />
RUNPERIOD 2015-03<br />
VERSION 15<br />
OUTPUT_TOPDIR /volatile/halld/offline_monitoring/RunPeriod-[RUNPERIOD]/ver[VERSION] # Example of other variables included in variable<br />
SCRIPTFILE /home/gxproj5/halld/hdswif/script.sh # Must specify full path<br />
ENVFILE /home/gxproj5/halld/hdswif/setup_jlab-2015-03.csh # Must specify full path<br />
<br />
3. Creating the workflow: Within SWIF jobs are registered into workflows. For offline monitoring, the workflow names are of the form <b>offline_monitoring_RunPeriod201Y_MM_verVV_hd_rawdata</b> with suitable replacements for the run period and version number. The command "swif list" will list all existing workflows. Also, for most simple SWIF commands hdswif also provides a wrapper. <br />
swif list<br />
<br />
For creation of workflows for offline monitoring the command:<br />
hdswif.py create [workflow] -c input.config<br />
should be used. When a config file (here: input.config) is passed in, hdswif will automatically create files that record the configuration of the current launch. These files are stored as for example:<br />
/group/halld/data_monitoring/run_conditions/jana_rawdata_comm_2015_03_ver15.conf<br />
/group/halld/data_monitoring/run_conditions/soft_comm_2015_03_ver15.xml<br />
<br />
The software packages stored in git (sim-recon and hdds) can have git tags applied to them, which makes it easier to find versions of the software than a SHA-1 hash. hdswif will ask if you would like to create a tag, and execute the following sequence: <pre>git tag -a offmon-201Y_MM-verVV -m "Used for offline monitoring 201Y-MM verVV started on 201y/mm/dd"</pre> <pre>git push offmon-201Y_MM-verVV</pre> This will only be invoked when the user name is gxprojN, and for the configuration files, the output directory will be /group/halld/data_monitoring/run_conditions/ for gxprojN accounts while it will be the current directory for other users.<br />
* To use the git-tagged software versions do for example <pre>cd $HALLD_HOME</pre> <pre>git checkout offmon-2015_03-ver15</pre><br />
<br />
4. Registering jobs in the workflow: To register jobs within the workflow, hdswif provides the use of config files. Jobs can be registered by specifying the workflow, config file (-c), run (-r) and file (-f) numbers if necessary. <br />
<span style="color:red">Note: Job configuration parameters can be set differently for jobs within the same workflow if necessary.</span><br />
Jobs can be added via <pre>hdswif.py add [workflow] -c input.config</pre><br />
By default, hdswif will add all files found within the directory /mss/halld/RunPeriod-201Y-MM/rawdata/ where 201Y-MM is specified by the RUNPERIOD parameter in the config file. If only some of the runs or files are needed, these can be specified for example with<br />
<pre>hdswif.py add [workflow] -c input.config -r 3180 -f '00[0-4]'</pre><br />
to specify to register running only over run 3180 files 000 - 004 (Unix-style brackets and wildcards can be used).<br />
<br />
5. Running the workflow: To run the workflow, simply use the hdswif wrapper:<br />
hdswif.py run [workflow]<br />
<br />
<b>It is recommended that some jobs be tested to make sure that everything is working rather than fail thousands of jobs.</b><br><b>Check the configuration setup when creating a workflow, and what is in script.sh. Also check the following:<br />
* Check stderr files. Are they small (<kB)?<br />
* Check stdout files. Are they very large (>MB)?<br />
* Check output ROOT files. Are they larger than several MB?<br />
* Check output REST files. Are they larger than several tens of MB?<br />
</b><br />
<br> For this purpose, hdswif will take an additional parameter to run which limits the number of jobs to submit: <pre>hdswif.py run [workflow] 10</pre><br />
in which case only 10 jobs will be submitted.<br />
<br />
To submit all jobs after checking the results, do <pre>hdswif.py run [workflow]</pre><br />
<br />
=== Checking the Status and Resubmitting ===<br />
1. The status of jobs can be checked on the terminal with <pre>jobstat -u gxprojN</pre> For the status of jobs on Auger see http://scicomp.jlab.org/scicomp/#/auger/jobs and for SWIF use <pre>swif list</pre> or for more information, <pre>swif status [workflow] -summary</pre> Note that "swif status" tends to be out of date sometimes, so don't panic if your workflow/jobs aren't showing up right away. Also see the Auger [http://scicomp.jlab.org/scicomp/#/auger/jobs job website].<br />
<br />
2. For failed jobs, SWIF can resubmit jobs based on the problem. For resubmission for failed jobs with the same resources, <pre>swif retry-jobs [workflow] -problems [problem name]</pre> can be used, and for jobs to be submitted with more resources, e.g., use <pre>swif modify-jobs [workflow] -ram add 2gb -problems AUGER-OVER_RLIMIT</pre> <br />
This only re-stages the jobs, be sure to resubmit them with:<br />
<pre>swif run -workflow [workflow] -errorlimit none</pre><br />
<br />
hdswif has a wrapper for both of these: <pre>hdswif.py resubmit [workflow] [problem]</pre> In this case [problem] can be one of <b>SYSTEM, TIMEOUT, RLIMIT</b>. If SYSTEM is specified, the jobs will be retried. For TIMEOUT and RLIMIT, the jobs will be modified by default with 2 additional hours or GB of RAM. If one more number is added as an option, then that many hours or GB of RAM will be added., e.g., <pre>hdswif.py resubmit [workflow] TIMEOUT 5</pre> will add 5 hours of processing time. You can wait until almost all jobs finish before resubmitting failed jobs since the number should be relatively small. <b>Even if jobs are resubmitted for one type of failure, jobs that later fail with that failure will not be automatically resubmitted.</b><br />
<br />
3. For information on swif, use the "swif help" commands and for hdswif see the attached documentaion in https://halldsvn.jlab.org/repos/trunk/scripts/monitoring/hdswif/manual_hdswif.pdf<br />
<br />
4. Below is a table describing the various errors that can occur.<br />
{| border="1" cellpadding="0" valign="left" style="text-align: left;"<br />
!width="150"| ERROR NAME<br />
!width="400"| Description<br />
!width="400"| Resolution<br />
!width="400"| hdswif command<br />
|-<br />
| AUGER-SUBMIT<br />
||<br />
SWIF’s attempt to submit jobs to Auger failed. Includes server-side problems as well as user failing to provide valid job parameters (e.g. incorrect project name, too many resources, etc.)<br />
||<br />
If requested resources are known to be correct resubmit. Otherwise modify job resources using swif directly.<br />
||<br />
<span style="color:purple"><b>hdswif.py resubmit [workflow] SYSTEM</b></span><br />
|-<br />
| AUGER-FAILED<br />
||<br />
Auger reports the job FAILED with no specific details.<br />
||<br />
Resubmit jobs. If problem persists, contact Chris Larrieu or SciComp.<br />
||<br />
<span style="color:purple"><b>hdswif.py resubmit [workflow] SYSTEM</b></span><br />
|-<br />
| AUGER-OUTPUT-FAIL<br />
||<br />
Failure to copy one or more output files.Can be due to permission problem, quota problem, system error, etc.<br />
||<br />
Check if output files will exist after job execution and that output directory exists, resubmit jobs. If problem persists, contact Chris Larrieu or SciComp.<br />
||<br />
<span style="color:purple"><b>hdswif.py resubmit [workflow] SYSTEM</b></span><br />
|-<br />
| AUGER-INPUT-FAIL<br />
||<br />
Auger failed to copy one or more of the requested input files, similar to output failures. Can also happen if tape file is unavailable (e.g. missing/damaged tape)<br />
||<br />
Check if input file exists, resubmit jobs. If problem persists, contact Chris Larrieu or SciComp.<br />
||<br />
<span style="color:purple"><b>hdswif.py resubmit [workflow] SYSTEM</b></span><br />
|-<br />
| AUGER-TIMEOUT<br />
||<br />
Job timed out.<br />
||<br />
If more time is needed for job add more resources.<br />
Default is to add 2 hrs of processing time. Also check whether code is hanging.<br />
||<br />
<span style="color:red"><b>hdswif.py resubmit [workflow] TIMEOUT</b></span><br><br />
Default is to add 2 hours. Optionally specify number of hours at end.<br />
|-<br />
| AUGER-OVER_RLIMIT<br />
||<br />
Not enough resources, RAM or disk space.<br />
||<br />
Add more resources for job.<br />
||<br />
<span style="color:blue"><b>hdswif.py resubmit [workflow] RLIMIT</b></span><br><br />
Default is to add 2 GB of RAM. Optionally specify GB at end. To add more disk space use SWIF directly.<br />
|-<br />
| SWIF-MISSING-OUTPUT<br />
||<br />
Output file specified by user was not found.<br />
||<br />
Check if output file exists at end of job.<br />
||<br />
<br />
|-<br />
| SWIF-USER-NON-ZERO <br />
||<br />
User script exited with non-zero status code.<br />
||<br />
Your script exited with non-zero status. Check the code you are running.<br />
||<br />
<br />
|-<br />
| SWIF-SYSTEM-ERROR <br />
||<br />
Job failed owing to a problem with swif (e.g. network connection timeout)<br />
||<br />
Resubmit jobs. If problem persists, contact Chris Larrieu or SciComp.<br />
||<br />
<span style="color:purple"><b>hdswif.py resubmit [workflow] SYSTEM</b></span><br />
|}<br />
<br style="clear:both;"/><br />
<br />
=== Post-analysis of statistics of the launch ===<br />
<br />
After jobs have been submitted, it will usually take a few days for all of the jobs to be processed.<br />
The next step is to check the resource usage for the current launch and publish the results online.<br />
<br />
# <b>Create summary XML, HTML files</b><br> The status and results of jobs are saved within the SWIF internal server, and are available via the command <pre>swif status [workflow] -summary -runs</pre> where the arguments -summary and -runs show summary statistics and statistics for individual jobs, respectively. hdswif has a command that takes this output in XML output and creates an HTML webpage showing results of the launch. To do this, do <pre>hdswif.py summary [workflow]</pre> This will create the XML output file from SWIF called swif_output_[workflow].xml and create a webpage containing png figure files. If the XML file already exists, hdswif will ask whether to overwrite the existing file.<br />
# <b>Publish output files online</b><br> At this stage the html output and figure files are created and ready to be put online. The html output capabilities of hdswif is useful for any user using SWIF, but since publication of the html output is specific to the offline monitoring, the script to do this is contained in the jproj scripts directory at https://halldsvn.jlab.org/repos/trunk/scripts/monitoring/jproj. For the gxprojN accounts this directory should exist as ~/halld/jproj. To publish the results online do for example <pre>python ~/halld/jproj/projects/templates/publish_offmon_results.py 2015_03 18</pre> The script simply copies the html output and corresponding figures to /group/halld/www/halldweb/html/data_monitoring/launch_analysis/<br />
# <b>Editing the summary HTML page</b><br> The top page for is offline monitoring https://halldweb.jlab.org/data_monitoring/launch_analysis/index.html and has links to the summary page for each run period. The summary files are /group/halld/www/halldweb/html/data_monitoring/launch_analysis/[run period].html Edit the file to<br />
## Add a new line to the first table which contains the version number, date, and comments for the current launch<br />
## Create a link to the webpage for the current launch. Simply copy and paste, modify the previous launch's link to have the correct launch ver.<br />
# <b>Freezing SWIF tables</b><br> Since we are now finished with the SWIF workflow that we used, the workflow should be "frozen" so that it cannot be mistakenly altered or modified. Do <pre>swif freeze [workflow]</pre><br />
# <b>Backing up SWIF output</b><br> With the workflow frozen we should be able to reproduce the XML output from SWIF, but we will backup the XML output just in case. Do <pre>cp ~/halld/hdswif/swif_output_[workflow].xml /group/halld/data_monitoring/swif_xml_backup/ </pre><br />
<br />
=== Cross Analysis of Launches ===<br />
<br />
The purpose of the cross analysis is to correlate how resource usage changed for the same files across different launches.<br />
To do this it is useful to create MySQL tables that contain information on each launch, and do queries across tables.<br />
<br />
#The scripts to do this are maintained in the svn directory https://halldsvn.jlab.org/repos/trunk/scripts/monitoring/jproj Do <pre> svn co https://halldsvn.jlab.org/repos/trunk/scripts/monitoring/jproj </pre> For the gxprojN accounts used for offline monitoring the directory should be ~/halld/jproj<br />
# The jproj directory contains two subdirectories, scripts and projects. The scripts directory contains useful scripts for processing the jobs registered in the jproj system, and each of the offline monitoring launches will be handled in the projects directory.<br />
# Go to the projects directory<pre>cd ~/halld/jproj/projects</pre> and use the script create_project.sh to create a new directory that contains the processing scripts for the current launch <pre>./create_project.sh [workflow]</pre> This should create a directory such as offline_monitoring_RunPeriod2015_03_ver15_hd_rawdata (same as the workflow name). The script uses the template files in the directory templates and by substitution creates script files for the current launch. Now go to the newly created analysis directory: <pre>cd [workflow]/analysis</pre><br />
which for the gxprojN accounts should have the full path /home/gxprojN/halld/jproj/projects/[workflow]/analysis.<br><br><br />
<span style="color:red">All of the analysis commands including arguments are contained in run_analysis.sh, but it is <b>strongly</b> recommended that all commands are run manually to check for errors.</span><br><br><br />
# Next, we need to create a few MySQL tables for the current launch. The MySQL tables are useful for comparing run/file combinations for different launches. For the SWIF launches, there are two tables needed, and will be named<br />
#* [workflow]Job<br />
#* [workflow]_aux<br />
# This naming scheme of tables and their roles are the same as from the jproj only launches. The [workflow]Job table will contain information gathered from SWIF about each job (which node it went to, start time of each stage, memory usage, etc.). The [workflow]_aux table will contain information gathered from the output stdout files from each job. First, create the Job table using <pre>python create_jproj_job_table.py [run period] [version]</pre> Note that this script uses the XML output from hdswif summary and inserts the contents into the MySQL table, so the XML output file must exist.<br />
# To check the contents of this MySQL table, do <pre>mysql -hhalldb -ufarmer farming</pre> <pre>mysql> describe offline_monitoring_RunPeriod2015_03_ver11_hd_rawdataJob;</pre><br />
# <b>Backing up tables</b><br> Tables created in MySQL should be backed up.<br />
<br />
== Explanation of Scripts ==<br />
Below are explanations of each script used in the offline monitoring system and a breif explanation of how they work.<br />
<br />
=== hdswif scripts ===<br />
<b>Summary:</b> hdswif.py is the main script that calls the other utility scripts. For the utility scripts, they can be run standalone by giving the appropriate parameters. Visual graphics are made using the PyROOT extension of ROOT. To use this, the environment variable PYTHONPATH must include ROOTSYS. If ROOTSYS has been set, adding it to PYTHONPATH will be done by the script, but if ROOTSYS is not set, then the scripts will abort.<br />
<br />
{| border="1" cellpadding="0" valign="left" style="text-align: left;"<br />
!width="150"| file name<br />
!width="800"| Description<br />
|-<br />
| hdswif.py ||<br />
Main script to control the behavior of SWIF. Most commands follow the form hdswif.py [command] [workflow] (options)<br />
|-<br />
| parse_swif.py ||<br />
Called within hdswif.py to create html output from SWIF results.<br />
|-<br />
| createXMLfiles.py ||<br />
* Called by the "create" option of hdswif.py<br />
* Creates XML files for logging information about launch. Must specify config file with option -c.<br />
* Also adds tags to git repositories of sim-recon and hdds.<br />
* For XML file creation, the file will be written out to /group/halld/data_monitoring/run_conditions/ if the user is gxprojN (used for offline monitoring). If not, to avoid general users adding things to the above directory, the output files will be the current directory.<br />
* To write out the versions of each package, environment variables such as HDDS_HOME will need to be set.<br />
* Versions for each software package are extracted using the directory structure, so if these are changed the scripts must change accordingly.<br />
* For each launch, <b>the output soft_comm_[RUNPERIOD]_ver[VER].xml file should be checked that all version numbers have been extracted.</b><br />
|-<br />
| read_config.py ||<br />
* Called by the "add" option of hdswif.py<br />
* Takes in config file name, optionally set verbose<br />
* Return a dictionary between config parameter names and values (e.g., 'PROJECT' : 'gluex', 'NCORES' : 6, ...)<br />
* Prints the config parameters at the end. For parameters changed from default, a '*' will be printed<br />
|-<br />
| output_job_details.py ||<br />
* Called by the "details" option of hdswif.py<br />
* Takes in workflow name, run and file. Run and file must be numbers, no wildcards<br />
* Finds ids for jobs specified by the run and file number and returns info on each one<br />
* Job info is retrieved from pbs farm system and shows configuration parameters for that job<br />
|-<br />
| results_by_resources.py ||<br />
* Called within parse_swif.py<br />
* Takes in XML output from SWIF and creates 2 plots<br />
* Creates html table showing results of jobs by resources requested for that job<br />
* This table is shown under "Status by Resources" in the output html file of hdswif.py summary [workflow]<br />
|-<br />
| create_ordered_hists.py ||<br />
* Called within parse_swif.py<br />
* Takes in XML output from SWIF and creates 2 plots<br />
* Plots are dependency time and pending time of each job, ordered by submission.<br />
* Different colors represent jobs submitted at different times. For jobs submitted at the same time, jobs are ordered in increasing time.<br />
|-<br />
| create_stacked_times.py ||<br />
* Called within parse_swif.py<br />
* Takes in XML output from SWIF and creates 2 plots<br />
* Plots show total job time divided into colors for different stages<br />
* One shows all jobs in order of Auger ID (roughly submission order), the other one shows in order of total job time<br />
* The jobs show at a glance which stage contributes how much of the job's time<br />
|}<br />
<br />
=== Utility scripts ===<br />
{| border="1" cellpadding="0" valign="left" style="text-align: left;"<br />
!width="150"| file name<br />
!width="800"| Description<br />
|-<br />
| stderr_by_size.py ||<br />
* Independent of all other scripts in hdswif directory<br />
* When diagnosing problems it is useful to check the stderr/stdout files. Frequently, different problems are easier to find based on the size of stderr files<br />
* Takes in run period and version as arguments, creates a directory /volatile/halld/offline_monitoring/RunPeriod-[RUNPERIOD]/ver[VER]/log/bysize that contains soft links to all stdout and stderr files from the specified launch, <b>in separate directories given by stderr file size</b>.<br />
|}<br />
<br />
== Hall D Job Management System ==<br />
<br />
This section details instructions on how to create and launch a set of jobs using the Hall-D Job Management System developed by Mark Ito. These instructions are generic: this system can be used for the weekly monitoring jobs, but can also be used for other sets of job launches as well. <br />
<br />
=== Database Table Overview ===<br />
<br />
* Job management database table (<project_name>): For each input file, keeps track of whether or not a job for it has been submitted, along with other optional fields. <br />
<br />
* Job status database table (<project_name>Job (no space)): For each job, keeps track of the job-id, the job status, memory used, cpu & wall time, time taken to complete various stages (e.g. pending, dependency, active), and others. <br />
<br />
* Job metrics database table (<project_name>_aux (no space)): For each job, keeps track of the job-id, how many events were processed, the time it took to copy the cache file, and the time it took to run the plugin. This information is culled from the log files of each job, and is done within the analysis directory of each launch.<br />
<br />
=== Initialize Project Management ===<br />
<br />
* Log into the ifarm machine with one of the gxproj accounts. For this example we will use gxproj1.<br />
<pre><br />
ssh gxproj1@ifarm -Y<br />
</pre><br />
<br />
* Go to a directory to do the launch. In principle, any directory will work, but for gxproj1 this is usually done in /home/gxproj1/halld/<br />
<br />
* Check out the necessary scripts <pre>svn co https://halldsvn.jlab.org/repos/trunk/scripts/monitoring/jproj</pre> This will get all necessary scripts for launching. Once checked out, <pre>cd projects</pre><br />
<br />
* The script create_project.sh can be used to create a new project. It will take a single argument, the project name. It is assumed that the project name is of the form offline_monitoring_RunPeriod20YY_MM_verVV_hd_rawdata where this string will be parsed to give the run period 20YY_MM and version number VV. One thing to do BEFORE creation of a new project is to editthe conditions of the launch (plugins to run over, memory requested, disk space requested) within templates/template.jsub . This<br />
information is saved automatically at project creation time into files in /group/halld/data_monitoring/run_conditions .<br />
To create a project do <pre>./create_project.sh offline_monitoring_RunPeriod20YY_MM_verVV_hd_rawdata</pre><br />
The name has been chosen to be as consistent as possible with other directory structures.<br />
However, mysql requires that "-" be escaped in table names, so unfortunately run periods will be<br />
given as 2YYY_MM instead of 2YYY-MM.<br />
<br />
* A new directory with the project name (offline_monitoring_RunPeriod20YY_MM_verVV_hd_rawdata) and files will be copied and modified from the template directory to reflect the run period, the user, the directory that it was created in, project name, etc.<br />
<br />
* For each project, cd into the new directory<br />
* The script clear.sh will remove any existing tables for the current project name, then recreate it. Do<br />
<pre>./clear.sh</pre><br />
* To use the jproj.pl script that was checked in, add the directory to your path with<br />
<pre>source ../../scripts/setup.csh</pre><br />
or always specify the full path<br />
<pre>../../scripts/jproj.pl</pre><br />
* Now update the table of runs with<br />
<pre>jproj.pl <project name> update</pre><br />
This will fill the table with all files within /mss that are of the same form as what is in <project name>.jproj . If you want to register only on a subset of all such files, you can edit this file directly.<br />
* Once you have registered all of the files you would like to run over, do<br />
<pre>jproj.pl <project name> submit [max # of jobs] [run number]</pre><br />
where the additional options specify how many jobs to submit and which run number to run on. Without these options all files that are registered and have not been submitted yet will be submitted.<br />
<br />
At this stage you are ready to submit all files. It is a good idea to submit a few test jobs<br />
at first to check that all scripts are working and that the plugins do not crash. Once you are<br />
sure that this does not happen, you can send all jobs in. The remaining jobs are then the monitoring<br />
which will (among other things) put the results on the online webpage for the collaboration to view, and<br />
the analysis of the launch.<br />
<br />
=== Project File Overview ===<br />
<br />
An overview of each project file:<br />
* '''clear.sh:''' For the current project, deletes the job status and management database tables (if any), and creates new, empty ones. <br />
* '''<project_name>.jproj:''' Contains the path and file name format for the input files for the jobs. <br />
* '''<project_name>.jsub:''' The xml job submission script. The run number and file number variables are set during job submission for each input file. <br />
* '''script.sh:''' The script that is executed during the job. If output job directories are not pre-created manually, they should be created in this script with the proper permissions:<br />
<pre><br />
mkdir -p -m 775 my_directory<br />
</pre><br />
* '''setup_jlab-[run period].csh:''' The environment that is sourced at the beginning of the job execution. <br />
* '''status.sh:''' Updates the job status database table, and prints some of its columns to screen.<br />
<br />
=== Project Management ===<br />
<br />
* Delete (if any) and create the database table(s) for the current set of job submissions:<br />
<pre><br />
./clear.sh<br />
</pre><br />
Also, if testing was done with jobs, it is best to delete the output directory and the configuration files:<br />
<pre><br />
rm -frv /volatile/halld/offline_monitoring/RunPeriod-20YY-MM/verVV /group/halld/data_monitoring/run_conditions/soft_comm_20YY_MM_verVV.xml /group/halld/data_monitoring/run_conditions/jana_rawdata_comm_20YY_MM_verVV.conf<br />
</pre><br />
<br />
* Search for input files matching the string in the .jproj file, and create a row for each in the job management database table (called <project_name>). You can test by adding an optional argument at the end, which only selects files with a specific file number:<br />
<pre><br />
jproj.pl <project_name> update <optional_file_number><br />
</pre><br />
<br />
* Confirm that the job management database is accurate by printing it's contents to screen:<br />
<pre><br />
mysql -hhallddb -ufarmer farming -e "select * from <project_name>"<br />
</pre><br />
<br />
* ONLY if a mistake was made, to delete the tables from the database and recreate new, empty ones, run: <br />
<pre><br />
./clear.sh<br />
</pre><br />
<br />
* To look at the status of the submitted jobs, first query auger and update the job status database:<br />
<pre><br />
fill_in_job_details.pl <project_name><br />
</pre><br />
<br />
* The job status can then be viewed by submitting a query to the job status database (called <project_name>Job (no space in between)):<br />
<pre><br />
mysql -hhallddb -ufarmer farming -e "select id,run,file,jobId,hostname,status,timeSubmitted,timeActive,walltime,cput,timeComplete,result,error from <project_name>Job"<br />
</pre><br />
<br />
* These last two commands can instead be executed simultaneously by running:<br />
<pre><br />
./status.sh<br />
</pre><br />
<br />
=== Handy mysql Instructions ===<br />
<br />
* Handy mysql instructions:<br />
<pre><br />
mysql -hhallddb -ufarmer farming # Enter the "farming" mysql database on "hallddb" as user "farmer"<br />
quit; # Exit mysql<br />
show tables; # Show a list of the tables in the current database<br />
show columns from <project_name>; # show all of the columns for the given table<br />
select * from <project_name>; # show the contents of all rows from the given table<br />
</pre><br />
<br />
=== Backing Up Offline Monitoring Tables ===<br />
Tables created for offline monitoring can be backed up using the script backup_tables.sh which<br />
can be checked out with the other files from https://halldsvn.jlab.org/repos/trunk/scripts/monitoring/jproj/projects<br />
<br />
The script uses the command mysqldump to print out a file that can be executed to recreate the tables.<br />
Since executing this output file will drop the table when it exists, caution is advised.<br />
Example usage to backup all three tables created for run period 2014_10 ver 17:<br />
<pre>backup_tables.sh 2014_10 17</pre><br />
<br />
== Running Over Data As It Comes In ==<br />
<br />
A special user gxproj1 will have a cron job set up to run the plugins as new data appears on /mss.<br />
During the week, gxproj1 will submit offline plugin jobs with the same setup as the weekly jobs<br />
run the previous Friday. The procedure for this is shown below.<br />
<br />
<!--<br />
=== Setting up the environment ===<br />
The file<br />
/home/gxproj1/setup_jlab.csh<br />
is sourced through .tcshrc.<br />
This file is the same as what is linked to by<br />
/home/gluex/setup_jlab_commissioning.csh,<br />
except HALLD_HOME, HDDS_HOME, and JANA_CALIB_URL are set separately so that this<br />
user can have a separate build.<br />
<br />
To obtain the builds from the previous Friday's runs,<br />
execute<br />
/home/gxproj1/halld/monitoring/newruns/setup_previous.sh [year] [month] [day]<br />
The build revisions from the previous Friday are archived in files<br />
/work/halld/data_monitoring/run_conditions/soft_comm_[year]_[month]_[day].xml<br />
and the script will build libraries based on those stored revision numbers.<br />
--><br />
<br />
=== Running the cron job ===<br />
<br />
'''IMPORTANT:''' The cron job should not be running while you are manually submitting jobs using the jproj.pl script for the same project, or else you will probably multiply-submit a job. <br />
<br />
* Go to the cron job directory: <br />
<pre><br />
cd /u/home/gxproj1/halld/monitoring/newruns<br />
</pre><br />
<br />
* The cron_plugins file is the cronjob that will be executed. During execution, it runs the exec.sh command in the same folder. This command takes two arguments: the project name, and the maximum file number for each run. These fields should be updated in the cron_plugins file before running. <br />
<br />
* The exec.sh command updates the job management database table with any data that has arrived on tape since it was last updated, ignoring file numbers greater than the maximum file number. It then submits jobs for these files. <br />
<br />
* To start the cron job, run:<br />
<pre><br />
crontab cron_plugins<br />
</pre><br />
<br />
* To check whether the cron job is running, do<br />
<pre><br />
crontab -l<br />
</pre><br />
<br />
* To remove the cron job do<br />
<pre><br />
crontab -r<br />
</pre><br />
<br />
<!--<br />
The cron job will run the script scan_for_jobs.sh,<br />
which runs generatejobs_plugins_rawdata.sh for any<br />
new runs that it had not seen before. All previous<br />
runs are recorded in the file filelists/files_current.txt<br />
so clear this to run over runs, or set the parameters<br />
MINRUN and MAXRUN which will set the range of runs submitted.<br />
--><br />
<br />
==Post-Processing Procedures==<br />
<br />
To visualize the monitoring data, we save images of selected histograms and store time series of selected quantities in a database, which are then displayed on a web page. This section describes how to generate the monitoring images and database information.<br />
<br />
The scripts used to generate this summary data are primarily run from /home/gxprojN/halld/monitoring/process . If you want a new copy of the scripts, e.g., for a new monitoring run, you should check the scripts out from SVN:<br />
<syntaxhighlight><br />
svn co https://halldsvn.jlab.org/repos/trunk/scripts/monitoring/process<br />
</syntaxhighlight><br />
Note that these scripts currently have some parameters which must be periodically set by hand.<br />
<br />
The default python version on most JLab machine does not have the modules to allow these scripts to connect to the MySQL database. To run these scripts, load the environment with the following command<br />
<syntaxhighlight><br />
source /home/gxproj1/halld/monitoring/process/monitoring_env.csh<br />
</syntaxhighlight><br />
<br />
===Online Monitoring===<br />
<br />
There are two primary scripts for running over the monitoring data generated by the online system and offline reconstruction. The online script can be run with either of the following commands:<br />
<syntaxhighlight><br />
/home/gluex/halld/monitoring/process/check_new_runs.py<br />
<br />
OR <br />
<br />
/home/gluex/halld/monitoring/process/check_new_runs.csh<br />
</syntaxhighlight><br />
The shell script sets up the environment properly to run the python script. To connect to the monitoring database on the JLab CUE, modules included in the local installation of python >= 2.7 are needed. The shell script is appropriate to use in a cron job. The cronjob is currently run under the "gluex" account.<br />
<br />
The online monitoring system copies a ROOT file containing the results of the online monitoring, and other configuration files into a directory accessible outside the counting house. This python script automatically checks for new ROOT files, which it will then automatically process. It contains several configuration variables that must be correctly set, which contains the location of input/output directories, etc...<br />
<br />
===Offline Monitoring===<br />
<br />
After the data is run over, the results should be processed, so that summary data is entered into the monitoring database and plots are made for the monitoring webpages. Currently, this processing is controlled by a cronjob that runs the following script:<br />
<syntaxhighlight><br />
/home/gxproj1/halld/monitoring/process/check_monitoring_data.csh <br />
</syntaxhighlight><br />
This script checks for new ROOT files, and only runs over those it hasn't processed yet. Since one monitoring ROOT file is produced for each EVIO file, whenever a new file is produced, the plots for the corresponding run are recreated and all the ROOT files for a run are combined into one file. Information is stored in the database on a per-file basis. <br />
<br />
Plots for the monitoring web page can be made from single histograms or multiple histograms using RootSpy macros. If you want to change the list of plots made, you must modify one of the following files:<br />
* histograms_to_monitor - specify either the name of the histogram or its the full ROOT path <br />
* macros_to_monitor - specify the full path to the RootSpy macro .C file<br />
<br />
When a new monitoring run is started, or the conditions are changed, the following steps should be taken to process the new files:<br />
# Add a new data version, as described below:<br />
# Change the following parameters in check_monitoring_data.csh:<br />
## JOBDATE should correspond to the ouptut date used by the job submission script<br />
## OUTPUTDIR should correspond to the directory corresponding to the run period and revision corresponding to the new version you just submitted. Presumably, this directory will be empty at the beginning.<br />
## Once you create a new data version as defined below, you should pass the needed information as a command line option. Currently this is done by the ARGS variable. For example, the argument "-v RunPeriod-2014-10,8" tells the monitoring scripts to look up the version corresponding to revision 8 of RunPeriod-2014-10 in the monitoring DB and to use to store the results.<br />
<br />
<syntaxhighlight><br />
Example configuration parameters:<br />
set JOBDATE=2015-01-09<br />
set INPUTDIR=/volatile/halld/RunPeriod-2014-10/offline_monitoring<br />
set OUTPUTDIR=/w/halld-scifs1a/data_monitoring/RunPeriod-2014-10/ver08<br />
set ARGS=" -v RunPeriod-2014-10,8 "<br />
</syntaxhighlight><br />
If you want to process the results manually, the data is processed using the following script:<br />
<syntaxhighlight><br />
./process_new_offline_data.py <input directory> <output directory><br />
<br />
EXAMPLE:<br />
<br />
./process_new_offline_data.py 2014-11-14 /volatile/halld/RunPeriod-2014-10/offline_monitoring/ /w/halld-scifs1a/data_monitoring/RunPeriod-2014-10/ver02<br />
</syntaxhighlight><br />
The python script takes several options to enable/disable various steps in the processing. Of interest is the "--force" option, which will run over all monitoring ROOT files, whether or not they've been previously identified.<br />
<br />
Every time a new reconstruction pass is performed, a new version number must be generated. To do this, prepare a version file as described below. Then run the register_new_version.py script to store the information in the database. The script will return a version number, which then should be set by hand in process_new_offline_data.py - future versions of the script will streamline this part of the procedure. An example of how to generate a new version is:<br />
<syntaxhighlight><br />
./register_new_version.py add /home/gxproj1/halld/monitoring/process/versions/vers_RunPeriod-2014-10_pass1.txt<br />
</syntaxhighlight><br />
<br />
<b>If you are running the offline monitoring by checking out the files in trunk/scripts/monitoring/jproj/projects/<br />
of the svn repository</b>, and created a project with<br />
create_project.sh [project name] hd_rawdata<br />
Then go to the directory [project name]/processing/<br />
and execute<br />
./run_processing.sh<br />
which will run register_new_version.py as well as check_monitoring_data.csh for that project.<br />
<br />
===Step-by-Step Instructions For Processing a New Monitoring Run===<br />
<br />
The monitoring runs are current run out of the gxproj1 and gxproj5 accounts. After an offline monitoring run has been successfully started on the batch farm, the following steps should be followed to setup the post-processing for these runs.<br />
<br />
# The post-processing scripts are stored in $HOME/halld/monitoring/process and are automatically run by cron.<br />
# Run "svn update" to bring any changes in. Be sure that the list of histograms and macros to plot are current.<br />
# Add a new data version<br />
# Edit check_monitoring_data.csh to point to the current revisions/directories<br />
#* VERSION<br />
#* ARGS<br />
#* Note that the environment depends on a standard script - $HOME/setup_jlab.csh<br />
# Update files in the web directory, so that the results are displayed on the web pages: /group/halld/www/halldweb/html/data_monitoring/textdata<br />
# Copy the REST files to more permanent locations:<br />
#* cp -a /volatile/halld/offline_monitoring/RunPeriod-YYYY-MM/verVV/REST /work/halld/data_monitoring/RunPeriod-YYYY-MM/REST/verVV<br />
#* cp -a /volatile/halld/offline_monitoring/RunPeriod-YYYY-MM/verVV/REST /cache/halld/RunPeriod-YYYY-MM/REST/verVV [under testing]<br />
<br />
Check log files in $HOME/halld/monitoring/process/log for more information on how each run went. If there are problems, check log files, and modify check_monitoring_data.csh to vary the verbosity of the output.<br />
<br />
==Data Versions==<br />
<br />
To document the conditions of the monitoring data that is created, for the sake of reproducability and further analysis we save several pieces of information. The format is intended to be comprehensive enough to document not just monitoring data, but versions of raw and reconstructed data, so that this database table can be used for the event database as well.<br />
<br />
We store one record per pass through one run period, with the following structure:<br />
<br />
{| class="wikitable"<br />
! Field !! Description<br />
|-<br />
| data_type || The level of data we are processing. For the purposes of monitoring, "rawdata" is the online monitoring, "recon" is the offline monitoring<br />
|- <br />
| run_period || The run period of the data<br />
|- <br />
| revision || An integer specifying which pass through the run period this data corresponds to<br />
|- <br />
| software_version || The name of the XML file that specifies the different software versions used<br />
|-<br />
| jana_config || The name of the text file that specifies which JANA options were passed to the reconstruction program<br />
|-<br />
| ccdb_context || The value of JANA_CALIB_CONTEXT, which specifies the version of calibration constants that were used<br />
|-<br />
| production_time || The data at which monitoring/reconstruction began<br />
|-<br />
| dataVersionString || A convenient string for identifying this version of the data<br />
|}<br />
<br />
<br />
An example file used as as input to ./register_new_version.py is:<br />
<syntaxhighlight><br />
data_type = recon<br />
run_period = RunPeriod-2014-10<br />
revision = 1<br />
software_version = soft_comm_2014_11_06.xml<br />
jana_config = jana_rawdata_comm_2014_11_06.conf<br />
ccdb_context = calibtime=2014-11-10<br />
production_time = 2014-11-10<br />
dataVersionString = recon_RunPeriod-2014-10_20141110_ver01<br />
</syntaxhighlight></div>Kmoriyahttps://halldweb.jlab.org/wiki/index.php?title=Data_Monitoring_Procedures&diff=71723Data Monitoring Procedures2015-12-05T02:36:56Z<p>Kmoriya: /* hdswif scripts */</p>
<hr />
<div>__TOC__<br />
<br />
== Master List of File / Database / Webpage Locations ==<br />
=== Run Conditions ===<br />
* Online Run-by-run condition files (B-field, current, etc.): /work/halld/online_monitoring/conditions/<br />
* Offline monitoring run conditions (software versions, jana config): /group/halld/data_monitoring/run_conditions/<br />
*[http://www.jlab.org/Hall-D/test/RunInfo/ Run Info vers. 1]<br />
*[https://halldweb.jlab.org/cgi-bin/data_monitoring/run_conditions.pl Run Info vers. 2]<br />
<br />
=== Monitoring Output Files ===<br />
* Run Periods 201Y-MM is for example 2015-03, launch ver verVV is for example ver15<br />
* Online monitoring histograms: /work/halld/online_monitoring/root/<br />
* Offline monitoring histogram ROOT files: /work/halld/data_monitoring/RunPeriod-201Y-MM/verVV/rootfiles<br />
* REST files (most recent launch only): /work/halld/data_monitoring/RunPeriod-201Y-MM/REST/verVV<br />
* individual files for each job (ROOT, log, etc.): /volatile/halld/offline_monitoring/RunPeriod-201Y-MM/verVV/<br />
<br />
=== Monitoring Database ===<br />
* Accessing monitoring database (on ifarm): mysql -u datmon -h hallddb.jlab.org data_monitoring<br />
<br />
=== Monitoring Webpages ===<br />
*[https://halldweb.jlab.org/cgi-bin/data_monitoring/monitoring/plotBrowser.py Plot Browser]<br />
*[https://halldweb.jlab.org/cgi-bin/data_monitoring/monitoring/runBrowser.py Run Browser]<br />
*[https://halldweb.jlab.org/cgi-bin/data_monitoring/monitoring/timeSeries.py Time Series]<br />
*[https://halldweb.jlab.org/data_monitoring/launch_analysis/ Launch Analysis]<br />
<br />
== Job Monitoring Links ==<br />
* [http://scicomp.jlab.org/farm2/job.html Completed Job History]<br />
* [http://scicomp.jlab.org/farm2/project.html Job Stats By Project]<br />
* [http://scicomp.jlab.org/farm2/trackOrg.html Job Stats By Track]<br />
* [http://scicomp.jlab.org/farm2/report.html Cluster Report]<br />
* [http://scicomp.jlab.org/farm2/walltime.html Walltime Distribution]<br />
<br />
== Saving Online Monitoring Data ==<br />
<br />
The procedure for writing the data out is given in, e.g.,<br />
[https://halldweb.jlab.org/wiki/index.php/Raid-to-Silo_Transfer_Strategy Raid-to-Silo Transfer Strategy].<br />
<br />
Once the DAQ writes out the data to the raid disk, cron jobs will copy the file to tape,<br />
and within ~20 min., we will have access to the file on tape at<br />
/mss/halld/$RUN_PERIOD/rawdata/RunXXXXXX.<br />
<br />
All online monitoring plugins will be run as data is taken.<br />
They will be accessible within the counting house via RootSpy, and<br />
for each run and file, a ROOT file containing the histograms will be saved<br />
within a subdirectory for each run.<br />
<br />
For immediate access to these files, the raid disk files may be accessed directly<br />
from the counting house, or the tape files will be available within ~20 min. of the<br />
file being written out.<br />
<br />
== Offline Monitoring: Running Over Archived Data ==<br />
<br />
Once files are written to tape we run the online plugins on these files to confirm what we were seeing in the online monitoring, and also to update the results from the latest calibration and software. <br />
Manual scripts and cron jobs are set up to look for new data and run the plugins over a sample of files.<br />
<br />
Every other Friday (usually the Friday before the offline meetings) jobs will be started to run the newest software on all previous runs,and allows everybody to see improvements in each detector.<br />
For each launch, independent builds of hdds, sim-recon, the monitoring plugins, and an sqlite file will be generated.<br />
<br />
Below the procedures are described for<br />
# Preparing the software for the launch<br />
# Starting the launch (using hdswif)<br />
# Post-analysis of statistics of the launch<br />
<br />
Processing the results and making them available to the collaboration<br />
is handled in the section [[#Post-Processing_Procedures | Post-Processing Procedures]] below.<br />
<br />
<!--<br />
==== Generating an offline plugin job ====<br />
<br />
The user gxproj1 should be used for official offline monitoring jobs.<br />
Within /home/gxproj1/halld/monitoring/batch/ there will be scripts to run the online monitoring plugins over tape files.<br />
The main script is generatejobs_plugins_rawdata.sh, which can be used as<br />
generatejobs_plugins_rawdata.sh [minrun] [maxrun] (minfile) (maxfile)<br />
where minrun, maxrun specify the range of the run #, and minfile, maxfile (optional) specify<br />
the file range for that run.<br />
<br />
This will generate a script run_rawdata_XXXXXX.sh, where the run # has now been formatted to be 6 digits.<br />
Executing this script will send the monitoring plugins job to the Auger batch system.<br />
<br />
There is also a script clean.sh which can be used as<br />
clean.sh XXX<br />
This will clean up all associated files created in association with run XXX.<br />
<br />
Internally, the xml file used to submit the job will be created, and the job to run<br />
will be given within script.sh. All run parameters should be specified in at the beginning<br />
of generatejobs_plugins_rawdata.sh<br />
Since we are running on tape, the tape file will first be copied over to the cache disk, and the job will run<br />
over this cached file.<br />
<br />
==== Using cron to run automatically ====<br />
Within /home/gluex/halld/monitoring/cron/ there is a file cron_plugins<br />
that can be executed via<br />
crontab cron_plugins<br />
This will set up a cron job to call the script scan_for_jobs.sh, which will<br />
check in the rawdata directory and call generatejobs_plugins_rawdata.sh for<br />
any run that is more than 5 min old. The cron job is set up to run every 10 min.<br />
<br />
==== Magnetic Field Settings ====<br />
For tracking, it is necessary to set the correct magnetic field settings.<br />
The below table is now obsolete. Please refer to the [https://halldweb.jlab.org/cgi-bin/data_monitoring/run_conditions.pl GlueX Run Conditions] webpage.<br />
<br />
The actual field settings for each run have not been documented well.<br />
Below are values based on going through entries in the [https://logbooks.jlab.org/book/halld halld log].<br />
<br />
{| border="1" cellpadding="1" style="text-align: center;"<br />
!width="100"| Run #s<br />
!width="150"| Solenoid Current (A)<br />
!width="300"| JANA option<br />
!width="300"| notes<br />
|-<br />
| 940 - 996 || 1000 || -PBFIELD_MAP=Magnets/Solenoid/solenoid_1000A_poisson_20141104 || At run 997, the solenoid started ramping down<br />
|-<br />
| 998 - 1448 || 0 || -PBFIELD_TYPE=NoField -PDEFTAG:DTrackCandidate=StraightLine || Solenoid started ramping up around run 1431. Runs 1432 - 1448 should not be used for tracking. See [https://logbooks.jlab.org/entry/3309039 this] and [https://logbooks.jlab.org/entry/3309092 this] entry.<br />
|-<br />
| 1449 - 1620 || 1200 || -PBFIELD_MAP=Magnets/Solenoid/solenoid_1200A_poisson_20140520 || <br />
|-<br />
|}<br />
<br />
'''Note:''' The following run ranges were taken with a 300A field, and could be analyzed with the magnetic field map Magnets/Solenoid/solenoid_300A_poisson_20140819 , more study is needed to see if the straight line track fitter is needed or not: 1036 - 1053, 1065 - 1121, 1212? - 1254, 1309 - 1318 [1307,1308,1319-1329 were taken while the magnet was ramping].<br />
--><br />
<br />
=== General Information on Procedures ===<br />
This section explains how the offline monitoring should be run.<br />
Since we may want to simultaneously run offline monitoring for different run periods that require<br />
different environment variables, the scripts are set up so that a generic user can download the<br />
scripts and run them from anywhere. Most output directories for offline monitoring are created<br />
with group read/write permissions so that any Hall D group user has access to the contents,<br />
but there are some cases where use of the account that created the launch is necessary.<br />
<br />
The accounts used for offline monitoring are the gxprojN accounts created and maintained by<br />
Mark Ito (see [https://halldweb.jlab.org/wiki/index.php/GlueX-related_shared_accounts_on_the_JLab_CUE here] for how each account is used).<br />
As of October 2015, the following are used:<br />
* gxproj1 for running over Fall 2014 data (deprecated since June 2015)<br />
* gxproj5 for running over Spring 2015 data<br />
<br />
Since the summer of 2015 we have transitioned from a system using Mark Ito's jproj scripts<br />
to integrating the swif system that Chris Larrieu (SciComp) has been developing. For offline monitoring,<br />
the hdswif system that Kei developed is used for launching the jobs, and the jproj system is used<br />
for meta-analysis of launch statistics.<br />
<br />
Both hdswif and jproj are maintained in svn:<br />
* hdswif: https://halldsvn.jlab.org/repos/trunk/scripts/monitoring/hdswif<br />
* jproj : https://halldsvn.jlab.org/repos/trunk/scripts/monitoring/jproj<br />
<br />
To run the offline monitoring each package should be checked out and all necessary<br />
scripts are included.<br />
<br />
=== Preparing the software for the launch ===<br />
<br />
1. Setup the environment: <pre>source ~/env_monitoring_launch</pre><br />
<br />
2. Updating & building hdds: <br />
<pre><br />
cd $HDDS_HOME<br />
git pull # Get latest software<br />
scons -c install # Clean out the old install: EXTREMELY IMPORTANT for cleaning out stale headers<br />
scons install -j4 # Rebuild and re-install with 4 threads<br />
</pre><br />
<br />
3. Updating & building sim-recon: <br />
<pre><br />
cd $HALLD_HOME/src<br />
git pull # Get latest software<br />
scons -c install # Clean out the old install: EXTREMELY IMPORTANT for cleaning out stale headers<br />
scons install -j4 # Rebuild and re-install with 4 threads<br />
</pre><br />
<br />
4. Create a new sqlite file containing the very latest calibration constants. Original documentation on creating sqlite files are [https://halldweb.jlab.org/wiki/index.php/SQLite-form_of_the_CCDB_database here]. <br />
<pre><br />
cd $GLUEX_MYTOP/../sqlite/<br />
$CCDB_HOME/scripts/mysql2sqlite/mysql2sqlite.sh -hhallddb.jlab.org -uccdb_user ccdb | sqlite3 ccdb.sqlite<br />
mv ccdb.sqlite ccdb_monitoring_launch.sqlite #replacing the old file<br />
</pre><br />
<br />
5. Note that the above steps must be done <b>BEFORE</b> launch project creation. This is because we will track the revisions of the libraries used, and this is done by extracting the svn information in each directory. Also, note that the system assumes that we have the topmost build directory (usually called GLUEX_TOP) to be $HOME/builds . Such an assumption is necessary to be able to extract information about the library locations automatically.<br />
<br />
Create the appropriate project(s) and submit the jobs using hdswif, as detailed in the section below.<br />
<br />
=== Starting the Launch and Submitting Jobs ===<br />
<br />
Until the summer of 2015 we relied solely on Mark Ito's jproj system for submitting and keeping track of jobs. We have since moved to the swif system and use the hdswif wrapper for this. Below are instructions for how to use these.<br />
<br />
1. Downloading hdswif: Download the hdswif directory from svn. For the gxprojN accounts, use the directory ~/halld/hdswif. <br />
<pre><br />
cd ~/halld<br />
svn co https://halldsvn.jlab.org/repos/trunk/scripts/monitoring/hdswif<br />
cd hdswif<br />
</pre><br />
<br />
2. Edit the job config file, input.config, which is used to register jobs in hdswif. A typical config file will look this:<br />
PROJECT gluex<br />
TRACK reconstruction<br />
OS centos65<br />
NCORES 6<br />
DISK 40<br />
RAM 8<br />
TIMELIMIT 8<br />
JOBNAMEBASE offmon_<br />
RUNPERIOD 2015-03<br />
VERSION 15<br />
OUTPUT_TOPDIR /volatile/halld/offline_monitoring/RunPeriod-[RUNPERIOD]/ver[VERSION] # Example of other variables included in variable<br />
SCRIPTFILE /home/gxproj5/halld/hdswif/script.sh # Must specify full path<br />
ENVFILE /home/gxproj5/halld/hdswif/setup_jlab-2015-03.csh # Must specify full path<br />
<br />
3. Creating the workflow: Within SWIF jobs are registered into workflows. For offline monitoring, the workflow names are of the form <b>offline_monitoring_RunPeriod201Y_MM_verVV_hd_rawdata</b> with suitable replacements for the run period and version number. The command "swif list" will list all existing workflows. Also, for most simple SWIF commands hdswif also provides a wrapper. <br />
swif list<br />
<br />
For creation of workflows for offline monitoring the command:<br />
hdswif.py create [workflow] -c input.config<br />
should be used. When a config file (here: input.config) is passed in, hdswif will automatically create files that record the configuration of the current launch. These files are stored as for example:<br />
/group/halld/data_monitoring/run_conditions/jana_rawdata_comm_2015_03_ver15.conf<br />
/group/halld/data_monitoring/run_conditions/soft_comm_2015_03_ver15.xml<br />
<br />
The software packages stored in git (sim-recon and hdds) can have git tags applied to them, which makes it easier to find versions of the software than a SHA-1 hash. hdswif will ask if you would like to create a tag, and execute the following sequence: <pre>git tag -a offmon-201Y_MM-verVV -m "Used for offline monitoring 201Y-MM verVV started on 201y/mm/dd"</pre> <pre>git push offmon-201Y_MM-verVV</pre> This will only be invoked when the user name is gxprojN, and for the configuration files, the output directory will be /group/halld/data_monitoring/run_conditions/ for gxprojN accounts while it will be the current directory for other users.<br />
* To use the git-tagged software versions do for example <pre>cd $HALLD_HOME</pre> <pre>git checkout offmon-2015_03-ver15</pre><br />
<br />
4. Registering jobs in the workflow: To register jobs within the workflow, hdswif provides the use of config files. Jobs can be registered by specifying the workflow, config file (-c), run (-r) and file (-f) numbers if necessary. <br />
<span style="color:red">Note: Job configuration parameters can be set differently for jobs within the same workflow if necessary.</span><br />
Jobs can be added via <pre>hdswif.py add [workflow] -c input.config</pre><br />
By default, hdswif will add all files found within the directory /mss/halld/RunPeriod-201Y-MM/rawdata/ where 201Y-MM is specified by the RUNPERIOD parameter in the config file. If only some of the runs or files are needed, these can be specified for example with<br />
<pre>hdswif.py add [workflow] -c input.config -r 3180 -f '00[0-4]'</pre><br />
to specify to register running only over run 3180 files 000 - 004 (Unix-style brackets and wildcards can be used).<br />
<br />
5. Running the workflow: To run the workflow, simply use the hdswif wrapper:<br />
hdswif.py run [workflow]<br />
<br />
<b>It is recommended that some jobs be tested to make sure that everything is working rather than fail thousands of jobs.</b><br><b>Check the configuration setup when creating a workflow, and what is in script.sh. Also check the following:<br />
* Check stderr files. Are they small (<kB)?<br />
* Check stdout files. Are they very large (>MB)?<br />
* Check output ROOT files. Are they larger than several MB?<br />
* Check output REST files. Are they larger than several tens of MB?<br />
</b><br />
<br> For this purpose, hdswif will take an additional parameter to run which limits the number of jobs to submit: <pre>hdswif.py run [workflow] 10</pre><br />
in which case only 10 jobs will be submitted.<br />
<br />
To submit all jobs after checking the results, do <pre>hdswif.py run [workflow]</pre><br />
<br />
=== Checking the Status and Resubmitting ===<br />
1. The status of jobs can be checked on the terminal with <pre>jobstat -u gxprojN</pre> For the status of jobs on Auger see http://scicomp.jlab.org/scicomp/#/auger/jobs and for SWIF use <pre>swif list</pre> or for more information, <pre>swif status [workflow] -summary</pre> Note that "swif status" tends to be out of date sometimes, so don't panic if your workflow/jobs aren't showing up right away. Also see the Auger [http://scicomp.jlab.org/scicomp/#/auger/jobs job website].<br />
<br />
2. For failed jobs, SWIF can resubmit jobs based on the problem. For resubmission for failed jobs with the same resources, <pre>swif retry-jobs [workflow] -problems [problem name]</pre> can be used, and for jobs to be submitted with more resources, e.g., use <pre>swif modify-jobs [workflow] -ram add 2gb -problems AUGER-OVER_RLIMIT</pre> <br />
This only re-stages the jobs, be sure to resubmit them with:<br />
<pre>swif run -workflow [workflow] -errorlimit none</pre><br />
<br />
hdswif has a wrapper for both of these: <pre>hdswif.py resubmit [workflow] [problem]</pre> In this case [problem] can be one of <b>SYSTEM, TIMEOUT, RLIMIT</b>. If SYSTEM is specified, the jobs will be retried. For TIMEOUT and RLIMIT, the jobs will be modified by default with 2 additional hours or GB of RAM. If one more number is added as an option, then that many hours or GB of RAM will be added., e.g., <pre>hdswif.py resubmit [workflow] TIMEOUT 5</pre> will add 5 hours of processing time. You can wait until almost all jobs finish before resubmitting failed jobs since the number should be relatively small. <b>Even if jobs are resubmitted for one type of failure, jobs that later fail with that failure will not be automatically resubmitted.</b><br />
<br />
3. For information on swif, use the "swif help" commands and for hdswif see the attached documentaion in https://halldsvn.jlab.org/repos/trunk/scripts/monitoring/hdswif/manual_hdswif.pdf<br />
<br />
4. Below is a table describing the various errors that can occur.<br />
{| border="1" cellpadding="0" valign="left" style="text-align: left;"<br />
!width="150"| ERROR NAME<br />
!width="400"| Description<br />
!width="400"| Resolution<br />
!width="400"| hdswif command<br />
|-<br />
| AUGER-SUBMIT<br />
||<br />
SWIF’s attempt to submit jobs to Auger failed. Includes server-side problems as well as user failing to provide valid job parameters (e.g. incorrect project name, too many resources, etc.)<br />
||<br />
If requested resources are known to be correct resubmit. Otherwise modify job resources using swif directly.<br />
||<br />
<span style="color:purple"><b>hdswif.py resubmit [workflow] SYSTEM</b></span><br />
|-<br />
| AUGER-FAILED<br />
||<br />
Auger reports the job FAILED with no specific details.<br />
||<br />
Resubmit jobs. If problem persists, contact Chris Larrieu or SciComp.<br />
||<br />
<span style="color:purple"><b>hdswif.py resubmit [workflow] SYSTEM</b></span><br />
|-<br />
| AUGER-OUTPUT-FAIL<br />
||<br />
Failure to copy one or more output files.Can be due to permission problem, quota problem, system error, etc.<br />
||<br />
Check if output files will exist after job execution and that output directory exists, resubmit jobs. If problem persists, contact Chris Larrieu or SciComp.<br />
||<br />
<span style="color:purple"><b>hdswif.py resubmit [workflow] SYSTEM</b></span><br />
|-<br />
| AUGER-INPUT-FAIL<br />
||<br />
Auger failed to copy one or more of the requested input files, similar to output failures. Can also happen if tape file is unavailable (e.g. missing/damaged tape)<br />
||<br />
Check if input file exists, resubmit jobs. If problem persists, contact Chris Larrieu or SciComp.<br />
||<br />
<span style="color:purple"><b>hdswif.py resubmit [workflow] SYSTEM</b></span><br />
|-<br />
| AUGER-TIMEOUT<br />
||<br />
Job timed out.<br />
||<br />
If more time is needed for job add more resources.<br />
Default is to add 2 hrs of processing time. Also check whether code is hanging.<br />
||<br />
<span style="color:red"><b>hdswif.py resubmit [workflow] TIMEOUT</b></span><br><br />
Default is to add 2 hours. Optionally specify number of hours at end.<br />
|-<br />
| AUGER-OVER_RLIMIT<br />
||<br />
Not enough resources, RAM or disk space.<br />
||<br />
Add more resources for job.<br />
||<br />
<span style="color:blue"><b>hdswif.py resubmit [workflow] RLIMIT</b></span><br><br />
Default is to add 2 GB of RAM. Optionally specify GB at end. To add more disk space use SWIF directly.<br />
|-<br />
| SWIF-MISSING-OUTPUT<br />
||<br />
Output file specified by user was not found.<br />
||<br />
Check if output file exists at end of job.<br />
||<br />
<br />
|-<br />
| SWIF-USER-NON-ZERO <br />
||<br />
User script exited with non-zero status code.<br />
||<br />
Your script exited with non-zero status. Check the code you are running.<br />
||<br />
<br />
|-<br />
| SWIF-SYSTEM-ERROR <br />
||<br />
Job failed owing to a problem with swif (e.g. network connection timeout)<br />
||<br />
Resubmit jobs. If problem persists, contact Chris Larrieu or SciComp.<br />
||<br />
<span style="color:purple"><b>hdswif.py resubmit [workflow] SYSTEM</b></span><br />
|}<br />
<br style="clear:both;"/><br />
<br />
=== Post-analysis of statistics of the launch ===<br />
<br />
After jobs have been submitted, it will usually take a few days for all of the jobs to be processed.<br />
The next step is to check the resource usage for the current launch and publish the results online.<br />
<br />
# <b>Create summary XML, HTML files</b><br> The status and results of jobs are saved within the SWIF internal server, and are available via the command <pre>swif status [workflow] -summary -runs</pre> where the arguments -summary and -runs show summary statistics and statistics for individual jobs, respectively. hdswif has a command that takes this output in XML output and creates an HTML webpage showing results of the launch. To do this, do <pre>hdswif.py summary [workflow]</pre> This will create the XML output file from SWIF called swif_output_[workflow].xml and create a webpage containing png figure files. If the XML file already exists, hdswif will ask whether to overwrite the existing file.<br />
# <b>Publish output files online</b><br> At this stage the html output and figure files are created and ready to be put online. The html output capabilities of hdswif is useful for any user using SWIF, but since publication of the html output is specific to the offline monitoring, the script to do this is contained in the jproj scripts directory at https://halldsvn.jlab.org/repos/trunk/scripts/monitoring/jproj. For the gxprojN accounts this directory should exist as ~/halld/jproj. To publish the results online do for example <pre>python ~/halld/jproj/projects/templates/publish_offmon_results.py 2015_03 18</pre> The script simply copies the html output and corresponding figures to /group/halld/www/halldweb/html/data_monitoring/launch_analysis/<br />
# <b>Editing the summary HTML page</b><br> The top page for is offline monitoring https://halldweb.jlab.org/data_monitoring/launch_analysis/index.html and has links to the summary page for each run period. The summary files are /group/halld/www/halldweb/html/data_monitoring/launch_analysis/[run period].html Edit the file to<br />
## Add a new line to the first table which contains the version number, date, and comments for the current launch<br />
## Create a link to the webpage for the current launch. Simply copy and paste, modify the previous launch's link to have the correct launch ver.<br />
# <b>Freezing SWIF tables</b><br> Since we are now finished with the SWIF workflow that we used, the workflow should be "frozen" so that it cannot be mistakenly altered or modified. Do <pre>swif freeze [workflow]</pre><br />
# <b>Backing up SWIF output</b><br> With the workflow frozen we should be able to reproduce the XML output from SWIF, but we will backup the XML output just in case. Do <pre>cp ~/halld/hdswif/swif_output_[workflow].xml /group/halld/data_monitoring/swif_xml_backup/ </pre><br />
<br />
=== Cross Analysis of Launches ===<br />
<br />
The purpose of the cross analysis is to correlate how resource usage changed for the same files across different launches.<br />
To do this it is useful to create MySQL tables that contain information on each launch, and do queries across tables.<br />
<br />
#The scripts to do this are maintained in the svn directory https://halldsvn.jlab.org/repos/trunk/scripts/monitoring/jproj Do <pre> svn co https://halldsvn.jlab.org/repos/trunk/scripts/monitoring/jproj </pre> For the gxprojN accounts used for offline monitoring the directory should be ~/halld/jproj<br />
# The jproj directory contains two subdirectories, scripts and projects. The scripts directory contains useful scripts for processing the jobs registered in the jproj system, and each of the offline monitoring launches will be handled in the projects directory.<br />
# Go to the projects directory<pre>cd ~/halld/jproj/projects</pre> and use the script create_project.sh to create a new directory that contains the processing scripts for the current launch <pre>./create_project.sh [workflow]</pre> This should create a directory such as offline_monitoring_RunPeriod2015_03_ver15_hd_rawdata (same as the workflow name). The script uses the template files in the directory templates and by substitution creates script files for the current launch. Now go to the newly created analysis directory: <pre>cd [workflow]/analysis</pre><br />
which for the gxprojN accounts should have the full path /home/gxprojN/halld/jproj/projects/[workflow]/analysis.<br><br><br />
<span style="color:red">All of the analysis commands including arguments are contained in run_analysis.sh, but it is <b>strongly</b> recommended that all commands are run manually to check for errors.</span><br><br><br />
# Next, we need to create a few MySQL tables for the current launch. The MySQL tables are useful for comparing run/file combinations for different launches. For the SWIF launches, there are two tables needed, and will be named<br />
#* [workflow]Job<br />
#* [workflow]_aux<br />
# This naming scheme of tables and their roles are the same as from the jproj only launches. The [workflow]Job table will contain information gathered from SWIF about each job (which node it went to, start time of each stage, memory usage, etc.). The [workflow]_aux table will contain information gathered from the output stdout files from each job. First, create the Job table using <pre>python create_jproj_job_table.py [run period] [version]</pre> Note that this script uses the XML output from hdswif summary and inserts the contents into the MySQL table, so the XML output file must exist.<br />
# To check the contents of this MySQL table, do <pre>mysql -hhalldb -ufarmer farming</pre> <pre>mysql> describe offline_monitoring_RunPeriod2015_03_ver11_hd_rawdataJob;</pre><br />
# <b>Backing up tables</b><br> Tables created in MySQL should be backed up.<br />
<br />
== Explanation of Scripts ==<br />
Below are explanations of each script used in the offline monitoring system and a breif explanation of how they work.<br />
<br />
=== hdswif scripts ===<br />
<b>Summary:</b> hdswif.py is the main script that calls the other utility scripts. For the utility scripts, they can be run standalone by giving the appropriate parameters. Visual graphics are made using the PyROOT extension of ROOT. To use this, the environment variable PYTHONPATH must include ROOTSYS. If ROOTSYS has been set, adding it to PYTHONPATH will be done by the script, but if ROOTSYS is not set, then the scripts will abort.<br />
<br />
{| border="1" cellpadding="0" valign="left" style="text-align: left;"<br />
!width="150"| file name<br />
!width="800"| Description<br />
|-<br />
| hdswif.py ||<br />
Main script to control the behavior of SWIF. Most commands follow the form hdswif.py [command] [workflow] (options)<br />
|-<br />
| parse_swif.py ||<br />
Called within hdswif.py to create html output from SWIF results.<br />
|-<br />
| createXMLfiles.py ||<br />
* Called by the "create" option of hdswif.py<br />
* Creates XML files for logging information about launch. Must specify config file with option -c.<br />
* Also adds tags to git repositories of sim-recon and hdds.<br />
* For XML file creation, the file will be written out to /group/halld/data_monitoring/run_conditions/ if the user is gxprojN (used for offline monitoring). If not, to avoid general users adding things to the above directory, the output files will be the current directory.<br />
* To write out the versions of each package, environment variables such as HDDS_HOME will need to be set.<br />
* Versions for each software package are extracted using the directory structure, so if these are changed the scripts must change accordingly.<br />
* For each launch, <b>the output soft_comm_[RUNPERIOD]_ver[VER].xml file should be checked that all version numbers have been extracted.</b><br />
|-<br />
| read_config.py ||<br />
* Called by the "add" option of hdswif.py<br />
* Takes in config file name, optionally set verbose<br />
* Return a dictionary between config parameter names and values (e.g., 'PROJECT' : 'gluex', 'NCORES' : 6, ...)<br />
* Prints the config parameters at the end. For parameters changed from default, a '*' will be printed<br />
|-<br />
| output_job_details.py ||<br />
* Called by the "details" option of hdswif.py<br />
* Takes in workflow name, run and file. Run and file must be numbers, no wildcards<br />
* Finds ids for jobs specified by the run and file number and returns info on each one<br />
* Job info is retrieved from pbs farm system and shows configuration parameters for that job<br />
|-<br />
| results_by_resources.py ||<br />
* Called within parse_swif.py<br />
* Takes in XML output from SWIF and creates 2 plots<br />
* Creates html table showing results of jobs by resources requested for that job<br />
* This table is shown under "Status by Resources" in the output html file of hdswif.py summary [workflow]<br />
|-<br />
| create_ordered_hists.py ||<br />
* Called within parse_swif.py<br />
* Takes in XML output from SWIF and creates 2 plots<br />
* Plots are dependency time and pending time of each job, ordered by submission.<br />
* Different colors represent jobs submitted at different times. For jobs submitted at the same time, jobs are ordered in increasing time.<br />
|-<br />
| create_stacked_times.py ||<br />
* Called within parse_swif.py<br />
* Takes in XML output from SWIF and creates 2 plots<br />
* Plots show total job time divided into colors for different stages<br />
* One shows all jobs in order of Auger ID (roughly submission order), the other one shows in order of total job time<br />
* The jobs show at a glance which stage contributes how much of the job's time<br />
|}<br />
<br />
== Hall D Job Management System ==<br />
<br />
This section details instructions on how to create and launch a set of jobs using the Hall-D Job Management System developed by Mark Ito. These instructions are generic: this system can be used for the weekly monitoring jobs, but can also be used for other sets of job launches as well. <br />
<br />
=== Database Table Overview ===<br />
<br />
* Job management database table (<project_name>): For each input file, keeps track of whether or not a job for it has been submitted, along with other optional fields. <br />
<br />
* Job status database table (<project_name>Job (no space)): For each job, keeps track of the job-id, the job status, memory used, cpu & wall time, time taken to complete various stages (e.g. pending, dependency, active), and others. <br />
<br />
* Job metrics database table (<project_name>_aux (no space)): For each job, keeps track of the job-id, how many events were processed, the time it took to copy the cache file, and the time it took to run the plugin. This information is culled from the log files of each job, and is done within the analysis directory of each launch.<br />
<br />
=== Initialize Project Management ===<br />
<br />
* Log into the ifarm machine with one of the gxproj accounts. For this example we will use gxproj1.<br />
<pre><br />
ssh gxproj1@ifarm -Y<br />
</pre><br />
<br />
* Go to a directory to do the launch. In principle, any directory will work, but for gxproj1 this is usually done in /home/gxproj1/halld/<br />
<br />
* Check out the necessary scripts <pre>svn co https://halldsvn.jlab.org/repos/trunk/scripts/monitoring/jproj</pre> This will get all necessary scripts for launching. Once checked out, <pre>cd projects</pre><br />
<br />
* The script create_project.sh can be used to create a new project. It will take a single argument, the project name. It is assumed that the project name is of the form offline_monitoring_RunPeriod20YY_MM_verVV_hd_rawdata where this string will be parsed to give the run period 20YY_MM and version number VV. One thing to do BEFORE creation of a new project is to editthe conditions of the launch (plugins to run over, memory requested, disk space requested) within templates/template.jsub . This<br />
information is saved automatically at project creation time into files in /group/halld/data_monitoring/run_conditions .<br />
To create a project do <pre>./create_project.sh offline_monitoring_RunPeriod20YY_MM_verVV_hd_rawdata</pre><br />
The name has been chosen to be as consistent as possible with other directory structures.<br />
However, mysql requires that "-" be escaped in table names, so unfortunately run periods will be<br />
given as 2YYY_MM instead of 2YYY-MM.<br />
<br />
* A new directory with the project name (offline_monitoring_RunPeriod20YY_MM_verVV_hd_rawdata) and files will be copied and modified from the template directory to reflect the run period, the user, the directory that it was created in, project name, etc.<br />
<br />
* For each project, cd into the new directory<br />
* The script clear.sh will remove any existing tables for the current project name, then recreate it. Do<br />
<pre>./clear.sh</pre><br />
* To use the jproj.pl script that was checked in, add the directory to your path with<br />
<pre>source ../../scripts/setup.csh</pre><br />
or always specify the full path<br />
<pre>../../scripts/jproj.pl</pre><br />
* Now update the table of runs with<br />
<pre>jproj.pl <project name> update</pre><br />
This will fill the table with all files within /mss that are of the same form as what is in <project name>.jproj . If you want to register only on a subset of all such files, you can edit this file directly.<br />
* Once you have registered all of the files you would like to run over, do<br />
<pre>jproj.pl <project name> submit [max # of jobs] [run number]</pre><br />
where the additional options specify how many jobs to submit and which run number to run on. Without these options all files that are registered and have not been submitted yet will be submitted.<br />
<br />
At this stage you are ready to submit all files. It is a good idea to submit a few test jobs<br />
at first to check that all scripts are working and that the plugins do not crash. Once you are<br />
sure that this does not happen, you can send all jobs in. The remaining jobs are then the monitoring<br />
which will (among other things) put the results on the online webpage for the collaboration to view, and<br />
the analysis of the launch.<br />
<br />
=== Project File Overview ===<br />
<br />
An overview of each project file:<br />
* '''clear.sh:''' For the current project, deletes the job status and management database tables (if any), and creates new, empty ones. <br />
* '''<project_name>.jproj:''' Contains the path and file name format for the input files for the jobs. <br />
* '''<project_name>.jsub:''' The xml job submission script. The run number and file number variables are set during job submission for each input file. <br />
* '''script.sh:''' The script that is executed during the job. If output job directories are not pre-created manually, they should be created in this script with the proper permissions:<br />
<pre><br />
mkdir -p -m 775 my_directory<br />
</pre><br />
* '''setup_jlab-[run period].csh:''' The environment that is sourced at the beginning of the job execution. <br />
* '''status.sh:''' Updates the job status database table, and prints some of its columns to screen.<br />
<br />
=== Project Management ===<br />
<br />
* Delete (if any) and create the database table(s) for the current set of job submissions:<br />
<pre><br />
./clear.sh<br />
</pre><br />
Also, if testing was done with jobs, it is best to delete the output directory and the configuration files:<br />
<pre><br />
rm -frv /volatile/halld/offline_monitoring/RunPeriod-20YY-MM/verVV /group/halld/data_monitoring/run_conditions/soft_comm_20YY_MM_verVV.xml /group/halld/data_monitoring/run_conditions/jana_rawdata_comm_20YY_MM_verVV.conf<br />
</pre><br />
<br />
* Search for input files matching the string in the .jproj file, and create a row for each in the job management database table (called <project_name>). You can test by adding an optional argument at the end, which only selects files with a specific file number:<br />
<pre><br />
jproj.pl <project_name> update <optional_file_number><br />
</pre><br />
<br />
* Confirm that the job management database is accurate by printing it's contents to screen:<br />
<pre><br />
mysql -hhallddb -ufarmer farming -e "select * from <project_name>"<br />
</pre><br />
<br />
* ONLY if a mistake was made, to delete the tables from the database and recreate new, empty ones, run: <br />
<pre><br />
./clear.sh<br />
</pre><br />
<br />
* To look at the status of the submitted jobs, first query auger and update the job status database:<br />
<pre><br />
fill_in_job_details.pl <project_name><br />
</pre><br />
<br />
* The job status can then be viewed by submitting a query to the job status database (called <project_name>Job (no space in between)):<br />
<pre><br />
mysql -hhallddb -ufarmer farming -e "select id,run,file,jobId,hostname,status,timeSubmitted,timeActive,walltime,cput,timeComplete,result,error from <project_name>Job"<br />
</pre><br />
<br />
* These last two commands can instead be executed simultaneously by running:<br />
<pre><br />
./status.sh<br />
</pre><br />
<br />
=== Handy mysql Instructions ===<br />
<br />
* Handy mysql instructions:<br />
<pre><br />
mysql -hhallddb -ufarmer farming # Enter the "farming" mysql database on "hallddb" as user "farmer"<br />
quit; # Exit mysql<br />
show tables; # Show a list of the tables in the current database<br />
show columns from <project_name>; # show all of the columns for the given table<br />
select * from <project_name>; # show the contents of all rows from the given table<br />
</pre><br />
<br />
=== Backing Up Offline Monitoring Tables ===<br />
Tables created for offline monitoring can be backed up using the script backup_tables.sh which<br />
can be checked out with the other files from https://halldsvn.jlab.org/repos/trunk/scripts/monitoring/jproj/projects<br />
<br />
The script uses the command mysqldump to print out a file that can be executed to recreate the tables.<br />
Since executing this output file will drop the table when it exists, caution is advised.<br />
Example usage to backup all three tables created for run period 2014_10 ver 17:<br />
<pre>backup_tables.sh 2014_10 17</pre><br />
<br />
== Running Over Data As It Comes In ==<br />
<br />
A special user gxproj1 will have a cron job set up to run the plugins as new data appears on /mss.<br />
During the week, gxproj1 will submit offline plugin jobs with the same setup as the weekly jobs<br />
run the previous Friday. The procedure for this is shown below.<br />
<br />
<!--<br />
=== Setting up the environment ===<br />
The file<br />
/home/gxproj1/setup_jlab.csh<br />
is sourced through .tcshrc.<br />
This file is the same as what is linked to by<br />
/home/gluex/setup_jlab_commissioning.csh,<br />
except HALLD_HOME, HDDS_HOME, and JANA_CALIB_URL are set separately so that this<br />
user can have a separate build.<br />
<br />
To obtain the builds from the previous Friday's runs,<br />
execute<br />
/home/gxproj1/halld/monitoring/newruns/setup_previous.sh [year] [month] [day]<br />
The build revisions from the previous Friday are archived in files<br />
/work/halld/data_monitoring/run_conditions/soft_comm_[year]_[month]_[day].xml<br />
and the script will build libraries based on those stored revision numbers.<br />
--><br />
<br />
=== Running the cron job ===<br />
<br />
'''IMPORTANT:''' The cron job should not be running while you are manually submitting jobs using the jproj.pl script for the same project, or else you will probably multiply-submit a job. <br />
<br />
* Go to the cron job directory: <br />
<pre><br />
cd /u/home/gxproj1/halld/monitoring/newruns<br />
</pre><br />
<br />
* The cron_plugins file is the cronjob that will be executed. During execution, it runs the exec.sh command in the same folder. This command takes two arguments: the project name, and the maximum file number for each run. These fields should be updated in the cron_plugins file before running. <br />
<br />
* The exec.sh command updates the job management database table with any data that has arrived on tape since it was last updated, ignoring file numbers greater than the maximum file number. It then submits jobs for these files. <br />
<br />
* To start the cron job, run:<br />
<pre><br />
crontab cron_plugins<br />
</pre><br />
<br />
* To check whether the cron job is running, do<br />
<pre><br />
crontab -l<br />
</pre><br />
<br />
* To remove the cron job do<br />
<pre><br />
crontab -r<br />
</pre><br />
<br />
<!--<br />
The cron job will run the script scan_for_jobs.sh,<br />
which runs generatejobs_plugins_rawdata.sh for any<br />
new runs that it had not seen before. All previous<br />
runs are recorded in the file filelists/files_current.txt<br />
so clear this to run over runs, or set the parameters<br />
MINRUN and MAXRUN which will set the range of runs submitted.<br />
--><br />
<br />
==Post-Processing Procedures==<br />
<br />
To visualize the monitoring data, we save images of selected histograms and store time series of selected quantities in a database, which are then displayed on a web page. This section describes how to generate the monitoring images and database information.<br />
<br />
The scripts used to generate this summary data are primarily run from /home/gxprojN/halld/monitoring/process . If you want a new copy of the scripts, e.g., for a new monitoring run, you should check the scripts out from SVN:<br />
<syntaxhighlight><br />
svn co https://halldsvn.jlab.org/repos/trunk/scripts/monitoring/process<br />
</syntaxhighlight><br />
Note that these scripts currently have some parameters which must be periodically set by hand.<br />
<br />
The default python version on most JLab machine does not have the modules to allow these scripts to connect to the MySQL database. To run these scripts, load the environment with the following command<br />
<syntaxhighlight><br />
source /home/gxproj1/halld/monitoring/process/monitoring_env.csh<br />
</syntaxhighlight><br />
<br />
===Online Monitoring===<br />
<br />
There are two primary scripts for running over the monitoring data generated by the online system and offline reconstruction. The online script can be run with either of the following commands:<br />
<syntaxhighlight><br />
/home/gluex/halld/monitoring/process/check_new_runs.py<br />
<br />
OR <br />
<br />
/home/gluex/halld/monitoring/process/check_new_runs.csh<br />
</syntaxhighlight><br />
The shell script sets up the environment properly to run the python script. To connect to the monitoring database on the JLab CUE, modules included in the local installation of python >= 2.7 are needed. The shell script is appropriate to use in a cron job. The cronjob is currently run under the "gluex" account.<br />
<br />
The online monitoring system copies a ROOT file containing the results of the online monitoring, and other configuration files into a directory accessible outside the counting house. This python script automatically checks for new ROOT files, which it will then automatically process. It contains several configuration variables that must be correctly set, which contains the location of input/output directories, etc...<br />
<br />
===Offline Monitoring===<br />
<br />
After the data is run over, the results should be processed, so that summary data is entered into the monitoring database and plots are made for the monitoring webpages. Currently, this processing is controlled by a cronjob that runs the following script:<br />
<syntaxhighlight><br />
/home/gxproj1/halld/monitoring/process/check_monitoring_data.csh <br />
</syntaxhighlight><br />
This script checks for new ROOT files, and only runs over those it hasn't processed yet. Since one monitoring ROOT file is produced for each EVIO file, whenever a new file is produced, the plots for the corresponding run are recreated and all the ROOT files for a run are combined into one file. Information is stored in the database on a per-file basis. <br />
<br />
Plots for the monitoring web page can be made from single histograms or multiple histograms using RootSpy macros. If you want to change the list of plots made, you must modify one of the following files:<br />
* histograms_to_monitor - specify either the name of the histogram or its the full ROOT path <br />
* macros_to_monitor - specify the full path to the RootSpy macro .C file<br />
<br />
When a new monitoring run is started, or the conditions are changed, the following steps should be taken to process the new files:<br />
# Add a new data version, as described below:<br />
# Change the following parameters in check_monitoring_data.csh:<br />
## JOBDATE should correspond to the ouptut date used by the job submission script<br />
## OUTPUTDIR should correspond to the directory corresponding to the run period and revision corresponding to the new version you just submitted. Presumably, this directory will be empty at the beginning.<br />
## Once you create a new data version as defined below, you should pass the needed information as a command line option. Currently this is done by the ARGS variable. For example, the argument "-v RunPeriod-2014-10,8" tells the monitoring scripts to look up the version corresponding to revision 8 of RunPeriod-2014-10 in the monitoring DB and to use to store the results.<br />
<br />
<syntaxhighlight><br />
Example configuration parameters:<br />
set JOBDATE=2015-01-09<br />
set INPUTDIR=/volatile/halld/RunPeriod-2014-10/offline_monitoring<br />
set OUTPUTDIR=/w/halld-scifs1a/data_monitoring/RunPeriod-2014-10/ver08<br />
set ARGS=" -v RunPeriod-2014-10,8 "<br />
</syntaxhighlight><br />
If you want to process the results manually, the data is processed using the following script:<br />
<syntaxhighlight><br />
./process_new_offline_data.py <input directory> <output directory><br />
<br />
EXAMPLE:<br />
<br />
./process_new_offline_data.py 2014-11-14 /volatile/halld/RunPeriod-2014-10/offline_monitoring/ /w/halld-scifs1a/data_monitoring/RunPeriod-2014-10/ver02<br />
</syntaxhighlight><br />
The python script takes several options to enable/disable various steps in the processing. Of interest is the "--force" option, which will run over all monitoring ROOT files, whether or not they've been previously identified.<br />
<br />
Every time a new reconstruction pass is performed, a new version number must be generated. To do this, prepare a version file as described below. Then run the register_new_version.py script to store the information in the database. The script will return a version number, which then should be set by hand in process_new_offline_data.py - future versions of the script will streamline this part of the procedure. An example of how to generate a new version is:<br />
<syntaxhighlight><br />
./register_new_version.py add /home/gxproj1/halld/monitoring/process/versions/vers_RunPeriod-2014-10_pass1.txt<br />
</syntaxhighlight><br />
<br />
<b>If you are running the offline monitoring by checking out the files in trunk/scripts/monitoring/jproj/projects/<br />
of the svn repository</b>, and created a project with<br />
create_project.sh [project name] hd_rawdata<br />
Then go to the directory [project name]/processing/<br />
and execute<br />
./run_processing.sh<br />
which will run register_new_version.py as well as check_monitoring_data.csh for that project.<br />
<br />
===Step-by-Step Instructions For Processing a New Monitoring Run===<br />
<br />
The monitoring runs are current run out of the gxproj1 and gxproj5 accounts. After an offline monitoring run has been successfully started on the batch farm, the following steps should be followed to setup the post-processing for these runs.<br />
<br />
# The post-processing scripts are stored in $HOME/halld/monitoring/process and are automatically run by cron.<br />
# Run "svn update" to bring any changes in. Be sure that the list of histograms and macros to plot are current.<br />
# Add a new data version<br />
# Edit check_monitoring_data.csh to point to the current revisions/directories<br />
#* VERSION<br />
#* ARGS<br />
#* Note that the environment depends on a standard script - $HOME/setup_jlab.csh<br />
# Update files in the web directory, so that the results are displayed on the web pages: /group/halld/www/halldweb/html/data_monitoring/textdata<br />
# Copy the REST files to more permanent locations:<br />
#* cp -a /volatile/halld/offline_monitoring/RunPeriod-YYYY-MM/verVV/REST /work/halld/data_monitoring/RunPeriod-YYYY-MM/REST/verVV<br />
#* cp -a /volatile/halld/offline_monitoring/RunPeriod-YYYY-MM/verVV/REST /cache/halld/RunPeriod-YYYY-MM/REST/verVV [under testing]<br />
<br />
Check log files in $HOME/halld/monitoring/process/log for more information on how each run went. If there are problems, check log files, and modify check_monitoring_data.csh to vary the verbosity of the output.<br />
<br />
==Data Versions==<br />
<br />
To document the conditions of the monitoring data that is created, for the sake of reproducability and further analysis we save several pieces of information. The format is intended to be comprehensive enough to document not just monitoring data, but versions of raw and reconstructed data, so that this database table can be used for the event database as well.<br />
<br />
We store one record per pass through one run period, with the following structure:<br />
<br />
{| class="wikitable"<br />
! Field !! Description<br />
|-<br />
| data_type || The level of data we are processing. For the purposes of monitoring, "rawdata" is the online monitoring, "recon" is the offline monitoring<br />
|- <br />
| run_period || The run period of the data<br />
|- <br />
| revision || An integer specifying which pass through the run period this data corresponds to<br />
|- <br />
| software_version || The name of the XML file that specifies the different software versions used<br />
|-<br />
| jana_config || The name of the text file that specifies which JANA options were passed to the reconstruction program<br />
|-<br />
| ccdb_context || The value of JANA_CALIB_CONTEXT, which specifies the version of calibration constants that were used<br />
|-<br />
| production_time || The data at which monitoring/reconstruction began<br />
|-<br />
| dataVersionString || A convenient string for identifying this version of the data<br />
|}<br />
<br />
<br />
An example file used as as input to ./register_new_version.py is:<br />
<syntaxhighlight><br />
data_type = recon<br />
run_period = RunPeriod-2014-10<br />
revision = 1<br />
software_version = soft_comm_2014_11_06.xml<br />
jana_config = jana_rawdata_comm_2014_11_06.conf<br />
ccdb_context = calibtime=2014-11-10<br />
production_time = 2014-11-10<br />
dataVersionString = recon_RunPeriod-2014-10_20141110_ver01<br />
</syntaxhighlight></div>Kmoriyahttps://halldweb.jlab.org/wiki/index.php?title=Data_Monitoring_Procedures&diff=71720Data Monitoring Procedures2015-12-05T00:16:13Z<p>Kmoriya: </p>
<hr />
<div>__TOC__<br />
<br />
== Master List of File / Database / Webpage Locations ==<br />
=== Run Conditions ===<br />
* Online Run-by-run condition files (B-field, current, etc.): /work/halld/online_monitoring/conditions/<br />
* Offline monitoring run conditions (software versions, jana config): /group/halld/data_monitoring/run_conditions/<br />
*[http://www.jlab.org/Hall-D/test/RunInfo/ Run Info vers. 1]<br />
*[https://halldweb.jlab.org/cgi-bin/data_monitoring/run_conditions.pl Run Info vers. 2]<br />
<br />
=== Monitoring Output Files ===<br />
* Run Periods 201Y-MM is for example 2015-03, launch ver verVV is for example ver15<br />
* Online monitoring histograms: /work/halld/online_monitoring/root/<br />
* Offline monitoring histogram ROOT files: /work/halld/data_monitoring/RunPeriod-201Y-MM/verVV/rootfiles<br />
* REST files (most recent launch only): /work/halld/data_monitoring/RunPeriod-201Y-MM/REST/verVV<br />
* individual files for each job (ROOT, log, etc.): /volatile/halld/offline_monitoring/RunPeriod-201Y-MM/verVV/<br />
<br />
=== Monitoring Database ===<br />
* Accessing monitoring database (on ifarm): mysql -u datmon -h hallddb.jlab.org data_monitoring<br />
<br />
=== Monitoring Webpages ===<br />
*[https://halldweb.jlab.org/cgi-bin/data_monitoring/monitoring/plotBrowser.py Plot Browser]<br />
*[https://halldweb.jlab.org/cgi-bin/data_monitoring/monitoring/runBrowser.py Run Browser]<br />
*[https://halldweb.jlab.org/cgi-bin/data_monitoring/monitoring/timeSeries.py Time Series]<br />
*[https://halldweb.jlab.org/data_monitoring/launch_analysis/ Launch Analysis]<br />
<br />
== Job Monitoring Links ==<br />
* [http://scicomp.jlab.org/farm2/job.html Completed Job History]<br />
* [http://scicomp.jlab.org/farm2/project.html Job Stats By Project]<br />
* [http://scicomp.jlab.org/farm2/trackOrg.html Job Stats By Track]<br />
* [http://scicomp.jlab.org/farm2/report.html Cluster Report]<br />
* [http://scicomp.jlab.org/farm2/walltime.html Walltime Distribution]<br />
<br />
== Saving Online Monitoring Data ==<br />
<br />
The procedure for writing the data out is given in, e.g.,<br />
[https://halldweb.jlab.org/wiki/index.php/Raid-to-Silo_Transfer_Strategy Raid-to-Silo Transfer Strategy].<br />
<br />
Once the DAQ writes out the data to the raid disk, cron jobs will copy the file to tape,<br />
and within ~20 min., we will have access to the file on tape at<br />
/mss/halld/$RUN_PERIOD/rawdata/RunXXXXXX.<br />
<br />
All online monitoring plugins will be run as data is taken.<br />
They will be accessible within the counting house via RootSpy, and<br />
for each run and file, a ROOT file containing the histograms will be saved<br />
within a subdirectory for each run.<br />
<br />
For immediate access to these files, the raid disk files may be accessed directly<br />
from the counting house, or the tape files will be available within ~20 min. of the<br />
file being written out.<br />
<br />
== Offline Monitoring: Running Over Archived Data ==<br />
<br />
Once files are written to tape we run the online plugins on these files to confirm what we were seeing in the online monitoring, and also to update the results from the latest calibration and software. <br />
Manual scripts and cron jobs are set up to look for new data and run the plugins over a sample of files.<br />
<br />
Every other Friday (usually the Friday before the offline meetings) jobs will be started to run the newest software on all previous runs,and allows everybody to see improvements in each detector.<br />
For each launch, independent builds of hdds, sim-recon, the monitoring plugins, and an sqlite file will be generated.<br />
<br />
Below the procedures are described for<br />
# Preparing the software for the launch<br />
# Starting the launch (using hdswif)<br />
# Post-analysis of statistics of the launch<br />
<br />
Processing the results and making them available to the collaboration<br />
is handled in the section [[#Post-Processing_Procedures | Post-Processing Procedures]] below.<br />
<br />
<!--<br />
==== Generating an offline plugin job ====<br />
<br />
The user gxproj1 should be used for official offline monitoring jobs.<br />
Within /home/gxproj1/halld/monitoring/batch/ there will be scripts to run the online monitoring plugins over tape files.<br />
The main script is generatejobs_plugins_rawdata.sh, which can be used as<br />
generatejobs_plugins_rawdata.sh [minrun] [maxrun] (minfile) (maxfile)<br />
where minrun, maxrun specify the range of the run #, and minfile, maxfile (optional) specify<br />
the file range for that run.<br />
<br />
This will generate a script run_rawdata_XXXXXX.sh, where the run # has now been formatted to be 6 digits.<br />
Executing this script will send the monitoring plugins job to the Auger batch system.<br />
<br />
There is also a script clean.sh which can be used as<br />
clean.sh XXX<br />
This will clean up all associated files created in association with run XXX.<br />
<br />
Internally, the xml file used to submit the job will be created, and the job to run<br />
will be given within script.sh. All run parameters should be specified in at the beginning<br />
of generatejobs_plugins_rawdata.sh<br />
Since we are running on tape, the tape file will first be copied over to the cache disk, and the job will run<br />
over this cached file.<br />
<br />
==== Using cron to run automatically ====<br />
Within /home/gluex/halld/monitoring/cron/ there is a file cron_plugins<br />
that can be executed via<br />
crontab cron_plugins<br />
This will set up a cron job to call the script scan_for_jobs.sh, which will<br />
check in the rawdata directory and call generatejobs_plugins_rawdata.sh for<br />
any run that is more than 5 min old. The cron job is set up to run every 10 min.<br />
<br />
==== Magnetic Field Settings ====<br />
For tracking, it is necessary to set the correct magnetic field settings.<br />
The below table is now obsolete. Please refer to the [https://halldweb.jlab.org/cgi-bin/data_monitoring/run_conditions.pl GlueX Run Conditions] webpage.<br />
<br />
The actual field settings for each run have not been documented well.<br />
Below are values based on going through entries in the [https://logbooks.jlab.org/book/halld halld log].<br />
<br />
{| border="1" cellpadding="1" style="text-align: center;"<br />
!width="100"| Run #s<br />
!width="150"| Solenoid Current (A)<br />
!width="300"| JANA option<br />
!width="300"| notes<br />
|-<br />
| 940 - 996 || 1000 || -PBFIELD_MAP=Magnets/Solenoid/solenoid_1000A_poisson_20141104 || At run 997, the solenoid started ramping down<br />
|-<br />
| 998 - 1448 || 0 || -PBFIELD_TYPE=NoField -PDEFTAG:DTrackCandidate=StraightLine || Solenoid started ramping up around run 1431. Runs 1432 - 1448 should not be used for tracking. See [https://logbooks.jlab.org/entry/3309039 this] and [https://logbooks.jlab.org/entry/3309092 this] entry.<br />
|-<br />
| 1449 - 1620 || 1200 || -PBFIELD_MAP=Magnets/Solenoid/solenoid_1200A_poisson_20140520 || <br />
|-<br />
|}<br />
<br />
'''Note:''' The following run ranges were taken with a 300A field, and could be analyzed with the magnetic field map Magnets/Solenoid/solenoid_300A_poisson_20140819 , more study is needed to see if the straight line track fitter is needed or not: 1036 - 1053, 1065 - 1121, 1212? - 1254, 1309 - 1318 [1307,1308,1319-1329 were taken while the magnet was ramping].<br />
--><br />
<br />
=== General Information on Procedures ===<br />
This section explains how the offline monitoring should be run.<br />
Since we may want to simultaneously run offline monitoring for different run periods that require<br />
different environment variables, the scripts are set up so that a generic user can download the<br />
scripts and run them from anywhere. Most output directories for offline monitoring are created<br />
with group read/write permissions so that any Hall D group user has access to the contents,<br />
but there are some cases where use of the account that created the launch is necessary.<br />
<br />
The accounts used for offline monitoring are the gxprojN accounts created and maintained by<br />
Mark Ito (see [https://halldweb.jlab.org/wiki/index.php/GlueX-related_shared_accounts_on_the_JLab_CUE here] for how each account is used).<br />
As of October 2015, the following are used:<br />
* gxproj1 for running over Fall 2014 data (deprecated since June 2015)<br />
* gxproj5 for running over Spring 2015 data<br />
<br />
Since the summer of 2015 we have transitioned from a system using Mark Ito's jproj scripts<br />
to integrating the swif system that Chris Larrieu (SciComp) has been developing. For offline monitoring,<br />
the hdswif system that Kei developed is used for launching the jobs, and the jproj system is used<br />
for meta-analysis of launch statistics.<br />
<br />
Both hdswif and jproj are maintained in svn:<br />
* hdswif: https://halldsvn.jlab.org/repos/trunk/scripts/monitoring/hdswif<br />
* jproj : https://halldsvn.jlab.org/repos/trunk/scripts/monitoring/jproj<br />
<br />
To run the offline monitoring each package should be checked out and all necessary<br />
scripts are included.<br />
<br />
=== Preparing the software for the launch ===<br />
<br />
1. Setup the environment: <pre>source ~/env_monitoring_launch</pre><br />
<br />
2. Updating & building hdds: <br />
<pre><br />
cd $HDDS_HOME<br />
git pull # Get latest software<br />
scons -c install # Clean out the old install: EXTREMELY IMPORTANT for cleaning out stale headers<br />
scons install -j4 # Rebuild and re-install with 4 threads<br />
</pre><br />
<br />
3. Updating & building sim-recon: <br />
<pre><br />
cd $HALLD_HOME/src<br />
git pull # Get latest software<br />
scons -c install # Clean out the old install: EXTREMELY IMPORTANT for cleaning out stale headers<br />
scons install -j4 # Rebuild and re-install with 4 threads<br />
</pre><br />
<br />
4. Create a new sqlite file containing the very latest calibration constants. Original documentation on creating sqlite files are [https://halldweb.jlab.org/wiki/index.php/SQLite-form_of_the_CCDB_database here]. <br />
<pre><br />
cd $GLUEX_MYTOP/../sqlite/<br />
$CCDB_HOME/scripts/mysql2sqlite/mysql2sqlite.sh -hhallddb.jlab.org -uccdb_user ccdb | sqlite3 ccdb.sqlite<br />
mv ccdb.sqlite ccdb_monitoring_launch.sqlite #replacing the old file<br />
</pre><br />
<br />
5. Note that the above steps must be done <b>BEFORE</b> launch project creation. This is because we will track the revisions of the libraries used, and this is done by extracting the svn information in each directory. Also, note that the system assumes that we have the topmost build directory (usually called GLUEX_TOP) to be $HOME/builds . Such an assumption is necessary to be able to extract information about the library locations automatically.<br />
<br />
Create the appropriate project(s) and submit the jobs using hdswif, as detailed in the section below.<br />
<br />
=== Starting the Launch and Submitting Jobs ===<br />
<br />
Until the summer of 2015 we relied solely on Mark Ito's jproj system for submitting and keeping track of jobs. We have since moved to the swif system and use the hdswif wrapper for this. Below are instructions for how to use these.<br />
<br />
1. Downloading hdswif: Download the hdswif directory from svn. For the gxprojN accounts, use the directory ~/halld/hdswif. <br />
<pre><br />
cd ~/halld<br />
svn co https://halldsvn.jlab.org/repos/trunk/scripts/monitoring/hdswif<br />
cd hdswif<br />
</pre><br />
<br />
2. Edit the job config file, which is used to register jobs in hdswif. A typical config file will look this:<br />
PROJECT gluex<br />
TRACK reconstruction<br />
OS centos65<br />
NCORES 6<br />
DISK 40<br />
RAM 8<br />
TIMELIMIT 8<br />
JOBNAMEBASE offmon_<br />
RUNPERIOD 2015-03<br />
VERSION 15<br />
OUTPUT_TOPDIR /volatile/halld/offline_monitoring/RunPeriod-[RUNPERIOD]/ver[VERSION] # Example of other variables included in variable<br />
SCRIPTFILE /home/gxproj5/halld/hdswif/script.sh # Must specify full path<br />
ENVFILE /home/gxproj5/halld/hdswif/setup_jlab-2015-03.csh # Must specify full path<br />
<br />
3. Creating the workflow: Within SWIF jobs are registered into workflows. For offline monitoring, the workflow names are of the form <b>offline_monitoring_RunPeriod201Y_MM_verVV_hd_rawdata</b> with suitable replacements for the run period and version number. The command "swif list" will list all existing workflows. Also, for most simple SWIF commands hdswif also provides a wrapper. <br />
swif list<br />
<br />
For creation of workflows for offline monitoring the command:<br />
hdswif.py create [workflow] -c input.config<br />
should be used. When a config file (here: input.config) is passed in, hdswif will automatically create files that record the configuration of the current launch. These files are stored as for example:<br />
/group/halld/data_monitoring/run_conditions/jana_rawdata_comm_2015_03_ver15.conf<br />
/group/halld/data_monitoring/run_conditions/soft_comm_2015_03_ver15.xml<br />
<br />
The software packages stored in git (sim-recon and hdds) can have git tags applied to them, which makes it easier to find versions of the software than a SHA-1 hash. hdswif will ask if you would like to create a tag, and execute the following sequence: <pre>git tag -a offmon-201Y_MM-verVV -m "Used for offline monitoring 201Y-MM verVV started on 201y/mm/dd"</pre> <pre>git push offmon-201Y_MM-verVV</pre> This will only be invoked when the user name is gxprojN, and for the configuration files, the output directory will be /group/halld/data_monitoring/run_conditions/ for gxprojN accounts while it will be the current directory for other users.<br />
* To use the git-tagged software versions do for example <pre>cd $HALLD_HOME</pre> <pre>git checkout offmon-2015_03-ver15</pre><br />
<br />
4. Registering jobs in the workflow: To register jobs within the workflow, hdswif provides the use of config files. Jobs can be registered by specifying the workflow, config file (-c), run (-r) and file (-f) numbers if necessary. <br />
<span style="color:red">Note: Job configuration parameters can be set differently for jobs within the same workflow if necessary.</span><br />
Jobs can be added via <pre>hdswif.py add [workflow] -c input.config</pre><br />
By default, hdswif will add all files found within the directory /mss/halld/RunPeriod-201Y-MM/rawdata/ where 201Y-MM is specified by the RUNPERIOD parameter in the config file. If only some of the runs or files are needed, these can be specified for example with<br />
<pre>hdswif.py add [workflow] -c input.config -r 3180 -f '00[0-4]'</pre><br />
to specify to register running only over run 3180 files 000 - 004 (Unix-style brackets and wildcards can be used).<br />
<br />
5. Running the workflow: To run the workflow, simply use the hdswif wrapper:<br />
hdswif.py run [workflow]<br />
<br />
<b>It is recommended that some jobs be tested to make sure that everything is working rather than fail thousands of jobs.</b><br><b>Check the configuration setup when creating a workflow, and what is in script.sh. Also check the following:<br />
* Check stderr files. Are they small (<kB)?<br />
* Check stdout files. Are they very large (>MB)?<br />
* Check output ROOT files. Are they larger than several MB?<br />
* Check output REST files. Are they larger than several tens of MB?<br />
</b><br />
<br> For this purpose, hdswif will take an additional parameter to run which limits the number of jobs to submit: <pre>hdswif.py run [workflow] 10</pre><br />
in which case only 10 jobs will be submitted.<br />
<br />
To submit all jobs after checking the results, do <pre>hdswif.py run [workflow]</pre><br />
<br />
=== Checking the Status and Resubmitting ===<br />
1. The status of jobs can be checked on the terminal with <pre>jobstat -u gxprojN</pre> For the status of jobs on Auger see http://scicomp.jlab.org/scicomp/#/auger/jobs and for SWIF use <pre>swif list</pre> or for more information, <pre>swif status [workflow] -summary</pre> Note that "swif status" tends to be out of date sometimes, so don't panic if your workflow/jobs aren't showing up right away. Also see the Auger [http://scicomp.jlab.org/scicomp/#/auger/jobs job website].<br />
<br />
2. For failed jobs, SWIF can resubmit jobs based on the problem. For resubmission for failed jobs with the same resources, <pre>swif retry-jobs [workflow] -problems [problem name]</pre> can be used, and for jobs to be submitted with more resources, e.g., use <pre>swif modify-jobs [workflow] -ram add 2gb -problems AUGER-OVER_RLIMIT</pre> <br />
This only re-stages the jobs, be sure to resubmit them with:<br />
<pre>swif run -workflow [workflow] -errorlimit none</pre><br />
<br />
hdswif has a wrapper for both of these: <pre>hdswif.py resubmit [workflow] [problem]</pre> In this case [problem] can be one of <b>SYSTEM, TIMEOUT, RLIMIT</b>. If SYSTEM is specified, the jobs will be retried. For TIMEOUT and RLIMIT, the jobs will be modified by default with 2 additional hours or GB of RAM. If one more number is added as an option, then that many hours or GB of RAM will be added., e.g., <pre>hdswif.py resubmit [workflow] TIMEOUT 5</pre> will add 5 hours of processing time. You can wait until almost all jobs finish before resubmitting failed jobs since the number should be relatively small. <b>Even if jobs are resubmitted for one type of failure, jobs that later fail with that failure will not be automatically resubmitted.</b><br />
<br />
3. For information on swif, use the "swif help" commands and for hdswif see the attached documentaion in https://halldsvn.jlab.org/repos/trunk/scripts/monitoring/hdswif/manual_hdswif.pdf<br />
<br />
4. Below is a table describing the various errors that can occur.<br />
{| border="1" cellpadding="0" valign="left" style="text-align: left;"<br />
!width="150"| ERROR NAME<br />
!width="400"| Description<br />
!width="400"| Resolution<br />
!width="400"| hdswif command<br />
|-<br />
| AUGER-SUBMIT<br />
||<br />
SWIF’s attempt to submit jobs to Auger failed. Includes server-side problems as well as user failing to provide valid job parameters (e.g. incorrect project name, too many resources, etc.)<br />
||<br />
If requested resources are known to be correct resubmit. Otherwise modify job resources using swif directly.<br />
||<br />
<span style="color:purple"><b>hdswif.py resubmit [workflow] SYSTEM</b></span><br />
|-<br />
| AUGER-FAILED<br />
||<br />
Auger reports the job FAILED with no specific details.<br />
||<br />
Resubmit jobs. If problem persists, contact Chris Larrieu or SciComp.<br />
||<br />
<span style="color:purple"><b>hdswif.py resubmit [workflow] SYSTEM</b></span><br />
|-<br />
| AUGER-OUTPUT-FAIL<br />
||<br />
Failure to copy one or more output files.Can be due to permission problem, quota problem, system error, etc.<br />
||<br />
Check if output files will exist after job execution and that output directory exists, resubmit jobs. If problem persists, contact Chris Larrieu or SciComp.<br />
||<br />
<span style="color:purple"><b>hdswif.py resubmit [workflow] SYSTEM</b></span><br />
|-<br />
| AUGER-INPUT-FAIL<br />
||<br />
Auger failed to copy one or more of the requested input files, similar to output failures. Can also happen if tape file is unavailable (e.g. missing/damaged tape)<br />
||<br />
Check if input file exists, resubmit jobs. If problem persists, contact Chris Larrieu or SciComp.<br />
||<br />
<span style="color:purple"><b>hdswif.py resubmit [workflow] SYSTEM</b></span><br />
|-<br />
| AUGER-TIMEOUT<br />
||<br />
Job timed out.<br />
||<br />
If more time is needed for job add more resources.<br />
Default is to add 2 hrs of processing time. Also check whether code is hanging.<br />
||<br />
<span style="color:red"><b>hdswif.py resubmit [workflow] TIMEOUT</b></span><br><br />
Default is to add 2 hours. Optionally specify number of hours at end.<br />
|-<br />
| AUGER-OVER_RLIMIT<br />
||<br />
Not enough resources, RAM or disk space.<br />
||<br />
Add more resources for job.<br />
||<br />
<span style="color:blue"><b>hdswif.py resubmit [workflow] RLIMIT</b></span><br><br />
Default is to add 2 GB of RAM. Optionally specify GB at end. To add more disk space use SWIF directly.<br />
|-<br />
| SWIF-MISSING-OUTPUT<br />
||<br />
Output file specified by user was not found.<br />
||<br />
Check if output file exists at end of job.<br />
||<br />
<br />
|-<br />
| SWIF-USER-NON-ZERO <br />
||<br />
User script exited with non-zero status code.<br />
||<br />
Your script exited with non-zero status. Check the code you are running.<br />
||<br />
<br />
|-<br />
| SWIF-SYSTEM-ERROR <br />
||<br />
Job failed owing to a problem with swif (e.g. network connection timeout)<br />
||<br />
Resubmit jobs. If problem persists, contact Chris Larrieu or SciComp.<br />
||<br />
<span style="color:purple"><b>hdswif.py resubmit [workflow] SYSTEM</b></span><br />
|}<br />
<br style="clear:both;"/><br />
<br />
=== Post-analysis of statistics of the launch ===<br />
<br />
After jobs have been submitted, it will usually take a few days for all of the jobs to be processed.<br />
The next step is to check the resource usage for the current launch and publish the results online.<br />
<br />
# <b>Create summary XML, HTML files</b><br> The status and results of jobs are saved within the SWIF internal server, and are available via the command <pre>swif status [workflow] -summary -runs</pre> where the arguments -summary and -runs show summary statistics and statistics for individual jobs, respectively. hdswif has a command that takes this output in XML output and creates an HTML webpage showing results of the launch. To do this, do <pre>hdswif.py summary [workflow]</pre> This will create the XML output file from SWIF called swif_output_[workflow].xml and create a webpage containing png figure files. If the XML file already exists, hdswif will ask whether to overwrite the existing file.<br />
# <b>Publish output files online</b><br> At this stage the html output and figure files are created and ready to be put online. The html output capabilities of hdswif is useful for any user using SWIF, but since publication of the html output is specific to the offline monitoring, the script to do this is contained in the jproj scripts directory at https://halldsvn.jlab.org/repos/trunk/scripts/monitoring/jproj. For the gxprojN accounts this directory should exist as ~/halld/jproj. To publish the results online do for example <pre>python ~/halld/jproj/projects/templates/publish_offmon_results.py 2015_03 18</pre> The script simply copies the html output and corresponding figures to /group/halld/www/halldweb/html/data_monitoring/launch_analysis/<br />
# <b>Editing the summary HTML page</b><br> The top page for is offline monitoring https://halldweb.jlab.org/data_monitoring/launch_analysis/index.html and has links to the summary page for each run period. The summary files are /group/halld/www/halldweb/html/data_monitoring/launch_analysis/[run period].html Edit the file to<br />
## Add a new line to the first table which contains the version number, date, and comments for the current launch<br />
## Create a link to the webpage for the current launch. Simply copy and paste, modify the previous launch's link to have the correct launch ver.<br />
# <b>Freezing SWIF tables</b><br> Since we are now finished with the SWIF workflow that we used, the workflow should be "frozen" so that it cannot be mistakenly altered or modified. Do <pre>swif freeze [workflow]</pre><br />
# <b>Backing up SWIF output</b><br> With the workflow frozen we should be able to reproduce the XML output from SWIF, but we will backup the XML output just in case. Do <pre>cp ~/halld/hdswif/swif_output_[workflow].xml /group/halld/data_monitoring/swif_xml_backup/ </pre><br />
<br />
=== Cross Analysis of Launches ===<br />
<br />
The purpose of the cross analysis is to correlate how resource usage changed for the same files across different launches.<br />
To do this it is useful to create MySQL tables that contain information on each launch, and do queries across tables.<br />
<br />
#The scripts to do this are maintained in the svn directory https://halldsvn.jlab.org/repos/trunk/scripts/monitoring/jproj Do <pre> svn co https://halldsvn.jlab.org/repos/trunk/scripts/monitoring/jproj </pre> For the gxprojN accounts used for offline monitoring the directory should be ~/halld/jproj<br />
# The jproj directory contains two subdirectories, scripts and projects. The scripts directory contains useful scripts for processing the jobs registered in the jproj system, and each of the offline monitoring launches will be handled in the projects directory.<br />
# Go to the projects directory<pre>cd ~/halld/jproj/projects</pre> and use the script create_project.sh to create a new directory that contains the processing scripts for the current launch <pre>./create_project.sh [workflow]</pre> This should create a directory such as offline_monitoring_RunPeriod2015_03_ver15_hd_rawdata (same as the workflow name). The script uses the template files in the directory templates and by substitution creates script files for the current launch. Now go to the newly created analysis directory: <pre>cd [workflow]/analysis</pre><br />
which for the gxprojN accounts should have the full path /home/gxprojN/halld/jproj/projects/[workflow]/analysis.<br><br><br />
<span style="color:red">All of the analysis commands including arguments are contained in run_analysis.sh, but it is <b>strongly</b> recommended that all commands are run manually to check for errors.</span><br><br><br />
# Next, we need to create a few MySQL tables for the current launch. The MySQL tables are useful for comparing run/file combinations for different launches. For the SWIF launches, there are two tables needed, and will be named<br />
#* [workflow]Job<br />
#* [workflow]_aux<br />
# This naming scheme of tables and their roles are the same as from the jproj only launches. The [workflow]Job table will contain information gathered from SWIF about each job (which node it went to, start time of each stage, memory usage, etc.). The [workflow]_aux table will contain information gathered from the output stdout files from each job. First, create the Job table using <pre>python create_jproj_job_table.py [run period] [version]</pre> Note that this script uses the XML output from hdswif summary and inserts the contents into the MySQL table, so the XML output file must exist.<br />
# To check the contents of this MySQL table, do <pre>mysql -hhalldb -ufarmer farming</pre> <pre>mysql> describe offline_monitoring_RunPeriod2015_03_ver11_hd_rawdataJob;</pre><br />
# <b>Backing up tables</b><br> Tables created in MySQL should be backed up.<br />
<br />
== Explanation of Scripts ==<br />
Below are explanations of each script used in the offline monitoring system and a breif explanation of how they work.<br />
<br />
=== hdswif scripts ===<br />
<b>Summary:</b> hdswif.py is the main script that calls the other utility scripts. For the utility scripts, they can be run standalone by giving the appropriate parameters. Visual graphics are made using the PyROOT extension of ROOT. To use this, the environment variable PYTHONPATH must include ROOTSYS. If ROOTSYS has been set, adding it to PYTHONPATH will be done by the script, but if ROOTSYS is not set, then the scripts will abort.<br />
<br />
{| border="1" cellpadding="0" valign="left" style="text-align: left;"<br />
!width="150"| file name<br />
!width="800"| Description<br />
|-<br />
| hdswif.py ||<br />
Main script to control the behavior of SWIF. Most commands follow the form hdswif.py [command] [workflow] (options)<br />
|-<br />
| parse_swif.py ||<br />
Called within hdswif.py to create html output from SWIF results.<br />
|-<br />
| createXMLfiles.py ||<br />
* Creates XML files for logging information about launch. Must specify config file with option -c.<br />
* Also adds tags to git repositories of sim-recon and hdds.<br />
* For XML file creation, the file will be written out to /group/halld/data_monitoring/run_conditions/ if the user is gxprojN (used for offline monitoring). If not, to avoid general users adding things to the above directory, the output files will be the current directory.<br />
* To write out the versions of each package, environment variables such as HDDS_HOME will need to be set.<br />
* Versions for each software package are extracted using the directory structure, so if these are changed the scripts must change accordingly.<br />
* For each launch, <b>the output soft_comm_[RUNPERIOD]_ver[VER].xml file should be checked that all version numbers have been extracted.</b><br />
|-<br />
|}<br />
<br />
== Hall D Job Management System ==<br />
<br />
This section details instructions on how to create and launch a set of jobs using the Hall-D Job Management System developed by Mark Ito. These instructions are generic: this system can be used for the weekly monitoring jobs, but can also be used for other sets of job launches as well. <br />
<br />
=== Database Table Overview ===<br />
<br />
* Job management database table (<project_name>): For each input file, keeps track of whether or not a job for it has been submitted, along with other optional fields. <br />
<br />
* Job status database table (<project_name>Job (no space)): For each job, keeps track of the job-id, the job status, memory used, cpu & wall time, time taken to complete various stages (e.g. pending, dependency, active), and others. <br />
<br />
* Job metrics database table (<project_name>_aux (no space)): For each job, keeps track of the job-id, how many events were processed, the time it took to copy the cache file, and the time it took to run the plugin. This information is culled from the log files of each job, and is done within the analysis directory of each launch.<br />
<br />
=== Initialize Project Management ===<br />
<br />
* Log into the ifarm machine with one of the gxproj accounts. For this example we will use gxproj1.<br />
<pre><br />
ssh gxproj1@ifarm -Y<br />
</pre><br />
<br />
* Go to a directory to do the launch. In principle, any directory will work, but for gxproj1 this is usually done in /home/gxproj1/halld/<br />
<br />
* Check out the necessary scripts <pre>svn co https://halldsvn.jlab.org/repos/trunk/scripts/monitoring/jproj</pre> This will get all necessary scripts for launching. Once checked out, <pre>cd projects</pre><br />
<br />
* The script create_project.sh can be used to create a new project. It will take a single argument, the project name. It is assumed that the project name is of the form offline_monitoring_RunPeriod20YY_MM_verVV_hd_rawdata where this string will be parsed to give the run period 20YY_MM and version number VV. One thing to do BEFORE creation of a new project is to editthe conditions of the launch (plugins to run over, memory requested, disk space requested) within templates/template.jsub . This<br />
information is saved automatically at project creation time into files in /group/halld/data_monitoring/run_conditions .<br />
To create a project do <pre>./create_project.sh offline_monitoring_RunPeriod20YY_MM_verVV_hd_rawdata</pre><br />
The name has been chosen to be as consistent as possible with other directory structures.<br />
However, mysql requires that "-" be escaped in table names, so unfortunately run periods will be<br />
given as 2YYY_MM instead of 2YYY-MM.<br />
<br />
* A new directory with the project name (offline_monitoring_RunPeriod20YY_MM_verVV_hd_rawdata) and files will be copied and modified from the template directory to reflect the run period, the user, the directory that it was created in, project name, etc.<br />
<br />
* For each project, cd into the new directory<br />
* The script clear.sh will remove any existing tables for the current project name, then recreate it. Do<br />
<pre>./clear.sh</pre><br />
* To use the jproj.pl script that was checked in, add the directory to your path with<br />
<pre>source ../../scripts/setup.csh</pre><br />
or always specify the full path<br />
<pre>../../scripts/jproj.pl</pre><br />
* Now update the table of runs with<br />
<pre>jproj.pl <project name> update</pre><br />
This will fill the table with all files within /mss that are of the same form as what is in <project name>.jproj . If you want to register only on a subset of all such files, you can edit this file directly.<br />
* Once you have registered all of the files you would like to run over, do<br />
<pre>jproj.pl <project name> submit [max # of jobs] [run number]</pre><br />
where the additional options specify how many jobs to submit and which run number to run on. Without these options all files that are registered and have not been submitted yet will be submitted.<br />
<br />
At this stage you are ready to submit all files. It is a good idea to submit a few test jobs<br />
at first to check that all scripts are working and that the plugins do not crash. Once you are<br />
sure that this does not happen, you can send all jobs in. The remaining jobs are then the monitoring<br />
which will (among other things) put the results on the online webpage for the collaboration to view, and<br />
the analysis of the launch.<br />
<br />
=== Project File Overview ===<br />
<br />
An overview of each project file:<br />
* '''clear.sh:''' For the current project, deletes the job status and management database tables (if any), and creates new, empty ones. <br />
* '''<project_name>.jproj:''' Contains the path and file name format for the input files for the jobs. <br />
* '''<project_name>.jsub:''' The xml job submission script. The run number and file number variables are set during job submission for each input file. <br />
* '''script.sh:''' The script that is executed during the job. If output job directories are not pre-created manually, they should be created in this script with the proper permissions:<br />
<pre><br />
mkdir -p -m 775 my_directory<br />
</pre><br />
* '''setup_jlab-[run period].csh:''' The environment that is sourced at the beginning of the job execution. <br />
* '''status.sh:''' Updates the job status database table, and prints some of its columns to screen.<br />
<br />
=== Project Management ===<br />
<br />
* Delete (if any) and create the database table(s) for the current set of job submissions:<br />
<pre><br />
./clear.sh<br />
</pre><br />
Also, if testing was done with jobs, it is best to delete the output directory and the configuration files:<br />
<pre><br />
rm -frv /volatile/halld/offline_monitoring/RunPeriod-20YY-MM/verVV /group/halld/data_monitoring/run_conditions/soft_comm_20YY_MM_verVV.xml /group/halld/data_monitoring/run_conditions/jana_rawdata_comm_20YY_MM_verVV.conf<br />
</pre><br />
<br />
* Search for input files matching the string in the .jproj file, and create a row for each in the job management database table (called <project_name>). You can test by adding an optional argument at the end, which only selects files with a specific file number:<br />
<pre><br />
jproj.pl <project_name> update <optional_file_number><br />
</pre><br />
<br />
* Confirm that the job management database is accurate by printing it's contents to screen:<br />
<pre><br />
mysql -hhallddb -ufarmer farming -e "select * from <project_name>"<br />
</pre><br />
<br />
* ONLY if a mistake was made, to delete the tables from the database and recreate new, empty ones, run: <br />
<pre><br />
./clear.sh<br />
</pre><br />
<br />
* To look at the status of the submitted jobs, first query auger and update the job status database:<br />
<pre><br />
fill_in_job_details.pl <project_name><br />
</pre><br />
<br />
* The job status can then be viewed by submitting a query to the job status database (called <project_name>Job (no space in between)):<br />
<pre><br />
mysql -hhallddb -ufarmer farming -e "select id,run,file,jobId,hostname,status,timeSubmitted,timeActive,walltime,cput,timeComplete,result,error from <project_name>Job"<br />
</pre><br />
<br />
* These last two commands can instead be executed simultaneously by running:<br />
<pre><br />
./status.sh<br />
</pre><br />
<br />
=== Handy mysql Instructions ===<br />
<br />
* Handy mysql instructions:<br />
<pre><br />
mysql -hhallddb -ufarmer farming # Enter the "farming" mysql database on "hallddb" as user "farmer"<br />
quit; # Exit mysql<br />
show tables; # Show a list of the tables in the current database<br />
show columns from <project_name>; # show all of the columns for the given table<br />
select * from <project_name>; # show the contents of all rows from the given table<br />
</pre><br />
<br />
=== Backing Up Offline Monitoring Tables ===<br />
Tables created for offline monitoring can be backed up using the script backup_tables.sh which<br />
can be checked out with the other files from https://halldsvn.jlab.org/repos/trunk/scripts/monitoring/jproj/projects<br />
<br />
The script uses the command mysqldump to print out a file that can be executed to recreate the tables.<br />
Since executing this output file will drop the table when it exists, caution is advised.<br />
Example usage to backup all three tables created for run period 2014_10 ver 17:<br />
<pre>backup_tables.sh 2014_10 17</pre><br />
<br />
== Running Over Data As It Comes In ==<br />
<br />
A special user gxproj1 will have a cron job set up to run the plugins as new data appears on /mss.<br />
During the week, gxproj1 will submit offline plugin jobs with the same setup as the weekly jobs<br />
run the previous Friday. The procedure for this is shown below.<br />
<br />
<!--<br />
=== Setting up the environment ===<br />
The file<br />
/home/gxproj1/setup_jlab.csh<br />
is sourced through .tcshrc.<br />
This file is the same as what is linked to by<br />
/home/gluex/setup_jlab_commissioning.csh,<br />
except HALLD_HOME, HDDS_HOME, and JANA_CALIB_URL are set separately so that this<br />
user can have a separate build.<br />
<br />
To obtain the builds from the previous Friday's runs,<br />
execute<br />
/home/gxproj1/halld/monitoring/newruns/setup_previous.sh [year] [month] [day]<br />
The build revisions from the previous Friday are archived in files<br />
/work/halld/data_monitoring/run_conditions/soft_comm_[year]_[month]_[day].xml<br />
and the script will build libraries based on those stored revision numbers.<br />
--><br />
<br />
=== Running the cron job ===<br />
<br />
'''IMPORTANT:''' The cron job should not be running while you are manually submitting jobs using the jproj.pl script for the same project, or else you will probably multiply-submit a job. <br />
<br />
* Go to the cron job directory: <br />
<pre><br />
cd /u/home/gxproj1/halld/monitoring/newruns<br />
</pre><br />
<br />
* The cron_plugins file is the cronjob that will be executed. During execution, it runs the exec.sh command in the same folder. This command takes two arguments: the project name, and the maximum file number for each run. These fields should be updated in the cron_plugins file before running. <br />
<br />
* The exec.sh command updates the job management database table with any data that has arrived on tape since it was last updated, ignoring file numbers greater than the maximum file number. It then submits jobs for these files. <br />
<br />
* To start the cron job, run:<br />
<pre><br />
crontab cron_plugins<br />
</pre><br />
<br />
* To check whether the cron job is running, do<br />
<pre><br />
crontab -l<br />
</pre><br />
<br />
* To remove the cron job do<br />
<pre><br />
crontab -r<br />
</pre><br />
<br />
<!--<br />
The cron job will run the script scan_for_jobs.sh,<br />
which runs generatejobs_plugins_rawdata.sh for any<br />
new runs that it had not seen before. All previous<br />
runs are recorded in the file filelists/files_current.txt<br />
so clear this to run over runs, or set the parameters<br />
MINRUN and MAXRUN which will set the range of runs submitted.<br />
--><br />
<br />
==Post-Processing Procedures==<br />
<br />
To visualize the monitoring data, we save images of selected histograms and store time series of selected quantities in a database, which are then displayed on a web page. This section describes how to generate the monitoring images and database information.<br />
<br />
The scripts used to generate this summary data are primarily run from /home/gxprojN/halld/monitoring/process . If you want a new copy of the scripts, e.g., for a new monitoring run, you should check the scripts out from SVN:<br />
<syntaxhighlight><br />
svn co https://halldsvn.jlab.org/repos/trunk/scripts/monitoring/process<br />
</syntaxhighlight><br />
Note that these scripts currently have some parameters which must be periodically set by hand.<br />
<br />
The default python version on most JLab machine does not have the modules to allow these scripts to connect to the MySQL database. To run these scripts, load the environment with the following command<br />
<syntaxhighlight><br />
source /home/gxproj1/halld/monitoring/process/monitoring_env.csh<br />
</syntaxhighlight><br />
<br />
===Online Monitoring===<br />
<br />
There are two primary scripts for running over the monitoring data generated by the online system and offline reconstruction. The online script can be run with either of the following commands:<br />
<syntaxhighlight><br />
/home/gluex/halld/monitoring/process/check_new_runs.py<br />
<br />
OR <br />
<br />
/home/gluex/halld/monitoring/process/check_new_runs.csh<br />
</syntaxhighlight><br />
The shell script sets up the environment properly to run the python script. To connect to the monitoring database on the JLab CUE, modules included in the local installation of python >= 2.7 are needed. The shell script is appropriate to use in a cron job. The cronjob is currently run under the "gluex" account.<br />
<br />
The online monitoring system copies a ROOT file containing the results of the online monitoring, and other configuration files into a directory accessible outside the counting house. This python script automatically checks for new ROOT files, which it will then automatically process. It contains several configuration variables that must be correctly set, which contains the location of input/output directories, etc...<br />
<br />
===Offline Monitoring===<br />
<br />
After the data is run over, the results should be processed, so that summary data is entered into the monitoring database and plots are made for the monitoring webpages. Currently, this processing is controlled by a cronjob that runs the following script:<br />
<syntaxhighlight><br />
/home/gxproj1/halld/monitoring/process/check_monitoring_data.csh <br />
</syntaxhighlight><br />
This script checks for new ROOT files, and only runs over those it hasn't processed yet. Since one monitoring ROOT file is produced for each EVIO file, whenever a new file is produced, the plots for the corresponding run are recreated and all the ROOT files for a run are combined into one file. Information is stored in the database on a per-file basis. <br />
<br />
Plots for the monitoring web page can be made from single histograms or multiple histograms using RootSpy macros. If you want to change the list of plots made, you must modify one of the following files:<br />
* histograms_to_monitor - specify either the name of the histogram or its the full ROOT path <br />
* macros_to_monitor - specify the full path to the RootSpy macro .C file<br />
<br />
When a new monitoring run is started, or the conditions are changed, the following steps should be taken to process the new files:<br />
# Add a new data version, as described below:<br />
# Change the following parameters in check_monitoring_data.csh:<br />
## JOBDATE should correspond to the ouptut date used by the job submission script<br />
## OUTPUTDIR should correspond to the directory corresponding to the run period and revision corresponding to the new version you just submitted. Presumably, this directory will be empty at the beginning.<br />
## Once you create a new data version as defined below, you should pass the needed information as a command line option. Currently this is done by the ARGS variable. For example, the argument "-v RunPeriod-2014-10,8" tells the monitoring scripts to look up the version corresponding to revision 8 of RunPeriod-2014-10 in the monitoring DB and to use to store the results.<br />
<br />
<syntaxhighlight><br />
Example configuration parameters:<br />
set JOBDATE=2015-01-09<br />
set INPUTDIR=/volatile/halld/RunPeriod-2014-10/offline_monitoring<br />
set OUTPUTDIR=/w/halld-scifs1a/data_monitoring/RunPeriod-2014-10/ver08<br />
set ARGS=" -v RunPeriod-2014-10,8 "<br />
</syntaxhighlight><br />
If you want to process the results manually, the data is processed using the following script:<br />
<syntaxhighlight><br />
./process_new_offline_data.py <input directory> <output directory><br />
<br />
EXAMPLE:<br />
<br />
./process_new_offline_data.py 2014-11-14 /volatile/halld/RunPeriod-2014-10/offline_monitoring/ /w/halld-scifs1a/data_monitoring/RunPeriod-2014-10/ver02<br />
</syntaxhighlight><br />
The python script takes several options to enable/disable various steps in the processing. Of interest is the "--force" option, which will run over all monitoring ROOT files, whether or not they've been previously identified.<br />
<br />
Every time a new reconstruction pass is performed, a new version number must be generated. To do this, prepare a version file as described below. Then run the register_new_version.py script to store the information in the database. The script will return a version number, which then should be set by hand in process_new_offline_data.py - future versions of the script will streamline this part of the procedure. An example of how to generate a new version is:<br />
<syntaxhighlight><br />
./register_new_version.py add /home/gxproj1/halld/monitoring/process/versions/vers_RunPeriod-2014-10_pass1.txt<br />
</syntaxhighlight><br />
<br />
<b>If you are running the offline monitoring by checking out the files in trunk/scripts/monitoring/jproj/projects/<br />
of the svn repository</b>, and created a project with<br />
create_project.sh [project name] hd_rawdata<br />
Then go to the directory [project name]/processing/<br />
and execute<br />
./run_processing.sh<br />
which will run register_new_version.py as well as check_monitoring_data.csh for that project.<br />
<br />
===Step-by-Step Instructions For Processing a New Monitoring Run===<br />
<br />
The monitoring runs are current run out of the gxproj1 and gxproj5 accounts. After an offline monitoring run has been successfully started on the batch farm, the following steps should be followed to setup the post-processing for these runs.<br />
<br />
# The post-processing scripts are stored in $HOME/halld/monitoring/process and are automatically run by cron.<br />
# Run "svn update" to bring any changes in. Be sure that the list of histograms and macros to plot are current.<br />
# Add a new data version<br />
# Edit check_monitoring_data.csh to point to the current revisions/directories<br />
#* VERSION<br />
#* ARGS<br />
#* Note that the environment depends on a standard script - $HOME/setup_jlab.csh<br />
# Update files in the web directory, so that the results are displayed on the web pages: /group/halld/www/halldweb/html/data_monitoring/textdata<br />
# Copy the REST files to more permanent locations:<br />
#* cp -a /volatile/halld/offline_monitoring/RunPeriod-YYYY-MM/verVV/REST /work/halld/data_monitoring/RunPeriod-YYYY-MM/REST/verVV<br />
#* cp -a /volatile/halld/offline_monitoring/RunPeriod-YYYY-MM/verVV/REST /cache/halld/RunPeriod-YYYY-MM/REST/verVV [under testing]<br />
<br />
Check log files in $HOME/halld/monitoring/process/log for more information on how each run went. If there are problems, check log files, and modify check_monitoring_data.csh to vary the verbosity of the output.<br />
<br />
==Data Versions==<br />
<br />
To document the conditions of the monitoring data that is created, for the sake of reproducability and further analysis we save several pieces of information. The format is intended to be comprehensive enough to document not just monitoring data, but versions of raw and reconstructed data, so that this database table can be used for the event database as well.<br />
<br />
We store one record per pass through one run period, with the following structure:<br />
<br />
{| class="wikitable"<br />
! Field !! Description<br />
|-<br />
| data_type || The level of data we are processing. For the purposes of monitoring, "rawdata" is the online monitoring, "recon" is the offline monitoring<br />
|- <br />
| run_period || The run period of the data<br />
|- <br />
| revision || An integer specifying which pass through the run period this data corresponds to<br />
|- <br />
| software_version || The name of the XML file that specifies the different software versions used<br />
|-<br />
| jana_config || The name of the text file that specifies which JANA options were passed to the reconstruction program<br />
|-<br />
| ccdb_context || The value of JANA_CALIB_CONTEXT, which specifies the version of calibration constants that were used<br />
|-<br />
| production_time || The data at which monitoring/reconstruction began<br />
|-<br />
| dataVersionString || A convenient string for identifying this version of the data<br />
|}<br />
<br />
<br />
An example file used as as input to ./register_new_version.py is:<br />
<syntaxhighlight><br />
data_type = recon<br />
run_period = RunPeriod-2014-10<br />
revision = 1<br />
software_version = soft_comm_2014_11_06.xml<br />
jana_config = jana_rawdata_comm_2014_11_06.conf<br />
ccdb_context = calibtime=2014-11-10<br />
production_time = 2014-11-10<br />
dataVersionString = recon_RunPeriod-2014-10_20141110_ver01<br />
</syntaxhighlight></div>Kmoriyahttps://halldweb.jlab.org/wiki/index.php?title=Data_Monitoring_Procedures&diff=71476Data Monitoring Procedures2015-11-19T12:47:56Z<p>Kmoriya: /* Post-analysis of statistics of the launch */</p>
<hr />
<div>__TOC__<br />
<br />
== Master List of File / Database / Webpage Locations ==<br />
=== Run Conditions ===<br />
* Online Run-by-run condition files (B-field, current, etc.): /work/halld/online_monitoring/conditions/<br />
* Offline monitoring run conditions (software versions, jana config): /group/halld/data_monitoring/run_conditions/<br />
*[http://www.jlab.org/Hall-D/test/RunInfo/ Run Info vers. 1]<br />
*[https://halldweb.jlab.org/cgi-bin/data_monitoring/run_conditions.pl Run Info vers. 2]<br />
<br />
=== Monitoring Output Files ===<br />
* Run Periods 201Y-MM is for example 2015-03, launch ver verVV is for example ver15<br />
* Online monitoring histograms: /work/halld/online_monitoring/root/<br />
* Offline monitoring histogram ROOT files: /work/halld/data_monitoring/RunPeriod-201Y-MM/verVV/rootfiles<br />
* REST files (most recent launch only): /work/halld/data_monitoring/RunPeriod-201Y-MM/REST/verVV<br />
* individual files for each job (ROOT, log, etc.): /volatile/halld/offline_monitoring/RunPeriod-201Y-MM/verVV/<br />
<br />
=== Monitoring Database ===<br />
* Accessing monitoring database (on ifarm): mysql -u datmon -h hallddb.jlab.org data_monitoring<br />
<br />
=== Monitoring Webpages ===<br />
*[https://halldweb.jlab.org/cgi-bin/data_monitoring/monitoring/plotBrowser.py Plot Browser]<br />
*[https://halldweb.jlab.org/cgi-bin/data_monitoring/monitoring/runBrowser.py Run Browser]<br />
*[https://halldweb.jlab.org/cgi-bin/data_monitoring/monitoring/timeSeries.py Time Series]<br />
*[https://halldweb.jlab.org/data_monitoring/launch_analysis/ Launch Analysis]<br />
<br />
== Job Monitoring Links ==<br />
* [http://scicomp.jlab.org/farm2/job.html Completed Job History]<br />
* [http://scicomp.jlab.org/farm2/project.html Job Stats By Project]<br />
* [http://scicomp.jlab.org/farm2/trackOrg.html Job Stats By Track]<br />
* [http://scicomp.jlab.org/farm2/report.html Cluster Report]<br />
* [http://scicomp.jlab.org/farm2/walltime.html Walltime Distribution]<br />
<br />
== Saving Online Monitoring Data ==<br />
<br />
The procedure for writing the data out is given in, e.g.,<br />
[https://halldweb.jlab.org/wiki/index.php/Raid-to-Silo_Transfer_Strategy Raid-to-Silo Transfer Strategy].<br />
<br />
Once the DAQ writes out the data to the raid disk, cron jobs will copy the file to tape,<br />
and within ~20 min., we will have access to the file on tape at<br />
/mss/halld/$RUN_PERIOD/rawdata/RunXXXXXX.<br />
<br />
All online monitoring plugins will be run as data is taken.<br />
They will be accessible within the counting house via RootSpy, and<br />
for each run and file, a ROOT file containing the histograms will be saved<br />
within a subdirectory for each run.<br />
<br />
For immediate access to these files, the raid disk files may be accessed directly<br />
from the counting house, or the tape files will be available within ~20 min. of the<br />
file being written out.<br />
<br />
== Offline Monitoring: Running Over Archived Data ==<br />
<br />
Once files are written to tape we run the online plugins on these files to confirm what we were seeing in the online monitoring, and also to update the results from the latest calibration and software. <br />
Manual scripts and cron jobs are set up to look for new data and run the plugins over a sample of files.<br />
<br />
Every other Friday (usually the Friday before the offline meetings) jobs will be started to run the newest software on all previous runs,and allows everybody to see improvements in each detector.<br />
For each launch, independent builds of hdds, sim-recon, the monitoring plugins, and an sqlite file will be generated.<br />
<br />
Below the procedures are described for<br />
# Preparing the software for the launch<br />
# Starting the launch (using hdswif)<br />
# Post-analysis of statistics of the launch<br />
<br />
Processing the results and making them available to the collaboration<br />
is handled in the section [[#Post-Processing_Procedures | Post-Processing Procedures]] below.<br />
<br />
<!--<br />
==== Generating an offline plugin job ====<br />
<br />
The user gxproj1 should be used for official offline monitoring jobs.<br />
Within /home/gxproj1/halld/monitoring/batch/ there will be scripts to run the online monitoring plugins over tape files.<br />
The main script is generatejobs_plugins_rawdata.sh, which can be used as<br />
generatejobs_plugins_rawdata.sh [minrun] [maxrun] (minfile) (maxfile)<br />
where minrun, maxrun specify the range of the run #, and minfile, maxfile (optional) specify<br />
the file range for that run.<br />
<br />
This will generate a script run_rawdata_XXXXXX.sh, where the run # has now been formatted to be 6 digits.<br />
Executing this script will send the monitoring plugins job to the Auger batch system.<br />
<br />
There is also a script clean.sh which can be used as<br />
clean.sh XXX<br />
This will clean up all associated files created in association with run XXX.<br />
<br />
Internally, the xml file used to submit the job will be created, and the job to run<br />
will be given within script.sh. All run parameters should be specified in at the beginning<br />
of generatejobs_plugins_rawdata.sh<br />
Since we are running on tape, the tape file will first be copied over to the cache disk, and the job will run<br />
over this cached file.<br />
<br />
==== Using cron to run automatically ====<br />
Within /home/gluex/halld/monitoring/cron/ there is a file cron_plugins<br />
that can be executed via<br />
crontab cron_plugins<br />
This will set up a cron job to call the script scan_for_jobs.sh, which will<br />
check in the rawdata directory and call generatejobs_plugins_rawdata.sh for<br />
any run that is more than 5 min old. The cron job is set up to run every 10 min.<br />
<br />
==== Magnetic Field Settings ====<br />
For tracking, it is necessary to set the correct magnetic field settings.<br />
The below table is now obsolete. Please refer to the [https://halldweb.jlab.org/cgi-bin/data_monitoring/run_conditions.pl GlueX Run Conditions] webpage.<br />
<br />
The actual field settings for each run have not been documented well.<br />
Below are values based on going through entries in the [https://logbooks.jlab.org/book/halld halld log].<br />
<br />
{| border="1" cellpadding="1" style="text-align: center;"<br />
!width="100"| Run #s<br />
!width="150"| Solenoid Current (A)<br />
!width="300"| JANA option<br />
!width="300"| notes<br />
|-<br />
| 940 - 996 || 1000 || -PBFIELD_MAP=Magnets/Solenoid/solenoid_1000A_poisson_20141104 || At run 997, the solenoid started ramping down<br />
|-<br />
| 998 - 1448 || 0 || -PBFIELD_TYPE=NoField -PDEFTAG:DTrackCandidate=StraightLine || Solenoid started ramping up around run 1431. Runs 1432 - 1448 should not be used for tracking. See [https://logbooks.jlab.org/entry/3309039 this] and [https://logbooks.jlab.org/entry/3309092 this] entry.<br />
|-<br />
| 1449 - 1620 || 1200 || -PBFIELD_MAP=Magnets/Solenoid/solenoid_1200A_poisson_20140520 || <br />
|-<br />
|}<br />
<br />
'''Note:''' The following run ranges were taken with a 300A field, and could be analyzed with the magnetic field map Magnets/Solenoid/solenoid_300A_poisson_20140819 , more study is needed to see if the straight line track fitter is needed or not: 1036 - 1053, 1065 - 1121, 1212? - 1254, 1309 - 1318 [1307,1308,1319-1329 were taken while the magnet was ramping].<br />
--><br />
<br />
=== General Information on Procedures ===<br />
This section explains how the offline monitoring should be run.<br />
Since we may want to simultaneously run offline monitoring for different run periods that require<br />
different environment variables, the scripts are set up so that a generic user can download the<br />
scripts and run them from anywhere. Most output directories for offline monitoring are created<br />
with group read/write permissions so that any Hall D group user has access to the contents,<br />
but there are some cases where use of the account that created the launch is necessary.<br />
<br />
The accounts used for offline monitoring are the gxprojN accounts created and maintained by<br />
Mark Ito (see [https://halldweb.jlab.org/wiki/index.php/GlueX-related_shared_accounts_on_the_JLab_CUE here] for how each account is used).<br />
As of October 2015, the following are used:<br />
* gxproj1 for running over Fall 2014 data (deprecated since June 2015)<br />
* gxproj5 for running over Spring 2015 data<br />
<br />
Since the summer of 2015 we have transitioned from a system using Mark Ito's jproj scripts<br />
to integrating the swif system that Chris Larrieu (SciComp) has been developing. For offline monitoring,<br />
the hdswif system that Kei developed is used for launching the jobs, and the jproj system is used<br />
for meta-analysis of launch statistics.<br />
<br />
Both hdswif and jproj are maintained in svn:<br />
* hdswif: https://halldsvn.jlab.org/repos/trunk/scripts/monitoring/hdswif<br />
* jproj : https://halldsvn.jlab.org/repos/trunk/scripts/monitoring/jproj<br />
<br />
To run the offline monitoring each package should be checked out and all necessary<br />
scripts are included.<br />
<br />
=== Preparing the software for the launch ===<br />
<br />
To begin a new launch the software must be built to the latest versions.<br />
For the gxprojN user accounts used, all software builds are contained in the directory ~/builds<br />
(which are soft links to /work/halld/home/gxprojN/builds). When logging into these accounts<br />
the setup files ~/setup_jlab-2015-03.csh or similar files should be sourced.<br />
<br />
<b>Note that Mark Ito does <u>not</u> want you to change the contents of each .cshrc file.</b><br />
You should consult him if you feel the need.<br />
<br />
1. Setup the environment: <pre>source ~/setup_jlab-2015-03.csh</pre><br />
2. Building hdds: <br />
<pre><br />
cd ~/builds/hdds/hdds<br />
git pull # Get latest software<br />
scons -c install # Clean out the old install: EXTREMELY IMPORTANT for cleaning out stale headers<br />
scons install -j4 # Rebuild and re-install with 4 threads<br />
</pre><br />
3. Building sim-recon: <br />
<pre><br />
cd ~/builds/sim-recon/sim-recon/<br />
git pull<br />
cd src<br />
scons -c install # Clean out the old install: EXTREMELY IMPORTANT for cleaning out stale headers<br />
scons install -j4 # Rebuild and re-install with 4 threads<br />
</pre><br />
4. Prepare the latest sqlite file: The sqlite is set in the ~/setup_jlab-2015-03.csh script as <b>sqlite:////home/gxproj5/ccdb.sqlite</b> through the environment variables <b>JANA_CALIB_URL</b> and <b>CCDB_CONNECTION</b>. Therefore, go to this directory and create a new sqlite file. We create the sqlite file in a temporary directory since creating the sqlite file in a directory where the output file exists causes errors. Original documentation on creating sqlite files are [https://halldweb.jlab.org/wiki/index.php/SQLite-form_of_the_CCDB_database here]. <br><span style="color:red">NOTE: SQLITE FILES DO NOT WORK ON THE NEW /work DISK INSTALLED IN OCTOBER 2015 </span><br />
<pre><br />
cd ~/tmp<br />
$CCDB_HOME/scripts/mysql2sqlite/mysql2sqlite.sh -hhallddb.jlab.org -uccdb_user ccdb | sqlite3 ccdb.sqlite<br />
mv ccdb.sqlite ../<br />
</pre><br />
5. Note that the above steps must be done <b>BEFORE</b> launch project creation. This is because we will track the revisions of the libraries used, and this is done by extracting the svn information in each directory. Also, note that the system assumes that we have the topmost build directory (usually called GLUEX_TOP) to be $HOME/builds . Such an assumption is necessary to be able to extract information about the library locations automatically.<br />
<br />
Create the appropriate project(s) and submit the jobs using hdswif, as detailed in the section below.<br />
<br />
=== Starting the Launch and Submitting Jobs ===<br />
<br />
Until the summer of 2015 we relied solely on Mark Ito's jproj system for submitting and keeping track of jobs. We have since moved to the swif system and use the hdswif wrapper for this. Below are instructions for how to use these.<br />
<br />
# Downloading hdswif: Download the hdswif directory from svn. For the gxprojN accounts, use the directory ~/halld/hdswif. <pre>cd ~/halld </pre> <pre> svn co https://halldsvn.jlab.org/repos/trunk/scripts/monitoring/hdswif</pre> <pre>cd hdswif</pre><br />
# Creating the workflow: Within SWIF jobs are registered into workflows. First create the workflow. For offline monitoring, the workflow names are of the form <b>offline_monitoring_RunPeriod201Y_MM_verVV_hd_rawdata</b> with suitable replacements for the run period and version number. The command "swif list" will list all existing workflows. Also, for most simple SWIF commands hdswif also provides a wrapper. <pre>swif list</pre> For creation of workflows for offline monitoring the command <pre>hdswif.py create [workflow] -c [config file] </pre> should be used. As an example config file, see the input.config file in the folder (and update it). When a config file is passed in, hdswif will automatically create files that record the configuration of the current launch. These files are stored as for example<br />
#* /group/halld/data_monitoring/run_conditions/jana_rawdata_comm_2015_03_ver15.conf<br />
#* /group/halld/data_monitoring/run_conditions/soft_comm_2015_03_ver15.xml<br />
#* The software packages stored in git (sim-recon and hdds) can have git tags applied to them, which makes it easier to find versions of the software than a SHA-1 hash. hdswif will ask if you would like to create a tag, and execute the following sequence: <pre>git tag -a offmon-201Y_MM-verVV -m "Used for offline monitoring 201Y-MM verVV started on 201y/mm/dd"</pre> <pre>git push offmon-201Y_MM-verVV</pre> This will only be invoked when the user name is gxprojN, and for the configuration files, the output directory will be /group/halld/data_monitoring/run_conditions/ for gxprojN accounts while it will be the current directory for other users.<br />
#* To use the git-tagged software versions do for example <pre>cd $HALLD_HOME</pre> <pre>git checkout offmon-2015_03-ver15</pre><br />
# Registering jobs in the workflow: To register jobs within the workflow, hdswif provides the use of config files. Jobs can be registered by specifying the workflow, config file (-c), run (-r) and file (-f) numbers if necessary. A typical config file will look this:<br />
PROJECT gluex<br />
TRACK reconstruction<br />
OS centos65<br />
NCORES 6<br />
DISK 40<br />
RAM 8<br />
TIMELIMIT 8<br />
JOBNAMEBASE offmon_<br />
RUNPERIOD 2015-03<br />
VERSION 15<br />
OUTPUT_TOPDIR /volatile/halld/offline_monitoring/RunPeriod-[RUNPERIOD]/ver[VERSION] # Example of other variables included in variable<br />
SCRIPTFILE /home/gxproj5/halld/hdswif/script.sh # Must specify full path<br />
ENVFILE /home/gxproj5/halld/hdswif/setup_jlab-2015-03.csh # Must specify full path<br />
The config file contains configuration parameters for each of the jobs.<br><br />
<span style="color:red">Note: Job configuration parameters can be set differently for jobs within the same workflow if necessary.</span><br />
Edit the config file and save as a new file if necessary. Once the configuration is set, jobs can be added via <pre>hdswif.py add [workflow] -c input.config</pre><br />
By default, hdswif will add all files found within the directory /mss/halld/RunPeriod-201Y-MM/rawdata/ where 201Y-MM is specified by the RUNPERIOD parameter in the config file. If only some of the runs or files are needed, these can be specified for example with<br />
<pre>hdswif.py add [workflow] -c input.config -r 3180 -f '00[0-4]'</pre><br />
to specify to register running only over run 3180 files 000 - 004 (Unix-style brackets and wildcards can be used).<br />
<br />
Running the workflow: To run the workflow, simply use swif run: <pre>swif run -workflow [workflow] -errorlimit none</pre> <span style="color:red">MAKE SURE THE ERRORLIMIT IS SET TO NONE OR THE WORKFLOW WILL BE STOPPED AFTER ANY JOB FAILS</span><br> Or equivalently, using the hdswif wrapper (which has the errorlimit set by default), <pre>hdsswif.py run [workflow]</pre><br><br />
<b>It is recommended that some jobs be tested to make sure that everything is working rather than fail thousands of jobs.</b><br><b>Check the configuration setup when creating a workflow, and what is in script.sh. Also check the following:<br />
* Check stderr files. Are they small (<kB)?<br />
* Check stdout files. Are they very large (>MB)?<br />
* Check output ROOT files. Are they larger than several MB?<br />
* Check output REST files. Are they larger than several tens of MB?<br />
</b><br />
<br> For this purpose, hdswif will take an additional parameter to run which limits the number of jobs to submit: <pre>hdswif.py run [workflow] 10</pre><br />
in which case only 10 jobs will be submitted.<br />
<br />
To submit all jobs after checking the results, do <pre>hdswif.py run [workflow]</pre><br />
<br />
=== Checking the Status and Resubmitting ===<br />
1. The status of jobs can be checked on the terminal with <pre>jobstat -u gxprojN</pre> For the status of jobs on Auger see http://scicomp.jlab.org/scicomp/#/auger/jobs and for SWIF use <pre>swif list</pre> or for more information, <pre>swif status [workflow] -summary</pre> Note that "swif status" tends to be out of date sometimes, so don't panic if your workflow/jobs aren't showing up right away. Also see the Auger [http://scicomp.jlab.org/scicomp/#/auger/jobs job website].<br />
<br />
2. For failed jobs, SWIF can resubmit jobs based on the problem. For resubmission for failed jobs with the same resources, <pre>swif retry-jobs [workflow] -problems [problem name]</pre> can be used, and for jobs to be submitted with more resources, e.g., use <pre>swif modify-jobs -ram add 2gb -problems AUGER-OVER_RLIMIT</pre> <br />
This only re-stages the jobs, be sure to resubmit them with:<br />
<pre>swif run -workflow [workflow] -errorlimit none</pre><br />
<br />
hdswif has a wrapper for both of these: <pre>hdswif.py resubmit [workflow] [problem]</pre> In this case [problem] can be one of <b>SYSTEM, TIMEOUT, RLIMIT</b>. If SYSTEM is specified, the jobs will be retried. For TIMEOUT and RLIMIT, the jobs will be modified by default with 2 additional hours or GB of RAM. If one more number is added as an option, then that many hours or GB of RAM will be added., e.g., <pre>hdswif.py resubmit [workflow] TIMEOUT 5</pre> will add 5 hours of processing time. You can wait until almost all jobs finish before resubmitting failed jobs since the number should be relatively small. <b>Even if jobs are resubmitted for one type of failure, jobs that later fail with that failure will not be automatically resubmitted.</b><br />
<br />
3. For information on swif, use the "swif help" commands and for hdswif see the attached documentaion in https://halldsvn.jlab.org/repos/trunk/scripts/monitoring/hdswif/manual_hdswif.pdf<br />
<br />
4. Below is a table describing the various errors that can occur.<br />
{| border="1" cellpadding="0" valign="left" style="text-align: left;"<br />
!width="150"| ERROR NAME<br />
!width="400"| Description<br />
!width="400"| Resolution<br />
!width="400"| hdswif command<br />
|-<br />
| AUGER-SUBMIT<br />
||<br />
SWIF’s attempt to submit jobs to Auger failed. Includes server-side problems as well as user failing to provide valid job parameters (e.g. incorrect project name, too many resources, etc.)<br />
||<br />
If requested resources are known to be correct resubmit. Otherwise modify job resources using swif directly.<br />
||<br />
<span style="color:purple"><b>hdswif.py resubmit [workflow] SYSTEM</b></span><br />
|-<br />
| AUGER-FAILED<br />
||<br />
Auger reports the job FAILED with no specific details.<br />
||<br />
Resubmit jobs. If problem persists, contact Chris Larrieu or SciComp.<br />
||<br />
<span style="color:purple"><b>hdswif.py resubmit [workflow] SYSTEM</b></span><br />
|-<br />
| AUGER-OUTPUT-FAIL<br />
||<br />
Failure to copy one or more output files.Can be due to permission problem, quota problem, system error, etc.<br />
||<br />
Check if output files will exist after job execution and that output directory exists, resubmit jobs. If problem persists, contact Chris Larrieu or SciComp.<br />
||<br />
<span style="color:purple"><b>hdswif.py resubmit [workflow] SYSTEM</b></span><br />
|-<br />
| AUGER-INPUT-FAIL<br />
||<br />
Auger failed to copy one or more of the requested input files, similar to output failures. Can also happen if tape file is unavailable (e.g. missing/damaged tape)<br />
||<br />
Check if input file exists, resubmit jobs. If problem persists, contact Chris Larrieu or SciComp.<br />
||<br />
<span style="color:purple"><b>hdswif.py resubmit [workflow] SYSTEM</b></span><br />
|-<br />
| AUGER-TIMEOUT<br />
||<br />
Job timed out.<br />
||<br />
If more time is needed for job add more resources.<br />
Default is to add 2 hrs of processing time. Also check whether code is hanging.<br />
||<br />
<span style="color:red"><b>hdswif.py resubmit [workflow] TIMEOUT</b></span><br><br />
Default is to add 2 hours. Optionally specify number of hours at end.<br />
|-<br />
| AUGER-OVER_RLIMIT<br />
||<br />
Not enough resources, RAM or disk space.<br />
||<br />
Add more resources for job.<br />
||<br />
<span style="color:blue"><b>hdswif.py resubmit [workflow] RLIMIT</b></span><br><br />
Default is to add 2 GB of RAM. Optionally specify GB at end. To add more disk space use SWIF directly.<br />
|-<br />
| SWIF-MISSING-OUTPUT<br />
||<br />
Output file specified by user was not found.<br />
||<br />
Check if output file exists at end of job.<br />
||<br />
<br />
|-<br />
| SWIF-USER-NON-ZERO <br />
||<br />
User script exited with non-zero status code.<br />
||<br />
Your script exited with non-zero status. Check the code you are running.<br />
||<br />
<br />
|-<br />
| SWIF-SYSTEM-ERROR <br />
||<br />
Job failed owing to a problem with swif (e.g. network connection timeout)<br />
||<br />
Resubmit jobs. If problem persists, contact Chris Larrieu or SciComp.<br />
||<br />
<span style="color:purple"><b>hdswif.py resubmit [workflow] SYSTEM</b></span><br />
|}<br />
<br style="clear:both;"/><br />
<br />
=== Post-analysis of statistics of the launch ===<br />
<br />
After jobs have been submitted, it will usually take a few days for all of the jobs to be processed.<br />
The next step is to check the resource usage for the current launch and publish the results online.<br />
<br />
# <b>Create summary XML, HTML files</b><br> The status and results of jobs are saved within the SWIF internal server, and are available via the command <pre>swif status [workflow] -summary -runs</pre> where the arguments -summary and -runs show summary statistics and statistics for individual jobs, respectively. hdswif has a command that takes this output in XML output and creates an HTML webpage showing results of the launch. To do this, do <pre>hdswif.py summary [workflow]</pre> This will create the XML output file from SWIF called swif_output_[workflow].xml and create a webpage containing png figure files. If the XML file already exists, hdswif will ask whether to overwrite the existing file.<br />
# <b>Publish output files online</b><br> At this stage the html output and figure files are created and ready to be put online. The html output capabilities of hdswif is useful for any user using SWIF, but since publication of the html output is specific to the offline monitoring, the script to do this is contained in the jproj scripts directory at https://halldsvn.jlab.org/repos/trunk/scripts/monitoring/jproj. For the gxprojN accounts this directory should exist as ~/halld/jproj. To publish the results online do for example <pre>python ~/halld/jproj/projects/templates/publish_offmon_results.py 2015_03 18</pre> The script simply copies the html output and corresponding figures to /group/halld/www/halldweb/html/data_monitoring/launch_analysis/<br />
# <b>Editing the summary HTML page</b><br> The top page for is offline monitoring https://halldweb.jlab.org/data_monitoring/launch_analysis/index.html and has links to the summary page for each run period. The summary files are /group/halld/www/halldweb/html/data_monitoring/launch_analysis/[run period].html Edit the file to<br />
## Add a new line to the first table which contains the version number, date, and comments for the current launch<br />
## Create a link to the webpage for the current launch. Simply copy and paste, modify the previous launch's link to have the correct launch ver.<br />
# <b>Freezing SWIF tables</b><br> Since we are now finished with the SWIF workflow that we used, the workflow should be "frozen" so that it cannot be mistakenly altered or modified. Do <pre>swif freeze [workflow]</pre><br />
# <b>Backing up SWIF output</b><br> With the workflow frozen we should be able to reproduce the XML output from SWIF, but we will backup the XML output just in case. Do <pre>cp ~/halld/hdswif/swif_output_[workflow].xml /group/halld/data_monitoring/swif_xml_backup/ </pre><br />
<br />
=== Cross Analysis of Launches ===<br />
<br />
The purpose of the cross analysis is to correlate how resource usage changed for the same files across different launches.<br />
To do this it is useful to create MySQL tables that contain information on each launch, and do queries across tables.<br />
<br />
#The scripts to do this are maintained in the svn directory https://halldsvn.jlab.org/repos/trunk/scripts/monitoring/jproj Do <pre> svn co https://halldsvn.jlab.org/repos/trunk/scripts/monitoring/jproj </pre> For the gxprojN accounts used for offline monitoring the directory should be ~/halld/jproj<br />
# The jproj directory contains two subdirectories, scripts and projects. The scripts directory contains useful scripts for processing the jobs registered in the jproj system, and each of the offline monitoring launches will be handled in the projects directory.<br />
# Go to the projects directory<pre>cd ~/halld/jproj/projects</pre> and use the script create_project.sh to create a new directory that contains the processing scripts for the current launch <pre>./create_project.sh [workflow]</pre> This should create a directory such as offline_monitoring_RunPeriod2015_03_ver15_hd_rawdata (same as the workflow name). The script uses the template files in the directory templates and by substitution creates script files for the current launch. Now go to the newly created analysis directory: <pre>cd [workflow]/analysis</pre><br />
which for the gxprojN accounts should have the full path /home/gxprojN/halld/jproj/projects/[workflow]/analysis.<br><br><br />
<span style="color:red">All of the analysis commands including arguments are contained in run_analysis.sh, but it is <b>strongly</b> recommended that all commands are run manually to check for errors.</span><br><br><br />
# Next, we need to create a few MySQL tables for the current launch. The MySQL tables are useful for comparing run/file combinations for different launches. For the SWIF launches, there are two tables needed, and will be named<br />
#* [workflow]Job<br />
#* [workflow]_aux<br />
# This naming scheme of tables and their roles are the same as from the jproj only launches. The [workflow]Job table will contain information gathered from SWIF about each job (which node it went to, start time of each stage, memory usage, etc.). The [workflow]_aux table will contain information gathered from the output stdout files from each job. First, create the Job table using <pre>python create_jproj_job_table.py [run period] [version]</pre> Note that this script uses the XML output from hdswif summary and inserts the contents into the MySQL table, so the XML output file must exist.<br />
# To check the contents of this MySQL table, do <pre>mysql -hhalldb -ufarmer farming</pre> <pre>mysql> describe offline_monitoring_RunPeriod2015_03_ver11_hd_rawdataJob;</pre><br />
# <b>Backing up tables</b><br> Tables created in MySQL should be backed up.<br />
<br />
== Hall D Job Management System ==<br />
<br />
This section details instructions on how to create and launch a set of jobs using the Hall-D Job Management System developed by Mark Ito. These instructions are generic: this system can be used for the weekly monitoring jobs, but can also be used for other sets of job launches as well. <br />
<br />
=== Database Table Overview ===<br />
<br />
* Job management database table (<project_name>): For each input file, keeps track of whether or not a job for it has been submitted, along with other optional fields. <br />
<br />
* Job status database table (<project_name>Job (no space)): For each job, keeps track of the job-id, the job status, memory used, cpu & wall time, time taken to complete various stages (e.g. pending, dependency, active), and others. <br />
<br />
* Job metrics database table (<project_name>_aux (no space)): For each job, keeps track of the job-id, how many events were processed, the time it took to copy the cache file, and the time it took to run the plugin. This information is culled from the log files of each job, and is done within the analysis directory of each launch.<br />
<br />
=== Initialize Project Management ===<br />
<br />
* Log into the ifarm machine with one of the gxproj accounts. For this example we will use gxproj1.<br />
<pre><br />
ssh gxproj1@ifarm -Y<br />
</pre><br />
<br />
* Go to a directory to do the launch. In principle, any directory will work, but for gxproj1 this is usually done in /home/gxproj1/halld/<br />
<br />
* Check out the necessary scripts <pre>svn co https://halldsvn.jlab.org/repos/trunk/scripts/monitoring/jproj</pre> This will get all necessary scripts for launching. Once checked out, <pre>cd projects</pre><br />
<br />
* The script create_project.sh can be used to create a new project. It will take a single argument, the project name. It is assumed that the project name is of the form offline_monitoring_RunPeriod20YY_MM_verVV_hd_rawdata where this string will be parsed to give the run period 20YY_MM and version number VV. One thing to do BEFORE creation of a new project is to editthe conditions of the launch (plugins to run over, memory requested, disk space requested) within templates/template.jsub . This<br />
information is saved automatically at project creation time into files in /group/halld/data_monitoring/run_conditions .<br />
To create a project do <pre>./create_project.sh offline_monitoring_RunPeriod20YY_MM_verVV_hd_rawdata</pre><br />
The name has been chosen to be as consistent as possible with other directory structures.<br />
However, mysql requires that "-" be escaped in table names, so unfortunately run periods will be<br />
given as 2YYY_MM instead of 2YYY-MM.<br />
<br />
* A new directory with the project name (offline_monitoring_RunPeriod20YY_MM_verVV_hd_rawdata) and files will be copied and modified from the template directory to reflect the run period, the user, the directory that it was created in, project name, etc.<br />
<br />
* For each project, cd into the new directory<br />
* The script clear.sh will remove any existing tables for the current project name, then recreate it. Do<br />
<pre>./clear.sh</pre><br />
* To use the jproj.pl script that was checked in, add the directory to your path with<br />
<pre>source ../../scripts/setup.csh</pre><br />
or always specify the full path<br />
<pre>../../scripts/jproj.pl</pre><br />
* Now update the table of runs with<br />
<pre>jproj.pl <project name> update</pre><br />
This will fill the table with all files within /mss that are of the same form as what is in <project name>.jproj . If you want to register only on a subset of all such files, you can edit this file directly.<br />
* Once you have registered all of the files you would like to run over, do<br />
<pre>jproj.pl <project name> submit [max # of jobs] [run number]</pre><br />
where the additional options specify how many jobs to submit and which run number to run on. Without these options all files that are registered and have not been submitted yet will be submitted.<br />
<br />
At this stage you are ready to submit all files. It is a good idea to submit a few test jobs<br />
at first to check that all scripts are working and that the plugins do not crash. Once you are<br />
sure that this does not happen, you can send all jobs in. The remaining jobs are then the monitoring<br />
which will (among other things) put the results on the online webpage for the collaboration to view, and<br />
the analysis of the launch.<br />
<br />
=== Project File Overview ===<br />
<br />
An overview of each project file:<br />
* '''clear.sh:''' For the current project, deletes the job status and management database tables (if any), and creates new, empty ones. <br />
* '''<project_name>.jproj:''' Contains the path and file name format for the input files for the jobs. <br />
* '''<project_name>.jsub:''' The xml job submission script. The run number and file number variables are set during job submission for each input file. <br />
* '''script.sh:''' The script that is executed during the job. If output job directories are not pre-created manually, they should be created in this script with the proper permissions:<br />
<pre><br />
mkdir -p -m 775 my_directory<br />
</pre><br />
* '''setup_jlab-[run period].csh:''' The environment that is sourced at the beginning of the job execution. <br />
* '''status.sh:''' Updates the job status database table, and prints some of its columns to screen.<br />
<br />
=== Project Management ===<br />
<br />
* Delete (if any) and create the database table(s) for the current set of job submissions:<br />
<pre><br />
./clear.sh<br />
</pre><br />
Also, if testing was done with jobs, it is best to delete the output directory and the configuration files:<br />
<pre><br />
rm -frv /volatile/halld/offline_monitoring/RunPeriod-20YY-MM/verVV /group/halld/data_monitoring/run_conditions/soft_comm_20YY_MM_verVV.xml /group/halld/data_monitoring/run_conditions/jana_rawdata_comm_20YY_MM_verVV.conf<br />
</pre><br />
<br />
* Search for input files matching the string in the .jproj file, and create a row for each in the job management database table (called <project_name>). You can test by adding an optional argument at the end, which only selects files with a specific file number:<br />
<pre><br />
jproj.pl <project_name> update <optional_file_number><br />
</pre><br />
<br />
* Confirm that the job management database is accurate by printing it's contents to screen:<br />
<pre><br />
mysql -hhallddb -ufarmer farming -e "select * from <project_name>"<br />
</pre><br />
<br />
* ONLY if a mistake was made, to delete the tables from the database and recreate new, empty ones, run: <br />
<pre><br />
./clear.sh<br />
</pre><br />
<br />
* To look at the status of the submitted jobs, first query auger and update the job status database:<br />
<pre><br />
fill_in_job_details.pl <project_name><br />
</pre><br />
<br />
* The job status can then be viewed by submitting a query to the job status database (called <project_name>Job (no space in between)):<br />
<pre><br />
mysql -hhallddb -ufarmer farming -e "select id,run,file,jobId,hostname,status,timeSubmitted,timeActive,walltime,cput,timeComplete,result,error from <project_name>Job"<br />
</pre><br />
<br />
* These last two commands can instead be executed simultaneously by running:<br />
<pre><br />
./status.sh<br />
</pre><br />
<br />
=== Handy mysql Instructions ===<br />
<br />
* Handy mysql instructions:<br />
<pre><br />
mysql -hhallddb -ufarmer farming # Enter the "farming" mysql database on "hallddb" as user "farmer"<br />
quit; # Exit mysql<br />
show tables; # Show a list of the tables in the current database<br />
show columns from <project_name>; # show all of the columns for the given table<br />
select * from <project_name>; # show the contents of all rows from the given table<br />
</pre><br />
<br />
=== Backing Up Offline Monitoring Tables ===<br />
Tables created for offline monitoring can be backed up using the script backup_tables.sh which<br />
can be checked out with the other files from https://halldsvn.jlab.org/repos/trunk/scripts/monitoring/jproj/projects<br />
<br />
The script uses the command mysqldump to print out a file that can be executed to recreate the tables.<br />
Since executing this output file will drop the table when it exists, caution is advised.<br />
Example usage to backup all three tables created for run period 2014_10 ver 17:<br />
<pre>backup_tables.sh 2014_10 17</pre><br />
<br />
== Running Over Data As It Comes In ==<br />
<br />
A special user gxproj1 will have a cron job set up to run the plugins as new data appears on /mss.<br />
During the week, gxproj1 will submit offline plugin jobs with the same setup as the weekly jobs<br />
run the previous Friday. The procedure for this is shown below.<br />
<br />
<!--<br />
=== Setting up the environment ===<br />
The file<br />
/home/gxproj1/setup_jlab.csh<br />
is sourced through .tcshrc.<br />
This file is the same as what is linked to by<br />
/home/gluex/setup_jlab_commissioning.csh,<br />
except HALLD_HOME, HDDS_HOME, and JANA_CALIB_URL are set separately so that this<br />
user can have a separate build.<br />
<br />
To obtain the builds from the previous Friday's runs,<br />
execute<br />
/home/gxproj1/halld/monitoring/newruns/setup_previous.sh [year] [month] [day]<br />
The build revisions from the previous Friday are archived in files<br />
/work/halld/data_monitoring/run_conditions/soft_comm_[year]_[month]_[day].xml<br />
and the script will build libraries based on those stored revision numbers.<br />
--><br />
<br />
=== Running the cron job ===<br />
<br />
'''IMPORTANT:''' The cron job should not be running while you are manually submitting jobs using the jproj.pl script for the same project, or else you will probably multiply-submit a job. <br />
<br />
* Go to the cron job directory: <br />
<pre><br />
cd /u/home/gxproj1/halld/monitoring/newruns<br />
</pre><br />
<br />
* The cron_plugins file is the cronjob that will be executed. During execution, it runs the exec.sh command in the same folder. This command takes two arguments: the project name, and the maximum file number for each run. These fields should be updated in the cron_plugins file before running. <br />
<br />
* The exec.sh command updates the job management database table with any data that has arrived on tape since it was last updated, ignoring file numbers greater than the maximum file number. It then submits jobs for these files. <br />
<br />
* To start the cron job, run:<br />
<pre><br />
crontab cron_plugins<br />
</pre><br />
<br />
* To check whether the cron job is running, do<br />
<pre><br />
crontab -l<br />
</pre><br />
<br />
* To remove the cron job do<br />
<pre><br />
crontab -r<br />
</pre><br />
<br />
<!--<br />
The cron job will run the script scan_for_jobs.sh,<br />
which runs generatejobs_plugins_rawdata.sh for any<br />
new runs that it had not seen before. All previous<br />
runs are recorded in the file filelists/files_current.txt<br />
so clear this to run over runs, or set the parameters<br />
MINRUN and MAXRUN which will set the range of runs submitted.<br />
--><br />
<br />
==Post-Processing Procedures==<br />
<br />
To visualize the monitoring data, we save images of selected histograms and store time series of selected quantities in a database, which are then displayed on a web page. This section describes how to generate the monitoring images and database information.<br />
<br />
The scripts used to generate this summary data are primarily run from /home/gxprojN/halld/monitoring/process . If you want a new copy of the scripts, e.g., for a new monitoring run, you should check the scripts out from SVN:<br />
<syntaxhighlight><br />
svn co https://halldsvn.jlab.org/repos/trunk/scripts/monitoring/process<br />
</syntaxhighlight><br />
Note that these scripts currently have some parameters which must be periodically set by hand.<br />
<br />
The default python version on most JLab machine does not have the modules to allow these scripts to connect to the MySQL database. To run these scripts, load the environment with the following command<br />
<syntaxhighlight><br />
source /home/gxproj1/halld/monitoring/process/monitoring_env.csh<br />
</syntaxhighlight><br />
<br />
===Online Monitoring===<br />
<br />
There are two primary scripts for running over the monitoring data generated by the online system and offline reconstruction. The online script can be run with either of the following commands:<br />
<syntaxhighlight><br />
/home/gluex/halld/monitoring/process/check_new_runs.py<br />
<br />
OR <br />
<br />
/home/gluex/halld/monitoring/process/check_new_runs.csh<br />
</syntaxhighlight><br />
The shell script sets up the environment properly to run the python script. To connect to the monitoring database on the JLab CUE, modules included in the local installation of python >= 2.7 are needed. The shell script is appropriate to use in a cron job. The cronjob is currently run under the "gluex" account.<br />
<br />
The online monitoring system copies a ROOT file containing the results of the online monitoring, and other configuration files into a directory accessible outside the counting house. This python script automatically checks for new ROOT files, which it will then automatically process. It contains several configuration variables that must be correctly set, which contains the location of input/output directories, etc...<br />
<br />
===Offline Monitoring===<br />
<br />
After the data is run over, the results should be processed, so that summary data is entered into the monitoring database and plots are made for the monitoring webpages. Currently, this processing is controlled by a cronjob that runs the following script:<br />
<syntaxhighlight><br />
/home/gxproj1/halld/monitoring/process/check_monitoring_data.csh <br />
</syntaxhighlight><br />
This script checks for new ROOT files, and only runs over those it hasn't processed yet. Since one monitoring ROOT file is produced for each EVIO file, whenever a new file is produced, the plots for the corresponding run are recreated and all the ROOT files for a run are combined into one file. Information is stored in the database on a per-file basis. <br />
<br />
Plots for the monitoring web page can be made from single histograms or multiple histograms using RootSpy macros. If you want to change the list of plots made, you must modify one of the following files:<br />
* histograms_to_monitor - specify either the name of the histogram or its the full ROOT path <br />
* macros_to_monitor - specify the full path to the RootSpy macro .C file<br />
<br />
When a new monitoring run is started, or the conditions are changed, the following steps should be taken to process the new files:<br />
# Add a new data version, as described below:<br />
# Change the following parameters in check_monitoring_data.csh:<br />
## JOBDATE should correspond to the ouptut date used by the job submission script<br />
## OUTPUTDIR should correspond to the directory corresponding to the run period and revision corresponding to the new version you just submitted. Presumably, this directory will be empty at the beginning.<br />
## Once you create a new data version as defined below, you should pass the needed information as a command line option. Currently this is done by the ARGS variable. For example, the argument "-v RunPeriod-2014-10,8" tells the monitoring scripts to look up the version corresponding to revision 8 of RunPeriod-2014-10 in the monitoring DB and to use to store the results.<br />
<br />
<syntaxhighlight><br />
Example configuration parameters:<br />
set JOBDATE=2015-01-09<br />
set INPUTDIR=/volatile/halld/RunPeriod-2014-10/offline_monitoring<br />
set OUTPUTDIR=/w/halld-scifs1a/data_monitoring/RunPeriod-2014-10/ver08<br />
set ARGS=" -v RunPeriod-2014-10,8 "<br />
</syntaxhighlight><br />
If you want to process the results manually, the data is processed using the following script:<br />
<syntaxhighlight><br />
./process_new_offline_data.py <input directory> <output directory><br />
<br />
EXAMPLE:<br />
<br />
./process_new_offline_data.py 2014-11-14 /volatile/halld/RunPeriod-2014-10/offline_monitoring/ /w/halld-scifs1a/data_monitoring/RunPeriod-2014-10/ver02<br />
</syntaxhighlight><br />
The python script takes several options to enable/disable various steps in the processing. Of interest is the "--force" option, which will run over all monitoring ROOT files, whether or not they've been previously identified.<br />
<br />
Every time a new reconstruction pass is performed, a new version number must be generated. To do this, prepare a version file as described below. Then run the register_new_version.py script to store the information in the database. The script will return a version number, which then should be set by hand in process_new_offline_data.py - future versions of the script will streamline this part of the procedure. An example of how to generate a new version is:<br />
<syntaxhighlight><br />
./register_new_version.py add /home/gxproj1/halld/monitoring/process/versions/vers_RunPeriod-2014-10_pass1.txt<br />
</syntaxhighlight><br />
<br />
<b>If you are running the offline monitoring by checking out the files in trunk/scripts/monitoring/jproj/projects/<br />
of the svn repository</b>, and created a project with<br />
create_project.sh [project name] hd_rawdata<br />
Then go to the directory [project name]/processing/<br />
and execute<br />
./run_processing.sh<br />
which will run register_new_version.py as well as check_monitoring_data.csh for that project.<br />
<br />
===Step-by-Step Instructions For Processing a New Monitoring Run===<br />
<br />
The monitoring runs are current run out of the gxproj1 and gxproj5 accounts. After an offline monitoring run has been successfully started on the batch farm, the following steps should be followed to setup the post-processing for these runs.<br />
<br />
# The post-processing scripts are stored in $HOME/halld/monitoring/process and are automatically run by cron.<br />
# Run "svn update" to bring any changes in. Be sure that the list of histograms and macros to plot are current.<br />
# Add a new data version<br />
# Edit check_monitoring_data.csh to point to the current revisions/directories<br />
#* VERSION<br />
#* ARGS<br />
#* Note that the environment depends on a standard script - $HOME/setup_jlab.csh<br />
# Update files in the web directory, so that the results are displayed on the web pages: /group/halld/www/halldweb/html/data_monitoring/textdata<br />
# Copy the REST files to more permanent locations:<br />
#* cp -a /volatile/halld/offline_monitoring/RunPeriod-YYYY-MM/verVV/REST /work/halld/data_monitoring/RunPeriod-YYYY-MM/REST/verVV<br />
#* cp -a /volatile/halld/offline_monitoring/RunPeriod-YYYY-MM/verVV/REST /cache/halld/RunPeriod-YYYY-MM/REST/verVV [under testing]<br />
<br />
Check log files in $HOME/halld/monitoring/process/log for more information on how each run went. If there are problems, check log files, and modify check_monitoring_data.csh to vary the verbosity of the output.<br />
<br />
==Data Versions==<br />
<br />
To document the conditions of the monitoring data that is created, for the sake of reproducability and further analysis we save several pieces of information. The format is intended to be comprehensive enough to document not just monitoring data, but versions of raw and reconstructed data, so that this database table can be used for the event database as well.<br />
<br />
We store one record per pass through one run period, with the following structure:<br />
<br />
{| class="wikitable"<br />
! Field !! Description<br />
|-<br />
| data_type || The level of data we are processing. For the purposes of monitoring, "rawdata" is the online monitoring, "recon" is the offline monitoring<br />
|- <br />
| run_period || The run period of the data<br />
|- <br />
| revision || An integer specifying which pass through the run period this data corresponds to<br />
|- <br />
| software_version || The name of the XML file that specifies the different software versions used<br />
|-<br />
| jana_config || The name of the text file that specifies which JANA options were passed to the reconstruction program<br />
|-<br />
| ccdb_context || The value of JANA_CALIB_CONTEXT, which specifies the version of calibration constants that were used<br />
|-<br />
| production_time || The data at which monitoring/reconstruction began<br />
|-<br />
| dataVersionString || A convenient string for identifying this version of the data<br />
|}<br />
<br />
<br />
An example file used as as input to ./register_new_version.py is:<br />
<syntaxhighlight><br />
data_type = recon<br />
run_period = RunPeriod-2014-10<br />
revision = 1<br />
software_version = soft_comm_2014_11_06.xml<br />
jana_config = jana_rawdata_comm_2014_11_06.conf<br />
ccdb_context = calibtime=2014-11-10<br />
production_time = 2014-11-10<br />
dataVersionString = recon_RunPeriod-2014-10_20141110_ver01<br />
</syntaxhighlight></div>Kmoriyahttps://halldweb.jlab.org/wiki/index.php?title=GlueX_Offline_Meeting,_November_11,_2015&diff=71291GlueX Offline Meeting, November 11, 20152015-11-11T18:27:44Z<p>Kmoriya: /* Agenda */</p>
<hr />
<div>GlueX Offline Software Meeting<br><br />
Wednesday, November 11, 2015<br><br />
1:30 pm EST<br><br />
JLab: CEBAF Center F326/327<br />
<br />
==Agenda==<br />
<br />
# Announcements<br />
## New work disk: [https://mailman.jlab.org/pipermail/halld-offline/2015-November/002162.html /work/halld2]<br />
## [https://mailman.jlab.org/mailman/private/gluex-collaboration/2015-November/004139.html Private Wiki] open for business<br />
# Review of [[GlueX Offline Meeting, October 28, 2015#Minutes|minutes from October 28]] (all)<br />
# [https://halldweb.jlab.org/wiki/images/a/ab/2015-11-11-offline_monitoring.pdf Offline Monitoring] (Kei)<br />
# Geant4 Update (Richard, David)<br />
# [[Data Challenge 3]] update (Mark)<br />
# [[Sim1 Conditions|Future Commissioning Simulations]] (all)<br />
# [https://halldweb.jlab.org/talks/2015/fetch-dist_111115.pdf Binary Distributions of GlueX Software] (Nathan)<br />
# [[Automatic Tests of GlueX Software|b1pi results review]]<br />
# Review of [https://github.com/JeffersonLab/sim-recon/pulls?q=is%3Aopen+is%3Apr recent pull requests]<br />
# Action Item Review<br />
<br />
==Communication Information==<br />
<br />
===Remote Connection===<br />
<br />
* The BlueJeans meeting number is 968 592 007 .<br />
* [http://bluejeans.com/968592007 Join the Meeting] via BlueJeans<br />
<br />
===Slides===<br />
<br />
Talks can be deposited in the directory <code>/group/halld/www/halldweb/html/talks/2015</code> on the JLab CUE. This directory is accessible from the web at https://halldweb.jlab.org/talks/2015/ .</div>Kmoriyahttps://halldweb.jlab.org/wiki/index.php?title=File:2015-11-11-offline_monitoring.pdf&diff=71290File:2015-11-11-offline monitoring.pdf2015-11-11T18:27:25Z<p>Kmoriya: Talk by Kei on offline monitoring for offline meeting 2015/11/11</p>
<hr />
<div>Talk by Kei on offline monitoring for offline meeting 2015/11/11</div>Kmoriyahttps://halldweb.jlab.org/wiki/index.php?title=Data_Monitoring_Procedures&diff=71259Data Monitoring Procedures2015-11-09T15:30:39Z<p>Kmoriya: /* Offline Monitoring: Running Over Archived Data */</p>
<hr />
<div>__TOC__<br />
<br />
== Master List of File / Database / Webpage Locations ==<br />
=== Run Conditions ===<br />
* Online Run-by-run condition files (B-field, current, etc.): /work/halld/online_monitoring/conditions/<br />
* Offline monitoring run conditions (software versions, jana config): /group/halld/data_monitoring/run_conditions/<br />
*[http://www.jlab.org/Hall-D/test/RunInfo/ Run Info vers. 1]<br />
*[https://halldweb.jlab.org/cgi-bin/data_monitoring/run_conditions.pl Run Info vers. 2]<br />
<br />
=== Monitoring Output Files ===<br />
* Run Periods 201Y-MM is for example 2015-03, launch ver verVV is for example ver15<br />
* Online monitoring histograms: /work/halld/online_monitoring/root/<br />
* Offline monitoring histogram ROOT files: /work/halld/data_monitoring/RunPeriod-201Y-MM/verVV/rootfiles<br />
* REST files (most recent launch only): /work/halld/data_monitoring/RunPeriod-201Y-MM/REST/verVV<br />
* individual files for each job (ROOT, log, etc.): /volatile/halld/offline_monitoring/RunPeriod-201Y-MM/verVV/<br />
<br />
=== Monitoring Database ===<br />
* Accessing monitoring database (on ifarm): mysql -u datmon -h hallddb.jlab.org data_monitoring<br />
<br />
=== Monitoring Webpages ===<br />
*[https://halldweb.jlab.org/cgi-bin/data_monitoring/monitoring/plotBrowser.py Plot Browser]<br />
*[https://halldweb.jlab.org/cgi-bin/data_monitoring/monitoring/runBrowser.py Run Browser]<br />
*[https://halldweb.jlab.org/cgi-bin/data_monitoring/monitoring/timeSeries.py Time Series]<br />
*[https://halldweb.jlab.org/data_monitoring/launch_analysis/ Launch Analysis]<br />
<br />
== Job Monitoring Links ==<br />
* [http://scicomp.jlab.org/farm2/job.html Completed Job History]<br />
* [http://scicomp.jlab.org/farm2/project.html Job Stats By Project]<br />
* [http://scicomp.jlab.org/farm2/trackOrg.html Job Stats By Track]<br />
* [http://scicomp.jlab.org/farm2/report.html Cluster Report]<br />
* [http://scicomp.jlab.org/farm2/walltime.html Walltime Distribution]<br />
<br />
== Saving Online Monitoring Data ==<br />
<br />
The procedure for writing the data out is given in, e.g.,<br />
[https://halldweb.jlab.org/wiki/index.php/Raid-to-Silo_Transfer_Strategy Raid-to-Silo Transfer Strategy].<br />
<br />
Once the DAQ writes out the data to the raid disk, cron jobs will copy the file to tape,<br />
and within ~20 min., we will have access to the file on tape at<br />
/mss/halld/$RUN_PERIOD/rawdata/RunXXXXXX.<br />
<br />
All online monitoring plugins will be run as data is taken.<br />
They will be accessible within the counting house via RootSpy, and<br />
for each run and file, a ROOT file containing the histograms will be saved<br />
within a subdirectory for each run.<br />
<br />
For immediate access to these files, the raid disk files may be accessed directly<br />
from the counting house, or the tape files will be available within ~20 min. of the<br />
file being written out.<br />
<br />
== Offline Monitoring: Running Over Archived Data ==<br />
<br />
Once files are written to tape we run the online plugins on these files to confirm what we were seeing in the online monitoring, and also to update the results from the latest calibration and software. <br />
Manual scripts and cron jobs are set up to look for new data and run the plugins over a sample of files.<br />
<br />
Every other Friday (usually the Friday before the offline meetings) jobs will be started to run the newest software on all previous runs,and allows everybody to see improvements in each detector.<br />
For each launch, independent builds of hdds, sim-recon, the monitoring plugins, and an sqlite file will be generated.<br />
<br />
Below the procedures are described for<br />
# Preparing the software for the launch<br />
# Starting the launch (using hdswif)<br />
# Post-analysis of statistics of the launch<br />
<br />
Processing the results and making them available to the collaboration<br />
is handled in the section [[#Post-Processing_Procedures | Post-Processing Procedures]] below.<br />
<br />
<!--<br />
==== Generating an offline plugin job ====<br />
<br />
The user gxproj1 should be used for official offline monitoring jobs.<br />
Within /home/gxproj1/halld/monitoring/batch/ there will be scripts to run the online monitoring plugins over tape files.<br />
The main script is generatejobs_plugins_rawdata.sh, which can be used as<br />
generatejobs_plugins_rawdata.sh [minrun] [maxrun] (minfile) (maxfile)<br />
where minrun, maxrun specify the range of the run #, and minfile, maxfile (optional) specify<br />
the file range for that run.<br />
<br />
This will generate a script run_rawdata_XXXXXX.sh, where the run # has now been formatted to be 6 digits.<br />
Executing this script will send the monitoring plugins job to the Auger batch system.<br />
<br />
There is also a script clean.sh which can be used as<br />
clean.sh XXX<br />
This will clean up all associated files created in association with run XXX.<br />
<br />
Internally, the xml file used to submit the job will be created, and the job to run<br />
will be given within script.sh. All run parameters should be specified in at the beginning<br />
of generatejobs_plugins_rawdata.sh<br />
Since we are running on tape, the tape file will first be copied over to the cache disk, and the job will run<br />
over this cached file.<br />
<br />
==== Using cron to run automatically ====<br />
Within /home/gluex/halld/monitoring/cron/ there is a file cron_plugins<br />
that can be executed via<br />
crontab cron_plugins<br />
This will set up a cron job to call the script scan_for_jobs.sh, which will<br />
check in the rawdata directory and call generatejobs_plugins_rawdata.sh for<br />
any run that is more than 5 min old. The cron job is set up to run every 10 min.<br />
<br />
==== Magnetic Field Settings ====<br />
For tracking, it is necessary to set the correct magnetic field settings.<br />
The below table is now obsolete. Please refer to the [https://halldweb.jlab.org/cgi-bin/data_monitoring/run_conditions.pl GlueX Run Conditions] webpage.<br />
<br />
The actual field settings for each run have not been documented well.<br />
Below are values based on going through entries in the [https://logbooks.jlab.org/book/halld halld log].<br />
<br />
{| border="1" cellpadding="1" style="text-align: center;"<br />
!width="100"| Run #s<br />
!width="150"| Solenoid Current (A)<br />
!width="300"| JANA option<br />
!width="300"| notes<br />
|-<br />
| 940 - 996 || 1000 || -PBFIELD_MAP=Magnets/Solenoid/solenoid_1000A_poisson_20141104 || At run 997, the solenoid started ramping down<br />
|-<br />
| 998 - 1448 || 0 || -PBFIELD_TYPE=NoField -PDEFTAG:DTrackCandidate=StraightLine || Solenoid started ramping up around run 1431. Runs 1432 - 1448 should not be used for tracking. See [https://logbooks.jlab.org/entry/3309039 this] and [https://logbooks.jlab.org/entry/3309092 this] entry.<br />
|-<br />
| 1449 - 1620 || 1200 || -PBFIELD_MAP=Magnets/Solenoid/solenoid_1200A_poisson_20140520 || <br />
|-<br />
|}<br />
<br />
'''Note:''' The following run ranges were taken with a 300A field, and could be analyzed with the magnetic field map Magnets/Solenoid/solenoid_300A_poisson_20140819 , more study is needed to see if the straight line track fitter is needed or not: 1036 - 1053, 1065 - 1121, 1212? - 1254, 1309 - 1318 [1307,1308,1319-1329 were taken while the magnet was ramping].<br />
--><br />
<br />
=== General Information on Procedures ===<br />
This section explains how the offline monitoring should be run.<br />
Since we may want to simultaneously run offline monitoring for different run periods that require<br />
different environment variables, the scripts are set up so that a generic user can download the<br />
scripts and run them from anywhere. Most output directories for offline monitoring are created<br />
with group read/write permissions so that any Hall D group user has access to the contents,<br />
but there are some cases where use of the account that created the launch is necessary.<br />
<br />
The accounts used for offline monitoring are the gxprojN accounts created and maintained by<br />
Mark Ito (see [https://halldweb.jlab.org/wiki/index.php/GlueX-related_shared_accounts_on_the_JLab_CUE here] for how each account is used).<br />
As of October 2015, the following are used:<br />
* gxproj1 for running over Fall 2014 data (deprecated since June 2015)<br />
* gxproj5 for running over Spring 2015 data<br />
<br />
Since the summer of 2015 we have transitioned from a system using Mark Ito's jproj scripts<br />
to integrating the swif system that Chris Larrieu (SciComp) has been developing. For offline monitoring,<br />
the hdswif system that Kei developed is used for launching the jobs, and the jproj system is used<br />
for meta-analysis of launch statistics.<br />
<br />
Both hdswif and jproj are maintained in svn:<br />
* hdswif: https://halldsvn.jlab.org/repos/trunk/scripts/monitoring/hdswif<br />
* jproj : https://halldsvn.jlab.org/repos/trunk/scripts/monitoring/jproj<br />
<br />
To run the offline monitoring each package should be checked out and all necessary<br />
scripts are included.<br />
<br />
=== Preparing the software for the launch ===<br />
<br />
To begin a new launch the software must be built to the latest versions.<br />
For the gxprojN user accounts used, all software builds are contained in the directory ~/builds<br />
(which are soft links to /work/halld/home/gxprojN/builds). When logging into these accounts<br />
the setup files ~/setup_jlab-2015-03.csh or similar files should be sourced.<br />
<br />
<b>Note that Mark Ito does <u>not</u> want you to change the contents of each .cshrc file.</b><br />
You should consult him if you feel the need.<br />
<br />
1. Setup the environment: <pre>source ~/setup_jlab-2015-03.csh</pre><br />
2. Building hdds: <br />
<pre><br />
cd ~/builds/hdds/hdds<br />
git pull # Get latest software<br />
scons -c install # Clean out the old install: EXTREMELY IMPORTANT for cleaning out stale headers<br />
scons install -j4 # Rebuild and re-install with 4 threads<br />
</pre><br />
3. Building sim-recon: <br />
<pre><br />
cd ~/builds/sim-recon/sim-recon/<br />
git pull<br />
cd src<br />
scons -c install # Clean out the old install: EXTREMELY IMPORTANT for cleaning out stale headers<br />
scons install -j4 # Rebuild and re-install with 4 threads<br />
</pre><br />
4. Prepare the latest sqlite file: The sqlite is set in the ~/setup_jlab-2015-03.csh script as <b>sqlite:////home/gxproj5/ccdb.sqlite</b> through the environment variables <b>JANA_CALIB_URL</b> and <b>CCDB_CONNECTION</b>. Therefore, go to this directory and create a new sqlite file. We create the sqlite file in a temporary directory since creating the sqlite file in a directory where the output file exists causes errors. Original documentation on creating sqlite files are [https://halldweb.jlab.org/wiki/index.php/SQLite-form_of_the_CCDB_database here]. <br><span style="color:red">NOTE: SQLITE FILES DO NOT WORK ON THE NEW /work DISK INSTALLED IN OCTOBER 2015 </span><br />
<pre><br />
cd ~/tmp<br />
$CCDB_HOME/scripts/mysql2sqlite/mysql2sqlite.sh -hhallddb.jlab.org -uccdb_user ccdb | sqlite3 ccdb.sqlite<br />
mv ccdb.sqlite ../<br />
</pre><br />
5. Note that the above steps must be done <b>BEFORE</b> launch project creation. This is because we will track the revisions of the libraries used, and this is done by extracting the svn information in each directory. Also, note that the system assumes that we have the topmost build directory (usually called GLUEX_TOP) to be $HOME/builds . Such an assumption is necessary to be able to extract information about the library locations automatically.<br />
<br />
Create the appropriate project(s) and submit the jobs using hdswif, as detailed in the section below.<br />
<br />
=== Starting the Launch and Submitting Jobs ===<br />
<br />
Until the summer of 2015 we relied solely on Mark Ito's jproj system for submitting and keeping track of jobs. We have since moved to the swif system and use the hdswif wrapper for this. Below are instructions for how to use these.<br />
<br />
# Downloading hdswif: Download the hdswif directory from svn. For the gxprojN accounts, use the directory ~/halld/hdswif. <pre>cd ~/halld </pre> <pre> svn co https://halldsvn.jlab.org/repos/trunk/scripts/monitoring/hdswif</pre> <pre>cd hdswif</pre><br />
# Creating the workflow: Within SWIF jobs are registered into workflows. First create the workflow. For offline monitoring, the workflow names are of the form <b>offline_monitoring_RunPeriod201Y_MM_verVV_hd_rawdata</b> with suitable replacements for the run period and version number. The command "swif list" will list all existing workflows. Also, for most simple SWIF commands hdswif also provides a wrapper. <pre>swif list</pre> For creation of workflows for offline monitoring the command <pre>hdswif.py create [workflow] -c [config file] </pre> should be used. As an example config file, see the input.config file in the folder (and update it). When a config file is passed in, hdswif will automatically create files that record the configuration of the current launch. These files are stored as for example<br />
#* /group/halld/data_monitoring/run_conditions/jana_rawdata_comm_2015_03_ver15.conf<br />
#* /group/halld/data_monitoring/run_conditions/soft_comm_2015_03_ver15.xml<br />
#* The software packages stored in git (sim-recon and hdds) can have git tags applied to them, which makes it easier to find versions of the software than a SHA-1 hash. hdswif will ask if you would like to create a tag, and execute the following sequence: <pre>git tag -a offmon-201Y_MM-verVV -m "Used for offline monitoring 201Y-MM verVV started on 201y/mm/dd"</pre> <pre>git push offmon-201Y_MM-verVV</pre> This will only be invoked when the user name is gxprojN, and for the configuration files, the output directory will be /group/halld/data_monitoring/run_conditions/ for gxprojN accounts while it will be the current directory for other users.<br />
#* To use the git-tagged software versions do for example <pre>cd $HALLD_HOME</pre> <pre>git checkout offmon-2015_03-ver15</pre><br />
# Registering jobs in the workflow: To register jobs within the workflow, hdswif provides the use of config files. Jobs can be registered by specifying the workflow, config file (-c), run (-r) and file (-f) numbers if necessary. A typical config file will look this:<br />
PROJECT gluex<br />
TRACK reconstruction<br />
OS centos65<br />
NCORES 6<br />
DISK 40<br />
RAM 8<br />
TIMELIMIT 8<br />
JOBNAMEBASE offmon_<br />
RUNPERIOD 2015-03<br />
VERSION 15<br />
OUTPUT_TOPDIR /volatile/halld/offline_monitoring/RunPeriod-[RUNPERIOD]/ver[VERSION] # Example of other variables included in variable<br />
SCRIPTFILE /home/gxproj5/halld/hdswif/script.sh # Must specify full path<br />
ENVFILE /home/gxproj5/halld/hdswif/setup_jlab-2015-03.csh # Must specify full path<br />
The config file contains configuration parameters for each of the jobs.<br><br />
<span style="color:red">Note: Job configuration parameters can be set differently for jobs within the same workflow if necessary.</span><br />
Edit the config file and save as a new file if necessary. Once the configuration is set, jobs can be added via <pre>hdswif.py add [workflow] -c input.config</pre><br />
By default, hdswif will add all files found within the directory /mss/halld/RunPeriod-201Y-MM/rawdata/ where 201Y-MM is specified by the RUNPERIOD parameter in the config file. If only some of the runs or files are needed, these can be specified for example with<br />
<pre>hdswif.py add [workflow] -c input.config -r 3180 -f '00[0-4]'</pre><br />
to specify to register running only over run 3180 files 000 - 004 (Unix-style brackets and wildcards can be used).<br />
<br />
Running the workflow: To run the workflow, simply use swif run: <pre>swif run -workflow [workflow] -errorlimit none</pre> <span style="color:red">MAKE SURE THE ERRORLIMIT IS SET TO NONE OR THE WORKFLOW WILL BE STOPPED AFTER ANY JOB FAILS</span><br> Or equivalently, using the hdswif wrapper (which has the errorlimit set by default), <pre>hdsswif.py run [workflow]</pre><br><br />
<b>It is recommended that some jobs be tested to make sure that everything is working rather than fail thousands of jobs.</b><br><b>Check the configuration setup when creating a workflow, and what is in script.sh. Also check the following:<br />
* Check stderr files. Are they small (<kB)?<br />
* Check stdout files. Are they very large (>MB)?<br />
* Check output ROOT files. Are they larger than several MB?<br />
* Check output REST files. Are they larger than several tens of MB?<br />
</b><br />
<br> For this purpose, hdswif will take an additional parameter to run which limits the number of jobs to submit: <pre>hdswif.py run [workflow] 10</pre><br />
in which case only 10 jobs will be submitted.<br />
<br />
To submit all jobs after checking the results, do <pre>hdswif.py run [workflow]</pre><br />
<br />
=== Checking the Status and Resubmitting ===<br />
1. The status of jobs can be checked on the terminal with <pre>jobstat -u gxprojN</pre> For the status of jobs on Auger see http://scicomp.jlab.org/scicomp/#/auger/jobs and for SWIF use <pre>swif list</pre> or for more information, <pre>swif status [workflow] -summary</pre> Note that "swif status" tends to be out of date sometimes, so don't panic if your workflow/jobs aren't showing up right away. Also see the Auger [http://scicomp.jlab.org/scicomp/#/auger/jobs job website].<br />
<br />
2. For failed jobs, SWIF can resubmit jobs based on the problem. For resubmission for failed jobs with the same resources, <pre>swif retry-jobs [workflow] -problems [problem name]</pre> can be used, and for jobs to be submitted with more resources, e.g., use <pre>swif modify-jobs -ram add 2gb -problems AUGER-OVER_RLIMIT</pre> <br />
This only re-stages the jobs, be sure to resubmit them with:<br />
<pre>swif run -workflow [workflow] -errorlimit none</pre><br />
<br />
hdswif has a wrapper for both of these: <pre>hdswif.py resubmit [workflow] [problem]</pre> In this case [problem] can be one of <b>SYSTEM, TIMEOUT, RLIMIT</b>. If SYSTEM is specified, the jobs will be retried. For TIMEOUT and RLIMIT, the jobs will be modified by default with 2 additional hours or GB of RAM. If one more number is added as an option, then that many hours or GB of RAM will be added., e.g., <pre>hdswif.py resubmit [workflow] TIMEOUT 5</pre> will add 5 hours of processing time. You can wait until almost all jobs finish before resubmitting failed jobs since the number should be relatively small. <b>Even if jobs are resubmitted for one type of failure, jobs that later fail with that failure will not be automatically resubmitted.</b><br />
<br />
3. For information on swif, use the "swif help" commands and for hdswif see the attached documentaion in https://halldsvn.jlab.org/repos/trunk/scripts/monitoring/hdswif/manual_hdswif.pdf<br />
<br />
4. Below is a table describing the various errors that can occur.<br />
{| border="1" cellpadding="0" valign="left" style="text-align: left;"<br />
!width="150"| ERROR NAME<br />
!width="400"| Description<br />
!width="400"| Resolution<br />
!width="400"| hdswif command<br />
|-<br />
| AUGER-SUBMIT<br />
||<br />
SWIF’s attempt to submit jobs to Auger failed. Includes server-side problems as well as user failing to provide valid job parameters (e.g. incorrect project name, too many resources, etc.)<br />
||<br />
If requested resources are known to be correct resubmit. Otherwise modify job resources using swif directly.<br />
||<br />
<span style="color:purple"><b>hdswif.py resubmit [workflow] SYSTEM</b></span><br />
|-<br />
| AUGER-FAILED<br />
||<br />
Auger reports the job FAILED with no specific details.<br />
||<br />
Resubmit jobs. If problem persists, contact Chris Larrieu or SciComp.<br />
||<br />
<span style="color:purple"><b>hdswif.py resubmit [workflow] SYSTEM</b></span><br />
|-<br />
| AUGER-OUTPUT-FAIL<br />
||<br />
Failure to copy one or more output files.Can be due to permission problem, quota problem, system error, etc.<br />
||<br />
Check if output files will exist after job execution and that output directory exists, resubmit jobs. If problem persists, contact Chris Larrieu or SciComp.<br />
||<br />
<span style="color:purple"><b>hdswif.py resubmit [workflow] SYSTEM</b></span><br />
|-<br />
| AUGER-INPUT-FAIL<br />
||<br />
Auger failed to copy one or more of the requested input files, similar to output failures. Can also happen if tape file is unavailable (e.g. missing/damaged tape)<br />
||<br />
Check if input file exists, resubmit jobs. If problem persists, contact Chris Larrieu or SciComp.<br />
||<br />
<span style="color:purple"><b>hdswif.py resubmit [workflow] SYSTEM</b></span><br />
|-<br />
| AUGER-TIMEOUT<br />
||<br />
Job timed out.<br />
||<br />
If more time is needed for job add more resources.<br />
Default is to add 2 hrs of processing time. Also check whether code is hanging.<br />
||<br />
<span style="color:red"><b>hdswif.py resubmit [workflow] TIMEOUT</b></span><br><br />
Default is to add 2 hours. Optionally specify number of hours at end.<br />
|-<br />
| AUGER-OVER_RLIMIT<br />
||<br />
Not enough resources, RAM or disk space.<br />
||<br />
Add more resources for job.<br />
||<br />
<span style="color:blue"><b>hdswif.py resubmit [workflow] RLIMIT</b></span><br><br />
Default is to add 2 GB of RAM. Optionally specify GB at end. To add more disk space use SWIF directly.<br />
|-<br />
| SWIF-MISSING-OUTPUT<br />
||<br />
Output file specified by user was not found.<br />
||<br />
Check if output file exists at end of job.<br />
||<br />
<br />
|-<br />
| SWIF-USER-NON-ZERO <br />
||<br />
User script exited with non-zero status code.<br />
||<br />
Your script exited with non-zero status. Check the code you are running.<br />
||<br />
<br />
|-<br />
| SWIF-SYSTEM-ERROR <br />
||<br />
Job failed owing to a problem with swif (e.g. network connection timeout)<br />
||<br />
Resubmit jobs. If problem persists, contact Chris Larrieu or SciComp.<br />
||<br />
<span style="color:purple"><b>hdswif.py resubmit [workflow] SYSTEM</b></span><br />
|}<br />
<br style="clear:both;"/><br />
<br />
=== Post-analysis of statistics of the launch ===<br />
<br />
After jobs have been submitted, it will usually take a few days for all of the jobs to be processed.<br />
The next step is to check the resource usage for the current launch and publish the results online.<br />
<br />
# <b>Create summary XML, HTML files</b><br> The status and results of jobs are saved within the SWIF internal server, and are available via the command <pre>swif status [workflow] -summary -runs</pre> where the arguments -summary and -runs show summary statistics and statistics for individual jobs, respectively. hdswif has a command that takes this output in XML output and creates an HTML webpage showing results of the launch. To do this, do <pre>hdswif.py summary [workflow]</pre> This will create the XML output file from SWIF called swif_output_[workflow].xml and create a webpage containing png figure files. If the XML file already exists, hdswif will ask whether to overwrite the existing file.<br />
# <b>Publish output files online</b><br> At this stage the html output and figure files are created and ready to be put online. The html output capabilities of hdswif is useful for any user using SWIF, but since publication of the html output is specific to the offline monitoring, the script to do this is contained in the jproj scripts directory at https://halldsvn.jlab.org/repos/trunk/scripts/monitoring/jproj. For the gxprojN accounts this directory should exist as ~/halld/jproj. To publish the results online do for example <pre>python ~/halld/jproj/projects/templates/publish_offmon_results.py 2015_03 18</pre> The script simply copies the html output and corresponding figures to /group/halld/www/halldweb/html/data_monitoring/launch_analysis/<br />
# <b>Editing the summary HTML page</b><br> The top page for is offline monitoring https://halldweb.jlab.org/data_monitoring/launch_analysis/index.html and has links to the summary page for each run period. The summary files are /group/halld/www/halldweb/html/data_monitoring/launch_analysis/[run period].html Edit the file to<br />
## Add a new line to the first table which contains the version number, date, and comments for the current launch<br />
## Create a link to the webpage for the current launch. Simply copy and paste, modify the previous launch's link to have the correct launch ver.<br />
# <b>Freezing SWIF tables</b><br> Since we are now finished with the SWIF workflow that we used, the workflow should be "frozen" so that it cannot be mistakenly altered or modified. Do <pre>swif freeze [workflow]</pre><br />
=== Cross Analysis of Launches ===<br />
<br />
The purpose of the cross analysis is to correlate how resource usage changed for the same files across different launches.<br />
To do this it is useful to create MySQL tables that contain information on each launch, and do queries across tables.<br />
<br />
#The scripts to do this are maintained in the svn directory https://halldsvn.jlab.org/repos/trunk/scripts/monitoring/jproj Do <pre> svn co https://halldsvn.jlab.org/repos/trunk/scripts/monitoring/jproj </pre> For the gxprojN accounts used for offline monitoring the directory should be ~/halld/jproj<br />
# The jproj directory contains two subdirectories, scripts and projects. The scripts directory contains useful scripts for processing the jobs registered in the jproj system, and each of the offline monitoring launches will be handled in the projects directory.<br />
# Go to the projects directory<pre>cd ~/halld/jproj/projects</pre> and use the script create_project.sh to create a new directory that contains the processing scripts for the current launch <pre>./create_project.sh [workflow]</pre> This should create a directory such as offline_monitoring_RunPeriod2015_03_ver15_hd_rawdata (same as the workflow name). The script uses the template files in the directory templates and by substitution creates script files for the current launch. Now go to the newly created analysis directory: <pre>cd [workflow]/analysis</pre><br />
which for the gxprojN accounts should have the full path /home/gxprojN/halld/jproj/projects/[workflow]/analysis.<br><br><br />
<span style="color:red">All of the analysis commands including arguments are contained in run_analysis.sh, but it is <b>strongly</b> recommended that all commands are run manually to check for errors.</span><br><br><br />
# Next, we need to create a few MySQL tables for the current launch. The MySQL tables are useful for comparing run/file combinations for different launches. For the SWIF launches, there are two tables needed, and will be named<br />
#* [workflow]Job<br />
#* [workflow]_aux<br />
# This naming scheme of tables and their roles are the same as from the jproj only launches. The [workflow]Job table will contain information gathered from SWIF about each job (which node it went to, start time of each stage, memory usage, etc.). The [workflow]_aux table will contain information gathered from the output stdout files from each job. First, create the Job table using <pre>python create_jproj_job_table.py [run period] [version]</pre> Note that this script uses the XML output from hdswif summary and inserts the contents into the MySQL table, so the XML output file must exist.<br />
# To check the contents of this MySQL table, do <pre>mysql -hhalldb -ufarmer farming</pre> <pre>mysql> describe offline_monitoring_RunPeriod2015_03_ver11_hd_rawdataJob;</pre><br />
# <b>Backing up tables</b><br> Tables created in MySQL should be backed up.<br />
<br />
== Hall D Job Management System ==<br />
<br />
This section details instructions on how to create and launch a set of jobs using the Hall-D Job Management System developed by Mark Ito. These instructions are generic: this system can be used for the weekly monitoring jobs, but can also be used for other sets of job launches as well. <br />
<br />
=== Database Table Overview ===<br />
<br />
* Job management database table (<project_name>): For each input file, keeps track of whether or not a job for it has been submitted, along with other optional fields. <br />
<br />
* Job status database table (<project_name>Job (no space)): For each job, keeps track of the job-id, the job status, memory used, cpu & wall time, time taken to complete various stages (e.g. pending, dependency, active), and others. <br />
<br />
* Job metrics database table (<project_name>_aux (no space)): For each job, keeps track of the job-id, how many events were processed, the time it took to copy the cache file, and the time it took to run the plugin. This information is culled from the log files of each job, and is done within the analysis directory of each launch.<br />
<br />
=== Initialize Project Management ===<br />
<br />
* Log into the ifarm machine with one of the gxproj accounts. For this example we will use gxproj1.<br />
<pre><br />
ssh gxproj1@ifarm -Y<br />
</pre><br />
<br />
* Go to a directory to do the launch. In principle, any directory will work, but for gxproj1 this is usually done in /home/gxproj1/halld/<br />
<br />
* Check out the necessary scripts <pre>svn co https://halldsvn.jlab.org/repos/trunk/scripts/monitoring/jproj</pre> This will get all necessary scripts for launching. Once checked out, <pre>cd projects</pre><br />
<br />
* The script create_project.sh can be used to create a new project. It will take a single argument, the project name. It is assumed that the project name is of the form offline_monitoring_RunPeriod20YY_MM_verVV_hd_rawdata where this string will be parsed to give the run period 20YY_MM and version number VV. One thing to do BEFORE creation of a new project is to editthe conditions of the launch (plugins to run over, memory requested, disk space requested) within templates/template.jsub . This<br />
information is saved automatically at project creation time into files in /group/halld/data_monitoring/run_conditions .<br />
To create a project do <pre>./create_project.sh offline_monitoring_RunPeriod20YY_MM_verVV_hd_rawdata</pre><br />
The name has been chosen to be as consistent as possible with other directory structures.<br />
However, mysql requires that "-" be escaped in table names, so unfortunately run periods will be<br />
given as 2YYY_MM instead of 2YYY-MM.<br />
<br />
* A new directory with the project name (offline_monitoring_RunPeriod20YY_MM_verVV_hd_rawdata) and files will be copied and modified from the template directory to reflect the run period, the user, the directory that it was created in, project name, etc.<br />
<br />
* For each project, cd into the new directory<br />
* The script clear.sh will remove any existing tables for the current project name, then recreate it. Do<br />
<pre>./clear.sh</pre><br />
* To use the jproj.pl script that was checked in, add the directory to your path with<br />
<pre>source ../../scripts/setup.csh</pre><br />
or always specify the full path<br />
<pre>../../scripts/jproj.pl</pre><br />
* Now update the table of runs with<br />
<pre>jproj.pl <project name> update</pre><br />
This will fill the table with all files within /mss that are of the same form as what is in <project name>.jproj . If you want to register only on a subset of all such files, you can edit this file directly.<br />
* Once you have registered all of the files you would like to run over, do<br />
<pre>jproj.pl <project name> submit [max # of jobs] [run number]</pre><br />
where the additional options specify how many jobs to submit and which run number to run on. Without these options all files that are registered and have not been submitted yet will be submitted.<br />
<br />
At this stage you are ready to submit all files. It is a good idea to submit a few test jobs<br />
at first to check that all scripts are working and that the plugins do not crash. Once you are<br />
sure that this does not happen, you can send all jobs in. The remaining jobs are then the monitoring<br />
which will (among other things) put the results on the online webpage for the collaboration to view, and<br />
the analysis of the launch.<br />
<br />
=== Project File Overview ===<br />
<br />
An overview of each project file:<br />
* '''clear.sh:''' For the current project, deletes the job status and management database tables (if any), and creates new, empty ones. <br />
* '''<project_name>.jproj:''' Contains the path and file name format for the input files for the jobs. <br />
* '''<project_name>.jsub:''' The xml job submission script. The run number and file number variables are set during job submission for each input file. <br />
* '''script.sh:''' The script that is executed during the job. If output job directories are not pre-created manually, they should be created in this script with the proper permissions:<br />
<pre><br />
mkdir -p -m 775 my_directory<br />
</pre><br />
* '''setup_jlab-[run period].csh:''' The environment that is sourced at the beginning of the job execution. <br />
* '''status.sh:''' Updates the job status database table, and prints some of its columns to screen.<br />
<br />
=== Project Management ===<br />
<br />
* Delete (if any) and create the database table(s) for the current set of job submissions:<br />
<pre><br />
./clear.sh<br />
</pre><br />
Also, if testing was done with jobs, it is best to delete the output directory and the configuration files:<br />
<pre><br />
rm -frv /volatile/halld/offline_monitoring/RunPeriod-20YY-MM/verVV /group/halld/data_monitoring/run_conditions/soft_comm_20YY_MM_verVV.xml /group/halld/data_monitoring/run_conditions/jana_rawdata_comm_20YY_MM_verVV.conf<br />
</pre><br />
<br />
* Search for input files matching the string in the .jproj file, and create a row for each in the job management database table (called <project_name>). You can test by adding an optional argument at the end, which only selects files with a specific file number:<br />
<pre><br />
jproj.pl <project_name> update <optional_file_number><br />
</pre><br />
<br />
* Confirm that the job management database is accurate by printing it's contents to screen:<br />
<pre><br />
mysql -hhallddb -ufarmer farming -e "select * from <project_name>"<br />
</pre><br />
<br />
* ONLY if a mistake was made, to delete the tables from the database and recreate new, empty ones, run: <br />
<pre><br />
./clear.sh<br />
</pre><br />
<br />
* To look at the status of the submitted jobs, first query auger and update the job status database:<br />
<pre><br />
fill_in_job_details.pl <project_name><br />
</pre><br />
<br />
* The job status can then be viewed by submitting a query to the job status database (called <project_name>Job (no space in between)):<br />
<pre><br />
mysql -hhallddb -ufarmer farming -e "select id,run,file,jobId,hostname,status,timeSubmitted,timeActive,walltime,cput,timeComplete,result,error from <project_name>Job"<br />
</pre><br />
<br />
* These last two commands can instead be executed simultaneously by running:<br />
<pre><br />
./status.sh<br />
</pre><br />
<br />
=== Handy mysql Instructions ===<br />
<br />
* Handy mysql instructions:<br />
<pre><br />
mysql -hhallddb -ufarmer farming # Enter the "farming" mysql database on "hallddb" as user "farmer"<br />
quit; # Exit mysql<br />
show tables; # Show a list of the tables in the current database<br />
show columns from <project_name>; # show all of the columns for the given table<br />
select * from <project_name>; # show the contents of all rows from the given table<br />
</pre><br />
<br />
=== Backing Up Offline Monitoring Tables ===<br />
Tables created for offline monitoring can be backed up using the script backup_tables.sh which<br />
can be checked out with the other files from https://halldsvn.jlab.org/repos/trunk/scripts/monitoring/jproj/projects<br />
<br />
The script uses the command mysqldump to print out a file that can be executed to recreate the tables.<br />
Since executing this output file will drop the table when it exists, caution is advised.<br />
Example usage to backup all three tables created for run period 2014_10 ver 17:<br />
<pre>backup_tables.sh 2014_10 17</pre><br />
<br />
== Running Over Data As It Comes In ==<br />
<br />
A special user gxproj1 will have a cron job set up to run the plugins as new data appears on /mss.<br />
During the week, gxproj1 will submit offline plugin jobs with the same setup as the weekly jobs<br />
run the previous Friday. The procedure for this is shown below.<br />
<br />
<!--<br />
=== Setting up the environment ===<br />
The file<br />
/home/gxproj1/setup_jlab.csh<br />
is sourced through .tcshrc.<br />
This file is the same as what is linked to by<br />
/home/gluex/setup_jlab_commissioning.csh,<br />
except HALLD_HOME, HDDS_HOME, and JANA_CALIB_URL are set separately so that this<br />
user can have a separate build.<br />
<br />
To obtain the builds from the previous Friday's runs,<br />
execute<br />
/home/gxproj1/halld/monitoring/newruns/setup_previous.sh [year] [month] [day]<br />
The build revisions from the previous Friday are archived in files<br />
/work/halld/data_monitoring/run_conditions/soft_comm_[year]_[month]_[day].xml<br />
and the script will build libraries based on those stored revision numbers.<br />
--><br />
<br />
=== Running the cron job ===<br />
<br />
'''IMPORTANT:''' The cron job should not be running while you are manually submitting jobs using the jproj.pl script for the same project, or else you will probably multiply-submit a job. <br />
<br />
* Go to the cron job directory: <br />
<pre><br />
cd /u/home/gxproj1/halld/monitoring/newruns<br />
</pre><br />
<br />
* The cron_plugins file is the cronjob that will be executed. During execution, it runs the exec.sh command in the same folder. This command takes two arguments: the project name, and the maximum file number for each run. These fields should be updated in the cron_plugins file before running. <br />
<br />
* The exec.sh command updates the job management database table with any data that has arrived on tape since it was last updated, ignoring file numbers greater than the maximum file number. It then submits jobs for these files. <br />
<br />
* To start the cron job, run:<br />
<pre><br />
crontab cron_plugins<br />
</pre><br />
<br />
* To check whether the cron job is running, do<br />
<pre><br />
crontab -l<br />
</pre><br />
<br />
* To remove the cron job do<br />
<pre><br />
crontab -r<br />
</pre><br />
<br />
<!--<br />
The cron job will run the script scan_for_jobs.sh,<br />
which runs generatejobs_plugins_rawdata.sh for any<br />
new runs that it had not seen before. All previous<br />
runs are recorded in the file filelists/files_current.txt<br />
so clear this to run over runs, or set the parameters<br />
MINRUN and MAXRUN which will set the range of runs submitted.<br />
--><br />
<br />
==Post-Processing Procedures==<br />
<br />
To visualize the monitoring data, we save images of selected histograms and store time series of selected quantities in a database, which are then displayed on a web page. This section describes how to generate the monitoring images and database information.<br />
<br />
The scripts used to generate this summary data are primarily run from /home/gxprojN/halld/monitoring/process . If you want a new copy of the scripts, e.g., for a new monitoring run, you should check the scripts out from SVN:<br />
<syntaxhighlight><br />
svn co https://halldsvn.jlab.org/repos/trunk/scripts/monitoring/process<br />
</syntaxhighlight><br />
Note that these scripts currently have some parameters which must be periodically set by hand.<br />
<br />
The default python version on most JLab machine does not have the modules to allow these scripts to connect to the MySQL database. To run these scripts, load the environment with the following command<br />
<syntaxhighlight><br />
source /home/gxproj1/halld/monitoring/process/monitoring_env.csh<br />
</syntaxhighlight><br />
<br />
===Online Monitoring===<br />
<br />
There are two primary scripts for running over the monitoring data generated by the online system and offline reconstruction. The online script can be run with either of the following commands:<br />
<syntaxhighlight><br />
/home/gluex/halld/monitoring/process/check_new_runs.py<br />
<br />
OR <br />
<br />
/home/gluex/halld/monitoring/process/check_new_runs.csh<br />
</syntaxhighlight><br />
The shell script sets up the environment properly to run the python script. To connect to the monitoring database on the JLab CUE, modules included in the local installation of python >= 2.7 are needed. The shell script is appropriate to use in a cron job. The cronjob is currently run under the "gluex" account.<br />
<br />
The online monitoring system copies a ROOT file containing the results of the online monitoring, and other configuration files into a directory accessible outside the counting house. This python script automatically checks for new ROOT files, which it will then automatically process. It contains several configuration variables that must be correctly set, which contains the location of input/output directories, etc...<br />
<br />
===Offline Monitoring===<br />
<br />
After the data is run over, the results should be processed, so that summary data is entered into the monitoring database and plots are made for the monitoring webpages. Currently, this processing is controlled by a cronjob that runs the following script:<br />
<syntaxhighlight><br />
/home/gxproj1/halld/monitoring/process/check_monitoring_data.csh <br />
</syntaxhighlight><br />
This script checks for new ROOT files, and only runs over those it hasn't processed yet. Since one monitoring ROOT file is produced for each EVIO file, whenever a new file is produced, the plots for the corresponding run are recreated and all the ROOT files for a run are combined into one file. Information is stored in the database on a per-file basis. <br />
<br />
Plots for the monitoring web page can be made from single histograms or multiple histograms using RootSpy macros. If you want to change the list of plots made, you must modify one of the following files:<br />
* histograms_to_monitor - specify either the name of the histogram or its the full ROOT path <br />
* macros_to_monitor - specify the full path to the RootSpy macro .C file<br />
<br />
When a new monitoring run is started, or the conditions are changed, the following steps should be taken to process the new files:<br />
# Add a new data version, as described below:<br />
# Change the following parameters in check_monitoring_data.csh:<br />
## JOBDATE should correspond to the ouptut date used by the job submission script<br />
## OUTPUTDIR should correspond to the directory corresponding to the run period and revision corresponding to the new version you just submitted. Presumably, this directory will be empty at the beginning.<br />
## Once you create a new data version as defined below, you should pass the needed information as a command line option. Currently this is done by the ARGS variable. For example, the argument "-v RunPeriod-2014-10,8" tells the monitoring scripts to look up the version corresponding to revision 8 of RunPeriod-2014-10 in the monitoring DB and to use to store the results.<br />
<br />
<syntaxhighlight><br />
Example configuration parameters:<br />
set JOBDATE=2015-01-09<br />
set INPUTDIR=/volatile/halld/RunPeriod-2014-10/offline_monitoring<br />
set OUTPUTDIR=/w/halld-scifs1a/data_monitoring/RunPeriod-2014-10/ver08<br />
set ARGS=" -v RunPeriod-2014-10,8 "<br />
</syntaxhighlight><br />
If you want to process the results manually, the data is processed using the following script:<br />
<syntaxhighlight><br />
./process_new_offline_data.py <input directory> <output directory><br />
<br />
EXAMPLE:<br />
<br />
./process_new_offline_data.py 2014-11-14 /volatile/halld/RunPeriod-2014-10/offline_monitoring/ /w/halld-scifs1a/data_monitoring/RunPeriod-2014-10/ver02<br />
</syntaxhighlight><br />
The python script takes several options to enable/disable various steps in the processing. Of interest is the "--force" option, which will run over all monitoring ROOT files, whether or not they've been previously identified.<br />
<br />
Every time a new reconstruction pass is performed, a new version number must be generated. To do this, prepare a version file as described below. Then run the register_new_version.py script to store the information in the database. The script will return a version number, which then should be set by hand in process_new_offline_data.py - future versions of the script will streamline this part of the procedure. An example of how to generate a new version is:<br />
<syntaxhighlight><br />
./register_new_version.py add /home/gxproj1/halld/monitoring/process/versions/vers_RunPeriod-2014-10_pass1.txt<br />
</syntaxhighlight><br />
<br />
<b>If you are running the offline monitoring by checking out the files in trunk/scripts/monitoring/jproj/projects/<br />
of the svn repository</b>, and created a project with<br />
create_project.sh [project name] hd_rawdata<br />
Then go to the directory [project name]/processing/<br />
and execute<br />
./run_processing.sh<br />
which will run register_new_version.py as well as check_monitoring_data.csh for that project.<br />
<br />
===Step-by-Step Instructions For Processing a New Monitoring Run===<br />
<br />
The monitoring runs are current run out of the gxproj1 and gxproj5 accounts. After an offline monitoring run has been successfully started on the batch farm, the following steps should be followed to setup the post-processing for these runs.<br />
<br />
# The post-processing scripts are stored in $HOME/halld/monitoring/process and are automatically run by cron.<br />
# Run "svn update" to bring any changes in. Be sure that the list of histograms and macros to plot are current.<br />
# Add a new data version<br />
# Edit check_monitoring_data.csh to point to the current revisions/directories<br />
#* VERSION<br />
#* ARGS<br />
#* Note that the environment depends on a standard script - $HOME/setup_jlab.csh<br />
# Update files in the web directory, so that the results are displayed on the web pages: /group/halld/www/halldweb/html/data_monitoring/textdata<br />
# Copy the REST files to more permanent locations:<br />
#* cp -a /volatile/halld/offline_monitoring/RunPeriod-YYYY-MM/verVV/REST /work/halld/data_monitoring/RunPeriod-YYYY-MM/REST/verVV<br />
#* cp -a /volatile/halld/offline_monitoring/RunPeriod-YYYY-MM/verVV/REST /cache/halld/RunPeriod-YYYY-MM/REST/verVV [under testing]<br />
<br />
Check log files in $HOME/halld/monitoring/process/log for more information on how each run went. If there are problems, check log files, and modify check_monitoring_data.csh to vary the verbosity of the output.<br />
<br />
==Data Versions==<br />
<br />
To document the conditions of the monitoring data that is created, for the sake of reproducability and further analysis we save several pieces of information. The format is intended to be comprehensive enough to document not just monitoring data, but versions of raw and reconstructed data, so that this database table can be used for the event database as well.<br />
<br />
We store one record per pass through one run period, with the following structure:<br />
<br />
{| class="wikitable"<br />
! Field !! Description<br />
|-<br />
| data_type || The level of data we are processing. For the purposes of monitoring, "rawdata" is the online monitoring, "recon" is the offline monitoring<br />
|- <br />
| run_period || The run period of the data<br />
|- <br />
| revision || An integer specifying which pass through the run period this data corresponds to<br />
|- <br />
| software_version || The name of the XML file that specifies the different software versions used<br />
|-<br />
| jana_config || The name of the text file that specifies which JANA options were passed to the reconstruction program<br />
|-<br />
| ccdb_context || The value of JANA_CALIB_CONTEXT, which specifies the version of calibration constants that were used<br />
|-<br />
| production_time || The data at which monitoring/reconstruction began<br />
|-<br />
| dataVersionString || A convenient string for identifying this version of the data<br />
|}<br />
<br />
<br />
An example file used as as input to ./register_new_version.py is:<br />
<syntaxhighlight><br />
data_type = recon<br />
run_period = RunPeriod-2014-10<br />
revision = 1<br />
software_version = soft_comm_2014_11_06.xml<br />
jana_config = jana_rawdata_comm_2014_11_06.conf<br />
ccdb_context = calibtime=2014-11-10<br />
production_time = 2014-11-10<br />
dataVersionString = recon_RunPeriod-2014-10_20141110_ver01<br />
</syntaxhighlight></div>Kmoriyahttps://halldweb.jlab.org/wiki/index.php?title=Data_Monitoring_Procedures&diff=71253Data Monitoring Procedures2015-11-08T03:48:47Z<p>Kmoriya: /* Starting the Launch and Submitting Jobs */</p>
<hr />
<div>__TOC__<br />
<br />
== Master List of File / Database / Webpage Locations ==<br />
=== Run Conditions ===<br />
* Online Run-by-run condition files (B-field, current, etc.): /work/halld/online_monitoring/conditions/<br />
* Offline monitoring run conditions (software versions, jana config): /group/halld/data_monitoring/run_conditions/<br />
*[http://www.jlab.org/Hall-D/test/RunInfo/ Run Info vers. 1]<br />
*[https://halldweb.jlab.org/cgi-bin/data_monitoring/run_conditions.pl Run Info vers. 2]<br />
<br />
=== Monitoring Output Files ===<br />
* Run Periods 201Y-MM is for example 2015-03, launch ver verVV is for example ver15<br />
* Online monitoring histograms: /work/halld/online_monitoring/root/<br />
* Offline monitoring histogram ROOT files: /work/halld/data_monitoring/RunPeriod-201Y-MM/verVV/rootfiles<br />
* REST files (most recent launch only): /work/halld/data_monitoring/RunPeriod-201Y-MM/REST/verVV<br />
* individual files for each job (ROOT, log, etc.): /volatile/halld/offline_monitoring/RunPeriod-201Y-MM/verVV/<br />
<br />
=== Monitoring Database ===<br />
* Accessing monitoring database (on ifarm): mysql -u datmon -h hallddb.jlab.org data_monitoring<br />
<br />
=== Monitoring Webpages ===<br />
*[https://halldweb.jlab.org/cgi-bin/data_monitoring/monitoring/plotBrowser.py Plot Browser]<br />
*[https://halldweb.jlab.org/cgi-bin/data_monitoring/monitoring/runBrowser.py Run Browser]<br />
*[https://halldweb.jlab.org/cgi-bin/data_monitoring/monitoring/timeSeries.py Time Series]<br />
*[https://halldweb.jlab.org/data_monitoring/launch_analysis/ Launch Analysis]<br />
<br />
== Job Monitoring Links ==<br />
* [http://scicomp.jlab.org/farm2/job.html Completed Job History]<br />
* [http://scicomp.jlab.org/farm2/project.html Job Stats By Project]<br />
* [http://scicomp.jlab.org/farm2/trackOrg.html Job Stats By Track]<br />
* [http://scicomp.jlab.org/farm2/report.html Cluster Report]<br />
* [http://scicomp.jlab.org/farm2/walltime.html Walltime Distribution]<br />
<br />
== Saving Online Monitoring Data ==<br />
<br />
The procedure for writing the data out is given in, e.g.,<br />
[https://halldweb.jlab.org/wiki/index.php/Raid-to-Silo_Transfer_Strategy Raid-to-Silo Transfer Strategy].<br />
<br />
Once the DAQ writes out the data to the raid disk, cron jobs will copy the file to tape,<br />
and within ~20 min., we will have access to the file on tape at<br />
/mss/halld/$RUN_PERIOD/rawdata/RunXXXXXX.<br />
<br />
All online monitoring plugins will be run as data is taken.<br />
They will be accessible within the counting house via RootSpy, and<br />
for each run and file, a ROOT file containing the histograms will be saved<br />
within a subdirectory for each run.<br />
<br />
For immediate access to these files, the raid disk files may be accessed directly<br />
from the counting house, or the tape files will be available within ~20 min. of the<br />
file being written out.<br />
<br />
== Offline Monitoring: Running Over Archived Data ==<br />
<br />
Once files are written to tape we run the online plugins on these files to confirm what we were seeing in the online monitoring, and also to update the results from the latest calibration and software. <br />
Manual scripts and cron jobs are set up to look for new data and run the plugins over a sample of files.<br />
<br />
Every other Friday (usually the Friday before the offline meetings) jobs will be started to run the newest software on all previous runs,and allows everybody to see improvements in each detector.<br />
For each launch, independent builds of hdds, sim-recon, the monitoring plugins, and an sqlite file will be generated.<br />
<br />
Below the procedures are described for<br />
# Preparing the software for the launch<br />
# Starting the launch (using hdswif)<br />
# Post-analysis of statistics of the launch<br />
<br />
Processing the results and making them available to the collaboration<br />
is handled in the section [[#Post-Processing_Procedures | Post-Processing Procedures]] below.<br />
<br />
<!--<br />
==== Generating an offline plugin job ====<br />
<br />
The user gxproj1 should be used for official offline monitoring jobs.<br />
Within /home/gxproj1/halld/monitoring/batch/ there will be scripts to run the online monitoring plugins over tape files.<br />
The main script is generatejobs_plugins_rawdata.sh, which can be used as<br />
generatejobs_plugins_rawdata.sh [minrun] [maxrun] (minfile) (maxfile)<br />
where minrun, maxrun specify the range of the run #, and minfile, maxfile (optional) specify<br />
the file range for that run.<br />
<br />
This will generate a script run_rawdata_XXXXXX.sh, where the run # has now been formatted to be 6 digits.<br />
Executing this script will send the monitoring plugins job to the Auger batch system.<br />
<br />
There is also a script clean.sh which can be used as<br />
clean.sh XXX<br />
This will clean up all associated files created in association with run XXX.<br />
<br />
Internally, the xml file used to submit the job will be created, and the job to run<br />
will be given within script.sh. All run parameters should be specified in at the beginning<br />
of generatejobs_plugins_rawdata.sh<br />
Since we are running on tape, the tape file will first be copied over to the cache disk, and the job will run<br />
over this cached file.<br />
<br />
==== Using cron to run automatically ====<br />
Within /home/gluex/halld/monitoring/cron/ there is a file cron_plugins<br />
that can be executed via<br />
crontab cron_plugins<br />
This will set up a cron job to call the script scan_for_jobs.sh, which will<br />
check in the rawdata directory and call generatejobs_plugins_rawdata.sh for<br />
any run that is more than 5 min old. The cron job is set up to run every 10 min.<br />
<br />
==== Magnetic Field Settings ====<br />
For tracking, it is necessary to set the correct magnetic field settings.<br />
The below table is now obsolete. Please refer to the [https://halldweb.jlab.org/cgi-bin/data_monitoring/run_conditions.pl GlueX Run Conditions] webpage.<br />
<br />
The actual field settings for each run have not been documented well.<br />
Below are values based on going through entries in the [https://logbooks.jlab.org/book/halld halld log].<br />
<br />
{| border="1" cellpadding="1" style="text-align: center;"<br />
!width="100"| Run #s<br />
!width="150"| Solenoid Current (A)<br />
!width="300"| JANA option<br />
!width="300"| notes<br />
|-<br />
| 940 - 996 || 1000 || -PBFIELD_MAP=Magnets/Solenoid/solenoid_1000A_poisson_20141104 || At run 997, the solenoid started ramping down<br />
|-<br />
| 998 - 1448 || 0 || -PBFIELD_TYPE=NoField -PDEFTAG:DTrackCandidate=StraightLine || Solenoid started ramping up around run 1431. Runs 1432 - 1448 should not be used for tracking. See [https://logbooks.jlab.org/entry/3309039 this] and [https://logbooks.jlab.org/entry/3309092 this] entry.<br />
|-<br />
| 1449 - 1620 || 1200 || -PBFIELD_MAP=Magnets/Solenoid/solenoid_1200A_poisson_20140520 || <br />
|-<br />
|}<br />
<br />
'''Note:''' The following run ranges were taken with a 300A field, and could be analyzed with the magnetic field map Magnets/Solenoid/solenoid_300A_poisson_20140819 , more study is needed to see if the straight line track fitter is needed or not: 1036 - 1053, 1065 - 1121, 1212? - 1254, 1309 - 1318 [1307,1308,1319-1329 were taken while the magnet was ramping].<br />
--><br />
<br />
=== General Information on Procedures ===<br />
This section explains how the offline monitoring should be run.<br />
Since we may want to simultaneously run offline monitoring for different run periods that require<br />
different environment variables, the scripts are set up so that a generic user can download the<br />
scripts and run them from anywhere. Most output directories for offline monitoring are created<br />
with group read/write permissions so that any Hall D group user has access to the contents,<br />
but there are some cases where use of the account that created the launch is necessary.<br />
<br />
The accounts used for offline monitoring are the gxprojN accounts created and maintained by<br />
Mark Ito (see [https://halldweb.jlab.org/wiki/index.php/GlueX-related_shared_accounts_on_the_JLab_CUE here] for how each account is used).<br />
As of October 2015, the following are used:<br />
* gxproj1 for running over Fall 2014 data (deprecated since June 2015)<br />
* gxproj5 for running over Spring 2015 data<br />
<br />
Since the summer of 2015 we have transitioned from a system using Mark Ito's jproj scripts<br />
to integrating the swif system that Chris Larrieu (SciComp) has been developing. For offline monitoring,<br />
the hdswif system that Kei developed is used for launching the jobs, and the jproj system is used<br />
for meta-analysis of launch statistics.<br />
<br />
Both hdswif and jproj are maintained in svn:<br />
* hdswif: https://halldsvn.jlab.org/repos/trunk/scripts/monitoring/hdswif<br />
* jproj : https://halldsvn.jlab.org/repos/trunk/scripts/monitoring/jproj<br />
<br />
To run the offline monitoring each package should be checked out and all necessary<br />
scripts are included.<br />
<br />
=== Preparing the software for the launch ===<br />
<br />
To begin a new launch the software must be built to the latest versions.<br />
For the gxprojN user accounts used, all software builds are contained in the directory ~/builds<br />
(which are soft links to /work/halld/home/gxprojN/builds). When logging into these accounts<br />
the setup files ~/setup_jlab-2015-03.csh or similar files should be sourced.<br />
<br />
<b>Note that Mark Ito does <u>not</u> want you to change the contents of each .cshrc file.</b><br />
You should consult him if you feel the need.<br />
<br />
1. Setup the environment: <pre>source ~/setup_jlab-2015-03.csh</pre><br />
2. Building hdds: <br />
<pre><br />
cd ~/builds/hdds/hdds<br />
git pull # Get latest software<br />
scons -c install # Clean out the old install: EXTREMELY IMPORTANT for cleaning out stale headers<br />
scons install -j4 # Rebuild and re-install with 4 threads<br />
</pre><br />
3. Building sim-recon: <br />
<pre><br />
cd ~/builds/sim-recon/sim-recon/<br />
git pull<br />
cd src<br />
scons -c install # Clean out the old install: EXTREMELY IMPORTANT for cleaning out stale headers<br />
scons install -j4 # Rebuild and re-install with 4 threads<br />
</pre><br />
4. Prepare the latest sqlite file: The sqlite is set in the ~/setup_jlab-2015-03.csh script as <b>sqlite:////home/gxproj5/ccdb.sqlite</b> through the environment variables <b>JANA_CALIB_URL</b> and <b>CCDB_CONNECTION</b>. Therefore, go to this directory and create a new sqlite file. We create the sqlite file in a temporary directory since creating the sqlite file in a directory where the output file exists causes errors. Original documentation on creating sqlite files are [https://halldweb.jlab.org/wiki/index.php/SQLite-form_of_the_CCDB_database here]. <br><span style="color:red">NOTE: SQLITE FILES DO NOT WORK ON THE NEW /work DISK INSTALLED IN OCTOBER 2015 </span><br />
<pre><br />
cd ~/tmp<br />
$CCDB_HOME/scripts/mysql2sqlite/mysql2sqlite.sh -hhallddb.jlab.org -uccdb_user ccdb | sqlite3 ccdb.sqlite<br />
mv ccdb.sqlite ../<br />
</pre><br />
5. Note that the above steps must be done <b>BEFORE</b> launch project creation. This is because we will track the revisions of the libraries used, and this is done by extracting the svn information in each directory. Also, note that the system assumes that we have the topmost build directory (usually called GLUEX_TOP) to be $HOME/builds . Such an assumption is necessary to be able to extract information about the library locations automatically.<br />
<br />
Create the appropriate project(s) and submit the jobs using hdswif, as detailed in the section below.<br />
<br />
=== Starting the Launch and Submitting Jobs ===<br />
<br />
Until the summer of 2015 we relied solely on Mark Ito's jproj system for submitting and keeping track of jobs. We have since moved to the swif system and use the hdswif wrapper for this. Below are instructions for how to use these.<br />
<br />
# Downloading hdswif: Download the hdswif directory from svn. For the gxprojN accounts, use the directory ~/halld/hdswif. <pre>cd ~/halld </pre> <pre> svn co https://halldsvn.jlab.org/repos/trunk/scripts/monitoring/hdswif</pre> <pre>cd hdswif</pre><br />
# Creating the workflow: Within SWIF jobs are registered into workflows. First create the workflow. For offline monitoring, the workflow names are of the form <b>offline_monitoring_RunPeriod201Y_MM_verVV_hd_rawdata</b> with suitable replacements for the run period and version number. The command "swif list" will list all existing workflows. Also, for most simple SWIF commands hdswif also provides a wrapper. <pre>swif list</pre> For creation of workflows for offline monitoring the command <pre>hdswif.py create [workflow] -c [config file] </pre> should be used. As an example config file, see the input.config file in the folder (and update it). When a config file is passed in, hdswif will automatically create files that record the configuration of the current launch. These files are stored as for example<br />
#* /group/halld/data_monitoring/run_conditions/jana_rawdata_comm_2015_03_ver15.conf<br />
#* /group/halld/data_monitoring/run_conditions/soft_comm_2015_03_ver15.xml<br />
#* The software packages stored in git (sim-recon and hdds) can have git tags applied to them, which makes it easier to find versions of the software than a SHA-1 hash. hdswif will ask if you would like to create a tag, and execute the following sequence: <pre>git tag -a offmon-201Y_MM-verVV -m "Used for offline monitoring 201Y-MM verVV started on 201y/mm/dd"</pre> <pre>git push offmon-201Y_MM-verVV</pre> This will only be invoked when the user name is gxprojN, and for the configuration files, the output directory will be /group/halld/data_monitoring/run_conditions/ for gxprojN accounts while it will be the current directory for other users.<br />
#* To use the git-tagged software versions do for example <pre>cd $HALLD_HOME</pre> <pre>git checkout offmon-2015_03-ver15</pre><br />
# Registering jobs in the workflow: To register jobs within the workflow, hdswif provides the use of config files. Jobs can be registered by specifying the workflow, config file (-c), run (-r) and file (-f) numbers if necessary. A typical config file will look this:<br />
PROJECT gluex<br />
TRACK reconstruction<br />
OS centos65<br />
NCORES 6<br />
DISK 40<br />
RAM 8<br />
TIMELIMIT 8<br />
JOBNAMEBASE offmon_<br />
RUNPERIOD 2015-03<br />
VERSION 15<br />
OUTPUT_TOPDIR /volatile/halld/offline_monitoring/RunPeriod-[RUNPERIOD]/ver[VERSION] # Example of other variables included in variable<br />
SCRIPTFILE /home/gxproj5/halld/hdswif/script.sh # Must specify full path<br />
ENVFILE /home/gxproj5/halld/hdswif/setup_jlab-2015-03.csh # Must specify full path<br />
The config file contains configuration parameters for each of the jobs.<br><br />
<span style="color:red">Note: Job configuration parameters can be set differently for jobs within the same workflow if necessary.</span><br />
Edit the config file and save as a new file if necessary. Once the configuration is set, jobs can be added via <pre>hdswif.py add [workflow] -c input.config</pre><br />
By default, hdswif will add all files found within the directory /mss/halld/RunPeriod-201Y-MM/rawdata/ where 201Y-MM is specified by the RUNPERIOD parameter in the config file. If only some of the runs or files are needed, these can be specified for example with<br />
<pre>hdswif.py add [workflow] -c input.config -r 3180 -f '00[0-4]'</pre><br />
to specify to register running only over run 3180 files 000 - 004 (Unix-style brackets and wildcards can be used).<br />
<br />
Running the workflow: To run the workflow, simply use swif run: <pre>swif run -workflow [workflow] -errorlimit none</pre> <span style="color:red">MAKE SURE THE ERRORLIMIT IS SET TO NONE OR THE WORKFLOW WILL BE STOPPED AFTER ANY JOB FAILS</span><br> Or equivalently, using the hdswif wrapper (which has the errorlimit set by default), <pre>hdsswif.py run [workflow]</pre><br><br />
<b>It is recommended that some jobs be tested to make sure that everything is working rather than fail thousands of jobs.</b><br><b>Check the configuration setup when creating a workflow, and what is in script.sh. Also check the following:<br />
* Check stderr files. Are they small (<kB)?<br />
* Check stdout files. Are they very large (>MB)?<br />
* Check output ROOT files. Are they larger than several MB?<br />
* Check output REST files. Are they larger than several tens of MB?<br />
</b><br />
<br> For this purpose, hdswif will take an additional parameter to run which limits the number of jobs to submit: <pre>hdswif.py run [workflow] 10</pre><br />
in which case only 10 jobs will be submitted.<br />
<br />
To submit all jobs after checking the results, do <pre>hdswif.py run [workflow]</pre><br />
<br />
=== Checking the Status and Resubmitting ===<br />
1. The status of jobs can be checked on the terminal with <pre>jobstat -u gxprojN</pre> on Auger and for SWIF with <pre>swif list</pre> or for more information, <pre>swif status [workflow] -summary</pre> Note that "swif status" tends to be out of date sometimes, so don't panic if your workflow/jobs aren't showing up right away. Also see the Auger [http://scicomp.jlab.org/scicomp/#/auger/jobs job website].<br />
<br />
2. For failed jobs, SWIF can resubmit jobs based on the problem. For resubmission for failed jobs with the same resources, <pre>swif retry-jobs [workflow] -problems [problem name]</pre> can be used, and for jobs to be submitted with more resources, e.g., use <pre>swif modify-jobs -ram add 2gb -problems AUGER-OVER_RLIMIT</pre> <br />
This only re-stages the jobs, be sure to resubmit them with:<br />
<pre>swif run -workflow [workflow] -errorlimit none</pre><br />
<br />
hdswif has a wrapper for both of these: <pre>hdswif.py resubmit [workflow] [problem]</pre> In this case [problem] can be one of <b>SYSTEM, TIMEOUT, RLIMIT</b>. If SYSTEM is specified, the jobs will be retried. For TIMEOUT and RLIMIT, the jobs will be modified by default with 2 additional hours or GB of RAM. If one more number is added as an option, then that many hours or GB of RAM will be added., e.g., <pre>hdswif.py resubmit [workflow] TIMEOUT 5</pre> will add 5 hours of processing time. You can wait until almost all jobs finish before resubmitting failed jobs since the number should be relatively small. <b>Even if jobs are resubmitted for one type of failure, jobs that later fail with that failure will not be automatically resubmitted.</b><br />
<br />
3. For information on swif, use the "swif help" commands and for hdswif see the attached documentaion in https://halldsvn.jlab.org/repos/trunk/scripts/monitoring/hdswif/manual_hdswif.pdf<br />
<br />
4. Below is a table describing the various errors that can occur.<br />
{| border="1" cellpadding="0" valign="left" style="text-align: left;"<br />
!width="150"| ERROR NAME<br />
!width="400"| Description<br />
!width="400"| Resolution<br />
!width="400"| hdswif command<br />
|-<br />
| AUGER-SUBMIT<br />
||<br />
SWIF’s attempt to submit jobs to Auger failed. Includes server-side problems as well as user failing to provide valid job parameters (e.g. incorrect project name, too many resources, etc.)<br />
||<br />
If requested resources are known to be correct resubmit. Otherwise modify job resources using swif directly.<br />
||<br />
<span style="color:purple"><b>hdswif.py resubmit [workflow] SYSTEM</b></span><br />
|-<br />
| AUGER-FAILED<br />
||<br />
Auger reports the job FAILED with no specific details.<br />
||<br />
Resubmit jobs. If problem persists, contact Chris Larrieu or SciComp.<br />
||<br />
<span style="color:purple"><b>hdswif.py resubmit [workflow] SYSTEM</b></span><br />
|-<br />
| AUGER-OUTPUT-FAIL<br />
||<br />
Failure to copy one or more output files.Can be due to permission problem, quota problem, system error, etc.<br />
||<br />
Check if output files will exist after job execution and that output directory exists, resubmit jobs. If problem persists, contact Chris Larrieu or SciComp.<br />
||<br />
<span style="color:purple"><b>hdswif.py resubmit [workflow] SYSTEM</b></span><br />
|-<br />
| AUGER-INPUT-FAIL<br />
||<br />
Auger failed to copy one or more of the requested input files, similar to output failures. Can also happen if tape file is unavailable (e.g. missing/damaged tape)<br />
||<br />
Check if input file exists, resubmit jobs. If problem persists, contact Chris Larrieu or SciComp.<br />
||<br />
<span style="color:purple"><b>hdswif.py resubmit [workflow] SYSTEM</b></span><br />
|-<br />
| AUGER-TIMEOUT<br />
||<br />
Job timed out.<br />
||<br />
If more time is needed for job add more resources.<br />
Default is to add 2 hrs of processing time. Also check whether code is hanging.<br />
||<br />
<span style="color:red"><b>hdswif.py resubmit [workflow] TIMEOUT</b></span><br><br />
Default is to add 2 hours. Optionally specify number of hours at end.<br />
|-<br />
| AUGER-OVER_RLIMIT<br />
||<br />
Not enough resources, RAM or disk space.<br />
||<br />
Add more resources for job.<br />
||<br />
<span style="color:blue"><b>hdswif.py resubmit [workflow] RLIMIT</b></span><br><br />
Default is to add 2 GB of RAM. Optionally specify GB at end. To add more disk space use SWIF directly.<br />
|-<br />
| SWIF-MISSING-OUTPUT<br />
||<br />
Output file specified by user was not found.<br />
||<br />
Check if output file exists at end of job.<br />
||<br />
<br />
|-<br />
| SWIF-USER-NON-ZERO <br />
||<br />
User script exited with non-zero status code.<br />
||<br />
Your script exited with non-zero status. Check the code you are running.<br />
||<br />
<br />
|-<br />
| SWIF-SYSTEM-ERROR <br />
||<br />
Job failed owing to a problem with swif (e.g. network connection timeout)<br />
||<br />
Resubmit jobs. If problem persists, contact Chris Larrieu or SciComp.<br />
||<br />
<span style="color:purple"><b>hdswif.py resubmit [workflow] SYSTEM</b></span><br />
|}<br />
<br style="clear:both;"/><br />
<br />
=== Post-analysis of statistics of the launch ===<br />
<br />
# After jobs have been submitted, it will usually take a few days for all of the jobs to be processed.<br />
#* Status of Auger: http://scicomp.jlab.org/scicomp/#/auger/jobs (see also links above)<br />
#* Status of user jobs: <pre>jobstat -u [user name]</pre><br />
# The status and results of jobs are saved within the SWIF internal server, and are available via the command <pre>swif status [workflow] -summary -runs</pre> where the arguments -summary and -runs show summary statistics and statistics for individual jobs, respectively. hdswif has a command that takes this output in XML output and creates an HTML webpage showing results of the launch. To do this, do <pre>hdswif.py summary [workflow]</pre> This will create an XML file swif_output_[workflow].xml that contains all information from SWIF. If the file already exists, hdswif will ask whether to overwrite the existing file.<br />
# At this stage the html output and figure files are created and ready to be put online. For this step and other steps involving analysis of the statistics of the launch results, it is convenient to change to the jproj system.<br />
# The jproj scripts for offline monitoring are maintained in the svn directory https://halldsvn.jlab.org/repos/trunk/scripts/monitoring/jproj Do <pre> svn co https://halldsvn.jlab.org/repos/trunk/scripts/monitoring/jproj </pre> For the gxprojN accounts used for offline monitoring the directory should be ~/halld/jproj<br />
# The jproj directory contains two subdirectories, scripts and projects. The scripts directory contains useful scripts for processing the jobs registered in the jproj system, and each of the offline monitoring launches will be handled in the projects directory.<br />
# Go to the projects directory<pre>cd ~/halld/jproj/projects</pre> and use the script create_project.sh to create a new directory that contains the processing scripts for the current launch <pre>./create_project.sh [workflow]</pre> This should create a directory such as offline_monitoring_RunPeriod2015_03_ver15_hd_rawdata (same as the workflow name). The script uses the template files in the directory templates and by substitution creates script files for the current launch. Now go to the newly created analysis directory: <pre>cd [workflow]/analysis</pre><br />
which for the gxprojN accounts should have the full path /home/gxprojN/halld/jproj/projects/[workflow]/analysis.<br><br><br />
<span style="color:red">All of the analysis commands including arguments are contained in run_analysis.sh, but it is <b>strongly</b> recommended that all commands are run manually to check for errors.</span><br><br><br />
# The first thing to do is to make the html output from hdswif public. Copy the html file and related figures that were created from hdswif to the appropriate space within /group/halld/www/halldweb/html/data_monitoring/ : <pre>python publish_offmon_results.py [run period] [version]</pre> Note that the command with appropriate substitutions for arguments can be found within run_analysis.sh (same for all commands below).<br />
# Next, we need to create a few MySQL tables for the current launch. The MySQL tables are useful for comparing run/file combinations for different launches. For the SWIF launches, there are two tables needed, and will be named<br />
#* [workflow]Job<br />
#* [workflow]_aux<br />
# This naming scheme of tables and their roles are the same as from the jproj only launches. The [workflow]Job table will contain information gathered from SWIF about each job (which node it went to, start time of each stage, memory usage, etc.). The [workflow]_aux table will contain information gathered from the output stdout files from each job. First, create the Job table using <pre>python create_jproj_job_table.py [run period] [version]</pre> Note that this script uses the XML output from hdswif summary and inserts the contents into the MySQL table, so the XML output file must exist.<br />
# To check the contents of this MySQL table, do <pre>mysql -hhalldb -ufarmer farming</pre> <pre>mysql> describe offline_monitoring_RunPeriod2015_03_ver11_hd_rawdataJob;</pre><br />
<br />
== Hall D Job Management System ==<br />
<br />
This section details instructions on how to create and launch a set of jobs using the Hall-D Job Management System developed by Mark Ito. These instructions are generic: this system can be used for the weekly monitoring jobs, but can also be used for other sets of job launches as well. <br />
<br />
=== Database Table Overview ===<br />
<br />
* Job management database table (<project_name>): For each input file, keeps track of whether or not a job for it has been submitted, along with other optional fields. <br />
<br />
* Job status database table (<project_name>Job (no space)): For each job, keeps track of the job-id, the job status, memory used, cpu & wall time, time taken to complete various stages (e.g. pending, dependency, active), and others. <br />
<br />
* Job metrics database table (<project_name>_aux (no space)): For each job, keeps track of the job-id, how many events were processed, the time it took to copy the cache file, and the time it took to run the plugin. This information is culled from the log files of each job, and is done within the analysis directory of each launch.<br />
<br />
=== Initialize Project Management ===<br />
<br />
* Log into the ifarm machine with one of the gxproj accounts. For this example we will use gxproj1.<br />
<pre><br />
ssh gxproj1@ifarm -Y<br />
</pre><br />
<br />
* Go to a directory to do the launch. In principle, any directory will work, but for gxproj1 this is usually done in /home/gxproj1/halld/<br />
<br />
* Check out the necessary scripts <pre>svn co https://halldsvn.jlab.org/repos/trunk/scripts/monitoring/jproj</pre> This will get all necessary scripts for launching. Once checked out, <pre>cd projects</pre><br />
<br />
* The script create_project.sh can be used to create a new project. It will take a single argument, the project name. It is assumed that the project name is of the form offline_monitoring_RunPeriod20YY_MM_verVV_hd_rawdata where this string will be parsed to give the run period 20YY_MM and version number VV. One thing to do BEFORE creation of a new project is to editthe conditions of the launch (plugins to run over, memory requested, disk space requested) within templates/template.jsub . This<br />
information is saved automatically at project creation time into files in /group/halld/data_monitoring/run_conditions .<br />
To create a project do <pre>./create_project.sh offline_monitoring_RunPeriod20YY_MM_verVV_hd_rawdata</pre><br />
The name has been chosen to be as consistent as possible with other directory structures.<br />
However, mysql requires that "-" be escaped in table names, so unfortunately run periods will be<br />
given as 2YYY_MM instead of 2YYY-MM.<br />
<br />
* A new directory with the project name (offline_monitoring_RunPeriod20YY_MM_verVV_hd_rawdata) and files will be copied and modified from the template directory to reflect the run period, the user, the directory that it was created in, project name, etc.<br />
<br />
* For each project, cd into the new directory<br />
* The script clear.sh will remove any existing tables for the current project name, then recreate it. Do<br />
<pre>./clear.sh</pre><br />
* To use the jproj.pl script that was checked in, add the directory to your path with<br />
<pre>source ../../scripts/setup.csh</pre><br />
or always specify the full path<br />
<pre>../../scripts/jproj.pl</pre><br />
* Now update the table of runs with<br />
<pre>jproj.pl <project name> update</pre><br />
This will fill the table with all files within /mss that are of the same form as what is in <project name>.jproj . If you want to register only on a subset of all such files, you can edit this file directly.<br />
* Once you have registered all of the files you would like to run over, do<br />
<pre>jproj.pl <project name> submit [max # of jobs] [run number]</pre><br />
where the additional options specify how many jobs to submit and which run number to run on. Without these options all files that are registered and have not been submitted yet will be submitted.<br />
<br />
At this stage you are ready to submit all files. It is a good idea to submit a few test jobs<br />
at first to check that all scripts are working and that the plugins do not crash. Once you are<br />
sure that this does not happen, you can send all jobs in. The remaining jobs are then the monitoring<br />
which will (among other things) put the results on the online webpage for the collaboration to view, and<br />
the analysis of the launch.<br />
<br />
=== Project File Overview ===<br />
<br />
An overview of each project file:<br />
* '''clear.sh:''' For the current project, deletes the job status and management database tables (if any), and creates new, empty ones. <br />
* '''<project_name>.jproj:''' Contains the path and file name format for the input files for the jobs. <br />
* '''<project_name>.jsub:''' The xml job submission script. The run number and file number variables are set during job submission for each input file. <br />
* '''script.sh:''' The script that is executed during the job. If output job directories are not pre-created manually, they should be created in this script with the proper permissions:<br />
<pre><br />
mkdir -p -m 775 my_directory<br />
</pre><br />
* '''setup_jlab-[run period].csh:''' The environment that is sourced at the beginning of the job execution. <br />
* '''status.sh:''' Updates the job status database table, and prints some of its columns to screen.<br />
<br />
=== Project Management ===<br />
<br />
* Delete (if any) and create the database table(s) for the current set of job submissions:<br />
<pre><br />
./clear.sh<br />
</pre><br />
Also, if testing was done with jobs, it is best to delete the output directory and the configuration files:<br />
<pre><br />
rm -frv /volatile/halld/offline_monitoring/RunPeriod-20YY-MM/verVV /group/halld/data_monitoring/run_conditions/soft_comm_20YY_MM_verVV.xml /group/halld/data_monitoring/run_conditions/jana_rawdata_comm_20YY_MM_verVV.conf<br />
</pre><br />
<br />
* Search for input files matching the string in the .jproj file, and create a row for each in the job management database table (called <project_name>). You can test by adding an optional argument at the end, which only selects files with a specific file number:<br />
<pre><br />
jproj.pl <project_name> update <optional_file_number><br />
</pre><br />
<br />
* Confirm that the job management database is accurate by printing it's contents to screen:<br />
<pre><br />
mysql -hhallddb -ufarmer farming -e "select * from <project_name>"<br />
</pre><br />
<br />
* ONLY if a mistake was made, to delete the tables from the database and recreate new, empty ones, run: <br />
<pre><br />
./clear.sh<br />
</pre><br />
<br />
* To look at the status of the submitted jobs, first query auger and update the job status database:<br />
<pre><br />
fill_in_job_details.pl <project_name><br />
</pre><br />
<br />
* The job status can then be viewed by submitting a query to the job status database (called <project_name>Job (no space in between)):<br />
<pre><br />
mysql -hhallddb -ufarmer farming -e "select id,run,file,jobId,hostname,status,timeSubmitted,timeActive,walltime,cput,timeComplete,result,error from <project_name>Job"<br />
</pre><br />
<br />
* These last two commands can instead be executed simultaneously by running:<br />
<pre><br />
./status.sh<br />
</pre><br />
<br />
=== Handy mysql Instructions ===<br />
<br />
* Handy mysql instructions:<br />
<pre><br />
mysql -hhallddb -ufarmer farming # Enter the "farming" mysql database on "hallddb" as user "farmer"<br />
quit; # Exit mysql<br />
show tables; # Show a list of the tables in the current database<br />
show columns from <project_name>; # show all of the columns for the given table<br />
select * from <project_name>; # show the contents of all rows from the given table<br />
</pre><br />
<br />
=== Backing Up Offline Monitoring Tables ===<br />
Tables created for offline monitoring can be backed up using the script backup_tables.sh which<br />
can be checked out with the other files from https://halldsvn.jlab.org/repos/trunk/scripts/monitoring/jproj/projects<br />
<br />
The script uses the command mysqldump to print out a file that can be executed to recreate the tables.<br />
Since executing this output file will drop the table when it exists, caution is advised.<br />
Example usage to backup all three tables created for run period 2014_10 ver 17:<br />
<pre>backup_tables.sh 2014_10 17</pre><br />
<br />
== Running Over Data As It Comes In ==<br />
<br />
A special user gxproj1 will have a cron job set up to run the plugins as new data appears on /mss.<br />
During the week, gxproj1 will submit offline plugin jobs with the same setup as the weekly jobs<br />
run the previous Friday. The procedure for this is shown below.<br />
<br />
<!--<br />
=== Setting up the environment ===<br />
The file<br />
/home/gxproj1/setup_jlab.csh<br />
is sourced through .tcshrc.<br />
This file is the same as what is linked to by<br />
/home/gluex/setup_jlab_commissioning.csh,<br />
except HALLD_HOME, HDDS_HOME, and JANA_CALIB_URL are set separately so that this<br />
user can have a separate build.<br />
<br />
To obtain the builds from the previous Friday's runs,<br />
execute<br />
/home/gxproj1/halld/monitoring/newruns/setup_previous.sh [year] [month] [day]<br />
The build revisions from the previous Friday are archived in files<br />
/work/halld/data_monitoring/run_conditions/soft_comm_[year]_[month]_[day].xml<br />
and the script will build libraries based on those stored revision numbers.<br />
--><br />
<br />
=== Running the cron job ===<br />
<br />
'''IMPORTANT:''' The cron job should not be running while you are manually submitting jobs using the jproj.pl script for the same project, or else you will probably multiply-submit a job. <br />
<br />
* Go to the cron job directory: <br />
<pre><br />
cd /u/home/gxproj1/halld/monitoring/newruns<br />
</pre><br />
<br />
* The cron_plugins file is the cronjob that will be executed. During execution, it runs the exec.sh command in the same folder. This command takes two arguments: the project name, and the maximum file number for each run. These fields should be updated in the cron_plugins file before running. <br />
<br />
* The exec.sh command updates the job management database table with any data that has arrived on tape since it was last updated, ignoring file numbers greater than the maximum file number. It then submits jobs for these files. <br />
<br />
* To start the cron job, run:<br />
<pre><br />
crontab cron_plugins<br />
</pre><br />
<br />
* To check whether the cron job is running, do<br />
<pre><br />
crontab -l<br />
</pre><br />
<br />
* To remove the cron job do<br />
<pre><br />
crontab -r<br />
</pre><br />
<br />
<!--<br />
The cron job will run the script scan_for_jobs.sh,<br />
which runs generatejobs_plugins_rawdata.sh for any<br />
new runs that it had not seen before. All previous<br />
runs are recorded in the file filelists/files_current.txt<br />
so clear this to run over runs, or set the parameters<br />
MINRUN and MAXRUN which will set the range of runs submitted.<br />
--><br />
<br />
==Post-Processing Procedures==<br />
<br />
To visualize the monitoring data, we save images of selected histograms and store time series of selected quantities in a database, which are then displayed on a web page. This section describes how to generate the monitoring images and database information.<br />
<br />
The scripts used to generate this summary data are primarily run from /home/gxprojN/halld/monitoring/process . If you want a new copy of the scripts, e.g., for a new monitoring run, you should check the scripts out from SVN:<br />
<syntaxhighlight><br />
svn co https://halldsvn.jlab.org/repos/trunk/scripts/monitoring/process<br />
</syntaxhighlight><br />
Note that these scripts currently have some parameters which must be periodically set by hand.<br />
<br />
The default python version on most JLab machine does not have the modules to allow these scripts to connect to the MySQL database. To run these scripts, load the environment with the following command<br />
<syntaxhighlight><br />
source /home/gxproj1/halld/monitoring/process/monitoring_env.csh<br />
</syntaxhighlight><br />
<br />
===Online Monitoring===<br />
<br />
There are two primary scripts for running over the monitoring data generated by the online system and offline reconstruction. The online script can be run with either of the following commands:<br />
<syntaxhighlight><br />
/home/gluex/halld/monitoring/process/check_new_runs.py<br />
<br />
OR <br />
<br />
/home/gluex/halld/monitoring/process/check_new_runs.csh<br />
</syntaxhighlight><br />
The shell script sets up the environment properly to run the python script. To connect to the monitoring database on the JLab CUE, modules included in the local installation of python >= 2.7 are needed. The shell script is appropriate to use in a cron job. The cronjob is currently run under the "gluex" account.<br />
<br />
The online monitoring system copies a ROOT file containing the results of the online monitoring, and other configuration files into a directory accessible outside the counting house. This python script automatically checks for new ROOT files, which it will then automatically process. It contains several configuration variables that must be correctly set, which contains the location of input/output directories, etc...<br />
<br />
===Offline Monitoring===<br />
<br />
After the data is run over, the results should be processed, so that summary data is entered into the monitoring database and plots are made for the monitoring webpages. Currently, this processing is controlled by a cronjob that runs the following script:<br />
<syntaxhighlight><br />
/home/gxproj1/halld/monitoring/process/check_monitoring_data.csh <br />
</syntaxhighlight><br />
This script checks for new ROOT files, and only runs over those it hasn't processed yet. Since one monitoring ROOT file is produced for each EVIO file, whenever a new file is produced, the plots for the corresponding run are recreated and all the ROOT files for a run are combined into one file. Information is stored in the database on a per-file basis. <br />
<br />
Plots for the monitoring web page can be made from single histograms or multiple histograms using RootSpy macros. If you want to change the list of plots made, you must modify one of the following files:<br />
* histograms_to_monitor - specify either the name of the histogram or its the full ROOT path <br />
* macros_to_monitor - specify the full path to the RootSpy macro .C file<br />
<br />
When a new monitoring run is started, or the conditions are changed, the following steps should be taken to process the new files:<br />
# Add a new data version, as described below:<br />
# Change the following parameters in check_monitoring_data.csh:<br />
## JOBDATE should correspond to the ouptut date used by the job submission script<br />
## OUTPUTDIR should correspond to the directory corresponding to the run period and revision corresponding to the new version you just submitted. Presumably, this directory will be empty at the beginning.<br />
## Once you create a new data version as defined below, you should pass the needed information as a command line option. Currently this is done by the ARGS variable. For example, the argument "-v RunPeriod-2014-10,8" tells the monitoring scripts to look up the version corresponding to revision 8 of RunPeriod-2014-10 in the monitoring DB and to use to store the results.<br />
<br />
<syntaxhighlight><br />
Example configuration parameters:<br />
set JOBDATE=2015-01-09<br />
set INPUTDIR=/volatile/halld/RunPeriod-2014-10/offline_monitoring<br />
set OUTPUTDIR=/w/halld-scifs1a/data_monitoring/RunPeriod-2014-10/ver08<br />
set ARGS=" -v RunPeriod-2014-10,8 "<br />
</syntaxhighlight><br />
If you want to process the results manually, the data is processed using the following script:<br />
<syntaxhighlight><br />
./process_new_offline_data.py <input directory> <output directory><br />
<br />
EXAMPLE:<br />
<br />
./process_new_offline_data.py 2014-11-14 /volatile/halld/RunPeriod-2014-10/offline_monitoring/ /w/halld-scifs1a/data_monitoring/RunPeriod-2014-10/ver02<br />
</syntaxhighlight><br />
The python script takes several options to enable/disable various steps in the processing. Of interest is the "--force" option, which will run over all monitoring ROOT files, whether or not they've been previously identified.<br />
<br />
Every time a new reconstruction pass is performed, a new version number must be generated. To do this, prepare a version file as described below. Then run the register_new_version.py script to store the information in the database. The script will return a version number, which then should be set by hand in process_new_offline_data.py - future versions of the script will streamline this part of the procedure. An example of how to generate a new version is:<br />
<syntaxhighlight><br />
./register_new_version.py add /home/gxproj1/halld/monitoring/process/versions/vers_RunPeriod-2014-10_pass1.txt<br />
</syntaxhighlight><br />
<br />
<b>If you are running the offline monitoring by checking out the files in trunk/scripts/monitoring/jproj/projects/<br />
of the svn repository</b>, and created a project with<br />
create_project.sh [project name] hd_rawdata<br />
Then go to the directory [project name]/processing/<br />
and execute<br />
./run_processing.sh<br />
which will run register_new_version.py as well as check_monitoring_data.csh for that project.<br />
<br />
===Step-by-Step Instructions For Processing a New Monitoring Run===<br />
<br />
The monitoring runs are current run out of the gxproj1 and gxproj5 accounts. After an offline monitoring run has been successfully started on the batch farm, the following steps should be followed to setup the post-processing for these runs.<br />
<br />
# The post-processing scripts are stored in $HOME/halld/monitoring/process and are automatically run by cron.<br />
# Run "svn update" to bring any changes in. Be sure that the list of histograms and macros to plot are current.<br />
# Add a new data version<br />
# Edit check_monitoring_data.csh to point to the current revisions/directories<br />
#* VERSION<br />
#* ARGS<br />
#* Note that the environment depends on a standard script - $HOME/setup_jlab.csh<br />
# Update files in the web directory, so that the results are displayed on the web pages: /group/halld/www/halldweb/html/data_monitoring/textdata<br />
# Copy the REST files to more permanent locations:<br />
#* cp -a /volatile/halld/offline_monitoring/RunPeriod-YYYY-MM/verVV/REST /work/halld/data_monitoring/RunPeriod-YYYY-MM/REST/verVV<br />
#* cp -a /volatile/halld/offline_monitoring/RunPeriod-YYYY-MM/verVV/REST /cache/halld/RunPeriod-YYYY-MM/REST/verVV [under testing]<br />
<br />
Check log files in $HOME/halld/monitoring/process/log for more information on how each run went. If there are problems, check log files, and modify check_monitoring_data.csh to vary the verbosity of the output.<br />
<br />
==Data Versions==<br />
<br />
To document the conditions of the monitoring data that is created, for the sake of reproducability and further analysis we save several pieces of information. The format is intended to be comprehensive enough to document not just monitoring data, but versions of raw and reconstructed data, so that this database table can be used for the event database as well.<br />
<br />
We store one record per pass through one run period, with the following structure:<br />
<br />
{| class="wikitable"<br />
! Field !! Description<br />
|-<br />
| data_type || The level of data we are processing. For the purposes of monitoring, "rawdata" is the online monitoring, "recon" is the offline monitoring<br />
|- <br />
| run_period || The run period of the data<br />
|- <br />
| revision || An integer specifying which pass through the run period this data corresponds to<br />
|- <br />
| software_version || The name of the XML file that specifies the different software versions used<br />
|-<br />
| jana_config || The name of the text file that specifies which JANA options were passed to the reconstruction program<br />
|-<br />
| ccdb_context || The value of JANA_CALIB_CONTEXT, which specifies the version of calibration constants that were used<br />
|-<br />
| production_time || The data at which monitoring/reconstruction began<br />
|-<br />
| dataVersionString || A convenient string for identifying this version of the data<br />
|}<br />
<br />
<br />
An example file used as as input to ./register_new_version.py is:<br />
<syntaxhighlight><br />
data_type = recon<br />
run_period = RunPeriod-2014-10<br />
revision = 1<br />
software_version = soft_comm_2014_11_06.xml<br />
jana_config = jana_rawdata_comm_2014_11_06.conf<br />
ccdb_context = calibtime=2014-11-10<br />
production_time = 2014-11-10<br />
dataVersionString = recon_RunPeriod-2014-10_20141110_ver01<br />
</syntaxhighlight></div>Kmoriyahttps://halldweb.jlab.org/wiki/index.php?title=Data_Monitoring_Procedures&diff=71252Data Monitoring Procedures2015-11-08T02:40:28Z<p>Kmoriya: /* Starting the Launch and Submitting Jobs */</p>
<hr />
<div>__TOC__<br />
<br />
== Master List of File / Database / Webpage Locations ==<br />
=== Run Conditions ===<br />
* Online Run-by-run condition files (B-field, current, etc.): /work/halld/online_monitoring/conditions/<br />
* Offline monitoring run conditions (software versions, jana config): /group/halld/data_monitoring/run_conditions/<br />
*[http://www.jlab.org/Hall-D/test/RunInfo/ Run Info vers. 1]<br />
*[https://halldweb.jlab.org/cgi-bin/data_monitoring/run_conditions.pl Run Info vers. 2]<br />
<br />
=== Monitoring Output Files ===<br />
* Run Periods 201Y-MM is for example 2015-03, launch ver verVV is for example ver15<br />
* Online monitoring histograms: /work/halld/online_monitoring/root/<br />
* Offline monitoring histogram ROOT files: /work/halld/data_monitoring/RunPeriod-201Y-MM/verVV/rootfiles<br />
* REST files (most recent launch only): /work/halld/data_monitoring/RunPeriod-201Y-MM/REST/verVV<br />
* individual files for each job (ROOT, log, etc.): /volatile/halld/offline_monitoring/RunPeriod-201Y-MM/verVV/<br />
<br />
=== Monitoring Database ===<br />
* Accessing monitoring database (on ifarm): mysql -u datmon -h hallddb.jlab.org data_monitoring<br />
<br />
=== Monitoring Webpages ===<br />
*[https://halldweb.jlab.org/cgi-bin/data_monitoring/monitoring/plotBrowser.py Plot Browser]<br />
*[https://halldweb.jlab.org/cgi-bin/data_monitoring/monitoring/runBrowser.py Run Browser]<br />
*[https://halldweb.jlab.org/cgi-bin/data_monitoring/monitoring/timeSeries.py Time Series]<br />
*[https://halldweb.jlab.org/data_monitoring/launch_analysis/ Launch Analysis]<br />
<br />
== Job Monitoring Links ==<br />
* [http://scicomp.jlab.org/farm2/job.html Completed Job History]<br />
* [http://scicomp.jlab.org/farm2/project.html Job Stats By Project]<br />
* [http://scicomp.jlab.org/farm2/trackOrg.html Job Stats By Track]<br />
* [http://scicomp.jlab.org/farm2/report.html Cluster Report]<br />
* [http://scicomp.jlab.org/farm2/walltime.html Walltime Distribution]<br />
<br />
== Saving Online Monitoring Data ==<br />
<br />
The procedure for writing the data out is given in, e.g.,<br />
[https://halldweb.jlab.org/wiki/index.php/Raid-to-Silo_Transfer_Strategy Raid-to-Silo Transfer Strategy].<br />
<br />
Once the DAQ writes out the data to the raid disk, cron jobs will copy the file to tape,<br />
and within ~20 min., we will have access to the file on tape at<br />
/mss/halld/$RUN_PERIOD/rawdata/RunXXXXXX.<br />
<br />
All online monitoring plugins will be run as data is taken.<br />
They will be accessible within the counting house via RootSpy, and<br />
for each run and file, a ROOT file containing the histograms will be saved<br />
within a subdirectory for each run.<br />
<br />
For immediate access to these files, the raid disk files may be accessed directly<br />
from the counting house, or the tape files will be available within ~20 min. of the<br />
file being written out.<br />
<br />
== Offline Monitoring: Running Over Archived Data ==<br />
<br />
Once files are written to tape we run the online plugins on these files to confirm what we were seeing in the online monitoring, and also to update the results from the latest calibration and software. <br />
Manual scripts and cron jobs are set up to look for new data and run the plugins over a sample of files.<br />
<br />
Every other Friday (usually the Friday before the offline meetings) jobs will be started to run the newest software on all previous runs,and allows everybody to see improvements in each detector.<br />
For each launch, independent builds of hdds, sim-recon, the monitoring plugins, and an sqlite file will be generated.<br />
<br />
Below the procedures are described for<br />
# Preparing the software for the launch<br />
# Starting the launch (using hdswif)<br />
# Post-analysis of statistics of the launch<br />
<br />
Processing the results and making them available to the collaboration<br />
is handled in the section [[#Post-Processing_Procedures | Post-Processing Procedures]] below.<br />
<br />
<!--<br />
==== Generating an offline plugin job ====<br />
<br />
The user gxproj1 should be used for official offline monitoring jobs.<br />
Within /home/gxproj1/halld/monitoring/batch/ there will be scripts to run the online monitoring plugins over tape files.<br />
The main script is generatejobs_plugins_rawdata.sh, which can be used as<br />
generatejobs_plugins_rawdata.sh [minrun] [maxrun] (minfile) (maxfile)<br />
where minrun, maxrun specify the range of the run #, and minfile, maxfile (optional) specify<br />
the file range for that run.<br />
<br />
This will generate a script run_rawdata_XXXXXX.sh, where the run # has now been formatted to be 6 digits.<br />
Executing this script will send the monitoring plugins job to the Auger batch system.<br />
<br />
There is also a script clean.sh which can be used as<br />
clean.sh XXX<br />
This will clean up all associated files created in association with run XXX.<br />
<br />
Internally, the xml file used to submit the job will be created, and the job to run<br />
will be given within script.sh. All run parameters should be specified in at the beginning<br />
of generatejobs_plugins_rawdata.sh<br />
Since we are running on tape, the tape file will first be copied over to the cache disk, and the job will run<br />
over this cached file.<br />
<br />
==== Using cron to run automatically ====<br />
Within /home/gluex/halld/monitoring/cron/ there is a file cron_plugins<br />
that can be executed via<br />
crontab cron_plugins<br />
This will set up a cron job to call the script scan_for_jobs.sh, which will<br />
check in the rawdata directory and call generatejobs_plugins_rawdata.sh for<br />
any run that is more than 5 min old. The cron job is set up to run every 10 min.<br />
<br />
==== Magnetic Field Settings ====<br />
For tracking, it is necessary to set the correct magnetic field settings.<br />
The below table is now obsolete. Please refer to the [https://halldweb.jlab.org/cgi-bin/data_monitoring/run_conditions.pl GlueX Run Conditions] webpage.<br />
<br />
The actual field settings for each run have not been documented well.<br />
Below are values based on going through entries in the [https://logbooks.jlab.org/book/halld halld log].<br />
<br />
{| border="1" cellpadding="1" style="text-align: center;"<br />
!width="100"| Run #s<br />
!width="150"| Solenoid Current (A)<br />
!width="300"| JANA option<br />
!width="300"| notes<br />
|-<br />
| 940 - 996 || 1000 || -PBFIELD_MAP=Magnets/Solenoid/solenoid_1000A_poisson_20141104 || At run 997, the solenoid started ramping down<br />
|-<br />
| 998 - 1448 || 0 || -PBFIELD_TYPE=NoField -PDEFTAG:DTrackCandidate=StraightLine || Solenoid started ramping up around run 1431. Runs 1432 - 1448 should not be used for tracking. See [https://logbooks.jlab.org/entry/3309039 this] and [https://logbooks.jlab.org/entry/3309092 this] entry.<br />
|-<br />
| 1449 - 1620 || 1200 || -PBFIELD_MAP=Magnets/Solenoid/solenoid_1200A_poisson_20140520 || <br />
|-<br />
|}<br />
<br />
'''Note:''' The following run ranges were taken with a 300A field, and could be analyzed with the magnetic field map Magnets/Solenoid/solenoid_300A_poisson_20140819 , more study is needed to see if the straight line track fitter is needed or not: 1036 - 1053, 1065 - 1121, 1212? - 1254, 1309 - 1318 [1307,1308,1319-1329 were taken while the magnet was ramping].<br />
--><br />
<br />
=== General Information on Procedures ===<br />
This section explains how the offline monitoring should be run.<br />
Since we may want to simultaneously run offline monitoring for different run periods that require<br />
different environment variables, the scripts are set up so that a generic user can download the<br />
scripts and run them from anywhere. Most output directories for offline monitoring are created<br />
with group read/write permissions so that any Hall D group user has access to the contents,<br />
but there are some cases where use of the account that created the launch is necessary.<br />
<br />
The accounts used for offline monitoring are the gxprojN accounts created and maintained by<br />
Mark Ito (see [https://halldweb.jlab.org/wiki/index.php/GlueX-related_shared_accounts_on_the_JLab_CUE here] for how each account is used).<br />
As of October 2015, the following are used:<br />
* gxproj1 for running over Fall 2014 data (deprecated since June 2015)<br />
* gxproj5 for running over Spring 2015 data<br />
<br />
Since the summer of 2015 we have transitioned from a system using Mark Ito's jproj scripts<br />
to integrating the swif system that Chris Larrieu (SciComp) has been developing. For offline monitoring,<br />
the hdswif system that Kei developed is used for launching the jobs, and the jproj system is used<br />
for meta-analysis of launch statistics.<br />
<br />
Both hdswif and jproj are maintained in svn:<br />
* hdswif: https://halldsvn.jlab.org/repos/trunk/scripts/monitoring/hdswif<br />
* jproj : https://halldsvn.jlab.org/repos/trunk/scripts/monitoring/jproj<br />
<br />
To run the offline monitoring each package should be checked out and all necessary<br />
scripts are included.<br />
<br />
=== Preparing the software for the launch ===<br />
<br />
To begin a new launch the software must be built to the latest versions.<br />
For the gxprojN user accounts used, all software builds are contained in the directory ~/builds<br />
(which are soft links to /work/halld/home/gxprojN/builds). When logging into these accounts<br />
the setup files ~/setup_jlab-2015-03.csh or similar files should be sourced.<br />
<br />
<b>Note that Mark Ito does <u>not</u> want you to change the contents of each .cshrc file.</b><br />
You should consult him if you feel the need.<br />
<br />
1. Setup the environment: <pre>source ~/setup_jlab-2015-03.csh</pre><br />
2. Building hdds: <br />
<pre><br />
cd ~/builds/hdds/hdds<br />
git pull # Get latest software<br />
scons -c install # Clean out the old install: EXTREMELY IMPORTANT for cleaning out stale headers<br />
scons install -j4 # Rebuild and re-install with 4 threads<br />
</pre><br />
3. Building sim-recon: <br />
<pre><br />
cd ~/builds/sim-recon/sim-recon/<br />
git pull<br />
cd src<br />
scons -c install # Clean out the old install: EXTREMELY IMPORTANT for cleaning out stale headers<br />
scons install -j4 # Rebuild and re-install with 4 threads<br />
</pre><br />
4. Prepare the latest sqlite file: The sqlite is set in the ~/setup_jlab-2015-03.csh script as <b>sqlite:////home/gxproj5/ccdb.sqlite</b> through the environment variables <b>JANA_CALIB_URL</b> and <b>CCDB_CONNECTION</b>. Therefore, go to this directory and create a new sqlite file. We create the sqlite file in a temporary directory since creating the sqlite file in a directory where the output file exists causes errors. Original documentation on creating sqlite files are [https://halldweb.jlab.org/wiki/index.php/SQLite-form_of_the_CCDB_database here]. <br><span style="color:red">NOTE: SQLITE FILES DO NOT WORK ON THE NEW /work DISK INSTALLED IN OCTOBER 2015 </span><br />
<pre><br />
cd ~/tmp<br />
$CCDB_HOME/scripts/mysql2sqlite/mysql2sqlite.sh -hhallddb.jlab.org -uccdb_user ccdb | sqlite3 ccdb.sqlite<br />
mv ccdb.sqlite ../<br />
</pre><br />
5. Note that the above steps must be done <b>BEFORE</b> launch project creation. This is because we will track the revisions of the libraries used, and this is done by extracting the svn information in each directory. Also, note that the system assumes that we have the topmost build directory (usually called GLUEX_TOP) to be $HOME/builds . Such an assumption is necessary to be able to extract information about the library locations automatically.<br />
<br />
Create the appropriate project(s) and submit the jobs using hdswif, as detailed in the section below.<br />
<br />
=== Starting the Launch and Submitting Jobs ===<br />
<br />
Until the summer of 2015 we relied solely on Mark Ito's jproj system for submitting and keeping track of jobs. We have since moved to the swif system and use the hdswif wrapper for this. Below are instructions for how to use these.<br />
<br />
# Downloading hdswif: Download the hdswif directory from svn. For the gxprojN accounts, use the directory ~/halld/hdswif. <pre>cd ~/halld </pre> <pre> svn co https://halldsvn.jlab.org/repos/trunk/scripts/monitoring/hdswif</pre> <pre>cd hdswif</pre><br />
# Creating the workflow: Within SWIF jobs are registered into workflows. First create the workflow. For offline monitoring, the workflow names are of the form <b>offline_monitoring_RunPeriod201Y_MM_verVV_hd_rawdata</b> with suitable replacements for the run period and version number. The command "swif list" will list all existing workflows. Also, for most simple SWIF commands hdswif also provides a wrapper. <pre>swif list</pre> For creation of workflows for offline monitoring the command <pre>hdswif.py create [workflow] -c [config file] </pre> should be used. As an example config file, see the input.config file in the folder (and update it). When a config file is passed in, hdswif will automatically create files that record the configuration of the current launch. These files are stored as for example<br />
#* /group/halld/data_monitoring/run_conditions/jana_rawdata_comm_2015_03_ver15.conf<br />
#* /group/halld/data_monitoring/run_conditions/soft_comm_2015_03_ver15.xml<br />
#* The software packages stored in git (sim-recon and hdds) can have git tags applied to them, which makes it easier to find versions of the software than a SHA-1 hash. hdswif will ask if you would like to create a tag, and execute the following sequence: <pre>git tag -a offmon-201Y_MM-verVV -m "Used for offline monitoring 201Y-MM verVV started on 201y/mm/dd"</pre> <pre>git push offmon-201Y_MM-verVV</pre> This will only be invoked when the user name is gxprojN, and for the configuration files, the output directory will be /group/halld/data_monitoring/run_conditions/ for gxprojN accounts while it will be the current directory for other users.<br />
#* To use the git-tagged software versions do for example <pre>cd $HALLD_HOME</pre> <pre>git checkout offmon-2015_03-ver15</pre><br />
# Registering jobs in the workflow: To register jobs within the workflow, hdswif provides the use of config files. Jobs can be registered by specifying the workflow, config file (-c), run (-r) and file (-f) numbers if necessary. A typical config file will look this:<br />
PROJECT gluex<br />
TRACK reconstruction<br />
OS centos65<br />
NCORES 6<br />
DISK 40<br />
RAM 8<br />
TIMELIMIT 8<br />
JOBNAMEBASE offmon_<br />
RUNPERIOD 2015-03<br />
VERSION 15<br />
OUTPUT_TOPDIR /volatile/halld/offline_monitoring/RunPeriod-[RUNPERIOD]/ver[VERSION] # Example of other variables included in variable<br />
SCRIPTFILE /home/gxproj5/halld/hdswif/script.sh # Must specify full path<br />
ENVFILE /home/gxproj5/halld/hdswif/setup_jlab-2015-03.csh # Must specify full path<br />
The config file contains configuration parameters for each of the jobs.<br><br />
<span style="color:red">Note: Job configuration parameters can be set differently for jobs within the same workflow if necessary.</span><br />
Edit the config file and save as a new file if necessary. Once the configuration is set, jobs can be added via <pre>hdswif.py add [workflow] -c input.config</pre><br />
By default, hdswif will add all files found within the directory /mss/halld/RunPeriod-201Y-MM/rawdata/ where 201Y-MM is specified by the RUNPERIOD parameter in the config file. If only some of the runs or files are needed, these can be specified for example with<br />
<pre>hdswif.py add [workflow] -c input.config -r 3180 -f '00[0-4]'</pre><br />
to specify to register running only over run 3180 files 000 - 004 (Unix-style brackets and wildcards can be used).<br />
<br />
Running the workflow: To run the workflow, simply use swif run: <pre>swif run -workflow [workflow] -errorlimit none</pre> <span style="color:red">MAKE SURE THE ERRORLIMIT IS SET TO NONE OR THE WORKFLOW WILL BE STOPPED AFTER ANY JOB FAILS</span> or equivalently, using the hdswif wrapper (which has the errorlimit set by default), <pre>hdsswif.py run [workflow]</pre><br><br />
<b>It is recommended that some jobs be tested to make sure that everything is working rather than fail thousands of jobs.</b><br> For this purpose, hdswif will take an additional parameter to run which limits the number of jobs to submit: <pre>hdswif.py run [workflow] 10</pre><br />
in which case only 10 jobs will be submitted.<br />
<br />
<b>Checklist to make sure jobs are running correctly:</b><br />
* Check stderr files. Are they very large (>kB)?<br />
* Check stdout files. Are they very large (>MB)?<br />
* Check output ROOT files. Are they larger than several MB?<br />
* Check output REST files. Are they larger than several tens of MB?<br />
To submit all jobs after checking the results, do <pre>hdswif.py run [workflow]</pre><br />
<br />
=== Checking the Status and Resubmitting ===<br />
1. The status of jobs can be checked on the terminal with <pre>jobstat -u gxprojN</pre> on Auger and for SWIF with <pre>swif list</pre> or for more information, <pre>swif status [workflow] -summary</pre> Note that "swif status" tends to be out of date sometimes, so don't panic if your workflow/jobs aren't showing up right away. Also see the Auger [http://scicomp.jlab.org/scicomp/#/auger/jobs job website].<br />
<br />
2. For failed jobs, SWIF can resubmit jobs based on the problem. For resubmission for failed jobs with the same resources, <pre>swif retry-jobs [workflow] -problems [problem name]</pre> can be used, and for jobs to be submitted with more resources, e.g., use <pre>swif modify-jobs -ram add 2gb -problems AUGER-OVER_RLIMIT</pre> <br />
This only re-stages the jobs, be sure to resubmit them with:<br />
<pre>swif run -workflow [workflow] -errorlimit none</pre><br />
<br />
hdswif has a wrapper for both of these: <pre>hdswif.py resubmit [workflow] [problem]</pre> In this case [problem] can be one of <b>SYSTEM, TIMEOUT, RLIMIT</b>. If SYSTEM is specified, the jobs will be retried. For TIMEOUT and RLIMIT, the jobs will be modified by default with 2 additional hours or GB of RAM. If one more number is added as an option, then that many hours or GB of RAM will be added., e.g., <pre>hdswif.py resubmit [workflow] TIMEOUT 5</pre> will add 5 hours of processing time. You can wait until almost all jobs finish before resubmitting failed jobs since the number should be relatively small. <b>Even if jobs are resubmitted for one type of failure, jobs that later fail with that failure will not be automatically resubmitted.</b><br />
<br />
3. For information on swif, use the "swif help" commands and for hdswif see the attached documentaion in https://halldsvn.jlab.org/repos/trunk/scripts/monitoring/hdswif/manual_hdswif.pdf<br />
<br />
4. Below is a table describing the various errors that can occur.<br />
{| border="1" cellpadding="0" valign="left" style="text-align: left;"<br />
!width="150"| ERROR NAME<br />
!width="400"| Description<br />
!width="400"| Resolution<br />
!width="400"| hdswif command<br />
|-<br />
| AUGER-SUBMIT<br />
||<br />
SWIF’s attempt to submit jobs to Auger failed. Includes server-side problems as well as user failing to provide valid job parameters (e.g. incorrect project name, too many resources, etc.)<br />
||<br />
If requested resources are known to be correct resubmit. Otherwise modify job resources using swif directly.<br />
||<br />
<span style="color:purple"><b>hdswif.py resubmit [workflow] SYSTEM</b></span><br />
|-<br />
| AUGER-FAILED<br />
||<br />
Auger reports the job FAILED with no specific details.<br />
||<br />
Resubmit jobs. If problem persists, contact Chris Larrieu or SciComp.<br />
||<br />
<span style="color:purple"><b>hdswif.py resubmit [workflow] SYSTEM</b></span><br />
|-<br />
| AUGER-OUTPUT-FAIL<br />
||<br />
Failure to copy one or more output files.Can be due to permission problem, quota problem, system error, etc.<br />
||<br />
Check if output files will exist after job execution and that output directory exists, resubmit jobs. If problem persists, contact Chris Larrieu or SciComp.<br />
||<br />
<span style="color:purple"><b>hdswif.py resubmit [workflow] SYSTEM</b></span><br />
|-<br />
| AUGER-INPUT-FAIL<br />
||<br />
Auger failed to copy one or more of the requested input files, similar to output failures. Can also happen if tape file is unavailable (e.g. missing/damaged tape)<br />
||<br />
Check if input file exists, resubmit jobs. If problem persists, contact Chris Larrieu or SciComp.<br />
||<br />
<span style="color:purple"><b>hdswif.py resubmit [workflow] SYSTEM</b></span><br />
|-<br />
| AUGER-TIMEOUT<br />
||<br />
Job timed out.<br />
||<br />
If more time is needed for job add more resources.<br />
Default is to add 2 hrs of processing time. Also check whether code is hanging.<br />
||<br />
<span style="color:red"><b>hdswif.py resubmit [workflow] TIMEOUT</b></span><br><br />
Default is to add 2 hours. Optionally specify number of hours at end.<br />
|-<br />
| AUGER-OVER_RLIMIT<br />
||<br />
Not enough resources, RAM or disk space.<br />
||<br />
Add more resources for job.<br />
||<br />
<span style="color:blue"><b>hdswif.py resubmit [workflow] RLIMIT</b></span><br><br />
Default is to add 2 GB of RAM. Optionally specify GB at end. To add more disk space use SWIF directly.<br />
|-<br />
| SWIF-MISSING-OUTPUT<br />
||<br />
Output file specified by user was not found.<br />
||<br />
Check if output file exists at end of job.<br />
||<br />
<br />
|-<br />
| SWIF-USER-NON-ZERO <br />
||<br />
User script exited with non-zero status code.<br />
||<br />
Your script exited with non-zero status. Check the code you are running.<br />
||<br />
<br />
|-<br />
| SWIF-SYSTEM-ERROR <br />
||<br />
Job failed owing to a problem with swif (e.g. network connection timeout)<br />
||<br />
Resubmit jobs. If problem persists, contact Chris Larrieu or SciComp.<br />
||<br />
<span style="color:purple"><b>hdswif.py resubmit [workflow] SYSTEM</b></span><br />
|}<br />
<br style="clear:both;"/><br />
<br />
=== Post-analysis of statistics of the launch ===<br />
<br />
# After jobs have been submitted, it will usually take a few days for all of the jobs to be processed.<br />
#* Status of Auger: http://scicomp.jlab.org/scicomp/#/auger/jobs (see also links above)<br />
#* Status of user jobs: <pre>jobstat -u [user name]</pre><br />
# The status and results of jobs are saved within the SWIF internal server, and are available via the command <pre>swif status [workflow] -summary -runs</pre> where the arguments -summary and -runs show summary statistics and statistics for individual jobs, respectively. hdswif has a command that takes this output in XML output and creates an HTML webpage showing results of the launch. To do this, do <pre>hdswif.py summary [workflow]</pre> This will create an XML file swif_output_[workflow].xml that contains all information from SWIF. If the file already exists, hdswif will ask whether to overwrite the existing file.<br />
# At this stage the html output and figure files are created and ready to be put online. For this step and other steps involving analysis of the statistics of the launch results, it is convenient to change to the jproj system.<br />
# The jproj scripts for offline monitoring are maintained in the svn directory https://halldsvn.jlab.org/repos/trunk/scripts/monitoring/jproj Do <pre> svn co https://halldsvn.jlab.org/repos/trunk/scripts/monitoring/jproj </pre> For the gxprojN accounts used for offline monitoring the directory should be ~/halld/jproj<br />
# The jproj directory contains two subdirectories, scripts and projects. The scripts directory contains useful scripts for processing the jobs registered in the jproj system, and each of the offline monitoring launches will be handled in the projects directory.<br />
# Go to the projects directory<pre>cd ~/halld/jproj/projects</pre> and use the script create_project.sh to create a new directory that contains the processing scripts for the current launch <pre>./create_project.sh [workflow]</pre> This should create a directory such as offline_monitoring_RunPeriod2015_03_ver15_hd_rawdata (same as the workflow name). The script uses the template files in the directory templates and by substitution creates script files for the current launch. Now go to the newly created analysis directory: <pre>cd [workflow]/analysis</pre><br />
which for the gxprojN accounts should have the full path /home/gxprojN/halld/jproj/projects/[workflow]/analysis.<br><br><br />
<span style="color:red">All of the analysis commands including arguments are contained in run_analysis.sh, but it is <b>strongly</b> recommended that all commands are run manually to check for errors.</span><br><br><br />
# The first thing to do is to make the html output from hdswif public. Copy the html file and related figures that were created from hdswif to the appropriate space within /group/halld/www/halldweb/html/data_monitoring/ : <pre>python publish_offmon_results.py [run period] [version]</pre> Note that the command with appropriate substitutions for arguments can be found within run_analysis.sh (same for all commands below).<br />
# Next, we need to create a few MySQL tables for the current launch. The MySQL tables are useful for comparing run/file combinations for different launches. For the SWIF launches, there are two tables needed, and will be named<br />
#* [workflow]Job<br />
#* [workflow]_aux<br />
# This naming scheme of tables and their roles are the same as from the jproj only launches. The [workflow]Job table will contain information gathered from SWIF about each job (which node it went to, start time of each stage, memory usage, etc.). The [workflow]_aux table will contain information gathered from the output stdout files from each job. First, create the Job table using <pre>python create_jproj_job_table.py [run period] [version]</pre> Note that this script uses the XML output from hdswif summary and inserts the contents into the MySQL table, so the XML output file must exist.<br />
# To check the contents of this MySQL table, do <pre>mysql -hhalldb -ufarmer farming</pre> <pre>mysql> describe offline_monitoring_RunPeriod2015_03_ver11_hd_rawdataJob;</pre><br />
<br />
== Hall D Job Management System ==<br />
<br />
This section details instructions on how to create and launch a set of jobs using the Hall-D Job Management System developed by Mark Ito. These instructions are generic: this system can be used for the weekly monitoring jobs, but can also be used for other sets of job launches as well. <br />
<br />
=== Database Table Overview ===<br />
<br />
* Job management database table (<project_name>): For each input file, keeps track of whether or not a job for it has been submitted, along with other optional fields. <br />
<br />
* Job status database table (<project_name>Job (no space)): For each job, keeps track of the job-id, the job status, memory used, cpu & wall time, time taken to complete various stages (e.g. pending, dependency, active), and others. <br />
<br />
* Job metrics database table (<project_name>_aux (no space)): For each job, keeps track of the job-id, how many events were processed, the time it took to copy the cache file, and the time it took to run the plugin. This information is culled from the log files of each job, and is done within the analysis directory of each launch.<br />
<br />
=== Initialize Project Management ===<br />
<br />
* Log into the ifarm machine with one of the gxproj accounts. For this example we will use gxproj1.<br />
<pre><br />
ssh gxproj1@ifarm -Y<br />
</pre><br />
<br />
* Go to a directory to do the launch. In principle, any directory will work, but for gxproj1 this is usually done in /home/gxproj1/halld/<br />
<br />
* Check out the necessary scripts <pre>svn co https://halldsvn.jlab.org/repos/trunk/scripts/monitoring/jproj</pre> This will get all necessary scripts for launching. Once checked out, <pre>cd projects</pre><br />
<br />
* The script create_project.sh can be used to create a new project. It will take a single argument, the project name. It is assumed that the project name is of the form offline_monitoring_RunPeriod20YY_MM_verVV_hd_rawdata where this string will be parsed to give the run period 20YY_MM and version number VV. One thing to do BEFORE creation of a new project is to editthe conditions of the launch (plugins to run over, memory requested, disk space requested) within templates/template.jsub . This<br />
information is saved automatically at project creation time into files in /group/halld/data_monitoring/run_conditions .<br />
To create a project do <pre>./create_project.sh offline_monitoring_RunPeriod20YY_MM_verVV_hd_rawdata</pre><br />
The name has been chosen to be as consistent as possible with other directory structures.<br />
However, mysql requires that "-" be escaped in table names, so unfortunately run periods will be<br />
given as 2YYY_MM instead of 2YYY-MM.<br />
<br />
* A new directory with the project name (offline_monitoring_RunPeriod20YY_MM_verVV_hd_rawdata) and files will be copied and modified from the template directory to reflect the run period, the user, the directory that it was created in, project name, etc.<br />
<br />
* For each project, cd into the new directory<br />
* The script clear.sh will remove any existing tables for the current project name, then recreate it. Do<br />
<pre>./clear.sh</pre><br />
* To use the jproj.pl script that was checked in, add the directory to your path with<br />
<pre>source ../../scripts/setup.csh</pre><br />
or always specify the full path<br />
<pre>../../scripts/jproj.pl</pre><br />
* Now update the table of runs with<br />
<pre>jproj.pl <project name> update</pre><br />
This will fill the table with all files within /mss that are of the same form as what is in <project name>.jproj . If you want to register only on a subset of all such files, you can edit this file directly.<br />
* Once you have registered all of the files you would like to run over, do<br />
<pre>jproj.pl <project name> submit [max # of jobs] [run number]</pre><br />
where the additional options specify how many jobs to submit and which run number to run on. Without these options all files that are registered and have not been submitted yet will be submitted.<br />
<br />
At this stage you are ready to submit all files. It is a good idea to submit a few test jobs<br />
at first to check that all scripts are working and that the plugins do not crash. Once you are<br />
sure that this does not happen, you can send all jobs in. The remaining jobs are then the monitoring<br />
which will (among other things) put the results on the online webpage for the collaboration to view, and<br />
the analysis of the launch.<br />
<br />
=== Project File Overview ===<br />
<br />
An overview of each project file:<br />
* '''clear.sh:''' For the current project, deletes the job status and management database tables (if any), and creates new, empty ones. <br />
* '''<project_name>.jproj:''' Contains the path and file name format for the input files for the jobs. <br />
* '''<project_name>.jsub:''' The xml job submission script. The run number and file number variables are set during job submission for each input file. <br />
* '''script.sh:''' The script that is executed during the job. If output job directories are not pre-created manually, they should be created in this script with the proper permissions:<br />
<pre><br />
mkdir -p -m 775 my_directory<br />
</pre><br />
* '''setup_jlab-[run period].csh:''' The environment that is sourced at the beginning of the job execution. <br />
* '''status.sh:''' Updates the job status database table, and prints some of its columns to screen.<br />
<br />
=== Project Management ===<br />
<br />
* Delete (if any) and create the database table(s) for the current set of job submissions:<br />
<pre><br />
./clear.sh<br />
</pre><br />
Also, if testing was done with jobs, it is best to delete the output directory and the configuration files:<br />
<pre><br />
rm -frv /volatile/halld/offline_monitoring/RunPeriod-20YY-MM/verVV /group/halld/data_monitoring/run_conditions/soft_comm_20YY_MM_verVV.xml /group/halld/data_monitoring/run_conditions/jana_rawdata_comm_20YY_MM_verVV.conf<br />
</pre><br />
<br />
* Search for input files matching the string in the .jproj file, and create a row for each in the job management database table (called <project_name>). You can test by adding an optional argument at the end, which only selects files with a specific file number:<br />
<pre><br />
jproj.pl <project_name> update <optional_file_number><br />
</pre><br />
<br />
* Confirm that the job management database is accurate by printing it's contents to screen:<br />
<pre><br />
mysql -hhallddb -ufarmer farming -e "select * from <project_name>"<br />
</pre><br />
<br />
* ONLY if a mistake was made, to delete the tables from the database and recreate new, empty ones, run: <br />
<pre><br />
./clear.sh<br />
</pre><br />
<br />
* To look at the status of the submitted jobs, first query auger and update the job status database:<br />
<pre><br />
fill_in_job_details.pl <project_name><br />
</pre><br />
<br />
* The job status can then be viewed by submitting a query to the job status database (called <project_name>Job (no space in between)):<br />
<pre><br />
mysql -hhallddb -ufarmer farming -e "select id,run,file,jobId,hostname,status,timeSubmitted,timeActive,walltime,cput,timeComplete,result,error from <project_name>Job"<br />
</pre><br />
<br />
* These last two commands can instead be executed simultaneously by running:<br />
<pre><br />
./status.sh<br />
</pre><br />
<br />
=== Handy mysql Instructions ===<br />
<br />
* Handy mysql instructions:<br />
<pre><br />
mysql -hhallddb -ufarmer farming # Enter the "farming" mysql database on "hallddb" as user "farmer"<br />
quit; # Exit mysql<br />
show tables; # Show a list of the tables in the current database<br />
show columns from <project_name>; # show all of the columns for the given table<br />
select * from <project_name>; # show the contents of all rows from the given table<br />
</pre><br />
<br />
=== Backing Up Offline Monitoring Tables ===<br />
Tables created for offline monitoring can be backed up using the script backup_tables.sh which<br />
can be checked out with the other files from https://halldsvn.jlab.org/repos/trunk/scripts/monitoring/jproj/projects<br />
<br />
The script uses the command mysqldump to print out a file that can be executed to recreate the tables.<br />
Since executing this output file will drop the table when it exists, caution is advised.<br />
Example usage to backup all three tables created for run period 2014_10 ver 17:<br />
<pre>backup_tables.sh 2014_10 17</pre><br />
<br />
== Running Over Data As It Comes In ==<br />
<br />
A special user gxproj1 will have a cron job set up to run the plugins as new data appears on /mss.<br />
During the week, gxproj1 will submit offline plugin jobs with the same setup as the weekly jobs<br />
run the previous Friday. The procedure for this is shown below.<br />
<br />
<!--<br />
=== Setting up the environment ===<br />
The file<br />
/home/gxproj1/setup_jlab.csh<br />
is sourced through .tcshrc.<br />
This file is the same as what is linked to by<br />
/home/gluex/setup_jlab_commissioning.csh,<br />
except HALLD_HOME, HDDS_HOME, and JANA_CALIB_URL are set separately so that this<br />
user can have a separate build.<br />
<br />
To obtain the builds from the previous Friday's runs,<br />
execute<br />
/home/gxproj1/halld/monitoring/newruns/setup_previous.sh [year] [month] [day]<br />
The build revisions from the previous Friday are archived in files<br />
/work/halld/data_monitoring/run_conditions/soft_comm_[year]_[month]_[day].xml<br />
and the script will build libraries based on those stored revision numbers.<br />
--><br />
<br />
=== Running the cron job ===<br />
<br />
'''IMPORTANT:''' The cron job should not be running while you are manually submitting jobs using the jproj.pl script for the same project, or else you will probably multiply-submit a job. <br />
<br />
* Go to the cron job directory: <br />
<pre><br />
cd /u/home/gxproj1/halld/monitoring/newruns<br />
</pre><br />
<br />
* The cron_plugins file is the cronjob that will be executed. During execution, it runs the exec.sh command in the same folder. This command takes two arguments: the project name, and the maximum file number for each run. These fields should be updated in the cron_plugins file before running. <br />
<br />
* The exec.sh command updates the job management database table with any data that has arrived on tape since it was last updated, ignoring file numbers greater than the maximum file number. It then submits jobs for these files. <br />
<br />
* To start the cron job, run:<br />
<pre><br />
crontab cron_plugins<br />
</pre><br />
<br />
* To check whether the cron job is running, do<br />
<pre><br />
crontab -l<br />
</pre><br />
<br />
* To remove the cron job do<br />
<pre><br />
crontab -r<br />
</pre><br />
<br />
<!--<br />
The cron job will run the script scan_for_jobs.sh,<br />
which runs generatejobs_plugins_rawdata.sh for any<br />
new runs that it had not seen before. All previous<br />
runs are recorded in the file filelists/files_current.txt<br />
so clear this to run over runs, or set the parameters<br />
MINRUN and MAXRUN which will set the range of runs submitted.<br />
--><br />
<br />
==Post-Processing Procedures==<br />
<br />
To visualize the monitoring data, we save images of selected histograms and store time series of selected quantities in a database, which are then displayed on a web page. This section describes how to generate the monitoring images and database information.<br />
<br />
The scripts used to generate this summary data are primarily run from /home/gxprojN/halld/monitoring/process . If you want a new copy of the scripts, e.g., for a new monitoring run, you should check the scripts out from SVN:<br />
<syntaxhighlight><br />
svn co https://halldsvn.jlab.org/repos/trunk/scripts/monitoring/process<br />
</syntaxhighlight><br />
Note that these scripts currently have some parameters which must be periodically set by hand.<br />
<br />
The default python version on most JLab machine does not have the modules to allow these scripts to connect to the MySQL database. To run these scripts, load the environment with the following command<br />
<syntaxhighlight><br />
source /home/gxproj1/halld/monitoring/process/monitoring_env.csh<br />
</syntaxhighlight><br />
<br />
===Online Monitoring===<br />
<br />
There are two primary scripts for running over the monitoring data generated by the online system and offline reconstruction. The online script can be run with either of the following commands:<br />
<syntaxhighlight><br />
/home/gluex/halld/monitoring/process/check_new_runs.py<br />
<br />
OR <br />
<br />
/home/gluex/halld/monitoring/process/check_new_runs.csh<br />
</syntaxhighlight><br />
The shell script sets up the environment properly to run the python script. To connect to the monitoring database on the JLab CUE, modules included in the local installation of python >= 2.7 are needed. The shell script is appropriate to use in a cron job. The cronjob is currently run under the "gluex" account.<br />
<br />
The online monitoring system copies a ROOT file containing the results of the online monitoring, and other configuration files into a directory accessible outside the counting house. This python script automatically checks for new ROOT files, which it will then automatically process. It contains several configuration variables that must be correctly set, which contains the location of input/output directories, etc...<br />
<br />
===Offline Monitoring===<br />
<br />
After the data is run over, the results should be processed, so that summary data is entered into the monitoring database and plots are made for the monitoring webpages. Currently, this processing is controlled by a cronjob that runs the following script:<br />
<syntaxhighlight><br />
/home/gxproj1/halld/monitoring/process/check_monitoring_data.csh <br />
</syntaxhighlight><br />
This script checks for new ROOT files, and only runs over those it hasn't processed yet. Since one monitoring ROOT file is produced for each EVIO file, whenever a new file is produced, the plots for the corresponding run are recreated and all the ROOT files for a run are combined into one file. Information is stored in the database on a per-file basis. <br />
<br />
Plots for the monitoring web page can be made from single histograms or multiple histograms using RootSpy macros. If you want to change the list of plots made, you must modify one of the following files:<br />
* histograms_to_monitor - specify either the name of the histogram or its the full ROOT path <br />
* macros_to_monitor - specify the full path to the RootSpy macro .C file<br />
<br />
When a new monitoring run is started, or the conditions are changed, the following steps should be taken to process the new files:<br />
# Add a new data version, as described below:<br />
# Change the following parameters in check_monitoring_data.csh:<br />
## JOBDATE should correspond to the ouptut date used by the job submission script<br />
## OUTPUTDIR should correspond to the directory corresponding to the run period and revision corresponding to the new version you just submitted. Presumably, this directory will be empty at the beginning.<br />
## Once you create a new data version as defined below, you should pass the needed information as a command line option. Currently this is done by the ARGS variable. For example, the argument "-v RunPeriod-2014-10,8" tells the monitoring scripts to look up the version corresponding to revision 8 of RunPeriod-2014-10 in the monitoring DB and to use to store the results.<br />
<br />
<syntaxhighlight><br />
Example configuration parameters:<br />
set JOBDATE=2015-01-09<br />
set INPUTDIR=/volatile/halld/RunPeriod-2014-10/offline_monitoring<br />
set OUTPUTDIR=/w/halld-scifs1a/data_monitoring/RunPeriod-2014-10/ver08<br />
set ARGS=" -v RunPeriod-2014-10,8 "<br />
</syntaxhighlight><br />
If you want to process the results manually, the data is processed using the following script:<br />
<syntaxhighlight><br />
./process_new_offline_data.py <input directory> <output directory><br />
<br />
EXAMPLE:<br />
<br />
./process_new_offline_data.py 2014-11-14 /volatile/halld/RunPeriod-2014-10/offline_monitoring/ /w/halld-scifs1a/data_monitoring/RunPeriod-2014-10/ver02<br />
</syntaxhighlight><br />
The python script takes several options to enable/disable various steps in the processing. Of interest is the "--force" option, which will run over all monitoring ROOT files, whether or not they've been previously identified.<br />
<br />
Every time a new reconstruction pass is performed, a new version number must be generated. To do this, prepare a version file as described below. Then run the register_new_version.py script to store the information in the database. The script will return a version number, which then should be set by hand in process_new_offline_data.py - future versions of the script will streamline this part of the procedure. An example of how to generate a new version is:<br />
<syntaxhighlight><br />
./register_new_version.py add /home/gxproj1/halld/monitoring/process/versions/vers_RunPeriod-2014-10_pass1.txt<br />
</syntaxhighlight><br />
<br />
<b>If you are running the offline monitoring by checking out the files in trunk/scripts/monitoring/jproj/projects/<br />
of the svn repository</b>, and created a project with<br />
create_project.sh [project name] hd_rawdata<br />
Then go to the directory [project name]/processing/<br />
and execute<br />
./run_processing.sh<br />
which will run register_new_version.py as well as check_monitoring_data.csh for that project.<br />
<br />
===Step-by-Step Instructions For Processing a New Monitoring Run===<br />
<br />
The monitoring runs are current run out of the gxproj1 and gxproj5 accounts. After an offline monitoring run has been successfully started on the batch farm, the following steps should be followed to setup the post-processing for these runs.<br />
<br />
# The post-processing scripts are stored in $HOME/halld/monitoring/process and are automatically run by cron.<br />
# Run "svn update" to bring any changes in. Be sure that the list of histograms and macros to plot are current.<br />
# Add a new data version<br />
# Edit check_monitoring_data.csh to point to the current revisions/directories<br />
#* VERSION<br />
#* ARGS<br />
#* Note that the environment depends on a standard script - $HOME/setup_jlab.csh<br />
# Update files in the web directory, so that the results are displayed on the web pages: /group/halld/www/halldweb/html/data_monitoring/textdata<br />
# Copy the REST files to more permanent locations:<br />
#* cp -a /volatile/halld/offline_monitoring/RunPeriod-YYYY-MM/verVV/REST /work/halld/data_monitoring/RunPeriod-YYYY-MM/REST/verVV<br />
#* cp -a /volatile/halld/offline_monitoring/RunPeriod-YYYY-MM/verVV/REST /cache/halld/RunPeriod-YYYY-MM/REST/verVV [under testing]<br />
<br />
Check log files in $HOME/halld/monitoring/process/log for more information on how each run went. If there are problems, check log files, and modify check_monitoring_data.csh to vary the verbosity of the output.<br />
<br />
==Data Versions==<br />
<br />
To document the conditions of the monitoring data that is created, for the sake of reproducability and further analysis we save several pieces of information. The format is intended to be comprehensive enough to document not just monitoring data, but versions of raw and reconstructed data, so that this database table can be used for the event database as well.<br />
<br />
We store one record per pass through one run period, with the following structure:<br />
<br />
{| class="wikitable"<br />
! Field !! Description<br />
|-<br />
| data_type || The level of data we are processing. For the purposes of monitoring, "rawdata" is the online monitoring, "recon" is the offline monitoring<br />
|- <br />
| run_period || The run period of the data<br />
|- <br />
| revision || An integer specifying which pass through the run period this data corresponds to<br />
|- <br />
| software_version || The name of the XML file that specifies the different software versions used<br />
|-<br />
| jana_config || The name of the text file that specifies which JANA options were passed to the reconstruction program<br />
|-<br />
| ccdb_context || The value of JANA_CALIB_CONTEXT, which specifies the version of calibration constants that were used<br />
|-<br />
| production_time || The data at which monitoring/reconstruction began<br />
|-<br />
| dataVersionString || A convenient string for identifying this version of the data<br />
|}<br />
<br />
<br />
An example file used as as input to ./register_new_version.py is:<br />
<syntaxhighlight><br />
data_type = recon<br />
run_period = RunPeriod-2014-10<br />
revision = 1<br />
software_version = soft_comm_2014_11_06.xml<br />
jana_config = jana_rawdata_comm_2014_11_06.conf<br />
ccdb_context = calibtime=2014-11-10<br />
production_time = 2014-11-10<br />
dataVersionString = recon_RunPeriod-2014-10_20141110_ver01<br />
</syntaxhighlight></div>Kmoriyahttps://halldweb.jlab.org/wiki/index.php?title=Data_Monitoring_Procedures&diff=71251Data Monitoring Procedures2015-11-08T02:14:47Z<p>Kmoriya: /* Checking the Status and Resubmitting */</p>
<hr />
<div>__TOC__<br />
<br />
== Master List of File / Database / Webpage Locations ==<br />
=== Run Conditions ===<br />
* Online Run-by-run condition files (B-field, current, etc.): /work/halld/online_monitoring/conditions/<br />
* Offline monitoring run conditions (software versions, jana config): /group/halld/data_monitoring/run_conditions/<br />
*[http://www.jlab.org/Hall-D/test/RunInfo/ Run Info vers. 1]<br />
*[https://halldweb.jlab.org/cgi-bin/data_monitoring/run_conditions.pl Run Info vers. 2]<br />
<br />
=== Monitoring Output Files ===<br />
* Run Periods 201Y-MM is for example 2015-03, launch ver verVV is for example ver15<br />
* Online monitoring histograms: /work/halld/online_monitoring/root/<br />
* Offline monitoring histogram ROOT files: /work/halld/data_monitoring/RunPeriod-201Y-MM/verVV/rootfiles<br />
* REST files (most recent launch only): /work/halld/data_monitoring/RunPeriod-201Y-MM/REST/verVV<br />
* individual files for each job (ROOT, log, etc.): /volatile/halld/offline_monitoring/RunPeriod-201Y-MM/verVV/<br />
<br />
=== Monitoring Database ===<br />
* Accessing monitoring database (on ifarm): mysql -u datmon -h hallddb.jlab.org data_monitoring<br />
<br />
=== Monitoring Webpages ===<br />
*[https://halldweb.jlab.org/cgi-bin/data_monitoring/monitoring/plotBrowser.py Plot Browser]<br />
*[https://halldweb.jlab.org/cgi-bin/data_monitoring/monitoring/runBrowser.py Run Browser]<br />
*[https://halldweb.jlab.org/cgi-bin/data_monitoring/monitoring/timeSeries.py Time Series]<br />
*[https://halldweb.jlab.org/data_monitoring/launch_analysis/ Launch Analysis]<br />
<br />
== Job Monitoring Links ==<br />
* [http://scicomp.jlab.org/farm2/job.html Completed Job History]<br />
* [http://scicomp.jlab.org/farm2/project.html Job Stats By Project]<br />
* [http://scicomp.jlab.org/farm2/trackOrg.html Job Stats By Track]<br />
* [http://scicomp.jlab.org/farm2/report.html Cluster Report]<br />
* [http://scicomp.jlab.org/farm2/walltime.html Walltime Distribution]<br />
<br />
== Saving Online Monitoring Data ==<br />
<br />
The procedure for writing the data out is given in, e.g.,<br />
[https://halldweb.jlab.org/wiki/index.php/Raid-to-Silo_Transfer_Strategy Raid-to-Silo Transfer Strategy].<br />
<br />
Once the DAQ writes out the data to the raid disk, cron jobs will copy the file to tape,<br />
and within ~20 min., we will have access to the file on tape at<br />
/mss/halld/$RUN_PERIOD/rawdata/RunXXXXXX.<br />
<br />
All online monitoring plugins will be run as data is taken.<br />
They will be accessible within the counting house via RootSpy, and<br />
for each run and file, a ROOT file containing the histograms will be saved<br />
within a subdirectory for each run.<br />
<br />
For immediate access to these files, the raid disk files may be accessed directly<br />
from the counting house, or the tape files will be available within ~20 min. of the<br />
file being written out.<br />
<br />
== Offline Monitoring: Running Over Archived Data ==<br />
<br />
Once files are written to tape we run the online plugins on these files to confirm what we were seeing in the online monitoring, and also to update the results from the latest calibration and software. <br />
Manual scripts and cron jobs are set up to look for new data and run the plugins over a sample of files.<br />
<br />
Every other Friday (usually the Friday before the offline meetings) jobs will be started to run the newest software on all previous runs,and allows everybody to see improvements in each detector.<br />
For each launch, independent builds of hdds, sim-recon, the monitoring plugins, and an sqlite file will be generated.<br />
<br />
Below the procedures are described for<br />
# Preparing the software for the launch<br />
# Starting the launch (using hdswif)<br />
# Post-analysis of statistics of the launch<br />
<br />
Processing the results and making them available to the collaboration<br />
is handled in the section [[#Post-Processing_Procedures | Post-Processing Procedures]] below.<br />
<br />
<!--<br />
==== Generating an offline plugin job ====<br />
<br />
The user gxproj1 should be used for official offline monitoring jobs.<br />
Within /home/gxproj1/halld/monitoring/batch/ there will be scripts to run the online monitoring plugins over tape files.<br />
The main script is generatejobs_plugins_rawdata.sh, which can be used as<br />
generatejobs_plugins_rawdata.sh [minrun] [maxrun] (minfile) (maxfile)<br />
where minrun, maxrun specify the range of the run #, and minfile, maxfile (optional) specify<br />
the file range for that run.<br />
<br />
This will generate a script run_rawdata_XXXXXX.sh, where the run # has now been formatted to be 6 digits.<br />
Executing this script will send the monitoring plugins job to the Auger batch system.<br />
<br />
There is also a script clean.sh which can be used as<br />
clean.sh XXX<br />
This will clean up all associated files created in association with run XXX.<br />
<br />
Internally, the xml file used to submit the job will be created, and the job to run<br />
will be given within script.sh. All run parameters should be specified in at the beginning<br />
of generatejobs_plugins_rawdata.sh<br />
Since we are running on tape, the tape file will first be copied over to the cache disk, and the job will run<br />
over this cached file.<br />
<br />
==== Using cron to run automatically ====<br />
Within /home/gluex/halld/monitoring/cron/ there is a file cron_plugins<br />
that can be executed via<br />
crontab cron_plugins<br />
This will set up a cron job to call the script scan_for_jobs.sh, which will<br />
check in the rawdata directory and call generatejobs_plugins_rawdata.sh for<br />
any run that is more than 5 min old. The cron job is set up to run every 10 min.<br />
<br />
==== Magnetic Field Settings ====<br />
For tracking, it is necessary to set the correct magnetic field settings.<br />
The below table is now obsolete. Please refer to the [https://halldweb.jlab.org/cgi-bin/data_monitoring/run_conditions.pl GlueX Run Conditions] webpage.<br />
<br />
The actual field settings for each run have not been documented well.<br />
Below are values based on going through entries in the [https://logbooks.jlab.org/book/halld halld log].<br />
<br />
{| border="1" cellpadding="1" style="text-align: center;"<br />
!width="100"| Run #s<br />
!width="150"| Solenoid Current (A)<br />
!width="300"| JANA option<br />
!width="300"| notes<br />
|-<br />
| 940 - 996 || 1000 || -PBFIELD_MAP=Magnets/Solenoid/solenoid_1000A_poisson_20141104 || At run 997, the solenoid started ramping down<br />
|-<br />
| 998 - 1448 || 0 || -PBFIELD_TYPE=NoField -PDEFTAG:DTrackCandidate=StraightLine || Solenoid started ramping up around run 1431. Runs 1432 - 1448 should not be used for tracking. See [https://logbooks.jlab.org/entry/3309039 this] and [https://logbooks.jlab.org/entry/3309092 this] entry.<br />
|-<br />
| 1449 - 1620 || 1200 || -PBFIELD_MAP=Magnets/Solenoid/solenoid_1200A_poisson_20140520 || <br />
|-<br />
|}<br />
<br />
'''Note:''' The following run ranges were taken with a 300A field, and could be analyzed with the magnetic field map Magnets/Solenoid/solenoid_300A_poisson_20140819 , more study is needed to see if the straight line track fitter is needed or not: 1036 - 1053, 1065 - 1121, 1212? - 1254, 1309 - 1318 [1307,1308,1319-1329 were taken while the magnet was ramping].<br />
--><br />
<br />
=== General Information on Procedures ===<br />
This section explains how the offline monitoring should be run.<br />
Since we may want to simultaneously run offline monitoring for different run periods that require<br />
different environment variables, the scripts are set up so that a generic user can download the<br />
scripts and run them from anywhere. Most output directories for offline monitoring are created<br />
with group read/write permissions so that any Hall D group user has access to the contents,<br />
but there are some cases where use of the account that created the launch is necessary.<br />
<br />
The accounts used for offline monitoring are the gxprojN accounts created and maintained by<br />
Mark Ito (see [https://halldweb.jlab.org/wiki/index.php/GlueX-related_shared_accounts_on_the_JLab_CUE here] for how each account is used).<br />
As of October 2015, the following are used:<br />
* gxproj1 for running over Fall 2014 data (deprecated since June 2015)<br />
* gxproj5 for running over Spring 2015 data<br />
<br />
Since the summer of 2015 we have transitioned from a system using Mark Ito's jproj scripts<br />
to integrating the swif system that Chris Larrieu (SciComp) has been developing. For offline monitoring,<br />
the hdswif system that Kei developed is used for launching the jobs, and the jproj system is used<br />
for meta-analysis of launch statistics.<br />
<br />
Both hdswif and jproj are maintained in svn:<br />
* hdswif: https://halldsvn.jlab.org/repos/trunk/scripts/monitoring/hdswif<br />
* jproj : https://halldsvn.jlab.org/repos/trunk/scripts/monitoring/jproj<br />
<br />
To run the offline monitoring each package should be checked out and all necessary<br />
scripts are included.<br />
<br />
=== Preparing the software for the launch ===<br />
<br />
To begin a new launch the software must be built to the latest versions.<br />
For the gxprojN user accounts used, all software builds are contained in the directory ~/builds<br />
(which are soft links to /work/halld/home/gxprojN/builds). When logging into these accounts<br />
the setup files ~/setup_jlab-2015-03.csh or similar files should be sourced.<br />
<br />
<b>Note that Mark Ito does <u>not</u> want you to change the contents of each .cshrc file.</b><br />
You should consult him if you feel the need.<br />
<br />
1. Setup the environment: <pre>source ~/setup_jlab-2015-03.csh</pre><br />
2. Building hdds: <br />
<pre><br />
cd ~/builds/hdds/hdds<br />
git pull # Get latest software<br />
scons -c install # Clean out the old install: EXTREMELY IMPORTANT for cleaning out stale headers<br />
scons install -j4 # Rebuild and re-install with 4 threads<br />
</pre><br />
3. Building sim-recon: <br />
<pre><br />
cd ~/builds/sim-recon/sim-recon/<br />
git pull<br />
cd src<br />
scons -c install # Clean out the old install: EXTREMELY IMPORTANT for cleaning out stale headers<br />
scons install -j4 # Rebuild and re-install with 4 threads<br />
</pre><br />
4. Prepare the latest sqlite file: The sqlite is set in the ~/setup_jlab-2015-03.csh script as <b>sqlite:////home/gxproj5/ccdb.sqlite</b> through the environment variables <b>JANA_CALIB_URL</b> and <b>CCDB_CONNECTION</b>. Therefore, go to this directory and create a new sqlite file. We create the sqlite file in a temporary directory since creating the sqlite file in a directory where the output file exists causes errors. Original documentation on creating sqlite files are [https://halldweb.jlab.org/wiki/index.php/SQLite-form_of_the_CCDB_database here]. <br><span style="color:red">NOTE: SQLITE FILES DO NOT WORK ON THE NEW /work DISK INSTALLED IN OCTOBER 2015 </span><br />
<pre><br />
cd ~/tmp<br />
$CCDB_HOME/scripts/mysql2sqlite/mysql2sqlite.sh -hhallddb.jlab.org -uccdb_user ccdb | sqlite3 ccdb.sqlite<br />
mv ccdb.sqlite ../<br />
</pre><br />
5. Note that the above steps must be done <b>BEFORE</b> launch project creation. This is because we will track the revisions of the libraries used, and this is done by extracting the svn information in each directory. Also, note that the system assumes that we have the topmost build directory (usually called GLUEX_TOP) to be $HOME/builds . Such an assumption is necessary to be able to extract information about the library locations automatically.<br />
<br />
Create the appropriate project(s) and submit the jobs using hdswif, as detailed in the section below.<br />
<br />
=== Starting the Launch and Submitting Jobs ===<br />
<br />
Until the summer of 2015 we relied solely on Mark Ito's jproj system for submitting and keeping track of jobs. We have since moved to the swif system and use the hdswif wrapper for this. Below are instructions for how to use these.<br />
<br />
# Downloading hdswif: Download the hdswif directory from svn. For the gxprojN accounts, use the directory ~/halld/hdswif. <pre>cd ~/halld </pre> <pre> svn co https://halldsvn.jlab.org/repos/trunk/scripts/monitoring/hdswif</pre> <pre>cd hdswif</pre><br />
# Creating the workflow: Within SWIF jobs are registered into workflows. First create the workflow. For offline monitoring, the workflow names are of the form <b>offline_monitoring_RunPeriod201Y_MM_verVV_hd_rawdata</b> with suitable replacements for the run period and version number. The command "swif list" will list all existing workflows. Also, for most simple SWIF commands hdswif also provides a wrapper. <pre>swif list</pre> For creation of workflows for offline monitoring the command <pre>hdswif.py create [workflow] -c [config file] </pre> should be used. As an example config file, see the input.config file in the folder (and update it). When a config file is passed in, hdswif will automatically create files that record the configuration of the current launch. These files are stored as for example<br />
#* /group/halld/data_monitoring/run_conditions/jana_rawdata_comm_2015_03_ver15.conf<br />
#* /group/halld/data_monitoring/run_conditions/soft_comm_2015_03_ver15.xml<br />
#* The software packages stored in git (sim-recon and hdds) can have git tags applied to them, which makes it easier to find versions of the software than a SHA-1 hash. hdswif will ask if you would like to create a tag, and execute the following sequence: <pre>git tag -a offmon-201Y_MM-verVV -m "Used for offline monitoring 201Y-MM verVV started on 201y/mm/dd"</pre> <pre>git push offmon-201Y_MM-verVV</pre> This will only be invoked when the user name is gxprojN, and for the configuration files, the output directory will be /group/halld/data_monitoring/run_conditions/ for gxprojN accounts while it will be the current directory for other users.<br />
#* To use the git-tagged software versions do for example <pre>cd $HALLD_HOME</pre> <pre>git checkout offmon-2015_03-ver15</pre><br />
# Registering jobs in the workflow: To register jobs within the workflow, hdswif provides the use of config files. Jobs can be registered by specifying the workflow, config file (-c), run (-r) and file (-f) numbers if necessary. A typical config file will look this:<br />
PROJECT gluex<br />
TRACK reconstruction<br />
OS centos65<br />
NCORES 6<br />
DISK 40<br />
RAM 8<br />
TIMELIMIT 8<br />
JOBNAMEBASE offmon_<br />
RUNPERIOD 2015-03<br />
VERSION 15<br />
OUTPUT_TOPDIR /volatile/halld/offline_monitoring/RunPeriod-[RUNPERIOD]/ver[VERSION] # Example of other variables included in variable<br />
SCRIPTFILE /home/gxproj5/halld/hdswif/script.sh # Must specify full path<br />
ENVFILE /home/gxproj5/halld/hdswif/setup_jlab-2015-03.csh # Must specify full path<br />
The config file contains configuration parameters for each of the jobs.<br><br />
<span style="color:red">Note: Job configuration parameters can be set differently for jobs within the same workflow if necessary.</span><br />
Edit the config file and save as a new file if necessary. Once the configuration is set, jobs can be added via <pre>hdswif.py add [workflow] -c input.config</pre><br />
By default, hdswif will add all files found within the directory /mss/halld/RunPeriod-201Y-MM/rawdata/ where 201Y-MM is specified by the RUNPERIOD parameter in the config file. If only some of the runs or files are needed, these can be specified for example with<br />
<pre>hdswif.py add [workflow] -c input.config -r 3180 -f '00[0-4]'</pre><br />
to specify to register running only over run 3180 files 000 - 004 (Unix-style brackets and wildcards can be used).<br />
<br />
Running the workflow: To run the workflow, simply use swif run: <pre>swif run -workflow [workflow] -errorlimit none</pre> <span style="color:red">MAKE SURE THE ERRORLIMIT IS SET TO NONE OR THE WORKFLOW WILL BE STOPPED AFTER ANY JOB FAILS</span> or equivalently, using the hdswif wrapper (which has the errorlimit set by default), <pre>hdsswif.py run [workflow]</pre><br><br />
<b>It is recommended that some jobs be tested to make sure that everything is working rather than fail thousands of jobs.</b><br> For this purpose, hdswif will take an additional parameter to run which limits the number of jobs to submit: <pre>hdswif.py run [workflow] 10</pre><br />
in which case only 10 jobs will be submitted. To submit all jobs after checking the results, do <pre>hdswif.py run [workflow]</pre><br />
<br />
=== Checking the Status and Resubmitting ===<br />
1. The status of jobs can be checked on the terminal with <pre>jobstat -u gxprojN</pre> on Auger and for SWIF with <pre>swif list</pre> or for more information, <pre>swif status [workflow] -summary</pre> Note that "swif status" tends to be out of date sometimes, so don't panic if your workflow/jobs aren't showing up right away. Also see the Auger [http://scicomp.jlab.org/scicomp/#/auger/jobs job website].<br />
<br />
2. For failed jobs, SWIF can resubmit jobs based on the problem. For resubmission for failed jobs with the same resources, <pre>swif retry-jobs [workflow] -problems [problem name]</pre> can be used, and for jobs to be submitted with more resources, e.g., use <pre>swif modify-jobs -ram add 2gb -problems AUGER-OVER_RLIMIT</pre> <br />
This only re-stages the jobs, be sure to resubmit them with:<br />
<pre>swif run -workflow [workflow] -errorlimit none</pre><br />
<br />
hdswif has a wrapper for both of these: <pre>hdswif.py resubmit [workflow] [problem]</pre> In this case [problem] can be one of <b>SYSTEM, TIMEOUT, RLIMIT</b>. If SYSTEM is specified, the jobs will be retried. For TIMEOUT and RLIMIT, the jobs will be modified by default with 2 additional hours or GB of RAM. If one more number is added as an option, then that many hours or GB of RAM will be added., e.g., <pre>hdswif.py resubmit [workflow] TIMEOUT 5</pre> will add 5 hours of processing time. You can wait until almost all jobs finish before resubmitting failed jobs since the number should be relatively small. <b>Even if jobs are resubmitted for one type of failure, jobs that later fail with that failure will not be automatically resubmitted.</b><br />
<br />
3. For information on swif, use the "swif help" commands and for hdswif see the attached documentaion in https://halldsvn.jlab.org/repos/trunk/scripts/monitoring/hdswif/manual_hdswif.pdf<br />
<br />
4. Below is a table describing the various errors that can occur.<br />
{| border="1" cellpadding="0" valign="left" style="text-align: left;"<br />
!width="150"| ERROR NAME<br />
!width="400"| Description<br />
!width="400"| Resolution<br />
!width="400"| hdswif command<br />
|-<br />
| AUGER-SUBMIT<br />
||<br />
SWIF’s attempt to submit jobs to Auger failed. Includes server-side problems as well as user failing to provide valid job parameters (e.g. incorrect project name, too many resources, etc.)<br />
||<br />
If requested resources are known to be correct resubmit. Otherwise modify job resources using swif directly.<br />
||<br />
<span style="color:purple"><b>hdswif.py resubmit [workflow] SYSTEM</b></span><br />
|-<br />
| AUGER-FAILED<br />
||<br />
Auger reports the job FAILED with no specific details.<br />
||<br />
Resubmit jobs. If problem persists, contact Chris Larrieu or SciComp.<br />
||<br />
<span style="color:purple"><b>hdswif.py resubmit [workflow] SYSTEM</b></span><br />
|-<br />
| AUGER-OUTPUT-FAIL<br />
||<br />
Failure to copy one or more output files.Can be due to permission problem, quota problem, system error, etc.<br />
||<br />
Check if output files will exist after job execution and that output directory exists, resubmit jobs. If problem persists, contact Chris Larrieu or SciComp.<br />
||<br />
<span style="color:purple"><b>hdswif.py resubmit [workflow] SYSTEM</b></span><br />
|-<br />
| AUGER-INPUT-FAIL<br />
||<br />
Auger failed to copy one or more of the requested input files, similar to output failures. Can also happen if tape file is unavailable (e.g. missing/damaged tape)<br />
||<br />
Check if input file exists, resubmit jobs. If problem persists, contact Chris Larrieu or SciComp.<br />
||<br />
<span style="color:purple"><b>hdswif.py resubmit [workflow] SYSTEM</b></span><br />
|-<br />
| AUGER-TIMEOUT<br />
||<br />
Job timed out.<br />
||<br />
If more time is needed for job add more resources.<br />
Default is to add 2 hrs of processing time. Also check whether code is hanging.<br />
||<br />
<span style="color:red"><b>hdswif.py resubmit [workflow] TIMEOUT</b></span><br><br />
Default is to add 2 hours. Optionally specify number of hours at end.<br />
|-<br />
| AUGER-OVER_RLIMIT<br />
||<br />
Not enough resources, RAM or disk space.<br />
||<br />
Add more resources for job.<br />
||<br />
<span style="color:blue"><b>hdswif.py resubmit [workflow] RLIMIT</b></span><br><br />
Default is to add 2 GB of RAM. Optionally specify GB at end. To add more disk space use SWIF directly.<br />
|-<br />
| SWIF-MISSING-OUTPUT<br />
||<br />
Output file specified by user was not found.<br />
||<br />
Check if output file exists at end of job.<br />
||<br />
<br />
|-<br />
| SWIF-USER-NON-ZERO <br />
||<br />
User script exited with non-zero status code.<br />
||<br />
Your script exited with non-zero status. Check the code you are running.<br />
||<br />
<br />
|-<br />
| SWIF-SYSTEM-ERROR <br />
||<br />
Job failed owing to a problem with swif (e.g. network connection timeout)<br />
||<br />
Resubmit jobs. If problem persists, contact Chris Larrieu or SciComp.<br />
||<br />
<span style="color:purple"><b>hdswif.py resubmit [workflow] SYSTEM</b></span><br />
|}<br />
<br style="clear:both;"/><br />
<br />
=== Post-analysis of statistics of the launch ===<br />
<br />
# After jobs have been submitted, it will usually take a few days for all of the jobs to be processed.<br />
#* Status of Auger: http://scicomp.jlab.org/scicomp/#/auger/jobs (see also links above)<br />
#* Status of user jobs: <pre>jobstat -u [user name]</pre><br />
# The status and results of jobs are saved within the SWIF internal server, and are available via the command <pre>swif status [workflow] -summary -runs</pre> where the arguments -summary and -runs show summary statistics and statistics for individual jobs, respectively. hdswif has a command that takes this output in XML output and creates an HTML webpage showing results of the launch. To do this, do <pre>hdswif.py summary [workflow]</pre> This will create an XML file swif_output_[workflow].xml that contains all information from SWIF. If the file already exists, hdswif will ask whether to overwrite the existing file.<br />
# At this stage the html output and figure files are created and ready to be put online. For this step and other steps involving analysis of the statistics of the launch results, it is convenient to change to the jproj system.<br />
# The jproj scripts for offline monitoring are maintained in the svn directory https://halldsvn.jlab.org/repos/trunk/scripts/monitoring/jproj Do <pre> svn co https://halldsvn.jlab.org/repos/trunk/scripts/monitoring/jproj </pre> For the gxprojN accounts used for offline monitoring the directory should be ~/halld/jproj<br />
# The jproj directory contains two subdirectories, scripts and projects. The scripts directory contains useful scripts for processing the jobs registered in the jproj system, and each of the offline monitoring launches will be handled in the projects directory.<br />
# Go to the projects directory<pre>cd ~/halld/jproj/projects</pre> and use the script create_project.sh to create a new directory that contains the processing scripts for the current launch <pre>./create_project.sh [workflow]</pre> This should create a directory such as offline_monitoring_RunPeriod2015_03_ver15_hd_rawdata (same as the workflow name). The script uses the template files in the directory templates and by substitution creates script files for the current launch. Now go to the newly created analysis directory: <pre>cd [workflow]/analysis</pre><br />
which for the gxprojN accounts should have the full path /home/gxprojN/halld/jproj/projects/[workflow]/analysis.<br><br><br />
<span style="color:red">All of the analysis commands including arguments are contained in run_analysis.sh, but it is <b>strongly</b> recommended that all commands are run manually to check for errors.</span><br><br><br />
# The first thing to do is to make the html output from hdswif public. Copy the html file and related figures that were created from hdswif to the appropriate space within /group/halld/www/halldweb/html/data_monitoring/ : <pre>python publish_offmon_results.py [run period] [version]</pre> Note that the command with appropriate substitutions for arguments can be found within run_analysis.sh (same for all commands below).<br />
# Next, we need to create a few MySQL tables for the current launch. The MySQL tables are useful for comparing run/file combinations for different launches. For the SWIF launches, there are two tables needed, and will be named<br />
#* [workflow]Job<br />
#* [workflow]_aux<br />
# This naming scheme of tables and their roles are the same as from the jproj only launches. The [workflow]Job table will contain information gathered from SWIF about each job (which node it went to, start time of each stage, memory usage, etc.). The [workflow]_aux table will contain information gathered from the output stdout files from each job. First, create the Job table using <pre>python create_jproj_job_table.py [run period] [version]</pre> Note that this script uses the XML output from hdswif summary and inserts the contents into the MySQL table, so the XML output file must exist.<br />
# To check the contents of this MySQL table, do <pre>mysql -hhalldb -ufarmer farming</pre> <pre>mysql> describe offline_monitoring_RunPeriod2015_03_ver11_hd_rawdataJob;</pre><br />
<br />
== Hall D Job Management System ==<br />
<br />
This section details instructions on how to create and launch a set of jobs using the Hall-D Job Management System developed by Mark Ito. These instructions are generic: this system can be used for the weekly monitoring jobs, but can also be used for other sets of job launches as well. <br />
<br />
=== Database Table Overview ===<br />
<br />
* Job management database table (<project_name>): For each input file, keeps track of whether or not a job for it has been submitted, along with other optional fields. <br />
<br />
* Job status database table (<project_name>Job (no space)): For each job, keeps track of the job-id, the job status, memory used, cpu & wall time, time taken to complete various stages (e.g. pending, dependency, active), and others. <br />
<br />
* Job metrics database table (<project_name>_aux (no space)): For each job, keeps track of the job-id, how many events were processed, the time it took to copy the cache file, and the time it took to run the plugin. This information is culled from the log files of each job, and is done within the analysis directory of each launch.<br />
<br />
=== Initialize Project Management ===<br />
<br />
* Log into the ifarm machine with one of the gxproj accounts. For this example we will use gxproj1.<br />
<pre><br />
ssh gxproj1@ifarm -Y<br />
</pre><br />
<br />
* Go to a directory to do the launch. In principle, any directory will work, but for gxproj1 this is usually done in /home/gxproj1/halld/<br />
<br />
* Check out the necessary scripts <pre>svn co https://halldsvn.jlab.org/repos/trunk/scripts/monitoring/jproj</pre> This will get all necessary scripts for launching. Once checked out, <pre>cd projects</pre><br />
<br />
* The script create_project.sh can be used to create a new project. It will take a single argument, the project name. It is assumed that the project name is of the form offline_monitoring_RunPeriod20YY_MM_verVV_hd_rawdata where this string will be parsed to give the run period 20YY_MM and version number VV. One thing to do BEFORE creation of a new project is to editthe conditions of the launch (plugins to run over, memory requested, disk space requested) within templates/template.jsub . This<br />
information is saved automatically at project creation time into files in /group/halld/data_monitoring/run_conditions .<br />
To create a project do <pre>./create_project.sh offline_monitoring_RunPeriod20YY_MM_verVV_hd_rawdata</pre><br />
The name has been chosen to be as consistent as possible with other directory structures.<br />
However, mysql requires that "-" be escaped in table names, so unfortunately run periods will be<br />
given as 2YYY_MM instead of 2YYY-MM.<br />
<br />
* A new directory with the project name (offline_monitoring_RunPeriod20YY_MM_verVV_hd_rawdata) and files will be copied and modified from the template directory to reflect the run period, the user, the directory that it was created in, project name, etc.<br />
<br />
* For each project, cd into the new directory<br />
* The script clear.sh will remove any existing tables for the current project name, then recreate it. Do<br />
<pre>./clear.sh</pre><br />
* To use the jproj.pl script that was checked in, add the directory to your path with<br />
<pre>source ../../scripts/setup.csh</pre><br />
or always specify the full path<br />
<pre>../../scripts/jproj.pl</pre><br />
* Now update the table of runs with<br />
<pre>jproj.pl <project name> update</pre><br />
This will fill the table with all files within /mss that are of the same form as what is in <project name>.jproj . If you want to register only on a subset of all such files, you can edit this file directly.<br />
* Once you have registered all of the files you would like to run over, do<br />
<pre>jproj.pl <project name> submit [max # of jobs] [run number]</pre><br />
where the additional options specify how many jobs to submit and which run number to run on. Without these options all files that are registered and have not been submitted yet will be submitted.<br />
<br />
At this stage you are ready to submit all files. It is a good idea to submit a few test jobs<br />
at first to check that all scripts are working and that the plugins do not crash. Once you are<br />
sure that this does not happen, you can send all jobs in. The remaining jobs are then the monitoring<br />
which will (among other things) put the results on the online webpage for the collaboration to view, and<br />
the analysis of the launch.<br />
<br />
=== Project File Overview ===<br />
<br />
An overview of each project file:<br />
* '''clear.sh:''' For the current project, deletes the job status and management database tables (if any), and creates new, empty ones. <br />
* '''<project_name>.jproj:''' Contains the path and file name format for the input files for the jobs. <br />
* '''<project_name>.jsub:''' The xml job submission script. The run number and file number variables are set during job submission for each input file. <br />
* '''script.sh:''' The script that is executed during the job. If output job directories are not pre-created manually, they should be created in this script with the proper permissions:<br />
<pre><br />
mkdir -p -m 775 my_directory<br />
</pre><br />
* '''setup_jlab-[run period].csh:''' The environment that is sourced at the beginning of the job execution. <br />
* '''status.sh:''' Updates the job status database table, and prints some of its columns to screen.<br />
<br />
=== Project Management ===<br />
<br />
* Delete (if any) and create the database table(s) for the current set of job submissions:<br />
<pre><br />
./clear.sh<br />
</pre><br />
Also, if testing was done with jobs, it is best to delete the output directory and the configuration files:<br />
<pre><br />
rm -frv /volatile/halld/offline_monitoring/RunPeriod-20YY-MM/verVV /group/halld/data_monitoring/run_conditions/soft_comm_20YY_MM_verVV.xml /group/halld/data_monitoring/run_conditions/jana_rawdata_comm_20YY_MM_verVV.conf<br />
</pre><br />
<br />
* Search for input files matching the string in the .jproj file, and create a row for each in the job management database table (called <project_name>). You can test by adding an optional argument at the end, which only selects files with a specific file number:<br />
<pre><br />
jproj.pl <project_name> update <optional_file_number><br />
</pre><br />
<br />
* Confirm that the job management database is accurate by printing it's contents to screen:<br />
<pre><br />
mysql -hhallddb -ufarmer farming -e "select * from <project_name>"<br />
</pre><br />
<br />
* ONLY if a mistake was made, to delete the tables from the database and recreate new, empty ones, run: <br />
<pre><br />
./clear.sh<br />
</pre><br />
<br />
* To look at the status of the submitted jobs, first query auger and update the job status database:<br />
<pre><br />
fill_in_job_details.pl <project_name><br />
</pre><br />
<br />
* The job status can then be viewed by submitting a query to the job status database (called <project_name>Job (no space in between)):<br />
<pre><br />
mysql -hhallddb -ufarmer farming -e "select id,run,file,jobId,hostname,status,timeSubmitted,timeActive,walltime,cput,timeComplete,result,error from <project_name>Job"<br />
</pre><br />
<br />
* These last two commands can instead be executed simultaneously by running:<br />
<pre><br />
./status.sh<br />
</pre><br />
<br />
=== Handy mysql Instructions ===<br />
<br />
* Handy mysql instructions:<br />
<pre><br />
mysql -hhallddb -ufarmer farming # Enter the "farming" mysql database on "hallddb" as user "farmer"<br />
quit; # Exit mysql<br />
show tables; # Show a list of the tables in the current database<br />
show columns from <project_name>; # show all of the columns for the given table<br />
select * from <project_name>; # show the contents of all rows from the given table<br />
</pre><br />
<br />
=== Backing Up Offline Monitoring Tables ===<br />
Tables created for offline monitoring can be backed up using the script backup_tables.sh which<br />
can be checked out with the other files from https://halldsvn.jlab.org/repos/trunk/scripts/monitoring/jproj/projects<br />
<br />
The script uses the command mysqldump to print out a file that can be executed to recreate the tables.<br />
Since executing this output file will drop the table when it exists, caution is advised.<br />
Example usage to backup all three tables created for run period 2014_10 ver 17:<br />
<pre>backup_tables.sh 2014_10 17</pre><br />
<br />
== Running Over Data As It Comes In ==<br />
<br />
A special user gxproj1 will have a cron job set up to run the plugins as new data appears on /mss.<br />
During the week, gxproj1 will submit offline plugin jobs with the same setup as the weekly jobs<br />
run the previous Friday. The procedure for this is shown below.<br />
<br />
<!--<br />
=== Setting up the environment ===<br />
The file<br />
/home/gxproj1/setup_jlab.csh<br />
is sourced through .tcshrc.<br />
This file is the same as what is linked to by<br />
/home/gluex/setup_jlab_commissioning.csh,<br />
except HALLD_HOME, HDDS_HOME, and JANA_CALIB_URL are set separately so that this<br />
user can have a separate build.<br />
<br />
To obtain the builds from the previous Friday's runs,<br />
execute<br />
/home/gxproj1/halld/monitoring/newruns/setup_previous.sh [year] [month] [day]<br />
The build revisions from the previous Friday are archived in files<br />
/work/halld/data_monitoring/run_conditions/soft_comm_[year]_[month]_[day].xml<br />
and the script will build libraries based on those stored revision numbers.<br />
--><br />
<br />
=== Running the cron job ===<br />
<br />
'''IMPORTANT:''' The cron job should not be running while you are manually submitting jobs using the jproj.pl script for the same project, or else you will probably multiply-submit a job. <br />
<br />
* Go to the cron job directory: <br />
<pre><br />
cd /u/home/gxproj1/halld/monitoring/newruns<br />
</pre><br />
<br />
* The cron_plugins file is the cronjob that will be executed. During execution, it runs the exec.sh command in the same folder. This command takes two arguments: the project name, and the maximum file number for each run. These fields should be updated in the cron_plugins file before running. <br />
<br />
* The exec.sh command updates the job management database table with any data that has arrived on tape since it was last updated, ignoring file numbers greater than the maximum file number. It then submits jobs for these files. <br />
<br />
* To start the cron job, run:<br />
<pre><br />
crontab cron_plugins<br />
</pre><br />
<br />
* To check whether the cron job is running, do<br />
<pre><br />
crontab -l<br />
</pre><br />
<br />
* To remove the cron job do<br />
<pre><br />
crontab -r<br />
</pre><br />
<br />
<!--<br />
The cron job will run the script scan_for_jobs.sh,<br />
which runs generatejobs_plugins_rawdata.sh for any<br />
new runs that it had not seen before. All previous<br />
runs are recorded in the file filelists/files_current.txt<br />
so clear this to run over runs, or set the parameters<br />
MINRUN and MAXRUN which will set the range of runs submitted.<br />
--><br />
<br />
==Post-Processing Procedures==<br />
<br />
To visualize the monitoring data, we save images of selected histograms and store time series of selected quantities in a database, which are then displayed on a web page. This section describes how to generate the monitoring images and database information.<br />
<br />
The scripts used to generate this summary data are primarily run from /home/gxprojN/halld/monitoring/process . If you want a new copy of the scripts, e.g., for a new monitoring run, you should check the scripts out from SVN:<br />
<syntaxhighlight><br />
svn co https://halldsvn.jlab.org/repos/trunk/scripts/monitoring/process<br />
</syntaxhighlight><br />
Note that these scripts currently have some parameters which must be periodically set by hand.<br />
<br />
The default python version on most JLab machine does not have the modules to allow these scripts to connect to the MySQL database. To run these scripts, load the environment with the following command<br />
<syntaxhighlight><br />
source /home/gxproj1/halld/monitoring/process/monitoring_env.csh<br />
</syntaxhighlight><br />
<br />
===Online Monitoring===<br />
<br />
There are two primary scripts for running over the monitoring data generated by the online system and offline reconstruction. The online script can be run with either of the following commands:<br />
<syntaxhighlight><br />
/home/gluex/halld/monitoring/process/check_new_runs.py<br />
<br />
OR <br />
<br />
/home/gluex/halld/monitoring/process/check_new_runs.csh<br />
</syntaxhighlight><br />
The shell script sets up the environment properly to run the python script. To connect to the monitoring database on the JLab CUE, modules included in the local installation of python >= 2.7 are needed. The shell script is appropriate to use in a cron job. The cronjob is currently run under the "gluex" account.<br />
<br />
The online monitoring system copies a ROOT file containing the results of the online monitoring, and other configuration files into a directory accessible outside the counting house. This python script automatically checks for new ROOT files, which it will then automatically process. It contains several configuration variables that must be correctly set, which contains the location of input/output directories, etc...<br />
<br />
===Offline Monitoring===<br />
<br />
After the data is run over, the results should be processed, so that summary data is entered into the monitoring database and plots are made for the monitoring webpages. Currently, this processing is controlled by a cronjob that runs the following script:<br />
<syntaxhighlight><br />
/home/gxproj1/halld/monitoring/process/check_monitoring_data.csh <br />
</syntaxhighlight><br />
This script checks for new ROOT files, and only runs over those it hasn't processed yet. Since one monitoring ROOT file is produced for each EVIO file, whenever a new file is produced, the plots for the corresponding run are recreated and all the ROOT files for a run are combined into one file. Information is stored in the database on a per-file basis. <br />
<br />
Plots for the monitoring web page can be made from single histograms or multiple histograms using RootSpy macros. If you want to change the list of plots made, you must modify one of the following files:<br />
* histograms_to_monitor - specify either the name of the histogram or its the full ROOT path <br />
* macros_to_monitor - specify the full path to the RootSpy macro .C file<br />
<br />
When a new monitoring run is started, or the conditions are changed, the following steps should be taken to process the new files:<br />
# Add a new data version, as described below:<br />
# Change the following parameters in check_monitoring_data.csh:<br />
## JOBDATE should correspond to the ouptut date used by the job submission script<br />
## OUTPUTDIR should correspond to the directory corresponding to the run period and revision corresponding to the new version you just submitted. Presumably, this directory will be empty at the beginning.<br />
## Once you create a new data version as defined below, you should pass the needed information as a command line option. Currently this is done by the ARGS variable. For example, the argument "-v RunPeriod-2014-10,8" tells the monitoring scripts to look up the version corresponding to revision 8 of RunPeriod-2014-10 in the monitoring DB and to use to store the results.<br />
<br />
<syntaxhighlight><br />
Example configuration parameters:<br />
set JOBDATE=2015-01-09<br />
set INPUTDIR=/volatile/halld/RunPeriod-2014-10/offline_monitoring<br />
set OUTPUTDIR=/w/halld-scifs1a/data_monitoring/RunPeriod-2014-10/ver08<br />
set ARGS=" -v RunPeriod-2014-10,8 "<br />
</syntaxhighlight><br />
If you want to process the results manually, the data is processed using the following script:<br />
<syntaxhighlight><br />
./process_new_offline_data.py <input directory> <output directory><br />
<br />
EXAMPLE:<br />
<br />
./process_new_offline_data.py 2014-11-14 /volatile/halld/RunPeriod-2014-10/offline_monitoring/ /w/halld-scifs1a/data_monitoring/RunPeriod-2014-10/ver02<br />
</syntaxhighlight><br />
The python script takes several options to enable/disable various steps in the processing. Of interest is the "--force" option, which will run over all monitoring ROOT files, whether or not they've been previously identified.<br />
<br />
Every time a new reconstruction pass is performed, a new version number must be generated. To do this, prepare a version file as described below. Then run the register_new_version.py script to store the information in the database. The script will return a version number, which then should be set by hand in process_new_offline_data.py - future versions of the script will streamline this part of the procedure. An example of how to generate a new version is:<br />
<syntaxhighlight><br />
./register_new_version.py add /home/gxproj1/halld/monitoring/process/versions/vers_RunPeriod-2014-10_pass1.txt<br />
</syntaxhighlight><br />
<br />
<b>If you are running the offline monitoring by checking out the files in trunk/scripts/monitoring/jproj/projects/<br />
of the svn repository</b>, and created a project with<br />
create_project.sh [project name] hd_rawdata<br />
Then go to the directory [project name]/processing/<br />
and execute<br />
./run_processing.sh<br />
which will run register_new_version.py as well as check_monitoring_data.csh for that project.<br />
<br />
===Step-by-Step Instructions For Processing a New Monitoring Run===<br />
<br />
The monitoring runs are current run out of the gxproj1 and gxproj5 accounts. After an offline monitoring run has been successfully started on the batch farm, the following steps should be followed to setup the post-processing for these runs.<br />
<br />
# The post-processing scripts are stored in $HOME/halld/monitoring/process and are automatically run by cron.<br />
# Run "svn update" to bring any changes in. Be sure that the list of histograms and macros to plot are current.<br />
# Add a new data version<br />
# Edit check_monitoring_data.csh to point to the current revisions/directories<br />
#* VERSION<br />
#* ARGS<br />
#* Note that the environment depends on a standard script - $HOME/setup_jlab.csh<br />
# Update files in the web directory, so that the results are displayed on the web pages: /group/halld/www/halldweb/html/data_monitoring/textdata<br />
# Copy the REST files to more permanent locations:<br />
#* cp -a /volatile/halld/offline_monitoring/RunPeriod-YYYY-MM/verVV/REST /work/halld/data_monitoring/RunPeriod-YYYY-MM/REST/verVV<br />
#* cp -a /volatile/halld/offline_monitoring/RunPeriod-YYYY-MM/verVV/REST /cache/halld/RunPeriod-YYYY-MM/REST/verVV [under testing]<br />
<br />
Check log files in $HOME/halld/monitoring/process/log for more information on how each run went. If there are problems, check log files, and modify check_monitoring_data.csh to vary the verbosity of the output.<br />
<br />
==Data Versions==<br />
<br />
To document the conditions of the monitoring data that is created, for the sake of reproducability and further analysis we save several pieces of information. The format is intended to be comprehensive enough to document not just monitoring data, but versions of raw and reconstructed data, so that this database table can be used for the event database as well.<br />
<br />
We store one record per pass through one run period, with the following structure:<br />
<br />
{| class="wikitable"<br />
! Field !! Description<br />
|-<br />
| data_type || The level of data we are processing. For the purposes of monitoring, "rawdata" is the online monitoring, "recon" is the offline monitoring<br />
|- <br />
| run_period || The run period of the data<br />
|- <br />
| revision || An integer specifying which pass through the run period this data corresponds to<br />
|- <br />
| software_version || The name of the XML file that specifies the different software versions used<br />
|-<br />
| jana_config || The name of the text file that specifies which JANA options were passed to the reconstruction program<br />
|-<br />
| ccdb_context || The value of JANA_CALIB_CONTEXT, which specifies the version of calibration constants that were used<br />
|-<br />
| production_time || The data at which monitoring/reconstruction began<br />
|-<br />
| dataVersionString || A convenient string for identifying this version of the data<br />
|}<br />
<br />
<br />
An example file used as as input to ./register_new_version.py is:<br />
<syntaxhighlight><br />
data_type = recon<br />
run_period = RunPeriod-2014-10<br />
revision = 1<br />
software_version = soft_comm_2014_11_06.xml<br />
jana_config = jana_rawdata_comm_2014_11_06.conf<br />
ccdb_context = calibtime=2014-11-10<br />
production_time = 2014-11-10<br />
dataVersionString = recon_RunPeriod-2014-10_20141110_ver01<br />
</syntaxhighlight></div>Kmoriyahttps://halldweb.jlab.org/wiki/index.php?title=GlueX_Talks&diff=70933GlueX Talks2015-11-02T20:45:13Z<p>Kmoriya: /* Talks in 2014 */</p>
<hr />
<div>== Conference and Workshop Talks==<br />
<br />
=== Talks in 2015 ===<br />
* [http://argus.phys.uregina.ca/cgi-bin/private/DocDB/ShowDocument?docid=2872 A first look at reconstructed data from the GlueX detector] DNP15, October 28-31, 2015, Santa Fe, NW, presented by Simon Taylor.<br />
* [http://argus.phys.uregina.ca/cgi-bin/private/DocDB/ShowDocument?docid=2687 Analysis Plans in GlueX] Athos/PWA Meeting, April 13-17 2015, GWU, presented by Curtis A. Meyer.<br />
* [http://argus.phys.uregina.ca/cgi-bin/private/DocDB/ShowDocument?docid=2745 Light-quark meson spectroscopy with the GlueX experiment] CIPANP May 2015, presented by Mark M. Dalton.<br />
* [http://argus.phys.uregina.ca/cgi-bin/private/DocDB/ShowDocument?docid=2795 Electromagnetic Production of Strangeness at Jefferson Lab] [http://lambda.phys.tohoku.ac.jp/hyp2015/ HYP2015], September 2015, presented by Kei Moriya<br />
* [http://argus.phys.uregina.ca/cgi-bin/private/DocDB/ShowDocument?docid=2799 First GlueX Results] Hadron 2015, September 2015, presented by Curtis A. Meyer.<br />
<br />
=== Talks in 2014 ===<br />
* [http://argus.phys.uregina.ca/cgi-bin/private/DocDB/ShowDocument?docid=2506 Physics in Hall D] Jefferson Lab 2014 Users Group Meeting, June 2-4, 2014, presented by Curtis A. Meyer.<br />
* [http://argus.phys.uregina.ca/cgi-bin/private/DocDB/ShowDocument?docid=2534 The GlueX Experiment at Jefferson Lab], HaPhy-CLAS Workshop on Hadron Productions, August 2014, Kei Moriya<br />
* [http://argus.phys.uregina.ca/cgi-bin/private/DocDB/ShowDocument?docid=2532 The GlueX experiment and the search for exotic mesons], PANIC 2014, August 25-29, Will Levine<br />
* [http://argus.phys.uregina.ca/cgi-bin/private/DocDB/ShowDocument?docid=2541 Electromagnetic Strangeness Production at Jefferson Lab Energies,] XI Conference on Quark Confinement at the Hadron Spectrum, September 8-12, 2014, presented by R. A. Schumacher.<br />
* [http://argus.phys.uregina.ca/cgi-bin/private/DocDB/ShowDocument?docid=2603 Eta Decay Programm at GlueX], MesonNet Meeting, INFN Frascati, September 29, Alexander Somov <br />
* [[DNP-2014 abstract submitted by Richard Jones|Abstract submitted to Mini-symposium on Hybrid Mesons and Molecules]], DNP-2014, Oct. 7-12, 2014, by Richard Jones<br />
* [http://argus.phys.uregina.ca/cgi-bin/private/DocDB/ShowDocument?docid=2689 GlueX Analysis] presented at the Future Directions in Partial Wave Analysis Workshop at Jefferson Lab, November 2014.<br />
<br />
=== Talks in 2013 ===<br />
* [http://argus.phys.uregina.ca/cgi-bin/private/DocDB/ShowDocument?docid=2375 Hybrid Meson Lectures] presented by Curtis Meyer in December 2013.<br />
* Carnegie Mellon University undergraduate colloquium, November 2013. [http://argus.phys.uregina.ca/cgi-bin/private/DocDB/ShowDocument?docid=2373 Big Science] a talk on how large science projects are conceived, funded and built following the 12-GeV upgrade as an example (Curtis A. Meyer).<br />
* [https://www.jlab.org/conferences/dnp2013/dnp-13.html APS-DNP 2013] October 23-26, 2013, Newport News, VA<br />
** [http://argus.phys.uregina.ca/cgi-bin/private/DocDB/ShowDocument?docid=2355 Characteristics of Silicon Photomultipliers (SiPM) for GlueX], Yi Qiang<br />
** [http://argus.phys.uregina.ca/cgi-bin/private/DocDB/ShowDocument?docid=2546 Reconstruction of showers in the GlueX barrel calorimeter], Will Levine<br />
** [http://argus.phys.uregina.ca/cgi-bin/private/DocDB/ShowDocument?docid=2674 JLab Hall-D Photon Beamline], Alexander Somov<br />
* [http://www.int.washington.edu/NNPSS/2013/HOME.html Nuclear Science Summer School] July 15-26, 2013, Stony Brook University campus. [http://argus.phys.uregina.ca/cgi-bin/private/DocDB/ShowDocument?docid=2278 Colloquium] presented by Curtis A. Meyer.<br />
* [http://www.inpc2013.it INPC2013] 25th International Nuclear Physics Conference June 2-7, 2013 Florence, Italy<br />
* [https://hep.ustc.edu.cn/indico/conferenceOtherViews.py?view=standard&confId=1 The Fifth Workshop on Hadron Physics in China and Opportunities in US], July 2-6, Huangshan, China<br />
** [http://argus.phys.uregina.ca/gluex/DocDB/0022/002282/001/20130703_Physics_HallD.pdf Physics Program at Jefferson Lab Hall-D], Yi Qiang<br />
* [http://www.aps.org/meetings/april/index.cfm APS] American Physical Society Meeting, April Mtg., Apr. 13-16, 2013, Denver, CO<br />
** [http://argus.phys.uregina.ca/cgi-bin/private/DocDB/ShowDocument?docid=2283 Characteristics of S12045(X) photon sensors for GlueX], Elton Smith<br />
** [http://argus.phys.uregina.ca/cgi-bin/private/DocDB/ShowDocument?docid=2284 The Barrel Calorimeter for the GlueX Experiment at Jefferson Lab], Zisis Papandreou (presented by Elton Smith)<br />
* [https://sites.google.com/site/ghpworkshop/ GHP Workshop] Meeting of the APS Topical Group on Hadron Physics (GHP), Apr. 10-12, 2013, Denver, CO<br />
** [http://argus.phys.uregina.ca/cgi-bin/private/DocDB/ShowDocument?docid=2189 Exploration of Exotic Mesons with GlueX], Elton Smith<br />
<br />
=== Talks in 2012 ===<br />
<br />
* PANDA XLIII Collaboration Meeting, Dec 10-14, 2012 GSI, Germany<br />
** [http://argus.phys.uregina.ca/cgi-bin/private/DocDB/ShowDocument?docid=2122 GlueX: Photoproduction of Hybrid Mesons], Elton Smith<br />
* Spectroscopy at 12 GeV Workshop, Jefferson Lab<br />
** [http://argus.phys.uregina.ca/cgi-bin/private/DocDB/ShowDocument?docid=2192 Current PWA Projects in Hall D], Matt Shepherd<br />
*[http://www.aps.org/units/dnp/meetings/meeting.cfm?name=DNP12/ DNP 2012] 2012 Fall Meeting of the APS Division of Nuclear Physics, October 24 - Ocober 27, 2012 Newport Beach, California<br />
** [http://argus.phys.uregina.ca/cgi-bin/private/DocDB/ShowDocument?docid=2110 GlueX: Thin Diamond Radiators for the GlueX Experiment], Brendan Pratt<br />
** [http://argus.phys.uregina.ca/cgi-bin/private/DocDB/ShowDocument?docid=2111 Collimation and Tagging Instrumentation for the GlueX Photon Beamline], R.T. Jones<br />
* PANDA XLII Collaboration Meeting, June 26-29, 2012 Evanston, IL<br />
** [http://argus.phys.uregina.ca/cgi-bin/private/DocDB/ShowDocument?docid=2193 GlueX: A search for light quark exotic mesons at Jefferson Lab], Matt Shepherd<br />
* [http://www.ge.infn.it/~athos12/ATHOS/Welcome.html ATHOS 2012] International Workshop on partial wave analysis, June 20-23, 2012. <br />
** [http://argus.phys.uregina.ca/cgi-bin/private/DocDB/ShowDocument?docid=2029 Talk] presented by Curtis Meyer.<br />
* [http://www.cap.ca/en/congress/2012 CAP Congress 2012], University of Calgary (Calgary, Alberta), June 11-15, 2012<br />
** [http://argus.phys.uregina.ca/cgi-bin/private/DocDB/ShowDocument?docid=2026 The GlueX Large Area Silicon Photomultipliers], Z. Papandreou<br />
*[http://www.jlab.org/conferences/ugm/program.htm Jefferson Lab User's Meeting], June 4-6, 2012.<br />
** [http://argus.phys.uregina.ca/cgi-bin/private/DocDB/ShowDocument?docid=2025 Talk] presented by Curtis Meyer.<br />
*[http://meson.if.uj.edu.pl/ MESON 2012] 12th International Workshop on Meson Production, Properties and Interaction, May 31 - Jun 5, 2012 Krakow, Poland<br />
** [http://argus.phys.uregina.ca/cgi-bin/private/DocDB/ShowDocument?docid=2016 GlueX: Photoproduction of Hybrid Mesons], Elton Smith<br />
*[http://cipanp2012.triumf.ca/ CIPANP 2012] Eleventh Conference on the Intersections of Particle and Nuclear Physics, May 29 - Jun 3, 2012, St. Petersburg, Florida<br />
** [http://argus.phys.uregina.ca/cgi-bin/private/DocDB/ShowDocument?docid=2022 GlueX: Neutron Radiation Hardness of SiPM and Its Applications in Jefferson Lab Hall-D], Yi Qiang<br />
** [http://argus.phys.uregina.ca/cgi-bin/private/DocDB/ShowDocument?docid=2675 Development of Level-1 Triggers for Experiments at Jefferson Lab], Alexander Somov<br />
<br />
=== Talks in 2011 ===<br />
*[http://web.mit.edu/panic11/ PANIC 11] The 19th Particles and Nuclei International Conference, in Cambridge, Massachusetts at the Massachusetts Institute of Technology (MIT)<br />
** [http://argus.phys.uregina.ca/cgi-bin/private/DocDB/ShowDocument?docid=1792 GlueX: Detector Construction and Event Simulations], Naomi Jarvis.<br />
* [http://www.hadron2011.de/ Hadron 2011], Munich, Germany, June 13-17, 2011<br />
** [http://argus.phys.uregina.ca/cgi-bin/private/DocDB/ShowDocument?docid=1780 Search for Gluonic Excitations in Hadrons with GlueX], I. Senderovich<br />
* [http://www.cap.ca/en/congress/2011 CAP Congress 2011], Memorial University of Newfoundland (St. John's, Newfoundland), June 13-17, 2011<br />
** [http://argus.phys.uregina.ca/cgi-bin/private/DocDB/ShowDocument?docid=2123 The GlueX Electromagnetic Barrel Calorimeter], Z. Papandreou<br />
* [http://www.aps.org/meetings/april/info/index.cfm APS Physics - April Meeting], Hyatt Regency Garden Grove, Anaheim, CA, April 30 - May 1, 2011.<br />
** [http://argus.phys.uregina.ca/cgi-bin/private/DocDB/ShowDocument?docid=1729 The GlueX Electromagnetic Barrel Calorimeter], Z. Papandreou.<br />
** [http://argus.phys.uregina.ca/cgi-bin/private/DocDB/ShowDocument?docid=1731 The Meson Spectrum from Lattice QCD], J. Dudek.<br />
** [http://argus.phys.uregina.ca/cgi-bin/private/DocDB/ShowDocument?docid=1734 The Experimental Spectrum of Hadrons], M. Shepherd.<br />
** [http://argus.phys.uregina.ca/cgi-bin/private/DocDB/ShowDocument?docid=1767 Overview of GlueX Offline Computing], M. Ito.<br />
** [http://argus.phys.uregina.ca/cgi-bin/private/DocDB/ShowDocument?docid=1727 Charged particle tracking for the gluex detector], S. Taylor.<br />
* [https://sites.google.com/site/ghpworkshop/home The 4th Workshop of the APS Topical Group on Hadronic Physics], Hyatt Regency Garden Grove, Anaheim, CA, April 27-29, 2011.<br />
** [http://argus.phys.uregina.ca/cgi-bin/private/DocDB/ShowDocument?docid=1730 The Meson Spectrum from Lattice QCD], J. Dudek.<br />
* [http://www1.jlab.org/ul/calendar/calendar_date.cfm?date=23&month=2&year=2011 Workshop on Excited Hadronic States and the Deconfinement Transition], Jefferson Lab, Newport News, VA, February 23-25, 2011.<br />
** [http://www.curtismeyer.com/talks/C_Meyer_Hadron_Spectroscopy.pptx Spectroscopy: experimental status and prospects], Curtis A. Meyer.<br />
* [http://www.sfu.ca/~caa12/WNPPC11/ WNPPC 2011], Winter Nuclear and Particle Physics Conference, 18-20 Feb 2011, Banff, Canada<br />
** [http://argus.phys.uregina.ca/cgi-bin/private/DocDB/ShowDocument?docid=1711 The GlueX Barrel Calorimeter], Zisis Papandreou<br />
** [http://argus.phys.uregina.ca/cgi-bin/private/DocDB/ShowDocument?docid=1712 Large Area Multi-Pixel Photon Detectors for the GlueX Barrel Calorimeter], Mehrnoosh Tahani<br />
* [http://www.gsi.de/forschung/kp/had2/meeting/Hirschegg_2011.html Hirschegg 2011], The Structure and Dynamics of Hadrons, Hirschegg, Austria, January 16-22, 2011<br />
** [http://argus.phys.uregina.ca/gluex/DocDB/0017/001706/001/hirschegg_gluex.pdf The GlueX Experiment (and its Context)], Ryan Mitchell<br />
<br />
=== Talks in 2010 ===<br />
* [http://www.lanl.gov/dnp DNP 2010], Santa Fe, NM, Nov 2 - 6.<br />
** [http://argus.phys.uregina.ca/cgi-bin/private/DocDB/ShowDocument?docid=1639 Level-1 Trigger of the GlueX Experiment], Alexander Somov<br />
* Jefferson Lab PAC36, Jefferson Lab, Newport News, VA, August, 2010.<br />
** [http://www.curtismeyer.com/talks/GlueX_Pac_Presentation.pptx GlueX Presentation], Curtis A. Meyer.<br />
* [http://www.lnf.infn.it/public/ LNF Frascati], Frascati, Italy, June 30, 2010<br />
** [http://argus.phys.uregina.ca/cgi-bin/private/DocDB/ShowDocument?docid=2124 The GlueX Experiment: construction is under way], Z. Papandreou<br />
* [http://www.cap.ca/en/congress/2010 CAP Congress 2010], University of Toronto (Toronto, Ontario), June 7-11, 2010<br />
** [http://argus.phys.uregina.ca/cgi-bin/private/DocDB/ShowDocument?docid=2123 The GlueX Electromagnetic Barrel Calorimeter], Z. Papandreou<br />
* Jefferson Lab Users Group Meeting, Jefferson Lab, Newport News, VA, June 7-9, 2010. <br />
** [http://argus.phys.uregina.ca/cgi-bin/public/DocDB/ShowDocument?docid=1544 GlueX/Hall-D Physics], Curtis A. Meyer.<br />
* [https://www.jlab.org/conferences/MENU10 Meson-Nucleon Physics and the Structure of the Nucleon (MENU 2010)], Williamsburg, VA, May 31, 2010.<br />
** [http://argus.phys.uregina.ca/cgi-bin/private/DocDB/ShowDocument?docid=1540 Physics Prospects with GlueX], Alexander Somov<br />
<br />
== Seminar Talks ==<br />
* [http://argus.phys.uregina.ca/cgi-bin/private/DocDB/ShowDocument?docid=2432 GlueX Program at Hall-D], Pizza seminar at Jefferson Lab, Newport News, Feb 26, 2014, Yi Qiang<br />
* [http://argus.phys.uregina.ca/cgi-bin/private/DocDB/ShowDocument?docid=2288 Physics Program at Jefferson Lab Hall-D], Seminar talk at Argonne National Lab, Chicago, August 2013, Yi Qiang<br />
* [http://argus.phys.uregina.ca/cgi-bin/private/DocDB/ShowDocument?docid=1957 The Search for Gluonic Excitations in Light Mesons with the GlueX Experiment], Seminar Talk at CPHT, Ecole Polytechnique, France, April 2012, Igor Senderovich<br />
<br />
== Colloquium Talks ==<br />
* [http://www.curtismeyer.com/talks/UTK_Sep_11_Colloquium.pptx The Jefferson Lab 12-GeV Upgrade and the GlueX Experiment], Colloquium at The University of Tennessee, Knoxville, September 2011, Curtis A. Meyer.<br />
<br />
* [http://www.curtismeyer.com/talks/ASU_Colloquium.pptx Quarks, QCD and Confinement: What we hope to learn at Jefferson Lab], Colloquium at ASU, February 2010, Curtis Meyer.<br />
<br />
* [http://www.curtismeyer.com/talks/StVincent2.pptx Gluonic Hadrons as a Probe of Confinement], Undergraduate Colloquium at Saint Vincent College, Latrobe PA, September 2009, Curtis Meyer.</div>Kmoriyahttps://halldweb.jlab.org/wiki/index.php?title=GlueX_Talks&diff=70931GlueX Talks2015-11-02T20:42:20Z<p>Kmoriya: /* Talks in 2015 */</p>
<hr />
<div>== Conference and Workshop Talks==<br />
<br />
=== Talks in 2015 ===<br />
* [http://argus.phys.uregina.ca/cgi-bin/private/DocDB/ShowDocument?docid=2687 Analysis Plans in GlueX] Athos/PWA Meeting, April 13-17 2015, GWU, presented by Curtis A. Meyer.<br />
* [http://argus.phys.uregina.ca/cgi-bin/private/DocDB/ShowDocument?docid=2745 Light-quark meson spectroscopy with the GlueX experiment] CIPANP May 2015, presented by Mark M. Dalton.<br />
* [http://argus.phys.uregina.ca/cgi-bin/private/DocDB/ShowDocument?docid=2795 Electromagnetic Production of Strangeness at Jefferson Lab] [http://lambda.phys.tohoku.ac.jp/hyp2015/ HYP2015], September 2015, presented by Kei Moriya<br />
* [http://argus.phys.uregina.ca/cgi-bin/private/DocDB/ShowDocument?docid=2799 First GlueX Results] Hadron 2015, September 2015, presented by Curtis A. Meyer.<br />
<br />
=== Talks in 2014 ===<br />
* [http://argus.phys.uregina.ca/cgi-bin/private/DocDB/ShowDocument?docid=2506 Physics in Hall D] Jefferson Lab 2014 Users Group Meeting, June 2-4, 2014, presented by Curtis A. Meyer.<br />
* [http://argus.phys.uregina.ca/cgi-bin/private/DocDB/ShowDocument?docid=2532 The GlueX experiment and the search for exotic mesons], PANIC 2014, August 25-29, Will Levine<br />
* [http://argus.phys.uregina.ca/cgi-bin/private/DocDB/ShowDocument?docid=2541 Electromagnetic Strangeness Production at Jefferson Lab Energies,] XI Conference on Quark Confinement at the Hadron Spectrum, September 8-12, 2014, presented by R. A. Schumacher.<br />
* [http://argus.phys.uregina.ca/cgi-bin/private/DocDB/ShowDocument?docid=2603 Eta Decay Programm at GlueX], MesonNet Meeting, INFN Frascati, September 29, Alexander Somov <br />
* [[DNP-2014 abstract submitted by Richard Jones|Abstract submitted to Mini-symposium on Hybrid Mesons and Molecules]], DNP-2014, Oct. 7-12, 2014, by Richard Jones<br />
* [http://argus.phys.uregina.ca/cgi-bin/private/DocDB/ShowDocument?docid=2689 GlueX Analysis] presented at the Future Directions in Partial Wave Analysis Workshop at Jefferson Lab, November 2014.<br />
<br />
=== Talks in 2013 ===<br />
* [http://argus.phys.uregina.ca/cgi-bin/private/DocDB/ShowDocument?docid=2375 Hybrid Meson Lectures] presented by Curtis Meyer in December 2013.<br />
* Carnegie Mellon University undergraduate colloquium, November 2013. [http://argus.phys.uregina.ca/cgi-bin/private/DocDB/ShowDocument?docid=2373 Big Science] a talk on how large science projects are conceived, funded and built following the 12-GeV upgrade as an example (Curtis A. Meyer).<br />
* [https://www.jlab.org/conferences/dnp2013/dnp-13.html APS-DNP 2013] October 23-26, 2013, Newport News, VA<br />
** [http://argus.phys.uregina.ca/cgi-bin/private/DocDB/ShowDocument?docid=2355 Characteristics of Silicon Photomultipliers (SiPM) for GlueX], Yi Qiang<br />
** [http://argus.phys.uregina.ca/cgi-bin/private/DocDB/ShowDocument?docid=2546 Reconstruction of showers in the GlueX barrel calorimeter], Will Levine<br />
** [http://argus.phys.uregina.ca/cgi-bin/private/DocDB/ShowDocument?docid=2674 JLab Hall-D Photon Beamline], Alexander Somov<br />
* [http://www.int.washington.edu/NNPSS/2013/HOME.html Nuclear Science Summer School] July 15-26, 2013, Stony Brook University campus. [http://argus.phys.uregina.ca/cgi-bin/private/DocDB/ShowDocument?docid=2278 Colloquium] presented by Curtis A. Meyer.<br />
* [http://www.inpc2013.it INPC2013] 25th International Nuclear Physics Conference June 2-7, 2013 Florence, Italy<br />
* [https://hep.ustc.edu.cn/indico/conferenceOtherViews.py?view=standard&confId=1 The Fifth Workshop on Hadron Physics in China and Opportunities in US], July 2-6, Huangshan, China<br />
** [http://argus.phys.uregina.ca/gluex/DocDB/0022/002282/001/20130703_Physics_HallD.pdf Physics Program at Jefferson Lab Hall-D], Yi Qiang<br />
* [http://www.aps.org/meetings/april/index.cfm APS] American Physical Society Meeting, April Mtg., Apr. 13-16, 2013, Denver, CO<br />
** [http://argus.phys.uregina.ca/cgi-bin/private/DocDB/ShowDocument?docid=2283 Characteristics of S12045(X) photon sensors for GlueX], Elton Smith<br />
** [http://argus.phys.uregina.ca/cgi-bin/private/DocDB/ShowDocument?docid=2284 The Barrel Calorimeter for the GlueX Experiment at Jefferson Lab], Zisis Papandreou (presented by Elton Smith)<br />
* [https://sites.google.com/site/ghpworkshop/ GHP Workshop] Meeting of the APS Topical Group on Hadron Physics (GHP), Apr. 10-12, 2013, Denver, CO<br />
** [http://argus.phys.uregina.ca/cgi-bin/private/DocDB/ShowDocument?docid=2189 Exploration of Exotic Mesons with GlueX], Elton Smith<br />
<br />
=== Talks in 2012 ===<br />
<br />
* PANDA XLIII Collaboration Meeting, Dec 10-14, 2012 GSI, Germany<br />
** [http://argus.phys.uregina.ca/cgi-bin/private/DocDB/ShowDocument?docid=2122 GlueX: Photoproduction of Hybrid Mesons], Elton Smith<br />
* Spectroscopy at 12 GeV Workshop, Jefferson Lab<br />
** [http://argus.phys.uregina.ca/cgi-bin/private/DocDB/ShowDocument?docid=2192 Current PWA Projects in Hall D], Matt Shepherd<br />
*[http://www.aps.org/units/dnp/meetings/meeting.cfm?name=DNP12/ DNP 2012] 2012 Fall Meeting of the APS Division of Nuclear Physics, October 24 - Ocober 27, 2012 Newport Beach, California<br />
** [http://argus.phys.uregina.ca/cgi-bin/private/DocDB/ShowDocument?docid=2110 GlueX: Thin Diamond Radiators for the GlueX Experiment], Brendan Pratt<br />
** [http://argus.phys.uregina.ca/cgi-bin/private/DocDB/ShowDocument?docid=2111 Collimation and Tagging Instrumentation for the GlueX Photon Beamline], R.T. Jones<br />
* PANDA XLII Collaboration Meeting, June 26-29, 2012 Evanston, IL<br />
** [http://argus.phys.uregina.ca/cgi-bin/private/DocDB/ShowDocument?docid=2193 GlueX: A search for light quark exotic mesons at Jefferson Lab], Matt Shepherd<br />
* [http://www.ge.infn.it/~athos12/ATHOS/Welcome.html ATHOS 2012] International Workshop on partial wave analysis, June 20-23, 2012. <br />
** [http://argus.phys.uregina.ca/cgi-bin/private/DocDB/ShowDocument?docid=2029 Talk] presented by Curtis Meyer.<br />
* [http://www.cap.ca/en/congress/2012 CAP Congress 2012], University of Calgary (Calgary, Alberta), June 11-15, 2012<br />
** [http://argus.phys.uregina.ca/cgi-bin/private/DocDB/ShowDocument?docid=2026 The GlueX Large Area Silicon Photomultipliers], Z. Papandreou<br />
*[http://www.jlab.org/conferences/ugm/program.htm Jefferson Lab User's Meeting], June 4-6, 2012.<br />
** [http://argus.phys.uregina.ca/cgi-bin/private/DocDB/ShowDocument?docid=2025 Talk] presented by Curtis Meyer.<br />
*[http://meson.if.uj.edu.pl/ MESON 2012] 12th International Workshop on Meson Production, Properties and Interaction, May 31 - Jun 5, 2012 Krakow, Poland<br />
** [http://argus.phys.uregina.ca/cgi-bin/private/DocDB/ShowDocument?docid=2016 GlueX: Photoproduction of Hybrid Mesons], Elton Smith<br />
*[http://cipanp2012.triumf.ca/ CIPANP 2012] Eleventh Conference on the Intersections of Particle and Nuclear Physics, May 29 - Jun 3, 2012, St. Petersburg, Florida<br />
** [http://argus.phys.uregina.ca/cgi-bin/private/DocDB/ShowDocument?docid=2022 GlueX: Neutron Radiation Hardness of SiPM and Its Applications in Jefferson Lab Hall-D], Yi Qiang<br />
** [http://argus.phys.uregina.ca/cgi-bin/private/DocDB/ShowDocument?docid=2675 Development of Level-1 Triggers for Experiments at Jefferson Lab], Alexander Somov<br />
<br />
=== Talks in 2011 ===<br />
*[http://web.mit.edu/panic11/ PANIC 11] The 19th Particles and Nuclei International Conference, in Cambridge, Massachusetts at the Massachusetts Institute of Technology (MIT)<br />
** [http://argus.phys.uregina.ca/cgi-bin/private/DocDB/ShowDocument?docid=1792 GlueX: Detector Construction and Event Simulations], Naomi Jarvis.<br />
* [http://www.hadron2011.de/ Hadron 2011], Munich, Germany, June 13-17, 2011<br />
** [http://argus.phys.uregina.ca/cgi-bin/private/DocDB/ShowDocument?docid=1780 Search for Gluonic Excitations in Hadrons with GlueX], I. Senderovich<br />
* [http://www.cap.ca/en/congress/2011 CAP Congress 2011], Memorial University of Newfoundland (St. John's, Newfoundland), June 13-17, 2011<br />
** [http://argus.phys.uregina.ca/cgi-bin/private/DocDB/ShowDocument?docid=2123 The GlueX Electromagnetic Barrel Calorimeter], Z. Papandreou<br />
* [http://www.aps.org/meetings/april/info/index.cfm APS Physics - April Meeting], Hyatt Regency Garden Grove, Anaheim, CA, April 30 - May 1, 2011.<br />
** [http://argus.phys.uregina.ca/cgi-bin/private/DocDB/ShowDocument?docid=1729 The GlueX Electromagnetic Barrel Calorimeter], Z. Papandreou.<br />
** [http://argus.phys.uregina.ca/cgi-bin/private/DocDB/ShowDocument?docid=1731 The Meson Spectrum from Lattice QCD], J. Dudek.<br />
** [http://argus.phys.uregina.ca/cgi-bin/private/DocDB/ShowDocument?docid=1734 The Experimental Spectrum of Hadrons], M. Shepherd.<br />
** [http://argus.phys.uregina.ca/cgi-bin/private/DocDB/ShowDocument?docid=1767 Overview of GlueX Offline Computing], M. Ito.<br />
** [http://argus.phys.uregina.ca/cgi-bin/private/DocDB/ShowDocument?docid=1727 Charged particle tracking for the gluex detector], S. Taylor.<br />
* [https://sites.google.com/site/ghpworkshop/home The 4th Workshop of the APS Topical Group on Hadronic Physics], Hyatt Regency Garden Grove, Anaheim, CA, April 27-29, 2011.<br />
** [http://argus.phys.uregina.ca/cgi-bin/private/DocDB/ShowDocument?docid=1730 The Meson Spectrum from Lattice QCD], J. Dudek.<br />
* [http://www1.jlab.org/ul/calendar/calendar_date.cfm?date=23&month=2&year=2011 Workshop on Excited Hadronic States and the Deconfinement Transition], Jefferson Lab, Newport News, VA, February 23-25, 2011.<br />
** [http://www.curtismeyer.com/talks/C_Meyer_Hadron_Spectroscopy.pptx Spectroscopy: experimental status and prospects], Curtis A. Meyer.<br />
* [http://www.sfu.ca/~caa12/WNPPC11/ WNPPC 2011], Winter Nuclear and Particle Physics Conference, 18-20 Feb 2011, Banff, Canada<br />
** [http://argus.phys.uregina.ca/cgi-bin/private/DocDB/ShowDocument?docid=1711 The GlueX Barrel Calorimeter], Zisis Papandreou<br />
** [http://argus.phys.uregina.ca/cgi-bin/private/DocDB/ShowDocument?docid=1712 Large Area Multi-Pixel Photon Detectors for the GlueX Barrel Calorimeter], Mehrnoosh Tahani<br />
* [http://www.gsi.de/forschung/kp/had2/meeting/Hirschegg_2011.html Hirschegg 2011], The Structure and Dynamics of Hadrons, Hirschegg, Austria, January 16-22, 2011<br />
** [http://argus.phys.uregina.ca/gluex/DocDB/0017/001706/001/hirschegg_gluex.pdf The GlueX Experiment (and its Context)], Ryan Mitchell<br />
<br />
=== Talks in 2010 ===<br />
* [http://www.lanl.gov/dnp DNP 2010], Santa Fe, NM, Nov 2 - 6.<br />
** [http://argus.phys.uregina.ca/cgi-bin/private/DocDB/ShowDocument?docid=1639 Level-1 Trigger of the GlueX Experiment], Alexander Somov<br />
* Jefferson Lab PAC36, Jefferson Lab, Newport News, VA, August, 2010.<br />
** [http://www.curtismeyer.com/talks/GlueX_Pac_Presentation.pptx GlueX Presentation], Curtis A. Meyer.<br />
* [http://www.lnf.infn.it/public/ LNF Frascati], Frascati, Italy, June 30, 2010<br />
** [http://argus.phys.uregina.ca/cgi-bin/private/DocDB/ShowDocument?docid=2124 The GlueX Experiment: construction is under way], Z. Papandreou<br />
* [http://www.cap.ca/en/congress/2010 CAP Congress 2010], University of Toronto (Toronto, Ontario), June 7-11, 2010<br />
** [http://argus.phys.uregina.ca/cgi-bin/private/DocDB/ShowDocument?docid=2123 The GlueX Electromagnetic Barrel Calorimeter], Z. Papandreou<br />
* Jefferson Lab Users Group Meeting, Jefferson Lab, Newport News, VA, June 7-9, 2010. <br />
** [http://argus.phys.uregina.ca/cgi-bin/public/DocDB/ShowDocument?docid=1544 GlueX/Hall-D Physics], Curtis A. Meyer.<br />
* [https://www.jlab.org/conferences/MENU10 Meson-Nucleon Physics and the Structure of the Nucleon (MENU 2010)], Williamsburg, VA, May 31, 2010.<br />
** [http://argus.phys.uregina.ca/cgi-bin/private/DocDB/ShowDocument?docid=1540 Physics Prospects with GlueX], Alexander Somov<br />
<br />
== Seminar Talks ==<br />
* [http://argus.phys.uregina.ca/cgi-bin/private/DocDB/ShowDocument?docid=2432 GlueX Program at Hall-D], Pizza seminar at Jefferson Lab, Newport News, Feb 26, 2014, Yi Qiang<br />
* [http://argus.phys.uregina.ca/cgi-bin/private/DocDB/ShowDocument?docid=2288 Physics Program at Jefferson Lab Hall-D], Seminar talk at Argonne National Lab, Chicago, August 2013, Yi Qiang<br />
* [http://argus.phys.uregina.ca/cgi-bin/private/DocDB/ShowDocument?docid=1957 The Search for Gluonic Excitations in Light Mesons with the GlueX Experiment], Seminar Talk at CPHT, Ecole Polytechnique, France, April 2012, Igor Senderovich<br />
<br />
== Colloquium Talks ==<br />
* [http://www.curtismeyer.com/talks/UTK_Sep_11_Colloquium.pptx The Jefferson Lab 12-GeV Upgrade and the GlueX Experiment], Colloquium at The University of Tennessee, Knoxville, September 2011, Curtis A. Meyer.<br />
<br />
* [http://www.curtismeyer.com/talks/ASU_Colloquium.pptx Quarks, QCD and Confinement: What we hope to learn at Jefferson Lab], Colloquium at ASU, February 2010, Curtis Meyer.<br />
<br />
* [http://www.curtismeyer.com/talks/StVincent2.pptx Gluonic Hadrons as a Probe of Confinement], Undergraduate Colloquium at Saint Vincent College, Latrobe PA, September 2009, Curtis Meyer.</div>Kmoriyahttps://halldweb.jlab.org/wiki/index.php?title=GlueX_Offline_Meeting,_October_28,_2015&diff=70911GlueX Offline Meeting, October 28, 20152015-10-28T17:24:23Z<p>Kmoriya: /* Agenda */</p>
<hr />
<div>GlueX Offline Software Meeting<br><br />
Wednesday, October 28, 2015<br><br />
1:30 pm EDT<br><br />
JLab: CEBAF Center F326/327<br />
<br />
==Agenda==<br />
<br />
# Announcements<br />
## new Version Management System features: URL checking for Git, hash-specific check-outs, [http://argus.phys.uregina.ca/cgi-bin/public/DocDB/ShowDocument?docid=2793 document update]<br />
## [http://www.jlab.org/Hall-D/software/HDSoftware_Documentation/ Doxygen] fixed<br />
## [https://mailman.jlab.org/pipermail/halld-offline/2015-October/002155.html New releases: sim-recon 1.6.0 and HDDS 3.4]<br />
## [https://halldweb.jlab.org/wiki-private Private wiki] getting there<br />
## [https://halldweb.jlab.org/wiki/index.php?title=GlueX_Offline_FAQ&diff=70904&oldid=70249 Recent additions to the Offline FAQ]<br />
# Review of [[GlueX Offline Meeting, October 14, 2015#Minutes|minutes from October 14]] (all)<br />
# [https://halldweb1.jlab.org/wiki/images/1/1b/2015-10-27-offline_monitoring.pdf Offline Monitoring] (Kei)<br />
# Geant4 Update (Richard, David)<br />
# [[Data Challenge 3]] update (Mark)<br />
# [[Sim1 Conditions|Future Commissioning Simulations]] (all)<br />
# [[Automatic Tests of GlueX Software|b1pi results review]]<br />
# Review of [https://github.com/JeffersonLab/sim-recon/pulls?q=is%3Aopen+is%3Apr recent pull requests]<br />
#* comments on merge<br />
# Action Item Review<br />
<br />
==Communication Information==<br />
<br />
===Remote Connection===<br />
<br />
* The BlueJeans meeting number is 968 592 007 .<br />
* [http://bluejeans.com/968592007 Join the Meeting] via BlueJeans<br />
<br />
===Slides===<br />
<br />
Talks can be deposited in the directory <code>/group/halld/www/halldweb/html/talks/2015</code> on the JLab CUE. This directory is accessible from the web at https://halldweb.jlab.org/talks/2015/ .</div>Kmoriyahttps://halldweb.jlab.org/wiki/index.php?title=File:2015-10-27-offline_monitoring.pdf&diff=70910File:2015-10-27-offline monitoring.pdf2015-10-28T17:23:46Z<p>Kmoriya: Talk at offline meeting on Oct 28 2015 for offline monitoring by Kei</p>
<hr />
<div>Talk at offline meeting on Oct 28 2015 for offline monitoring by Kei</div>Kmoriyahttps://halldweb.jlab.org/wiki/index.php?title=Data_Monitoring_Procedures&diff=70884Data Monitoring Procedures2015-10-26T14:38:22Z<p>Kmoriya: /* Starting the Launch and Submitting Jobs */</p>
<hr />
<div>__TOC__<br />
<br />
== Master List of File / Database / Webpage Locations ==<br />
=== Run Conditions ===<br />
* Online Run-by-run condition files (B-field, current, etc.): /work/halld/online_monitoring/conditions/<br />
* Offline monitoring run conditions (software versions, jana config): /group/halld/data_monitoring/run_conditions/<br />
*[http://www.jlab.org/Hall-D/test/RunInfo/ Run Info vers. 1]<br />
*[https://halldweb.jlab.org/cgi-bin/data_monitoring/run_conditions.pl Run Info vers. 2]<br />
<br />
=== Monitoring Output Files ===<br />
* Run Periods 201Y-MM is for example 2015-03, launch ver verVV is for example ver15<br />
* Online monitoring histograms: /work/halld/online_monitoring/root/<br />
* Offline monitoring histogram ROOT files: /work/halld/data_monitoring/RunPeriod-201Y-MM/verVV/rootfiles<br />
* REST files (most recent launch only): /work/halld/data_monitoring/RunPeriod-201Y-MM/REST/verVV<br />
* individual files for each job (ROOT, log, etc.): /volatile/halld/offline_monitoring/RunPeriod-201Y-MM/verVV/<br />
<br />
=== Monitoring Database ===<br />
* Accessing monitoring database (on ifarm): mysql -u datmon -h hallddb.jlab.org data_monitoring<br />
<br />
=== Monitoring Webpages ===<br />
*[https://halldweb.jlab.org/cgi-bin/data_monitoring/monitoring/plotBrowser.py Plot Browser]<br />
*[https://halldweb.jlab.org/cgi-bin/data_monitoring/monitoring/runBrowser.py Run Browser]<br />
*[https://halldweb.jlab.org/cgi-bin/data_monitoring/monitoring/timeSeries.py Time Series]<br />
*[https://halldweb.jlab.org/data_monitoring/launch_analysis/ Launch Analysis]<br />
<br />
== Job Monitoring Links ==<br />
* [http://scicomp.jlab.org/farm2/job.html Completed Job History]<br />
* [http://scicomp.jlab.org/farm2/project.html Job Stats By Project]<br />
* [http://scicomp.jlab.org/farm2/trackOrg.html Job Stats By Track]<br />
* [http://scicomp.jlab.org/farm2/report.html Cluster Report]<br />
* [http://scicomp.jlab.org/farm2/walltime.html Walltime Distribution]<br />
<br />
== Saving Online Monitoring Data ==<br />
<br />
The procedure for writing the data out is given in, e.g.,<br />
[https://halldweb.jlab.org/wiki/index.php/Raid-to-Silo_Transfer_Strategy Raid-to-Silo Transfer Strategy].<br />
<br />
Once the DAQ writes out the data to the raid disk, cron jobs will copy the file to tape,<br />
and within ~20 min., we will have access to the file on tape at<br />
/mss/halld/$RUN_PERIOD/rawdata/RunXXXXXX.<br />
<br />
All online monitoring plugins will be run as data is taken.<br />
They will be accessible within the counting house via RootSpy, and<br />
for each run and file, a ROOT file containing the histograms will be saved<br />
within a subdirectory for each run.<br />
<br />
For immediate access to these files, the raid disk files may be accessed directly<br />
from the counting house, or the tape files will be available within ~20 min. of the<br />
file being written out.<br />
<br />
== Offline Monitoring: Running Over Archived Data ==<br />
<br />
Once files are written to tape we run the online plugins on these files to confirm what we were seeing in the online monitoring, and also to update the results from the latest calibration and software. <br />
Manual scripts and cron jobs are set up to look for new data and run the plugins over a sample of files.<br />
<br />
Every other Friday (usually the Friday before the offline meetings) jobs will be started to run the newest software on all previous runs,and allows everybody to see improvements in each detector.<br />
For each launch, independent builds of hdds, sim-recon, the monitoring plugins, and an sqlite file will be generated.<br />
<br />
Below the procedures are described for<br />
# Preparing the software for the launch<br />
# Starting the launch (using hdswif)<br />
# Post-analysis of statistics of the launch<br />
<br />
Processing the results and making them available to the collaboration<br />
is handled in the section [[#Post-Processing_Procedures | Post-Processing Procedures]] below.<br />
<br />
<!--<br />
==== Generating an offline plugin job ====<br />
<br />
The user gxproj1 should be used for official offline monitoring jobs.<br />
Within /home/gxproj1/halld/monitoring/batch/ there will be scripts to run the online monitoring plugins over tape files.<br />
The main script is generatejobs_plugins_rawdata.sh, which can be used as<br />
generatejobs_plugins_rawdata.sh [minrun] [maxrun] (minfile) (maxfile)<br />
where minrun, maxrun specify the range of the run #, and minfile, maxfile (optional) specify<br />
the file range for that run.<br />
<br />
This will generate a script run_rawdata_XXXXXX.sh, where the run # has now been formatted to be 6 digits.<br />
Executing this script will send the monitoring plugins job to the Auger batch system.<br />
<br />
There is also a script clean.sh which can be used as<br />
clean.sh XXX<br />
This will clean up all associated files created in association with run XXX.<br />
<br />
Internally, the xml file used to submit the job will be created, and the job to run<br />
will be given within script.sh. All run parameters should be specified in at the beginning<br />
of generatejobs_plugins_rawdata.sh<br />
Since we are running on tape, the tape file will first be copied over to the cache disk, and the job will run<br />
over this cached file.<br />
<br />
==== Using cron to run automatically ====<br />
Within /home/gluex/halld/monitoring/cron/ there is a file cron_plugins<br />
that can be executed via<br />
crontab cron_plugins<br />
This will set up a cron job to call the script scan_for_jobs.sh, which will<br />
check in the rawdata directory and call generatejobs_plugins_rawdata.sh for<br />
any run that is more than 5 min old. The cron job is set up to run every 10 min.<br />
<br />
==== Magnetic Field Settings ====<br />
For tracking, it is necessary to set the correct magnetic field settings.<br />
The below table is now obsolete. Please refer to the [https://halldweb.jlab.org/cgi-bin/data_monitoring/run_conditions.pl GlueX Run Conditions] webpage.<br />
<br />
The actual field settings for each run have not been documented well.<br />
Below are values based on going through entries in the [https://logbooks.jlab.org/book/halld halld log].<br />
<br />
{| border="1" cellpadding="1" style="text-align: center;"<br />
!width="100"| Run #s<br />
!width="150"| Solenoid Current (A)<br />
!width="300"| JANA option<br />
!width="300"| notes<br />
|-<br />
| 940 - 996 || 1000 || -PBFIELD_MAP=Magnets/Solenoid/solenoid_1000A_poisson_20141104 || At run 997, the solenoid started ramping down<br />
|-<br />
| 998 - 1448 || 0 || -PBFIELD_TYPE=NoField -PDEFTAG:DTrackCandidate=StraightLine || Solenoid started ramping up around run 1431. Runs 1432 - 1448 should not be used for tracking. See [https://logbooks.jlab.org/entry/3309039 this] and [https://logbooks.jlab.org/entry/3309092 this] entry.<br />
|-<br />
| 1449 - 1620 || 1200 || -PBFIELD_MAP=Magnets/Solenoid/solenoid_1200A_poisson_20140520 || <br />
|-<br />
|}<br />
<br />
'''Note:''' The following run ranges were taken with a 300A field, and could be analyzed with the magnetic field map Magnets/Solenoid/solenoid_300A_poisson_20140819 , more study is needed to see if the straight line track fitter is needed or not: 1036 - 1053, 1065 - 1121, 1212? - 1254, 1309 - 1318 [1307,1308,1319-1329 were taken while the magnet was ramping].<br />
--><br />
<br />
=== General Information on Procedures ===<br />
This section explains how the offline monitoring should be run.<br />
Since we may want to simultaneously run offline monitoring for different run periods that require<br />
different environment variables, the scripts are set up so that a generic user can download the<br />
scripts and run them from anywhere. Most output directories for offline monitoring are created<br />
with group read/write permissions so that any Hall D group user has access to the contents,<br />
but there are some cases where use of the account that created the launch is necessary.<br />
<br />
The accounts used for offline monitoring are the gxprojN accounts created and maintained by<br />
Mark Ito (see [https://halldweb.jlab.org/wiki/index.php/GlueX-related_shared_accounts_on_the_JLab_CUE here] for how each account is used).<br />
As of October 2015, the following are used:<br />
* gxproj1 for running over Fall 2014 data (deprecated since June 2015)<br />
* gxproj5 for running over Spring 2015 data<br />
<br />
Since the summer of 2015 we have transitioned from a system using Mark Ito's jproj scripts<br />
to integrating the swif system that Chris Larrieu (SciComp) has been developing. For offline monitoring,<br />
the hdswif system that Kei developed is used for launching the jobs, and the jproj system is used<br />
for meta-analysis of launch statistics.<br />
<br />
Both hdswif and jproj are maintained in svn:<br />
* hdswif: https://halldsvn.jlab.org/repos/trunk/scripts/monitoring/hdswif<br />
* jproj : https://halldsvn.jlab.org/repos/trunk/scripts/monitoring/jproj<br />
<br />
To run the offline monitoring each package should be checked out and all necessary<br />
scripts are included.<br />
<br />
=== Preparing the software for the launch ===<br />
<br />
To begin a new launch the software must be built to the latest versions.<br />
For the gxprojN user accounts used, all software builds are contained in the directory ~/builds<br />
(which are soft links to /work/halld/home/gxprojN/builds). When logging into these accounts<br />
the setup files ~/setup_jlab-2015-03.csh or similar files should be sourced.<br />
<br />
<b>Note that Mark Ito does <u>not</u> want you to change the contents of each .cshrc file.</b><br />
You should consult him if you feel the need.<br />
<br />
<span style="color:red">NOTE: FOR BUILDING SOFTWARE IT IS A GOOD IDEA TO DO A COMPLETE WIPEOUT/CHECKOUT EACH TIME TO AVOID STALE HEADER FILES.</span><br />
<br />
# Building hdds: Go to ~/builds/hdds. The directory hdds is the one from git. Delete the contents, then download the newest version from git and build: <pre>cd ~/builds/hdds/</pre> <pre>rm -frv hdds</pre> <pre>git clone https://github.com/JeffersonLab/hdds</pre> <pre>cd hdds</pre> <pre>scons install</pre><br />
# Building sim-recon: Go to ~/builds/sim-recon. The directory sim-recon is the one from git. Delete the contents, then download the newest version from git and build: <pre>cd ~/builds/sim-recon</pre> <pre>rm -frv sim-recon</pre> <pre>git clone https://github.com/JeffersonLab/sim-recon</pre> <pre>cd sim-recon/src</pre> <pre>scons install -j8</pre><br />
# Prepare the latest sqlite file: The sqlite is set in the ~/setup_jlab-2015-03.csh script as <b>sqlite:////home/gxproj5/ccdb.sqlite</b> through the environment variables <b>JANA_CALIB_URL</b> and <b>CCDB_CONNECTION</b>. Therefore, go to this directory and create a new sqlite file. We create the sqlite file in a temporary directory since creating the sqlite file in a directory where the output file exists causes errors. Original documentation on creating sqlite files are [https://halldweb.jlab.org/wiki/index.php/SQLite-form_of_the_CCDB_database here]. <br><span style="color:red">NOTE: SQLITE FILES DO NOT WORK ON THE NEW /work DISK INSTALLED IN OCTOBER 2015 </span> <pre>cd ~/tmp</pre> <pre>$CCDB_HOME/scripts/mysql2sqlite/mysql2sqlite.sh -hhallddb.jlab.org -uccdb_user ccdb | sqlite3 ccdb.sqlite</pre> <pre>mv ccdb.sqlite ../</pre><br />
# Note that the above steps must be done <b>BEFORE</b> launch project creation. This is because we will track the revisions of the libraries used, and this is done by extracting the svn information in each directory. Also, note that the system assumes that we have the topmost build directory (usually called GLUEX_TOP) to be $HOME/builds . Such an assumption is necessary to be able to extract information about the library locations automatically.<br />
<br />
Create the appropriate project(s) and submit the jobs using hdswif, as detailed in the section below.<br />
<br />
=== Starting the Launch and Submitting Jobs ===<br />
<br />
Until the summer of 2015 we relied solely on Mark Ito's jproj system for submitting and keeping track of jobs. We have since moved to the swif system and use the hdswif wrapper for this. Below are instructions for how to use these.<br />
<br />
# Downloading hdswif: Download the hdswif directory from svn. For the gxprojN accounts, use the directory ~/halld/hdswif. <pre>cd ~/halld </pre> <pre> svn co https://halldsvn.jlab.org/repos/trunk/scripts/monitoring/hdswif</pre> <pre>cd hdswif</pre><br />
# Creating the workflow: Within SWIF jobs are registered into workflows. First create the workflow. For offline monitoring, the workflow names are of the form <b>offline_monitoring_RunPeriod201Y_MM_verVV_hd_rawdata</b> with suitable replacements for the run period and version number. The command "swif list" will list all existing workflows. Also, for most simple SWIF commands hdswif also provides a wrapper. <pre>swif list</pre> For creation of workflows for offline monitoring the command <pre>hdswif.py create [workflow] -c [config file] </pre> should be used. As an example config file, see the input.config file in the folder (and update it). When a config file is passed in, hdswif will automatically create files that record the configuration of the current launch. These files are stored as for example<br />
#* /group/halld/data_monitoring/run_conditions/jana_rawdata_comm_2015_03_ver15.conf<br />
#* /group/halld/data_monitoring/run_conditions/soft_comm_2015_03_ver15.xml<br />
#* The software packages stored in git (sim-recon and hdds) can have git tags applied to them, which makes it easier to find versions of the software than a SHA-1 hash. hdswif will ask if you would like to create a tag, and execute the following sequence: <pre>git tag -a offmon-201Y_MM-verVV -m "Used for offline monitoring 201Y-MM verVV started on 201y/mm/dd"</pre> <pre>git push offmon-201Y_MM-verVV</pre> This will only be invoked when the user name is gxprojN, and for the configuration files, the output directory will be /group/halld/data_monitoring/run_conditions/ for gxprojN accounts while it will be the current directory for other users.<br />
#* To use the git-tagged software versions do for example <pre>cd $HALLD_HOME</pre> <pre>git checkout offmon-2015_03-ver15</pre><br />
# Registering jobs in the workflow: To register jobs within the workflow, hdswif provides the use of config files. Jobs can be registered by specifying the workflow, config file (-c), run (-r) and file (-f) numbers if necessary. A typical config file will look this:<br />
PROJECT gluex<br />
TRACK reconstruction<br />
OS centos65<br />
NCORES 6<br />
DISK 40<br />
RAM 8<br />
TIMELIMIT 8<br />
JOBNAMEBASE offmon_<br />
RUNPERIOD 2015-03<br />
VERSION 15<br />
OUTPUT_TOPDIR /volatile/halld/offline_monitoring/RunPeriod-[RUNPERIOD]/ver[VERSION] # Example of other variables included in variable<br />
SCRIPTFILE /home/gxproj5/halld/hdswif/script.sh # Must specify full path<br />
ENVFILE /home/gxproj5/halld/hdswif/setup_jlab-2015-03.csh # Must specify full path<br />
The config file contains configuration parameters for each of the jobs.<br><br />
<span style="color:red">Note: Job configuration parameters can be set differently for jobs within the same workflow if necessary.</span><br />
Edit the config file and save as a new file if necessary. Once the configuration is set, jobs can be added via <pre>hdswif.py add [workflow] -c input.config</pre><br />
By default, hdswif will add all files found within the directory /mss/halld/RunPeriod-201Y-MM/rawdata/ where 201Y-MM is specified by the RUNPERIOD parameter in the config file. If only some of the runs or files are needed, these can be specified for example with<br />
<pre>hdswif.py add [workflow] -c input.config -r 3180 -f '00[0-4]'</pre><br />
to specify to register running only over run 3180 files 000 - 004 (Unix-style brackets and wildcards can be used).<br />
<br />
Running the workflow: To run the workflow, simply use swif run: <pre>swif run -workflow [workflow] -errorlimit none</pre> <span style="color:red">MAKE SURE THE ERRORLIMIT IS SET TO NONE OR THE WORKFLOW WILL BE STOPPED AFTER ANY JOB FAILS</span> or equivalently, using the hdswif wrapper (which has the errorlimit set by default), <pre>hdsswif.py run [workflow]</pre><br><br />
<b>It is recommended that some jobs be tested to make sure that everything is working rather than fail thousands of jobs.</b><br> For this purpose, hdswif will take an additional parameter to run which limits the number of jobs to submit: <pre>hdswif.py run [workflow] 10</pre><br />
in which case only 10 jobs will be submitted. To submit all jobs after checking the results, do <pre>hdswif.py run [workflow]</pre><br />
<br />
=== Checking the Status and Resubmitting ===<br />
# The status of jobs can be checked on the terminal with <pre>jobstat -u gxprojN</pre> on Auger and for SWIF with <pre>swif list</pre> or for more information, <pre>swif status [workflow] -summary</pre> Also see the Auger [http://scicomp.jlab.org/scicomp/#/auger/jobs job website].<br />
# For failed jobs, SWIF can resubmit jobs based on the problem. For resubmission for failed jobs with the same resources, <pre>swif retry-jobs [workflow] -problems [problem name]</pre> can be used, and for jobs to be submitted with more resources, e.g., use <pre>swif modify-jobs -ram add 2gb -problems AUGER-OVER_RLIMIT</pre> hdswif has a wrapper for both of these: <pre>hdswif.py resubmit [workflow] [problem]</pre> In this case [problem] can be one of <b>SYSTEM, TIMEOUT, RLIMIT</b>. If SYSTEM is specified, the jobs will be retried. For TIMEOUT and RLIMIT, the jobs will be modified by default with 2 additional hours or GB of RAM. If one more number is added as an option, then that many hours or GB of RAM will be added., e.g., <pre>hdswif.py resubmit [workflow] TIMEOUT 5</pre> will add 5 hours of processing time. You can wait until almost all jobs finish before resubmitting failed jobs since the number should be relatively small. <b>Even if jobs are resubmitted for one type of failure, jobs that later fail with that failure will not be automatically resubmitted.</b><br />
# For information on swif, use the "swif help" commands and for hdswif see the attached documentaion in https://halldsvn.jlab.org/repos/trunk/scripts/monitoring/hdswif/manual_hdswif.pdf<br />
<br />
=== Post-analysis of statistics of the launch ===<br />
<br />
# After jobs have been submitted, it will usually take a few days for all of the jobs to be processed.<br />
#* Status of Auger: http://scicomp.jlab.org/scicomp/#/auger/jobs (see also links above)<br />
#* Status of user jobs: <pre>jobstat -u [user name]</pre><br />
# The status and results of jobs are saved within the SWIF internal server, and are available via the command <pre>swif status [workflow] -summary -runs</pre> where the arguments -summary and -runs show summary statistics and statistics for individual jobs, respectively. hdswif has a command that takes this output in XML output and creates an HTML webpage showing results of the launch. To do this, do <pre>hdswif.py summary [workflow]</pre> This will create an XML file swif_output_[workflow].xml that contains all information from SWIF. If the file already exists, hdswif will ask whether to overwrite the existing file.<br />
# At this stage the html output and figure files are created and ready to be put online. For this step and other steps involving analysis of the statistics of the launch results, it is convenient to change to the jproj system.<br />
# The jproj scripts for offline monitoring are maintained in the svn directory https://halldsvn.jlab.org/repos/trunk/scripts/monitoring/jproj Do <pre> svn co https://halldsvn.jlab.org/repos/trunk/scripts/monitoring/jproj </pre> For the gxprojN accounts used for offline monitoring the directory should be ~/halld/jproj<br />
# The jproj directory contains two subdirectories, scripts and projects. The scripts directory contains useful scripts for processing the jobs registered in the jproj system, and each of the offline monitoring launches will be handled in the projects directory.<br />
# Go to the projects directory<pre>cd ~/halld/jproj/projects</pre> and use the script create_project.sh to create a new directory that contains the processing scripts for the current launch <pre>./create_project.sh [workflow]</pre> This should create a directory such as offline_monitoring_RunPeriod2015_03_ver15_hd_rawdata (same as the workflow name). The script uses the template files in the directory templates and by substitution creates script files for the current launch. Now go to the newly created analysis directory: <pre>cd [workflow]/analysis</pre><br />
which for the gxprojN accounts should have the full path /home/gxprojN/halld/jproj/projects/[workflow]/analysis.<br><br><br />
<span style="color:red">All of the analysis commands including arguments are contained in run_analysis.sh, but it is <b>strongly</b> recommended that all commands are run manually to check for errors.</span><br><br><br />
# The first thing to do is to make the html output from hdswif public. Copy the html file and related figures that were created from hdswif to the appropriate space within /group/halld/www/halldweb/html/data_monitoring/ : <pre>python publish_offmon_results.py [run period] [version]</pre> Note that the command with appropriate substitutions for arguments can be found within run_analysis.sh (same for all commands below).<br />
# Next, we need to create a few MySQL tables for the current launch. The MySQL tables are useful for comparing run/file combinations for different launches. For the SWIF launches, there are two tables needed, and will be named<br />
#* [workflow]Job<br />
#* [workflow]_aux<br />
# This naming scheme of tables and their roles are the same as from the jproj only launches. The [workflow]Job table will contain information gathered from SWIF about each job (which node it went to, start time of each stage, memory usage, etc.). The [workflow]_aux table will contain information gathered from the output stdout files from each job. First, create the Job table using <pre>python create_jproj_job_table.py [run period] [version]</pre> Note that this script uses the XML output from hdswif summary and inserts the contents into the MySQL table, so the XML output file must exist.<br />
# To check the contents of this MySQL table, do <pre>mysql -hhalldb -ufarmer farming</pre> <pre>mysql> describe offline_monitoring_RunPeriod2015_03_ver11_hd_rawdataJob;</pre><br />
<br />
== Hall D Job Management System ==<br />
<br />
This section details instructions on how to create and launch a set of jobs using the Hall-D Job Management System developed by Mark Ito. These instructions are generic: this system can be used for the weekly monitoring jobs, but can also be used for other sets of job launches as well. <br />
<br />
=== Database Table Overview ===<br />
<br />
* Job management database table (<project_name>): For each input file, keeps track of whether or not a job for it has been submitted, along with other optional fields. <br />
<br />
* Job status database table (<project_name>Job (no space)): For each job, keeps track of the job-id, the job status, memory used, cpu & wall time, time taken to complete various stages (e.g. pending, dependency, active), and others. <br />
<br />
* Job metrics database table (<project_name>_aux (no space)): For each job, keeps track of the job-id, how many events were processed, the time it took to copy the cache file, and the time it took to run the plugin. This information is culled from the log files of each job, and is done within the analysis directory of each launch.<br />
<br />
=== Initialize Project Management ===<br />
<br />
* Log into the ifarm machine with one of the gxproj accounts. For this example we will use gxproj1.<br />
<pre><br />
ssh gxproj1@ifarm -Y<br />
</pre><br />
<br />
* Go to a directory to do the launch. In principle, any directory will work, but for gxproj1 this is usually done in /home/gxproj1/halld/<br />
<br />
* Check out the necessary scripts <pre>svn co https://halldsvn.jlab.org/repos/trunk/scripts/monitoring/jproj</pre> This will get all necessary scripts for launching. Once checked out, <pre>cd projects</pre><br />
<br />
* The script create_project.sh can be used to create a new project. It will take a single argument, the project name. It is assumed that the project name is of the form offline_monitoring_RunPeriod20YY_MM_verVV_hd_rawdata where this string will be parsed to give the run period 20YY_MM and version number VV. One thing to do BEFORE creation of a new project is to editthe conditions of the launch (plugins to run over, memory requested, disk space requested) within templates/template.jsub . This<br />
information is saved automatically at project creation time into files in /group/halld/data_monitoring/run_conditions .<br />
To create a project do <pre>./create_project.sh offline_monitoring_RunPeriod20YY_MM_verVV_hd_rawdata</pre><br />
The name has been chosen to be as consistent as possible with other directory structures.<br />
However, mysql requires that "-" be escaped in table names, so unfortunately run periods will be<br />
given as 2YYY_MM instead of 2YYY-MM.<br />
<br />
* A new directory with the project name (offline_monitoring_RunPeriod20YY_MM_verVV_hd_rawdata) and files will be copied and modified from the template directory to reflect the run period, the user, the directory that it was created in, project name, etc.<br />
<br />
* For each project, cd into the new directory<br />
* The script clear.sh will remove any existing tables for the current project name, then recreate it. Do<br />
<pre>./clear.sh</pre><br />
* To use the jproj.pl script that was checked in, add the directory to your path with<br />
<pre>source ../../scripts/setup.csh</pre><br />
or always specify the full path<br />
<pre>../../scripts/jproj.pl</pre><br />
* Now update the table of runs with<br />
<pre>jproj.pl <project name> update</pre><br />
This will fill the table with all files within /mss that are of the same form as what is in <project name>.jproj . If you want to register only on a subset of all such files, you can edit this file directly.<br />
* Once you have registered all of the files you would like to run over, do<br />
<pre>jproj.pl <project name> submit [max # of jobs] [run number]</pre><br />
where the additional options specify how many jobs to submit and which run number to run on. Without these options all files that are registered and have not been submitted yet will be submitted.<br />
<br />
At this stage you are ready to submit all files. It is a good idea to submit a few test jobs<br />
at first to check that all scripts are working and that the plugins do not crash. Once you are<br />
sure that this does not happen, you can send all jobs in. The remaining jobs are then the monitoring<br />
which will (among other things) put the results on the online webpage for the collaboration to view, and<br />
the analysis of the launch.<br />
<br />
=== Project File Overview ===<br />
<br />
An overview of each project file:<br />
* '''clear.sh:''' For the current project, deletes the job status and management database tables (if any), and creates new, empty ones. <br />
* '''<project_name>.jproj:''' Contains the path and file name format for the input files for the jobs. <br />
* '''<project_name>.jsub:''' The xml job submission script. The run number and file number variables are set during job submission for each input file. <br />
* '''script.sh:''' The script that is executed during the job. If output job directories are not pre-created manually, they should be created in this script with the proper permissions:<br />
<pre><br />
mkdir -p -m 775 my_directory<br />
</pre><br />
* '''setup_jlab-[run period].csh:''' The environment that is sourced at the beginning of the job execution. <br />
* '''status.sh:''' Updates the job status database table, and prints some of its columns to screen.<br />
<br />
=== Project Management ===<br />
<br />
* Delete (if any) and create the database table(s) for the current set of job submissions:<br />
<pre><br />
./clear.sh<br />
</pre><br />
Also, if testing was done with jobs, it is best to delete the output directory and the configuration files:<br />
<pre><br />
rm -frv /volatile/halld/offline_monitoring/RunPeriod-20YY-MM/verVV /group/halld/data_monitoring/run_conditions/soft_comm_20YY_MM_verVV.xml /group/halld/data_monitoring/run_conditions/jana_rawdata_comm_20YY_MM_verVV.conf<br />
</pre><br />
<br />
* Search for input files matching the string in the .jproj file, and create a row for each in the job management database table (called <project_name>). You can test by adding an optional argument at the end, which only selects files with a specific file number:<br />
<pre><br />
jproj.pl <project_name> update <optional_file_number><br />
</pre><br />
<br />
* Confirm that the job management database is accurate by printing it's contents to screen:<br />
<pre><br />
mysql -hhallddb -ufarmer farming -e "select * from <project_name>"<br />
</pre><br />
<br />
* ONLY if a mistake was made, to delete the tables from the database and recreate new, empty ones, run: <br />
<pre><br />
./clear.sh<br />
</pre><br />
<br />
* To look at the status of the submitted jobs, first query auger and update the job status database:<br />
<pre><br />
fill_in_job_details.pl <project_name><br />
</pre><br />
<br />
* The job status can then be viewed by submitting a query to the job status database (called <project_name>Job (no space in between)):<br />
<pre><br />
mysql -hhallddb -ufarmer farming -e "select id,run,file,jobId,hostname,status,timeSubmitted,timeActive,walltime,cput,timeComplete,result,error from <project_name>Job"<br />
</pre><br />
<br />
* These last two commands can instead be executed simultaneously by running:<br />
<pre><br />
./status.sh<br />
</pre><br />
<br />
=== Handy mysql Instructions ===<br />
<br />
* Handy mysql instructions:<br />
<pre><br />
mysql -hhallddb -ufarmer farming # Enter the "farming" mysql database on "hallddb" as user "farmer"<br />
quit; # Exit mysql<br />
show tables; # Show a list of the tables in the current database<br />
show columns from <project_name>; # show all of the columns for the given table<br />
select * from <project_name>; # show the contents of all rows from the given table<br />
</pre><br />
<br />
=== Backing Up Offline Monitoring Tables ===<br />
Tables created for offline monitoring can be backed up using the script backup_tables.sh which<br />
can be checked out with the other files from https://halldsvn.jlab.org/repos/trunk/scripts/monitoring/jproj/projects<br />
<br />
The script uses the command mysqldump to print out a file that can be executed to recreate the tables.<br />
Since executing this output file will drop the table when it exists, caution is advised.<br />
Example usage to backup all three tables created for run period 2014_10 ver 17:<br />
<pre>backup_tables.sh 2014_10 17</pre><br />
<br />
== Running Over Data As It Comes In ==<br />
<br />
A special user gxproj1 will have a cron job set up to run the plugins as new data appears on /mss.<br />
During the week, gxproj1 will submit offline plugin jobs with the same setup as the weekly jobs<br />
run the previous Friday. The procedure for this is shown below.<br />
<br />
<!--<br />
=== Setting up the environment ===<br />
The file<br />
/home/gxproj1/setup_jlab.csh<br />
is sourced through .tcshrc.<br />
This file is the same as what is linked to by<br />
/home/gluex/setup_jlab_commissioning.csh,<br />
except HALLD_HOME, HDDS_HOME, and JANA_CALIB_URL are set separately so that this<br />
user can have a separate build.<br />
<br />
To obtain the builds from the previous Friday's runs,<br />
execute<br />
/home/gxproj1/halld/monitoring/newruns/setup_previous.sh [year] [month] [day]<br />
The build revisions from the previous Friday are archived in files<br />
/work/halld/data_monitoring/run_conditions/soft_comm_[year]_[month]_[day].xml<br />
and the script will build libraries based on those stored revision numbers.<br />
--><br />
<br />
=== Running the cron job ===<br />
<br />
'''IMPORTANT:''' The cron job should not be running while you are manually submitting jobs using the jproj.pl script for the same project, or else you will probably multiply-submit a job. <br />
<br />
* Go to the cron job directory: <br />
<pre><br />
cd /u/home/gxproj1/halld/monitoring/newruns<br />
</pre><br />
<br />
* The cron_plugins file is the cronjob that will be executed. During execution, it runs the exec.sh command in the same folder. This command takes two arguments: the project name, and the maximum file number for each run. These fields should be updated in the cron_plugins file before running. <br />
<br />
* The exec.sh command updates the job management database table with any data that has arrived on tape since it was last updated, ignoring file numbers greater than the maximum file number. It then submits jobs for these files. <br />
<br />
* To start the cron job, run:<br />
<pre><br />
crontab cron_plugins<br />
</pre><br />
<br />
* To check whether the cron job is running, do<br />
<pre><br />
crontab -l<br />
</pre><br />
<br />
* To remove the cron job do<br />
<pre><br />
crontab -r<br />
</pre><br />
<br />
<!--<br />
The cron job will run the script scan_for_jobs.sh,<br />
which runs generatejobs_plugins_rawdata.sh for any<br />
new runs that it had not seen before. All previous<br />
runs are recorded in the file filelists/files_current.txt<br />
so clear this to run over runs, or set the parameters<br />
MINRUN and MAXRUN which will set the range of runs submitted.<br />
--><br />
<br />
==Post-Processing Procedures==<br />
<br />
To visualize the monitoring data, we save images of selected histograms and store time series of selected quantities in a database, which are then displayed on a web page. This section describes how to generate the monitoring images and database information.<br />
<br />
The scripts used to generate this summary data are primarily run from /home/gxprojN/halld/monitoring/process . If you want a new copy of the scripts, e.g., for a new monitoring run, you should check the scripts out from SVN:<br />
<syntaxhighlight><br />
svn co https://halldsvn.jlab.org/repos/trunk/scripts/monitoring/process<br />
</syntaxhighlight><br />
Note that these scripts currently have some parameters which must be periodically set by hand.<br />
<br />
The default python version on most JLab machine does not have the modules to allow these scripts to connect to the MySQL database. To run these scripts, load the environment with the following command<br />
<syntaxhighlight><br />
source /home/gxproj1/halld/monitoring/process/monitoring_env.csh<br />
</syntaxhighlight><br />
<br />
===Online Monitoring===<br />
<br />
There are two primary scripts for running over the monitoring data generated by the online system and offline reconstruction. The online script can be run with either of the following commands:<br />
<syntaxhighlight><br />
/home/gluex/halld/monitoring/process/check_new_runs.py<br />
<br />
OR <br />
<br />
/home/gluex/halld/monitoring/process/check_new_runs.csh<br />
</syntaxhighlight><br />
The shell script sets up the environment properly to run the python script. To connect to the monitoring database on the JLab CUE, modules included in the local installation of python >= 2.7 are needed. The shell script is appropriate to use in a cron job. The cronjob is currently run under the "gluex" account.<br />
<br />
The online monitoring system copies a ROOT file containing the results of the online monitoring, and other configuration files into a directory accessible outside the counting house. This python script automatically checks for new ROOT files, which it will then automatically process. It contains several configuration variables that must be correctly set, which contains the location of input/output directories, etc...<br />
<br />
===Offline Monitoring===<br />
<br />
After the data is run over, the results should be processed, so that summary data is entered into the monitoring database and plots are made for the monitoring webpages. Currently, this processing is controlled by a cronjob that runs the following script:<br />
<syntaxhighlight><br />
/home/gxproj1/halld/monitoring/process/check_monitoring_data.csh <br />
</syntaxhighlight><br />
This script checks for new ROOT files, and only runs over those it hasn't processed yet. Since one monitoring ROOT file is produced for each EVIO file, whenever a new file is produced, the plots for the corresponding run are recreated and all the ROOT files for a run are combined into one file. Information is stored in the database on a per-file basis. <br />
<br />
Plots for the monitoring web page can be made from single histograms or multiple histograms using RootSpy macros. If you want to change the list of plots made, you must modify one of the following files:<br />
* histograms_to_monitor - specify either the name of the histogram or its the full ROOT path <br />
* macros_to_monitor - specify the full path to the RootSpy macro .C file<br />
<br />
When a new monitoring run is started, or the conditions are changed, the following steps should be taken to process the new files:<br />
# Add a new data version, as described below:<br />
# Change the following parameters in check_monitoring_data.csh:<br />
## JOBDATE should correspond to the ouptut date used by the job submission script<br />
## OUTPUTDIR should correspond to the directory corresponding to the run period and revision corresponding to the new version you just submitted. Presumably, this directory will be empty at the beginning.<br />
## Once you create a new data version as defined below, you should pass the needed information as a command line option. Currently this is done by the ARGS variable. For example, the argument "-v RunPeriod-2014-10,8" tells the monitoring scripts to look up the version corresponding to revision 8 of RunPeriod-2014-10 in the monitoring DB and to use to store the results.<br />
<br />
<syntaxhighlight><br />
Example configuration parameters:<br />
set JOBDATE=2015-01-09<br />
set INPUTDIR=/volatile/halld/RunPeriod-2014-10/offline_monitoring<br />
set OUTPUTDIR=/w/halld-scifs1a/data_monitoring/RunPeriod-2014-10/ver08<br />
set ARGS=" -v RunPeriod-2014-10,8 "<br />
</syntaxhighlight><br />
If you want to process the results manually, the data is processed using the following script:<br />
<syntaxhighlight><br />
./process_new_offline_data.py <input directory> <output directory><br />
<br />
EXAMPLE:<br />
<br />
./process_new_offline_data.py 2014-11-14 /volatile/halld/RunPeriod-2014-10/offline_monitoring/ /w/halld-scifs1a/data_monitoring/RunPeriod-2014-10/ver02<br />
</syntaxhighlight><br />
The python script takes several options to enable/disable various steps in the processing. Of interest is the "--force" option, which will run over all monitoring ROOT files, whether or not they've been previously identified.<br />
<br />
Every time a new reconstruction pass is performed, a new version number must be generated. To do this, prepare a version file as described below. Then run the register_new_version.py script to store the information in the database. The script will return a version number, which then should be set by hand in process_new_offline_data.py - future versions of the script will streamline this part of the procedure. An example of how to generate a new version is:<br />
<syntaxhighlight><br />
./register_new_version.py add /home/gxproj1/halld/monitoring/process/versions/vers_RunPeriod-2014-10_pass1.txt<br />
</syntaxhighlight><br />
<br />
<b>If you are running the offline monitoring by checking out the files in trunk/scripts/monitoring/jproj/projects/<br />
of the svn repository</b>, and created a project with<br />
create_project.sh [project name] hd_rawdata<br />
Then go to the directory [project name]/processing/<br />
and execute<br />
./run_processing.sh<br />
which will run register_new_version.py as well as check_monitoring_data.csh for that project.<br />
<br />
===Step-by-Step Instructions For Processing a New Monitoring Run===<br />
<br />
The monitoring runs are current run out of the gxproj1 and gxproj5 accounts. After an offline monitoring run has been successfully started on the batch farm, the following steps should be followed to setup the post-processing for these runs.<br />
<br />
# The post-processing scripts are stored in $HOME/halld/monitoring/process and are automatically run by cron.<br />
# Run "svn update" to bring any changes in. Be sure that the list of histograms and macros to plot are current.<br />
# Add a new data version<br />
# Edit check_monitoring_data.csh to point to the current revisions/directories<br />
#* VERSION<br />
#* ARGS<br />
#* Note that the environment depends on a standard script - $HOME/setup_jlab.csh<br />
# Update files in the web directory, so that the results are displayed on the web pages: /group/halld/www/halldweb/html/data_monitoring/textdata<br />
# Copy the REST files to more permanent locations:<br />
#* cp -a /volatile/halld/offline_monitoring/RunPeriod-YYYY-MM/verVV/REST /work/halld/data_monitoring/RunPeriod-YYYY-MM/REST/verVV<br />
#* cp -a /volatile/halld/offline_monitoring/RunPeriod-YYYY-MM/verVV/REST /cache/halld/RunPeriod-YYYY-MM/REST/verVV [under testing]<br />
<br />
Check log files in $HOME/halld/monitoring/process/log for more information on how each run went. If there are problems, check log files, and modify check_monitoring_data.csh to vary the verbosity of the output.<br />
<br />
==Data Versions==<br />
<br />
To document the conditions of the monitoring data that is created, for the sake of reproducability and further analysis we save several pieces of information. The format is intended to be comprehensive enough to document not just monitoring data, but versions of raw and reconstructed data, so that this database table can be used for the event database as well.<br />
<br />
We store one record per pass through one run period, with the following structure:<br />
<br />
{| class="wikitable"<br />
! Field !! Description<br />
|-<br />
| data_type || The level of data we are processing. For the purposes of monitoring, "rawdata" is the online monitoring, "recon" is the offline monitoring<br />
|- <br />
| run_period || The run period of the data<br />
|- <br />
| revision || An integer specifying which pass through the run period this data corresponds to<br />
|- <br />
| software_version || The name of the XML file that specifies the different software versions used<br />
|-<br />
| jana_config || The name of the text file that specifies which JANA options were passed to the reconstruction program<br />
|-<br />
| ccdb_context || The value of JANA_CALIB_CONTEXT, which specifies the version of calibration constants that were used<br />
|-<br />
| production_time || The data at which monitoring/reconstruction began<br />
|-<br />
| dataVersionString || A convenient string for identifying this version of the data<br />
|}<br />
<br />
<br />
An example file used as as input to ./register_new_version.py is:<br />
<syntaxhighlight><br />
data_type = recon<br />
run_period = RunPeriod-2014-10<br />
revision = 1<br />
software_version = soft_comm_2014_11_06.xml<br />
jana_config = jana_rawdata_comm_2014_11_06.conf<br />
ccdb_context = calibtime=2014-11-10<br />
production_time = 2014-11-10<br />
dataVersionString = recon_RunPeriod-2014-10_20141110_ver01<br />
</syntaxhighlight></div>Kmoriyahttps://halldweb.jlab.org/wiki/index.php?title=Data_Monitoring_Procedures&diff=70871Data Monitoring Procedures2015-10-23T23:28:30Z<p>Kmoriya: /* Checking the Status and Resubmitting */</p>
<hr />
<div>__TOC__<br />
<br />
== Master List of File / Database / Webpage Locations ==<br />
=== Run Conditions ===<br />
* Online Run-by-run condition files (B-field, current, etc.): /work/halld/online_monitoring/conditions/<br />
* Offline monitoring run conditions (software versions, jana config): /group/halld/data_monitoring/run_conditions/<br />
*[http://www.jlab.org/Hall-D/test/RunInfo/ Run Info vers. 1]<br />
*[https://halldweb.jlab.org/cgi-bin/data_monitoring/run_conditions.pl Run Info vers. 2]<br />
<br />
=== Monitoring Output Files ===<br />
* Run Periods 201Y-MM is for example 2015-03, launch ver verVV is for example ver15<br />
* Online monitoring histograms: /work/halld/online_monitoring/root/<br />
* Offline monitoring histogram ROOT files: /work/halld/data_monitoring/RunPeriod-201Y-MM/verVV/rootfiles<br />
* REST files (most recent launch only): /work/halld/data_monitoring/RunPeriod-201Y-MM/REST/verVV<br />
* individual files for each job (ROOT, log, etc.): /volatile/halld/offline_monitoring/RunPeriod-201Y-MM/verVV/<br />
<br />
=== Monitoring Database ===<br />
* Accessing monitoring database (on ifarm): mysql -u datmon -h hallddb.jlab.org data_monitoring<br />
<br />
=== Monitoring Webpages ===<br />
*[https://halldweb.jlab.org/cgi-bin/data_monitoring/monitoring/plotBrowser.py Plot Browser]<br />
*[https://halldweb.jlab.org/cgi-bin/data_monitoring/monitoring/runBrowser.py Run Browser]<br />
*[https://halldweb.jlab.org/cgi-bin/data_monitoring/monitoring/timeSeries.py Time Series]<br />
*[https://halldweb.jlab.org/data_monitoring/launch_analysis/ Launch Analysis]<br />
<br />
== Job Monitoring Links ==<br />
* [http://scicomp.jlab.org/farm2/job.html Completed Job History]<br />
* [http://scicomp.jlab.org/farm2/project.html Job Stats By Project]<br />
* [http://scicomp.jlab.org/farm2/trackOrg.html Job Stats By Track]<br />
* [http://scicomp.jlab.org/farm2/report.html Cluster Report]<br />
* [http://scicomp.jlab.org/farm2/walltime.html Walltime Distribution]<br />
<br />
== Saving Online Monitoring Data ==<br />
<br />
The procedure for writing the data out is given in, e.g.,<br />
[https://halldweb.jlab.org/wiki/index.php/Raid-to-Silo_Transfer_Strategy Raid-to-Silo Transfer Strategy].<br />
<br />
Once the DAQ writes out the data to the raid disk, cron jobs will copy the file to tape,<br />
and within ~20 min., we will have access to the file on tape at<br />
/mss/halld/$RUN_PERIOD/rawdata/RunXXXXXX.<br />
<br />
All online monitoring plugins will be run as data is taken.<br />
They will be accessible within the counting house via RootSpy, and<br />
for each run and file, a ROOT file containing the histograms will be saved<br />
within a subdirectory for each run.<br />
<br />
For immediate access to these files, the raid disk files may be accessed directly<br />
from the counting house, or the tape files will be available within ~20 min. of the<br />
file being written out.<br />
<br />
== Offline Monitoring: Running Over Archived Data ==<br />
<br />
Once files are written to tape we run the online plugins on these files to confirm what we were seeing in the online monitoring, and also to update the results from the latest calibration and software. <br />
Manual scripts and cron jobs are set up to look for new data and run the plugins over a sample of files.<br />
<br />
Every other Friday (usually the Friday before the offline meetings) jobs will be started to run the newest software on all previous runs,and allows everybody to see improvements in each detector.<br />
For each launch, independent builds of hdds, sim-recon, the monitoring plugins, and an sqlite file will be generated.<br />
<br />
Below the procedures are described for<br />
# Preparing the software for the launch<br />
# Starting the launch (using hdswif)<br />
# Post-analysis of statistics of the launch<br />
<br />
Processing the results and making them available to the collaboration<br />
is handled in the section [[#Post-Processing_Procedures | Post-Processing Procedures]] below.<br />
<br />
<!--<br />
==== Generating an offline plugin job ====<br />
<br />
The user gxproj1 should be used for official offline monitoring jobs.<br />
Within /home/gxproj1/halld/monitoring/batch/ there will be scripts to run the online monitoring plugins over tape files.<br />
The main script is generatejobs_plugins_rawdata.sh, which can be used as<br />
generatejobs_plugins_rawdata.sh [minrun] [maxrun] (minfile) (maxfile)<br />
where minrun, maxrun specify the range of the run #, and minfile, maxfile (optional) specify<br />
the file range for that run.<br />
<br />
This will generate a script run_rawdata_XXXXXX.sh, where the run # has now been formatted to be 6 digits.<br />
Executing this script will send the monitoring plugins job to the Auger batch system.<br />
<br />
There is also a script clean.sh which can be used as<br />
clean.sh XXX<br />
This will clean up all associated files created in association with run XXX.<br />
<br />
Internally, the xml file used to submit the job will be created, and the job to run<br />
will be given within script.sh. All run parameters should be specified in at the beginning<br />
of generatejobs_plugins_rawdata.sh<br />
Since we are running on tape, the tape file will first be copied over to the cache disk, and the job will run<br />
over this cached file.<br />
<br />
==== Using cron to run automatically ====<br />
Within /home/gluex/halld/monitoring/cron/ there is a file cron_plugins<br />
that can be executed via<br />
crontab cron_plugins<br />
This will set up a cron job to call the script scan_for_jobs.sh, which will<br />
check in the rawdata directory and call generatejobs_plugins_rawdata.sh for<br />
any run that is more than 5 min old. The cron job is set up to run every 10 min.<br />
<br />
==== Magnetic Field Settings ====<br />
For tracking, it is necessary to set the correct magnetic field settings.<br />
The below table is now obsolete. Please refer to the [https://halldweb.jlab.org/cgi-bin/data_monitoring/run_conditions.pl GlueX Run Conditions] webpage.<br />
<br />
The actual field settings for each run have not been documented well.<br />
Below are values based on going through entries in the [https://logbooks.jlab.org/book/halld halld log].<br />
<br />
{| border="1" cellpadding="1" style="text-align: center;"<br />
!width="100"| Run #s<br />
!width="150"| Solenoid Current (A)<br />
!width="300"| JANA option<br />
!width="300"| notes<br />
|-<br />
| 940 - 996 || 1000 || -PBFIELD_MAP=Magnets/Solenoid/solenoid_1000A_poisson_20141104 || At run 997, the solenoid started ramping down<br />
|-<br />
| 998 - 1448 || 0 || -PBFIELD_TYPE=NoField -PDEFTAG:DTrackCandidate=StraightLine || Solenoid started ramping up around run 1431. Runs 1432 - 1448 should not be used for tracking. See [https://logbooks.jlab.org/entry/3309039 this] and [https://logbooks.jlab.org/entry/3309092 this] entry.<br />
|-<br />
| 1449 - 1620 || 1200 || -PBFIELD_MAP=Magnets/Solenoid/solenoid_1200A_poisson_20140520 || <br />
|-<br />
|}<br />
<br />
'''Note:''' The following run ranges were taken with a 300A field, and could be analyzed with the magnetic field map Magnets/Solenoid/solenoid_300A_poisson_20140819 , more study is needed to see if the straight line track fitter is needed or not: 1036 - 1053, 1065 - 1121, 1212? - 1254, 1309 - 1318 [1307,1308,1319-1329 were taken while the magnet was ramping].<br />
--><br />
<br />
=== General Information on Procedures ===<br />
This section explains how the offline monitoring should be run.<br />
Since we may want to simultaneously run offline monitoring for different run periods that require<br />
different environment variables, the scripts are set up so that a generic user can download the<br />
scripts and run them from anywhere. Most output directories for offline monitoring are created<br />
with group read/write permissions so that any Hall D group user has access to the contents,<br />
but there are some cases where use of the account that created the launch is necessary.<br />
<br />
The accounts used for offline monitoring are the gxprojN accounts created and maintained by<br />
Mark Ito (see [https://halldweb.jlab.org/wiki/index.php/GlueX-related_shared_accounts_on_the_JLab_CUE here] for how each account is used).<br />
As of October 2015, the following are used:<br />
* gxproj1 for running over Fall 2014 data (deprecated since June 2015)<br />
* gxproj5 for running over Spring 2015 data<br />
<br />
Since the summer of 2015 we have transitioned from a system using Mark Ito's jproj scripts<br />
to integrating the swif system that Chris Larrieu (SciComp) has been developing. For offline monitoring,<br />
the hdswif system that Kei developed is used for launching the jobs, and the jproj system is used<br />
for meta-analysis of launch statistics.<br />
<br />
Both hdswif and jproj are maintained in svn:<br />
* hdswif: https://halldsvn.jlab.org/repos/trunk/scripts/monitoring/hdswif<br />
* jproj : https://halldsvn.jlab.org/repos/trunk/scripts/monitoring/jproj<br />
<br />
To run the offline monitoring each package should be checked out and all necessary<br />
scripts are included.<br />
<br />
=== Preparing the software for the launch ===<br />
<br />
To begin a new launch the software must be built to the latest versions.<br />
For the gxprojN user accounts used, all software builds are contained in the directory ~/builds<br />
(which are soft links to /work/halld/home/gxprojN/builds). When logging into these accounts<br />
the setup files ~/setup_jlab-2015-03.csh or similar files should be sourced.<br />
<br />
<b>Note that Mark Ito does <u>not</u> want you to change the contents of each .cshrc file.</b><br />
You should consult him if you feel the need.<br />
<br />
<span style="color:red">NOTE: FOR BUILDING SOFTWARE IT IS A GOOD IDEA TO DO A COMPLETE WIPEOUT/CHECKOUT EACH TIME TO AVOID STALE HEADER FILES.</span><br />
<br />
# Building hdds: Go to ~/builds/hdds. The directory hdds is the one from git. Delete the contents, then download the newest version from git and build: <pre>cd ~/builds/hdds/</pre> <pre>rm -frv hdds</pre> <pre>git clone https://github.com/JeffersonLab/hdds</pre> <pre>cd hdds</pre> <pre>scons install</pre><br />
# Building sim-recon: Go to ~/builds/sim-recon. The directory sim-recon is the one from git. Delete the contents, then download the newest version from git and build: <pre>cd ~/builds/sim-recon</pre> <pre>rm -frv sim-recon</pre> <pre>git clone https://github.com/JeffersonLab/sim-recon</pre> <pre>cd sim-recon/src</pre> <pre>scons install -j8</pre><br />
# Prepare the latest sqlite file: The sqlite is set in the ~/setup_jlab-2015-03.csh script as <b>sqlite:////home/gxproj5/ccdb.sqlite</b> through the environment variables <b>JANA_CALIB_URL</b> and <b>CCDB_CONNECTION</b>. Therefore, go to this directory and create a new sqlite file. We create the sqlite file in a temporary directory since creating the sqlite file in a directory where the output file exists causes errors. Original documentation on creating sqlite files are [https://halldweb.jlab.org/wiki/index.php/SQLite-form_of_the_CCDB_database here]. <br><span style="color:red">NOTE: SQLITE FILES DO NOT WORK ON THE NEW /work DISK INSTALLED IN OCTOBER 2015 </span> <pre>cd ~/tmp</pre> <pre>$CCDB_HOME/scripts/mysql2sqlite/mysql2sqlite.sh -hhallddb.jlab.org -uccdb_user ccdb | sqlite3 ccdb.sqlite</pre> <pre>mv ccdb.sqlite ../</pre><br />
# Note that the above steps must be done <b>BEFORE</b> launch project creation. This is because we will track the revisions of the libraries used, and this is done by extracting the svn information in each directory. Also, note that the system assumes that we have the topmost build directory (usually called GLUEX_TOP) to be $HOME/builds . Such an assumption is necessary to be able to extract information about the library locations automatically.<br />
<br />
Create the appropriate project(s) and submit the jobs using hdswif, as detailed in the section below.<br />
<br />
=== Starting the Launch and Submitting Jobs ===<br />
<br />
Until the summer of 2015 we relied solely on Mark Ito's jproj system for submitting and keeping track of jobs. We have since moved to the swif system and use the hdswif wrapper for this. Below are instructions for how to use these.<br />
<br />
# Downloading hdswif: Download the hdswif directory from svn. For the gxprojN accounts, use the directory ~/halld/hdswif. <pre>cd ~/halld </pre> <pre> svn co https://halldsvn.jlab.org/repos/trunk/scripts/monitoring/hdswif</pre> <pre>cd hdswif</pre><br />
# Creating the workflow: Within SWIF jobs are registered into workflows. First create the workflow. For offline monitoring, the workflow names are of the form <b>offline_monitoring_RunPeriod201Y_MM_verVV_hd_rawdata</b> with suitable replacements for the run period and version number. The command "swif list" will list all existing workflows. Also, for most simple SWIF commands hdswif also provides a wrapper. <pre>swif list</pre> For creation of workflows for offline monitoring the command <pre>hdswif.py create [workflow] -c [config file] </pre> should be used. As an example config file, see the input.config file in the folder (and update it). When a config file is passed in, hdswif will automatically create files that record the configuration of the current launch. These files are stored as for example<br />
#* /group/halld/data_monitoring/run_conditions/jana_rawdata_comm_2015_03_ver15.conf<br />
#* /group/halld/data_monitoring/run_conditions/soft_comm_2015_03_ver15.xml<br />
#* The software packages stored in git (sim-recon and hdds) can have git tags applied to them, which makes it easier to find versions of the software than a SHA-1 hash. hdswif will ask if you would like to create a tag, and execute the following sequence: <pre>git tag -a offmon-201Y_MM-verVV -m "Used for offline monitoring 201Y-MM verVV started on 201y/mm/dd"</pre> <pre>git push offmon-201Y_MM-verVV</pre> This will only be invoked when the user name is gxprojN, and for the configuration files, the output directory will be /group/halld/data_monitoring/run_conditions/ for gxprojN accounts while it will be the current directory for other users.<br />
#* To use the git-tagged software versions do for example <pre>cd $HALLD_HOME</pre> <pre>git checkout offmon-2015_03-ver15</pre><br />
# Registering jobs in the workflow: To register jobs within the workflow, hdswif provides the use of config files. Jobs can be registered by specifying the workflow, config file (-c), run (-r) and file (-f) numbers if necessary. A typical config file will look this:<br />
PROJECT gluex<br />
TRACK reconstruction<br />
OS centos65<br />
NCORES 6<br />
DISK 40<br />
RAM 8<br />
TIMELIMIT 8<br />
JOBNAMEBASE offmon_<br />
RUNPERIOD 2015-03<br />
VERSION 15<br />
OUTPUT_TOPDIR /volatile/halld/offline_monitoring/RunPeriod-[RUNPERIOD]/ver[VERSION] # Example of other variables included in variable<br />
SCRIPTFILE /home/gxproj5/halld/hdswif/script.sh # Must specify full path<br />
ENVFILE /home/gxproj5/halld/hdswif/setup_jlab-2015-03.csh # Must specify full path<br />
The config file contains configuration parameters for each of the jobs.<br><br />
<span style="color:red">Note: Job configuration parameters can be set differently for jobs within the same workflow if necessary.</span><br />
Edit the config file and save as a new file if necessary. Once the configuration is set, jobs can be added via <pre>hdswif.py add [workflow] -c input.config</pre><br />
By default, hdswif will add all files found within the directory /mss/halld/RunPeriod-201Y-MM/rawdata/ where 201Y-MM is specified by the RUNPERIOD parameter in the config file. If only some of the runs or files are needed, these can be specified for example with<br />
<pre>hdswif.py add [workflow] -c input.config -r 3180 -f '00[0-4]'</pre><br />
to specify to register running only over run 3180 files 000 - 004 (Unix-style brackets and wildcards can be used).<br />
<br />
Running the workflow: To run the workflow, simply use swif run: <pre>swif run -workflow [workflow]</pre> or equivalently, using the hdswif wrapper, <pre>hdsswif.py run [workflow]</pre><br><br />
<b>It is recommended that some jobs be tested to make sure that everything is working rather than fail thousands of jobs.</b><br> For this purpose, hdswif will take an additional parameter to run which limits the number of jobs to submit: <pre>hdswif.py run [workflow] 10</pre><br />
in which case only 10 jobs will be submitted. To submit all jobs after checking the results, do <pre>hdswif.py run [workflow]</pre><br />
<br />
=== Checking the Status and Resubmitting ===<br />
# The status of jobs can be checked on the terminal with <pre>jobstat -u gxprojN</pre> on Auger and for SWIF with <pre>swif list</pre> or for more information, <pre>swif status [workflow] -summary</pre> Also see the Auger [http://scicomp.jlab.org/scicomp/#/auger/jobs job website].<br />
# For failed jobs, SWIF can resubmit jobs based on the problem. For resubmission for failed jobs with the same resources, <pre>swif retry-jobs [workflow] -problems [problem name]</pre> can be used, and for jobs to be submitted with more resources, e.g., use <pre>swif modify-jobs -ram add 2gb -problems AUGER-OVER_RLIMIT</pre> hdswif has a wrapper for both of these: <pre>hdswif.py resubmit [workflow] [problem]</pre> In this case [problem] can be one of <b>SYSTEM, TIMEOUT, RLIMIT</b>. If SYSTEM is specified, the jobs will be retried. For TIMEOUT and RLIMIT, the jobs will be modified by default with 2 additional hours or GB of RAM. If one more number is added as an option, then that many hours or GB of RAM will be added., e.g., <pre>hdswif.py resubmit [workflow] TIMEOUT 5</pre> will add 5 hours of processing time. You can wait until almost all jobs finish before resubmitting failed jobs since the number should be relatively small. <b>Even if jobs are resubmitted for one type of failure, jobs that later fail with that failure will not be automatically resubmitted.</b><br />
# For information on swif, use the "swif help" commands and for hdswif see the attached documentaion in https://halldsvn.jlab.org/repos/trunk/scripts/monitoring/hdswif/manual_hdswif.pdf<br />
<br />
=== Post-analysis of statistics of the launch ===<br />
<br />
# After jobs have been submitted, it will usually take a few days for all of the jobs to be processed.<br />
#* Status of Auger: http://scicomp.jlab.org/scicomp/#/auger/jobs (see also links above)<br />
#* Status of user jobs: <pre>jobstat -u [user name]</pre><br />
# The status and results of jobs are saved within the SWIF internal server, and are available via the command <pre>swif status [workflow] -summary -runs</pre> where the arguments -summary and -runs show summary statistics and statistics for individual jobs, respectively. hdswif has a command that takes this output in XML output and creates an HTML webpage showing results of the launch. To do this, do <pre>hdswif.py summary [workflow]</pre> This will create an XML file swif_output_[workflow].xml that contains all information from SWIF. If the file already exists, hdswif will ask whether to overwrite the existing file.<br />
# At this stage the html output and figure files are created and ready to be put online. For this step and other steps involving analysis of the statistics of the launch results, it is convenient to change to the jproj system.<br />
# The jproj scripts for offline monitoring are maintained in the svn directory https://halldsvn.jlab.org/repos/trunk/scripts/monitoring/jproj Do <pre> svn co https://halldsvn.jlab.org/repos/trunk/scripts/monitoring/jproj </pre> For the gxprojN accounts used for offline monitoring the directory should be ~/halld/jproj<br />
# The jproj directory contains two subdirectories, scripts and projects. The scripts directory contains useful scripts for processing the jobs registered in the jproj system, and each of the offline monitoring launches will be handled in the projects directory.<br />
# Go to the projects directory<pre>cd ~/halld/jproj/projects</pre> and use the script create_project.sh to create a new directory that contains the processing scripts for the current launch <pre>./create_project.sh [workflow]</pre> This should create a directory such as offline_monitoring_RunPeriod2015_03_ver15_hd_rawdata (same as the workflow name). The script uses the template files in the directory templates and by substitution creates script files for the current launch. Now go to the newly created analysis directory: <pre>cd [workflow]/analysis</pre><br />
which for the gxprojN accounts should have the full path /home/gxprojN/halld/jproj/projects/[workflow]/analysis.<br><br><br />
<span style="color:red">All of the analysis commands including arguments are contained in run_analysis.sh, but it is <b>strongly</b> recommended that all commands are run manually to check for errors.</span><br><br><br />
# The first thing to do is to make the html output from hdswif public. Copy the html file and related figures that were created from hdswif to the appropriate space within /group/halld/www/halldweb/html/data_monitoring/ : <pre>python publish_offmon_results.py [run period] [version]</pre> Note that the command with appropriate substitutions for arguments can be found within run_analysis.sh (same for all commands below).<br />
# Next, we need to create a few MySQL tables for the current launch. The MySQL tables are useful for comparing run/file combinations for different launches. For the SWIF launches, there are two tables needed, and will be named<br />
#* [workflow]Job<br />
#* [workflow]_aux<br />
# This naming scheme of tables and their roles are the same as from the jproj only launches. The [workflow]Job table will contain information gathered from SWIF about each job (which node it went to, start time of each stage, memory usage, etc.). The [workflow]_aux table will contain information gathered from the output stdout files from each job. First, create the Job table using <pre>python create_jproj_job_table.py [run period] [version]</pre> Note that this script uses the XML output from hdswif summary and inserts the contents into the MySQL table, so the XML output file must exist.<br />
# To check the contents of this MySQL table, do <pre>mysql -hhalldb -ufarmer farming</pre> <pre>mysql> describe offline_monitoring_RunPeriod2015_03_ver11_hd_rawdataJob;</pre><br />
<br />
== Hall D Job Management System ==<br />
<br />
This section details instructions on how to create and launch a set of jobs using the Hall-D Job Management System developed by Mark Ito. These instructions are generic: this system can be used for the weekly monitoring jobs, but can also be used for other sets of job launches as well. <br />
<br />
=== Database Table Overview ===<br />
<br />
* Job management database table (<project_name>): For each input file, keeps track of whether or not a job for it has been submitted, along with other optional fields. <br />
<br />
* Job status database table (<project_name>Job (no space)): For each job, keeps track of the job-id, the job status, memory used, cpu & wall time, time taken to complete various stages (e.g. pending, dependency, active), and others. <br />
<br />
* Job metrics database table (<project_name>_aux (no space)): For each job, keeps track of the job-id, how many events were processed, the time it took to copy the cache file, and the time it took to run the plugin. This information is culled from the log files of each job, and is done within the analysis directory of each launch.<br />
<br />
=== Initialize Project Management ===<br />
<br />
* Log into the ifarm machine with one of the gxproj accounts. For this example we will use gxproj1.<br />
<pre><br />
ssh gxproj1@ifarm -Y<br />
</pre><br />
<br />
* Go to a directory to do the launch. In principle, any directory will work, but for gxproj1 this is usually done in /home/gxproj1/halld/<br />
<br />
* Check out the necessary scripts <pre>svn co https://halldsvn.jlab.org/repos/trunk/scripts/monitoring/jproj</pre> This will get all necessary scripts for launching. Once checked out, <pre>cd projects</pre><br />
<br />
* The script create_project.sh can be used to create a new project. It will take a single argument, the project name. It is assumed that the project name is of the form offline_monitoring_RunPeriod20YY_MM_verVV_hd_rawdata where this string will be parsed to give the run period 20YY_MM and version number VV. One thing to do BEFORE creation of a new project is to editthe conditions of the launch (plugins to run over, memory requested, disk space requested) within templates/template.jsub . This<br />
information is saved automatically at project creation time into files in /group/halld/data_monitoring/run_conditions .<br />
To create a project do <pre>./create_project.sh offline_monitoring_RunPeriod20YY_MM_verVV_hd_rawdata</pre><br />
The name has been chosen to be as consistent as possible with other directory structures.<br />
However, mysql requires that "-" be escaped in table names, so unfortunately run periods will be<br />
given as 2YYY_MM instead of 2YYY-MM.<br />
<br />
* A new directory with the project name (offline_monitoring_RunPeriod20YY_MM_verVV_hd_rawdata) and files will be copied and modified from the template directory to reflect the run period, the user, the directory that it was created in, project name, etc.<br />
<br />
* For each project, cd into the new directory<br />
* The script clear.sh will remove any existing tables for the current project name, then recreate it. Do<br />
<pre>./clear.sh</pre><br />
* To use the jproj.pl script that was checked in, add the directory to your path with<br />
<pre>source ../../scripts/setup.csh</pre><br />
or always specify the full path<br />
<pre>../../scripts/jproj.pl</pre><br />
* Now update the table of runs with<br />
<pre>jproj.pl <project name> update</pre><br />
This will fill the table with all files within /mss that are of the same form as what is in <project name>.jproj . If you want to register only on a subset of all such files, you can edit this file directly.<br />
* Once you have registered all of the files you would like to run over, do<br />
<pre>jproj.pl <project name> submit [max # of jobs] [run number]</pre><br />
where the additional options specify how many jobs to submit and which run number to run on. Without these options all files that are registered and have not been submitted yet will be submitted.<br />
<br />
At this stage you are ready to submit all files. It is a good idea to submit a few test jobs<br />
at first to check that all scripts are working and that the plugins do not crash. Once you are<br />
sure that this does not happen, you can send all jobs in. The remaining jobs are then the monitoring<br />
which will (among other things) put the results on the online webpage for the collaboration to view, and<br />
the analysis of the launch.<br />
<br />
=== Project File Overview ===<br />
<br />
An overview of each project file:<br />
* '''clear.sh:''' For the current project, deletes the job status and management database tables (if any), and creates new, empty ones. <br />
* '''<project_name>.jproj:''' Contains the path and file name format for the input files for the jobs. <br />
* '''<project_name>.jsub:''' The xml job submission script. The run number and file number variables are set during job submission for each input file. <br />
* '''script.sh:''' The script that is executed during the job. If output job directories are not pre-created manually, they should be created in this script with the proper permissions:<br />
<pre><br />
mkdir -p -m 775 my_directory<br />
</pre><br />
* '''setup_jlab-[run period].csh:''' The environment that is sourced at the beginning of the job execution. <br />
* '''status.sh:''' Updates the job status database table, and prints some of its columns to screen.<br />
<br />
=== Project Management ===<br />
<br />
* Delete (if any) and create the database table(s) for the current set of job submissions:<br />
<pre><br />
./clear.sh<br />
</pre><br />
Also, if testing was done with jobs, it is best to delete the output directory and the configuration files:<br />
<pre><br />
rm -frv /volatile/halld/offline_monitoring/RunPeriod-20YY-MM/verVV /group/halld/data_monitoring/run_conditions/soft_comm_20YY_MM_verVV.xml /group/halld/data_monitoring/run_conditions/jana_rawdata_comm_20YY_MM_verVV.conf<br />
</pre><br />
<br />
* Search for input files matching the string in the .jproj file, and create a row for each in the job management database table (called <project_name>). You can test by adding an optional argument at the end, which only selects files with a specific file number:<br />
<pre><br />
jproj.pl <project_name> update <optional_file_number><br />
</pre><br />
<br />
* Confirm that the job management database is accurate by printing it's contents to screen:<br />
<pre><br />
mysql -hhallddb -ufarmer farming -e "select * from <project_name>"<br />
</pre><br />
<br />
* ONLY if a mistake was made, to delete the tables from the database and recreate new, empty ones, run: <br />
<pre><br />
./clear.sh<br />
</pre><br />
<br />
* To look at the status of the submitted jobs, first query auger and update the job status database:<br />
<pre><br />
fill_in_job_details.pl <project_name><br />
</pre><br />
<br />
* The job status can then be viewed by submitting a query to the job status database (called <project_name>Job (no space in between)):<br />
<pre><br />
mysql -hhallddb -ufarmer farming -e "select id,run,file,jobId,hostname,status,timeSubmitted,timeActive,walltime,cput,timeComplete,result,error from <project_name>Job"<br />
</pre><br />
<br />
* These last two commands can instead be executed simultaneously by running:<br />
<pre><br />
./status.sh<br />
</pre><br />
<br />
=== Handy mysql Instructions ===<br />
<br />
* Handy mysql instructions:<br />
<pre><br />
mysql -hhallddb -ufarmer farming # Enter the "farming" mysql database on "hallddb" as user "farmer"<br />
quit; # Exit mysql<br />
show tables; # Show a list of the tables in the current database<br />
show columns from <project_name>; # show all of the columns for the given table<br />
select * from <project_name>; # show the contents of all rows from the given table<br />
</pre><br />
<br />
=== Backing Up Offline Monitoring Tables ===<br />
Tables created for offline monitoring can be backed up using the script backup_tables.sh which<br />
can be checked out with the other files from https://halldsvn.jlab.org/repos/trunk/scripts/monitoring/jproj/projects<br />
<br />
The script uses the command mysqldump to print out a file that can be executed to recreate the tables.<br />
Since executing this output file will drop the table when it exists, caution is advised.<br />
Example usage to backup all three tables created for run period 2014_10 ver 17:<br />
<pre>backup_tables.sh 2014_10 17</pre><br />
<br />
== Running Over Data As It Comes In ==<br />
<br />
A special user gxproj1 will have a cron job set up to run the plugins as new data appears on /mss.<br />
During the week, gxproj1 will submit offline plugin jobs with the same setup as the weekly jobs<br />
run the previous Friday. The procedure for this is shown below.<br />
<br />
<!--<br />
=== Setting up the environment ===<br />
The file<br />
/home/gxproj1/setup_jlab.csh<br />
is sourced through .tcshrc.<br />
This file is the same as what is linked to by<br />
/home/gluex/setup_jlab_commissioning.csh,<br />
except HALLD_HOME, HDDS_HOME, and JANA_CALIB_URL are set separately so that this<br />
user can have a separate build.<br />
<br />
To obtain the builds from the previous Friday's runs,<br />
execute<br />
/home/gxproj1/halld/monitoring/newruns/setup_previous.sh [year] [month] [day]<br />
The build revisions from the previous Friday are archived in files<br />
/work/halld/data_monitoring/run_conditions/soft_comm_[year]_[month]_[day].xml<br />
and the script will build libraries based on those stored revision numbers.<br />
--><br />
<br />
=== Running the cron job ===<br />
<br />
'''IMPORTANT:''' The cron job should not be running while you are manually submitting jobs using the jproj.pl script for the same project, or else you will probably multiply-submit a job. <br />
<br />
* Go to the cron job directory: <br />
<pre><br />
cd /u/home/gxproj1/halld/monitoring/newruns<br />
</pre><br />
<br />
* The cron_plugins file is the cronjob that will be executed. During execution, it runs the exec.sh command in the same folder. This command takes two arguments: the project name, and the maximum file number for each run. These fields should be updated in the cron_plugins file before running. <br />
<br />
* The exec.sh command updates the job management database table with any data that has arrived on tape since it was last updated, ignoring file numbers greater than the maximum file number. It then submits jobs for these files. <br />
<br />
* To start the cron job, run:<br />
<pre><br />
crontab cron_plugins<br />
</pre><br />
<br />
* To check whether the cron job is running, do<br />
<pre><br />
crontab -l<br />
</pre><br />
<br />
* To remove the cron job do<br />
<pre><br />
crontab -r<br />
</pre><br />
<br />
<!--<br />
The cron job will run the script scan_for_jobs.sh,<br />
which runs generatejobs_plugins_rawdata.sh for any<br />
new runs that it had not seen before. All previous<br />
runs are recorded in the file filelists/files_current.txt<br />
so clear this to run over runs, or set the parameters<br />
MINRUN and MAXRUN which will set the range of runs submitted.<br />
--><br />
<br />
==Post-Processing Procedures==<br />
<br />
To visualize the monitoring data, we save images of selected histograms and store time series of selected quantities in a database, which are then displayed on a web page. This section describes how to generate the monitoring images and database information.<br />
<br />
The scripts used to generate this summary data are primarily run from /home/gxprojN/halld/monitoring/process . If you want a new copy of the scripts, e.g., for a new monitoring run, you should check the scripts out from SVN:<br />
<syntaxhighlight><br />
svn co https://halldsvn.jlab.org/repos/trunk/scripts/monitoring/process<br />
</syntaxhighlight><br />
Note that these scripts currently have some parameters which must be periodically set by hand.<br />
<br />
The default python version on most JLab machine does not have the modules to allow these scripts to connect to the MySQL database. To run these scripts, load the environment with the following command<br />
<syntaxhighlight><br />
source /home/gxproj1/halld/monitoring/process/monitoring_env.csh<br />
</syntaxhighlight><br />
<br />
===Online Monitoring===<br />
<br />
There are two primary scripts for running over the monitoring data generated by the online system and offline reconstruction. The online script can be run with either of the following commands:<br />
<syntaxhighlight><br />
/home/gluex/halld/monitoring/process/check_new_runs.py<br />
<br />
OR <br />
<br />
/home/gluex/halld/monitoring/process/check_new_runs.csh<br />
</syntaxhighlight><br />
The shell script sets up the environment properly to run the python script. To connect to the monitoring database on the JLab CUE, modules included in the local installation of python >= 2.7 are needed. The shell script is appropriate to use in a cron job. The cronjob is currently run under the "gluex" account.<br />
<br />
The online monitoring system copies a ROOT file containing the results of the online monitoring, and other configuration files into a directory accessible outside the counting house. This python script automatically checks for new ROOT files, which it will then automatically process. It contains several configuration variables that must be correctly set, which contains the location of input/output directories, etc...<br />
<br />
===Offline Monitoring===<br />
<br />
After the data is run over, the results should be processed, so that summary data is entered into the monitoring database and plots are made for the monitoring webpages. Currently, this processing is controlled by a cronjob that runs the following script:<br />
<syntaxhighlight><br />
/home/gxproj1/halld/monitoring/process/check_monitoring_data.csh <br />
</syntaxhighlight><br />
This script checks for new ROOT files, and only runs over those it hasn't processed yet. Since one monitoring ROOT file is produced for each EVIO file, whenever a new file is produced, the plots for the corresponding run are recreated and all the ROOT files for a run are combined into one file. Information is stored in the database on a per-file basis. <br />
<br />
Plots for the monitoring web page can be made from single histograms or multiple histograms using RootSpy macros. If you want to change the list of plots made, you must modify one of the following files:<br />
* histograms_to_monitor - specify either the name of the histogram or its the full ROOT path <br />
* macros_to_monitor - specify the full path to the RootSpy macro .C file<br />
<br />
When a new monitoring run is started, or the conditions are changed, the following steps should be taken to process the new files:<br />
# Add a new data version, as described below:<br />
# Change the following parameters in check_monitoring_data.csh:<br />
## JOBDATE should correspond to the ouptut date used by the job submission script<br />
## OUTPUTDIR should correspond to the directory corresponding to the run period and revision corresponding to the new version you just submitted. Presumably, this directory will be empty at the beginning.<br />
## Once you create a new data version as defined below, you should pass the needed information as a command line option. Currently this is done by the ARGS variable. For example, the argument "-v RunPeriod-2014-10,8" tells the monitoring scripts to look up the version corresponding to revision 8 of RunPeriod-2014-10 in the monitoring DB and to use to store the results.<br />
<br />
<syntaxhighlight><br />
Example configuration parameters:<br />
set JOBDATE=2015-01-09<br />
set INPUTDIR=/volatile/halld/RunPeriod-2014-10/offline_monitoring<br />
set OUTPUTDIR=/w/halld-scifs1a/data_monitoring/RunPeriod-2014-10/ver08<br />
set ARGS=" -v RunPeriod-2014-10,8 "<br />
</syntaxhighlight><br />
If you want to process the results manually, the data is processed using the following script:<br />
<syntaxhighlight><br />
./process_new_offline_data.py <input directory> <output directory><br />
<br />
EXAMPLE:<br />
<br />
./process_new_offline_data.py 2014-11-14 /volatile/halld/RunPeriod-2014-10/offline_monitoring/ /w/halld-scifs1a/data_monitoring/RunPeriod-2014-10/ver02<br />
</syntaxhighlight><br />
The python script takes several options to enable/disable various steps in the processing. Of interest is the "--force" option, which will run over all monitoring ROOT files, whether or not they've been previously identified.<br />
<br />
Every time a new reconstruction pass is performed, a new version number must be generated. To do this, prepare a version file as described below. Then run the register_new_version.py script to store the information in the database. The script will return a version number, which then should be set by hand in process_new_offline_data.py - future versions of the script will streamline this part of the procedure. An example of how to generate a new version is:<br />
<syntaxhighlight><br />
./register_new_version.py add /home/gxproj1/halld/monitoring/process/versions/vers_RunPeriod-2014-10_pass1.txt<br />
</syntaxhighlight><br />
<br />
<b>If you are running the offline monitoring by checking out the files in trunk/scripts/monitoring/jproj/projects/<br />
of the svn repository</b>, and created a project with<br />
create_project.sh [project name] hd_rawdata<br />
Then go to the directory [project name]/processing/<br />
and execute<br />
./run_processing.sh<br />
which will run register_new_version.py as well as check_monitoring_data.csh for that project.<br />
<br />
===Step-by-Step Instructions For Processing a New Monitoring Run===<br />
<br />
The monitoring runs are current run out of the gxproj1 and gxproj5 accounts. After an offline monitoring run has been successfully started on the batch farm, the following steps should be followed to setup the post-processing for these runs.<br />
<br />
# The post-processing scripts are stored in $HOME/halld/monitoring/process and are automatically run by cron.<br />
# Run "svn update" to bring any changes in. Be sure that the list of histograms and macros to plot are current.<br />
# Edit check_monitoring_data.csh to point to the current revisions/directories<br />
#* VERSION<br />
#* ARGS<br />
#* Note that the environment depends on a standard script - $HOME/setup_jlab.csh<br />
# Update files in the web directory, so that the results are displayed on the web pages: /group/halld/www/halldweb/html/data_monitoring/textdata<br />
# Copy the REST files to more permanent locations:<br />
#* cp -a /volatile/halld/offline_monitoring/RunPeriod-YYYY-MM/verVV/REST /work/halld/data_monitoring/RunPeriod-YYYY-MM/REST/verVV<br />
#* cp -a /volatile/halld/offline_monitoring/RunPeriod-YYYY-MM/verVV/REST /cache/halld/RunPeriod-YYYY-MM/REST/verVV [under testing]<br />
<br />
Check log files in $HOME/halld/monitoring/process/log for more information on how each run went. If there are problems, check log files, and modify check_monitoring_data.csh to vary the verbosity of the output.<br />
<br />
==Data Versions==<br />
<br />
To document the conditions of the monitoring data that is created, for the sake of reproducability and further analysis we save several pieces of information. The format is intended to be comprehensive enough to document not just monitoring data, but versions of raw and reconstructed data, so that this database table can be used for the event database as well.<br />
<br />
We store one record per pass through one run period, with the following structure:<br />
<br />
{| class="wikitable"<br />
! Field !! Description<br />
|-<br />
| data_type || The level of data we are processing. For the purposes of monitoring, "rawdata" is the online monitoring, "recon" is the offline monitoring<br />
|- <br />
| run_period || The run period of the data<br />
|- <br />
| revision || An integer specifying which pass through the run period this data corresponds to<br />
|- <br />
| software_version || The name of the XML file that specifies the different software versions used<br />
|-<br />
| jana_config || The name of the text file that specifies which JANA options were passed to the reconstruction program<br />
|-<br />
| ccdb_context || The value of JANA_CALIB_CONTEXT, which specifies the version of calibration constants that were used<br />
|-<br />
| production_time || The data at which monitoring/reconstruction began<br />
|-<br />
| dataVersionString || A convenient string for identifying this version of the data<br />
|}<br />
<br />
<br />
An example file used as as input to ./register_new_version.py is:<br />
<syntaxhighlight><br />
data_type = recon<br />
run_period = RunPeriod-2014-10<br />
revision = 1<br />
software_version = soft_comm_2014_11_06.xml<br />
jana_config = jana_rawdata_comm_2014_11_06.conf<br />
ccdb_context = calibtime=2014-11-10<br />
production_time = 2014-11-10<br />
dataVersionString = recon_RunPeriod-2014-10_20141110_ver01<br />
</syntaxhighlight></div>Kmoriyahttps://halldweb.jlab.org/wiki/index.php?title=Data_Monitoring_Procedures&diff=70870Data Monitoring Procedures2015-10-23T23:23:02Z<p>Kmoriya: /* Starting the Launch and Submitting Jobs */</p>
<hr />
<div>__TOC__<br />
<br />
== Master List of File / Database / Webpage Locations ==<br />
=== Run Conditions ===<br />
* Online Run-by-run condition files (B-field, current, etc.): /work/halld/online_monitoring/conditions/<br />
* Offline monitoring run conditions (software versions, jana config): /group/halld/data_monitoring/run_conditions/<br />
*[http://www.jlab.org/Hall-D/test/RunInfo/ Run Info vers. 1]<br />
*[https://halldweb.jlab.org/cgi-bin/data_monitoring/run_conditions.pl Run Info vers. 2]<br />
<br />
=== Monitoring Output Files ===<br />
* Run Periods 201Y-MM is for example 2015-03, launch ver verVV is for example ver15<br />
* Online monitoring histograms: /work/halld/online_monitoring/root/<br />
* Offline monitoring histogram ROOT files: /work/halld/data_monitoring/RunPeriod-201Y-MM/verVV/rootfiles<br />
* REST files (most recent launch only): /work/halld/data_monitoring/RunPeriod-201Y-MM/REST/verVV<br />
* individual files for each job (ROOT, log, etc.): /volatile/halld/offline_monitoring/RunPeriod-201Y-MM/verVV/<br />
<br />
=== Monitoring Database ===<br />
* Accessing monitoring database (on ifarm): mysql -u datmon -h hallddb.jlab.org data_monitoring<br />
<br />
=== Monitoring Webpages ===<br />
*[https://halldweb.jlab.org/cgi-bin/data_monitoring/monitoring/plotBrowser.py Plot Browser]<br />
*[https://halldweb.jlab.org/cgi-bin/data_monitoring/monitoring/runBrowser.py Run Browser]<br />
*[https://halldweb.jlab.org/cgi-bin/data_monitoring/monitoring/timeSeries.py Time Series]<br />
*[https://halldweb.jlab.org/data_monitoring/launch_analysis/ Launch Analysis]<br />
<br />
== Job Monitoring Links ==<br />
* [http://scicomp.jlab.org/farm2/job.html Completed Job History]<br />
* [http://scicomp.jlab.org/farm2/project.html Job Stats By Project]<br />
* [http://scicomp.jlab.org/farm2/trackOrg.html Job Stats By Track]<br />
* [http://scicomp.jlab.org/farm2/report.html Cluster Report]<br />
* [http://scicomp.jlab.org/farm2/walltime.html Walltime Distribution]<br />
<br />
== Saving Online Monitoring Data ==<br />
<br />
The procedure for writing the data out is given in, e.g.,<br />
[https://halldweb.jlab.org/wiki/index.php/Raid-to-Silo_Transfer_Strategy Raid-to-Silo Transfer Strategy].<br />
<br />
Once the DAQ writes out the data to the raid disk, cron jobs will copy the file to tape,<br />
and within ~20 min., we will have access to the file on tape at<br />
/mss/halld/$RUN_PERIOD/rawdata/RunXXXXXX.<br />
<br />
All online monitoring plugins will be run as data is taken.<br />
They will be accessible within the counting house via RootSpy, and<br />
for each run and file, a ROOT file containing the histograms will be saved<br />
within a subdirectory for each run.<br />
<br />
For immediate access to these files, the raid disk files may be accessed directly<br />
from the counting house, or the tape files will be available within ~20 min. of the<br />
file being written out.<br />
<br />
== Offline Monitoring: Running Over Archived Data ==<br />
<br />
Once files are written to tape we run the online plugins on these files to confirm what we were seeing in the online monitoring, and also to update the results from the latest calibration and software. <br />
Manual scripts and cron jobs are set up to look for new data and run the plugins over a sample of files.<br />
<br />
Every other Friday (usually the Friday before the offline meetings) jobs will be started to run the newest software on all previous runs,and allows everybody to see improvements in each detector.<br />
For each launch, independent builds of hdds, sim-recon, the monitoring plugins, and an sqlite file will be generated.<br />
<br />
Below the procedures are described for<br />
# Preparing the software for the launch<br />
# Starting the launch (using hdswif)<br />
# Post-analysis of statistics of the launch<br />
<br />
Processing the results and making them available to the collaboration<br />
is handled in the section [[#Post-Processing_Procedures | Post-Processing Procedures]] below.<br />
<br />
<!--<br />
==== Generating an offline plugin job ====<br />
<br />
The user gxproj1 should be used for official offline monitoring jobs.<br />
Within /home/gxproj1/halld/monitoring/batch/ there will be scripts to run the online monitoring plugins over tape files.<br />
The main script is generatejobs_plugins_rawdata.sh, which can be used as<br />
generatejobs_plugins_rawdata.sh [minrun] [maxrun] (minfile) (maxfile)<br />
where minrun, maxrun specify the range of the run #, and minfile, maxfile (optional) specify<br />
the file range for that run.<br />
<br />
This will generate a script run_rawdata_XXXXXX.sh, where the run # has now been formatted to be 6 digits.<br />
Executing this script will send the monitoring plugins job to the Auger batch system.<br />
<br />
There is also a script clean.sh which can be used as<br />
clean.sh XXX<br />
This will clean up all associated files created in association with run XXX.<br />
<br />
Internally, the xml file used to submit the job will be created, and the job to run<br />
will be given within script.sh. All run parameters should be specified in at the beginning<br />
of generatejobs_plugins_rawdata.sh<br />
Since we are running on tape, the tape file will first be copied over to the cache disk, and the job will run<br />
over this cached file.<br />
<br />
==== Using cron to run automatically ====<br />
Within /home/gluex/halld/monitoring/cron/ there is a file cron_plugins<br />
that can be executed via<br />
crontab cron_plugins<br />
This will set up a cron job to call the script scan_for_jobs.sh, which will<br />
check in the rawdata directory and call generatejobs_plugins_rawdata.sh for<br />
any run that is more than 5 min old. The cron job is set up to run every 10 min.<br />
<br />
==== Magnetic Field Settings ====<br />
For tracking, it is necessary to set the correct magnetic field settings.<br />
The below table is now obsolete. Please refer to the [https://halldweb.jlab.org/cgi-bin/data_monitoring/run_conditions.pl GlueX Run Conditions] webpage.<br />
<br />
The actual field settings for each run have not been documented well.<br />
Below are values based on going through entries in the [https://logbooks.jlab.org/book/halld halld log].<br />
<br />
{| border="1" cellpadding="1" style="text-align: center;"<br />
!width="100"| Run #s<br />
!width="150"| Solenoid Current (A)<br />
!width="300"| JANA option<br />
!width="300"| notes<br />
|-<br />
| 940 - 996 || 1000 || -PBFIELD_MAP=Magnets/Solenoid/solenoid_1000A_poisson_20141104 || At run 997, the solenoid started ramping down<br />
|-<br />
| 998 - 1448 || 0 || -PBFIELD_TYPE=NoField -PDEFTAG:DTrackCandidate=StraightLine || Solenoid started ramping up around run 1431. Runs 1432 - 1448 should not be used for tracking. See [https://logbooks.jlab.org/entry/3309039 this] and [https://logbooks.jlab.org/entry/3309092 this] entry.<br />
|-<br />
| 1449 - 1620 || 1200 || -PBFIELD_MAP=Magnets/Solenoid/solenoid_1200A_poisson_20140520 || <br />
|-<br />
|}<br />
<br />
'''Note:''' The following run ranges were taken with a 300A field, and could be analyzed with the magnetic field map Magnets/Solenoid/solenoid_300A_poisson_20140819 , more study is needed to see if the straight line track fitter is needed or not: 1036 - 1053, 1065 - 1121, 1212? - 1254, 1309 - 1318 [1307,1308,1319-1329 were taken while the magnet was ramping].<br />
--><br />
<br />
=== General Information on Procedures ===<br />
This section explains how the offline monitoring should be run.<br />
Since we may want to simultaneously run offline monitoring for different run periods that require<br />
different environment variables, the scripts are set up so that a generic user can download the<br />
scripts and run them from anywhere. Most output directories for offline monitoring are created<br />
with group read/write permissions so that any Hall D group user has access to the contents,<br />
but there are some cases where use of the account that created the launch is necessary.<br />
<br />
The accounts used for offline monitoring are the gxprojN accounts created and maintained by<br />
Mark Ito (see [https://halldweb.jlab.org/wiki/index.php/GlueX-related_shared_accounts_on_the_JLab_CUE here] for how each account is used).<br />
As of October 2015, the following are used:<br />
* gxproj1 for running over Fall 2014 data (deprecated since June 2015)<br />
* gxproj5 for running over Spring 2015 data<br />
<br />
Since the summer of 2015 we have transitioned from a system using Mark Ito's jproj scripts<br />
to integrating the swif system that Chris Larrieu (SciComp) has been developing. For offline monitoring,<br />
the hdswif system that Kei developed is used for launching the jobs, and the jproj system is used<br />
for meta-analysis of launch statistics.<br />
<br />
Both hdswif and jproj are maintained in svn:<br />
* hdswif: https://halldsvn.jlab.org/repos/trunk/scripts/monitoring/hdswif<br />
* jproj : https://halldsvn.jlab.org/repos/trunk/scripts/monitoring/jproj<br />
<br />
To run the offline monitoring each package should be checked out and all necessary<br />
scripts are included.<br />
<br />
=== Preparing the software for the launch ===<br />
<br />
To begin a new launch the software must be built to the latest versions.<br />
For the gxprojN user accounts used, all software builds are contained in the directory ~/builds<br />
(which are soft links to /work/halld/home/gxprojN/builds). When logging into these accounts<br />
the setup files ~/setup_jlab-2015-03.csh or similar files should be sourced.<br />
<br />
<b>Note that Mark Ito does <u>not</u> want you to change the contents of each .cshrc file.</b><br />
You should consult him if you feel the need.<br />
<br />
<span style="color:red">NOTE: FOR BUILDING SOFTWARE IT IS A GOOD IDEA TO DO A COMPLETE WIPEOUT/CHECKOUT EACH TIME TO AVOID STALE HEADER FILES.</span><br />
<br />
# Building hdds: Go to ~/builds/hdds. The directory hdds is the one from git. Delete the contents, then download the newest version from git and build: <pre>cd ~/builds/hdds/</pre> <pre>rm -frv hdds</pre> <pre>git clone https://github.com/JeffersonLab/hdds</pre> <pre>cd hdds</pre> <pre>scons install</pre><br />
# Building sim-recon: Go to ~/builds/sim-recon. The directory sim-recon is the one from git. Delete the contents, then download the newest version from git and build: <pre>cd ~/builds/sim-recon</pre> <pre>rm -frv sim-recon</pre> <pre>git clone https://github.com/JeffersonLab/sim-recon</pre> <pre>cd sim-recon/src</pre> <pre>scons install -j8</pre><br />
# Prepare the latest sqlite file: The sqlite is set in the ~/setup_jlab-2015-03.csh script as <b>sqlite:////home/gxproj5/ccdb.sqlite</b> through the environment variables <b>JANA_CALIB_URL</b> and <b>CCDB_CONNECTION</b>. Therefore, go to this directory and create a new sqlite file. We create the sqlite file in a temporary directory since creating the sqlite file in a directory where the output file exists causes errors. Original documentation on creating sqlite files are [https://halldweb.jlab.org/wiki/index.php/SQLite-form_of_the_CCDB_database here]. <br><span style="color:red">NOTE: SQLITE FILES DO NOT WORK ON THE NEW /work DISK INSTALLED IN OCTOBER 2015 </span> <pre>cd ~/tmp</pre> <pre>$CCDB_HOME/scripts/mysql2sqlite/mysql2sqlite.sh -hhallddb.jlab.org -uccdb_user ccdb | sqlite3 ccdb.sqlite</pre> <pre>mv ccdb.sqlite ../</pre><br />
# Note that the above steps must be done <b>BEFORE</b> launch project creation. This is because we will track the revisions of the libraries used, and this is done by extracting the svn information in each directory. Also, note that the system assumes that we have the topmost build directory (usually called GLUEX_TOP) to be $HOME/builds . Such an assumption is necessary to be able to extract information about the library locations automatically.<br />
<br />
Create the appropriate project(s) and submit the jobs using hdswif, as detailed in the section below.<br />
<br />
=== Starting the Launch and Submitting Jobs ===<br />
<br />
Until the summer of 2015 we relied solely on Mark Ito's jproj system for submitting and keeping track of jobs. We have since moved to the swif system and use the hdswif wrapper for this. Below are instructions for how to use these.<br />
<br />
# Downloading hdswif: Download the hdswif directory from svn. For the gxprojN accounts, use the directory ~/halld/hdswif. <pre>cd ~/halld </pre> <pre> svn co https://halldsvn.jlab.org/repos/trunk/scripts/monitoring/hdswif</pre> <pre>cd hdswif</pre><br />
# Creating the workflow: Within SWIF jobs are registered into workflows. First create the workflow. For offline monitoring, the workflow names are of the form <b>offline_monitoring_RunPeriod201Y_MM_verVV_hd_rawdata</b> with suitable replacements for the run period and version number. The command "swif list" will list all existing workflows. Also, for most simple SWIF commands hdswif also provides a wrapper. <pre>swif list</pre> For creation of workflows for offline monitoring the command <pre>hdswif.py create [workflow] -c [config file] </pre> should be used. As an example config file, see the input.config file in the folder (and update it). When a config file is passed in, hdswif will automatically create files that record the configuration of the current launch. These files are stored as for example<br />
#* /group/halld/data_monitoring/run_conditions/jana_rawdata_comm_2015_03_ver15.conf<br />
#* /group/halld/data_monitoring/run_conditions/soft_comm_2015_03_ver15.xml<br />
#* The software packages stored in git (sim-recon and hdds) can have git tags applied to them, which makes it easier to find versions of the software than a SHA-1 hash. hdswif will ask if you would like to create a tag, and execute the following sequence: <pre>git tag -a offmon-201Y_MM-verVV -m "Used for offline monitoring 201Y-MM verVV started on 201y/mm/dd"</pre> <pre>git push offmon-201Y_MM-verVV</pre> This will only be invoked when the user name is gxprojN, and for the configuration files, the output directory will be /group/halld/data_monitoring/run_conditions/ for gxprojN accounts while it will be the current directory for other users.<br />
#* To use the git-tagged software versions do for example <pre>cd $HALLD_HOME</pre> <pre>git checkout offmon-2015_03-ver15</pre><br />
# Registering jobs in the workflow: To register jobs within the workflow, hdswif provides the use of config files. Jobs can be registered by specifying the workflow, config file (-c), run (-r) and file (-f) numbers if necessary. A typical config file will look this:<br />
PROJECT gluex<br />
TRACK reconstruction<br />
OS centos65<br />
NCORES 6<br />
DISK 40<br />
RAM 8<br />
TIMELIMIT 8<br />
JOBNAMEBASE offmon_<br />
RUNPERIOD 2015-03<br />
VERSION 15<br />
OUTPUT_TOPDIR /volatile/halld/offline_monitoring/RunPeriod-[RUNPERIOD]/ver[VERSION] # Example of other variables included in variable<br />
SCRIPTFILE /home/gxproj5/halld/hdswif/script.sh # Must specify full path<br />
ENVFILE /home/gxproj5/halld/hdswif/setup_jlab-2015-03.csh # Must specify full path<br />
The config file contains configuration parameters for each of the jobs.<br><br />
<span style="color:red">Note: Job configuration parameters can be set differently for jobs within the same workflow if necessary.</span><br />
Edit the config file and save as a new file if necessary. Once the configuration is set, jobs can be added via <pre>hdswif.py add [workflow] -c input.config</pre><br />
By default, hdswif will add all files found within the directory /mss/halld/RunPeriod-201Y-MM/rawdata/ where 201Y-MM is specified by the RUNPERIOD parameter in the config file. If only some of the runs or files are needed, these can be specified for example with<br />
<pre>hdswif.py add [workflow] -c input.config -r 3180 -f '00[0-4]'</pre><br />
to specify to register running only over run 3180 files 000 - 004 (Unix-style brackets and wildcards can be used).<br />
<br />
Running the workflow: To run the workflow, simply use swif run: <pre>swif run -workflow [workflow]</pre> or equivalently, using the hdswif wrapper, <pre>hdsswif.py run [workflow]</pre><br><br />
<b>It is recommended that some jobs be tested to make sure that everything is working rather than fail thousands of jobs.</b><br> For this purpose, hdswif will take an additional parameter to run which limits the number of jobs to submit: <pre>hdswif.py run [workflow] 10</pre><br />
in which case only 10 jobs will be submitted. To submit all jobs after checking the results, do <pre>hdswif.py run [workflow]</pre><br />
<br />
=== Checking the Status and Resubmitting ===<br />
# The status of jobs can be checked on the terminal with <pre>jobstat -u gxprojN</pre> on Auger and for SWIF with <pre>swif list</pre> or for more information, <pre>swif status [workflow] -summary</pre> Also see the Auger [http://scicomp.jlab.org/scicomp/#/auger/jobs job website].<br />
# For failed jobs, SWIF can resubmit jobs based on the problem. For resubmission for failed jobs with the same resources, <pre>swif retry-jobs [workflow] -problems [problem name]</pre> can be used, and for jobs to be submitted with more resources, e.g., use <pre>swif modify-jobs -ram add 2gb -problems AUGER-OVER_RLIMIT</pre> hdswif has a wrapper for both of these: <pre>hdswif.py resubmit [workflow] [problem]</pre> In this case [problem] can be one of <b>SYSTEM, TIMEOUT, RLIMIT</b>. If SYSTEM is specified, the jobs will be retried. For TIMEOUT and RLIMIT, the jobs will be modified by default with 2 additional hours or GB of RAM. If one more number is added as an option, then that many hours or GB of RAM will be added., e.g., <pre>hdswif.py resubmit [workflow] TIMEOUT 5</pre> will add 5 hours of processing time.<br />
# For information on swif, use the "swif help" commands and for hdswif see the attached documentaion in https://halldsvn.jlab.org/repos/trunk/scripts/monitoring/hdswif/manual_hdswif.pdf<br />
<br />
=== Post-analysis of statistics of the launch ===<br />
<br />
# After jobs have been submitted, it will usually take a few days for all of the jobs to be processed.<br />
#* Status of Auger: http://scicomp.jlab.org/scicomp/#/auger/jobs (see also links above)<br />
#* Status of user jobs: <pre>jobstat -u [user name]</pre><br />
# The status and results of jobs are saved within the SWIF internal server, and are available via the command <pre>swif status [workflow] -summary -runs</pre> where the arguments -summary and -runs show summary statistics and statistics for individual jobs, respectively. hdswif has a command that takes this output in XML output and creates an HTML webpage showing results of the launch. To do this, do <pre>hdswif.py summary [workflow]</pre> This will create an XML file swif_output_[workflow].xml that contains all information from SWIF. If the file already exists, hdswif will ask whether to overwrite the existing file.<br />
# At this stage the html output and figure files are created and ready to be put online. For this step and other steps involving analysis of the statistics of the launch results, it is convenient to change to the jproj system.<br />
# The jproj scripts for offline monitoring are maintained in the svn directory https://halldsvn.jlab.org/repos/trunk/scripts/monitoring/jproj Do <pre> svn co https://halldsvn.jlab.org/repos/trunk/scripts/monitoring/jproj </pre> For the gxprojN accounts used for offline monitoring the directory should be ~/halld/jproj<br />
# The jproj directory contains two subdirectories, scripts and projects. The scripts directory contains useful scripts for processing the jobs registered in the jproj system, and each of the offline monitoring launches will be handled in the projects directory.<br />
# Go to the projects directory<pre>cd ~/halld/jproj/projects</pre> and use the script create_project.sh to create a new directory that contains the processing scripts for the current launch <pre>./create_project.sh [workflow]</pre> This should create a directory such as offline_monitoring_RunPeriod2015_03_ver15_hd_rawdata (same as the workflow name). The script uses the template files in the directory templates and by substitution creates script files for the current launch. Now go to the newly created analysis directory: <pre>cd [workflow]/analysis</pre><br />
which for the gxprojN accounts should have the full path /home/gxprojN/halld/jproj/projects/[workflow]/analysis.<br><br><br />
<span style="color:red">All of the analysis commands including arguments are contained in run_analysis.sh, but it is <b>strongly</b> recommended that all commands are run manually to check for errors.</span><br><br><br />
# The first thing to do is to make the html output from hdswif public. Copy the html file and related figures that were created from hdswif to the appropriate space within /group/halld/www/halldweb/html/data_monitoring/ : <pre>python publish_offmon_results.py [run period] [version]</pre> Note that the command with appropriate substitutions for arguments can be found within run_analysis.sh (same for all commands below).<br />
# Next, we need to create a few MySQL tables for the current launch. The MySQL tables are useful for comparing run/file combinations for different launches. For the SWIF launches, there are two tables needed, and will be named<br />
#* [workflow]Job<br />
#* [workflow]_aux<br />
# This naming scheme of tables and their roles are the same as from the jproj only launches. The [workflow]Job table will contain information gathered from SWIF about each job (which node it went to, start time of each stage, memory usage, etc.). The [workflow]_aux table will contain information gathered from the output stdout files from each job. First, create the Job table using <pre>python create_jproj_job_table.py [run period] [version]</pre> Note that this script uses the XML output from hdswif summary and inserts the contents into the MySQL table, so the XML output file must exist.<br />
# To check the contents of this MySQL table, do <pre>mysql -hhalldb -ufarmer farming</pre> <pre>mysql> describe offline_monitoring_RunPeriod2015_03_ver11_hd_rawdataJob;</pre><br />
<br />
== Hall D Job Management System ==<br />
<br />
This section details instructions on how to create and launch a set of jobs using the Hall-D Job Management System developed by Mark Ito. These instructions are generic: this system can be used for the weekly monitoring jobs, but can also be used for other sets of job launches as well. <br />
<br />
=== Database Table Overview ===<br />
<br />
* Job management database table (<project_name>): For each input file, keeps track of whether or not a job for it has been submitted, along with other optional fields. <br />
<br />
* Job status database table (<project_name>Job (no space)): For each job, keeps track of the job-id, the job status, memory used, cpu & wall time, time taken to complete various stages (e.g. pending, dependency, active), and others. <br />
<br />
* Job metrics database table (<project_name>_aux (no space)): For each job, keeps track of the job-id, how many events were processed, the time it took to copy the cache file, and the time it took to run the plugin. This information is culled from the log files of each job, and is done within the analysis directory of each launch.<br />
<br />
=== Initialize Project Management ===<br />
<br />
* Log into the ifarm machine with one of the gxproj accounts. For this example we will use gxproj1.<br />
<pre><br />
ssh gxproj1@ifarm -Y<br />
</pre><br />
<br />
* Go to a directory to do the launch. In principle, any directory will work, but for gxproj1 this is usually done in /home/gxproj1/halld/<br />
<br />
* Check out the necessary scripts <pre>svn co https://halldsvn.jlab.org/repos/trunk/scripts/monitoring/jproj</pre> This will get all necessary scripts for launching. Once checked out, <pre>cd projects</pre><br />
<br />
* The script create_project.sh can be used to create a new project. It will take a single argument, the project name. It is assumed that the project name is of the form offline_monitoring_RunPeriod20YY_MM_verVV_hd_rawdata where this string will be parsed to give the run period 20YY_MM and version number VV. One thing to do BEFORE creation of a new project is to editthe conditions of the launch (plugins to run over, memory requested, disk space requested) within templates/template.jsub . This<br />
information is saved automatically at project creation time into files in /group/halld/data_monitoring/run_conditions .<br />
To create a project do <pre>./create_project.sh offline_monitoring_RunPeriod20YY_MM_verVV_hd_rawdata</pre><br />
The name has been chosen to be as consistent as possible with other directory structures.<br />
However, mysql requires that "-" be escaped in table names, so unfortunately run periods will be<br />
given as 2YYY_MM instead of 2YYY-MM.<br />
<br />
* A new directory with the project name (offline_monitoring_RunPeriod20YY_MM_verVV_hd_rawdata) and files will be copied and modified from the template directory to reflect the run period, the user, the directory that it was created in, project name, etc.<br />
<br />
* For each project, cd into the new directory<br />
* The script clear.sh will remove any existing tables for the current project name, then recreate it. Do<br />
<pre>./clear.sh</pre><br />
* To use the jproj.pl script that was checked in, add the directory to your path with<br />
<pre>source ../../scripts/setup.csh</pre><br />
or always specify the full path<br />
<pre>../../scripts/jproj.pl</pre><br />
* Now update the table of runs with<br />
<pre>jproj.pl <project name> update</pre><br />
This will fill the table with all files within /mss that are of the same form as what is in <project name>.jproj . If you want to register only on a subset of all such files, you can edit this file directly.<br />
* Once you have registered all of the files you would like to run over, do<br />
<pre>jproj.pl <project name> submit [max # of jobs] [run number]</pre><br />
where the additional options specify how many jobs to submit and which run number to run on. Without these options all files that are registered and have not been submitted yet will be submitted.<br />
<br />
At this stage you are ready to submit all files. It is a good idea to submit a few test jobs<br />
at first to check that all scripts are working and that the plugins do not crash. Once you are<br />
sure that this does not happen, you can send all jobs in. The remaining jobs are then the monitoring<br />
which will (among other things) put the results on the online webpage for the collaboration to view, and<br />
the analysis of the launch.<br />
<br />
=== Project File Overview ===<br />
<br />
An overview of each project file:<br />
* '''clear.sh:''' For the current project, deletes the job status and management database tables (if any), and creates new, empty ones. <br />
* '''<project_name>.jproj:''' Contains the path and file name format for the input files for the jobs. <br />
* '''<project_name>.jsub:''' The xml job submission script. The run number and file number variables are set during job submission for each input file. <br />
* '''script.sh:''' The script that is executed during the job. If output job directories are not pre-created manually, they should be created in this script with the proper permissions:<br />
<pre><br />
mkdir -p -m 775 my_directory<br />
</pre><br />
* '''setup_jlab-[run period].csh:''' The environment that is sourced at the beginning of the job execution. <br />
* '''status.sh:''' Updates the job status database table, and prints some of its columns to screen.<br />
<br />
=== Project Management ===<br />
<br />
* Delete (if any) and create the database table(s) for the current set of job submissions:<br />
<pre><br />
./clear.sh<br />
</pre><br />
Also, if testing was done with jobs, it is best to delete the output directory and the configuration files:<br />
<pre><br />
rm -frv /volatile/halld/offline_monitoring/RunPeriod-20YY-MM/verVV /group/halld/data_monitoring/run_conditions/soft_comm_20YY_MM_verVV.xml /group/halld/data_monitoring/run_conditions/jana_rawdata_comm_20YY_MM_verVV.conf<br />
</pre><br />
<br />
* Search for input files matching the string in the .jproj file, and create a row for each in the job management database table (called <project_name>). You can test by adding an optional argument at the end, which only selects files with a specific file number:<br />
<pre><br />
jproj.pl <project_name> update <optional_file_number><br />
</pre><br />
<br />
* Confirm that the job management database is accurate by printing it's contents to screen:<br />
<pre><br />
mysql -hhallddb -ufarmer farming -e "select * from <project_name>"<br />
</pre><br />
<br />
* ONLY if a mistake was made, to delete the tables from the database and recreate new, empty ones, run: <br />
<pre><br />
./clear.sh<br />
</pre><br />
<br />
* To look at the status of the submitted jobs, first query auger and update the job status database:<br />
<pre><br />
fill_in_job_details.pl <project_name><br />
</pre><br />
<br />
* The job status can then be viewed by submitting a query to the job status database (called <project_name>Job (no space in between)):<br />
<pre><br />
mysql -hhallddb -ufarmer farming -e "select id,run,file,jobId,hostname,status,timeSubmitted,timeActive,walltime,cput,timeComplete,result,error from <project_name>Job"<br />
</pre><br />
<br />
* These last two commands can instead be executed simultaneously by running:<br />
<pre><br />
./status.sh<br />
</pre><br />
<br />
=== Handy mysql Instructions ===<br />
<br />
* Handy mysql instructions:<br />
<pre><br />
mysql -hhallddb -ufarmer farming # Enter the "farming" mysql database on "hallddb" as user "farmer"<br />
quit; # Exit mysql<br />
show tables; # Show a list of the tables in the current database<br />
show columns from <project_name>; # show all of the columns for the given table<br />
select * from <project_name>; # show the contents of all rows from the given table<br />
</pre><br />
<br />
=== Backing Up Offline Monitoring Tables ===<br />
Tables created for offline monitoring can be backed up using the script backup_tables.sh which<br />
can be checked out with the other files from https://halldsvn.jlab.org/repos/trunk/scripts/monitoring/jproj/projects<br />
<br />
The script uses the command mysqldump to print out a file that can be executed to recreate the tables.<br />
Since executing this output file will drop the table when it exists, caution is advised.<br />
Example usage to backup all three tables created for run period 2014_10 ver 17:<br />
<pre>backup_tables.sh 2014_10 17</pre><br />
<br />
== Running Over Data As It Comes In ==<br />
<br />
A special user gxproj1 will have a cron job set up to run the plugins as new data appears on /mss.<br />
During the week, gxproj1 will submit offline plugin jobs with the same setup as the weekly jobs<br />
run the previous Friday. The procedure for this is shown below.<br />
<br />
<!--<br />
=== Setting up the environment ===<br />
The file<br />
/home/gxproj1/setup_jlab.csh<br />
is sourced through .tcshrc.<br />
This file is the same as what is linked to by<br />
/home/gluex/setup_jlab_commissioning.csh,<br />
except HALLD_HOME, HDDS_HOME, and JANA_CALIB_URL are set separately so that this<br />
user can have a separate build.<br />
<br />
To obtain the builds from the previous Friday's runs,<br />
execute<br />
/home/gxproj1/halld/monitoring/newruns/setup_previous.sh [year] [month] [day]<br />
The build revisions from the previous Friday are archived in files<br />
/work/halld/data_monitoring/run_conditions/soft_comm_[year]_[month]_[day].xml<br />
and the script will build libraries based on those stored revision numbers.<br />
--><br />
<br />
=== Running the cron job ===<br />
<br />
'''IMPORTANT:''' The cron job should not be running while you are manually submitting jobs using the jproj.pl script for the same project, or else you will probably multiply-submit a job. <br />
<br />
* Go to the cron job directory: <br />
<pre><br />
cd /u/home/gxproj1/halld/monitoring/newruns<br />
</pre><br />
<br />
* The cron_plugins file is the cronjob that will be executed. During execution, it runs the exec.sh command in the same folder. This command takes two arguments: the project name, and the maximum file number for each run. These fields should be updated in the cron_plugins file before running. <br />
<br />
* The exec.sh command updates the job management database table with any data that has arrived on tape since it was last updated, ignoring file numbers greater than the maximum file number. It then submits jobs for these files. <br />
<br />
* To start the cron job, run:<br />
<pre><br />
crontab cron_plugins<br />
</pre><br />
<br />
* To check whether the cron job is running, do<br />
<pre><br />
crontab -l<br />
</pre><br />
<br />
* To remove the cron job do<br />
<pre><br />
crontab -r<br />
</pre><br />
<br />
<!--<br />
The cron job will run the script scan_for_jobs.sh,<br />
which runs generatejobs_plugins_rawdata.sh for any<br />
new runs that it had not seen before. All previous<br />
runs are recorded in the file filelists/files_current.txt<br />
so clear this to run over runs, or set the parameters<br />
MINRUN and MAXRUN which will set the range of runs submitted.<br />
--><br />
<br />
==Post-Processing Procedures==<br />
<br />
To visualize the monitoring data, we save images of selected histograms and store time series of selected quantities in a database, which are then displayed on a web page. This section describes how to generate the monitoring images and database information.<br />
<br />
The scripts used to generate this summary data are primarily run from /home/gxprojN/halld/monitoring/process . If you want a new copy of the scripts, e.g., for a new monitoring run, you should check the scripts out from SVN:<br />
<syntaxhighlight><br />
svn co https://halldsvn.jlab.org/repos/trunk/scripts/monitoring/process<br />
</syntaxhighlight><br />
Note that these scripts currently have some parameters which must be periodically set by hand.<br />
<br />
The default python version on most JLab machine does not have the modules to allow these scripts to connect to the MySQL database. To run these scripts, load the environment with the following command<br />
<syntaxhighlight><br />
source /home/gxproj1/halld/monitoring/process/monitoring_env.csh<br />
</syntaxhighlight><br />
<br />
===Online Monitoring===<br />
<br />
There are two primary scripts for running over the monitoring data generated by the online system and offline reconstruction. The online script can be run with either of the following commands:<br />
<syntaxhighlight><br />
/home/gluex/halld/monitoring/process/check_new_runs.py<br />
<br />
OR <br />
<br />
/home/gluex/halld/monitoring/process/check_new_runs.csh<br />
</syntaxhighlight><br />
The shell script sets up the environment properly to run the python script. To connect to the monitoring database on the JLab CUE, modules included in the local installation of python >= 2.7 are needed. The shell script is appropriate to use in a cron job. The cronjob is currently run under the "gluex" account.<br />
<br />
The online monitoring system copies a ROOT file containing the results of the online monitoring, and other configuration files into a directory accessible outside the counting house. This python script automatically checks for new ROOT files, which it will then automatically process. It contains several configuration variables that must be correctly set, which contains the location of input/output directories, etc...<br />
<br />
===Offline Monitoring===<br />
<br />
After the data is run over, the results should be processed, so that summary data is entered into the monitoring database and plots are made for the monitoring webpages. Currently, this processing is controlled by a cronjob that runs the following script:<br />
<syntaxhighlight><br />
/home/gxproj1/halld/monitoring/process/check_monitoring_data.csh <br />
</syntaxhighlight><br />
This script checks for new ROOT files, and only runs over those it hasn't processed yet. Since one monitoring ROOT file is produced for each EVIO file, whenever a new file is produced, the plots for the corresponding run are recreated and all the ROOT files for a run are combined into one file. Information is stored in the database on a per-file basis. <br />
<br />
Plots for the monitoring web page can be made from single histograms or multiple histograms using RootSpy macros. If you want to change the list of plots made, you must modify one of the following files:<br />
* histograms_to_monitor - specify either the name of the histogram or its the full ROOT path <br />
* macros_to_monitor - specify the full path to the RootSpy macro .C file<br />
<br />
When a new monitoring run is started, or the conditions are changed, the following steps should be taken to process the new files:<br />
# Add a new data version, as described below:<br />
# Change the following parameters in check_monitoring_data.csh:<br />
## JOBDATE should correspond to the ouptut date used by the job submission script<br />
## OUTPUTDIR should correspond to the directory corresponding to the run period and revision corresponding to the new version you just submitted. Presumably, this directory will be empty at the beginning.<br />
## Once you create a new data version as defined below, you should pass the needed information as a command line option. Currently this is done by the ARGS variable. For example, the argument "-v RunPeriod-2014-10,8" tells the monitoring scripts to look up the version corresponding to revision 8 of RunPeriod-2014-10 in the monitoring DB and to use to store the results.<br />
<br />
<syntaxhighlight><br />
Example configuration parameters:<br />
set JOBDATE=2015-01-09<br />
set INPUTDIR=/volatile/halld/RunPeriod-2014-10/offline_monitoring<br />
set OUTPUTDIR=/w/halld-scifs1a/data_monitoring/RunPeriod-2014-10/ver08<br />
set ARGS=" -v RunPeriod-2014-10,8 "<br />
</syntaxhighlight><br />
If you want to process the results manually, the data is processed using the following script:<br />
<syntaxhighlight><br />
./process_new_offline_data.py <input directory> <output directory><br />
<br />
EXAMPLE:<br />
<br />
./process_new_offline_data.py 2014-11-14 /volatile/halld/RunPeriod-2014-10/offline_monitoring/ /w/halld-scifs1a/data_monitoring/RunPeriod-2014-10/ver02<br />
</syntaxhighlight><br />
The python script takes several options to enable/disable various steps in the processing. Of interest is the "--force" option, which will run over all monitoring ROOT files, whether or not they've been previously identified.<br />
<br />
Every time a new reconstruction pass is performed, a new version number must be generated. To do this, prepare a version file as described below. Then run the register_new_version.py script to store the information in the database. The script will return a version number, which then should be set by hand in process_new_offline_data.py - future versions of the script will streamline this part of the procedure. An example of how to generate a new version is:<br />
<syntaxhighlight><br />
./register_new_version.py add /home/gxproj1/halld/monitoring/process/versions/vers_RunPeriod-2014-10_pass1.txt<br />
</syntaxhighlight><br />
<br />
<b>If you are running the offline monitoring by checking out the files in trunk/scripts/monitoring/jproj/projects/<br />
of the svn repository</b>, and created a project with<br />
create_project.sh [project name] hd_rawdata<br />
Then go to the directory [project name]/processing/<br />
and execute<br />
./run_processing.sh<br />
which will run register_new_version.py as well as check_monitoring_data.csh for that project.<br />
<br />
===Step-by-Step Instructions For Processing a New Monitoring Run===<br />
<br />
The monitoring runs are current run out of the gxproj1 and gxproj5 accounts. After an offline monitoring run has been successfully started on the batch farm, the following steps should be followed to setup the post-processing for these runs.<br />
<br />
# The post-processing scripts are stored in $HOME/halld/monitoring/process and are automatically run by cron.<br />
# Run "svn update" to bring any changes in. Be sure that the list of histograms and macros to plot are current.<br />
# Edit check_monitoring_data.csh to point to the current revisions/directories<br />
#* VERSION<br />
#* ARGS<br />
#* Note that the environment depends on a standard script - $HOME/setup_jlab.csh<br />
# Update files in the web directory, so that the results are displayed on the web pages: /group/halld/www/halldweb/html/data_monitoring/textdata<br />
# Copy the REST files to more permanent locations:<br />
#* cp -a /volatile/halld/offline_monitoring/RunPeriod-YYYY-MM/verVV/REST /work/halld/data_monitoring/RunPeriod-YYYY-MM/REST/verVV<br />
#* cp -a /volatile/halld/offline_monitoring/RunPeriod-YYYY-MM/verVV/REST /cache/halld/RunPeriod-YYYY-MM/REST/verVV [under testing]<br />
<br />
Check log files in $HOME/halld/monitoring/process/log for more information on how each run went. If there are problems, check log files, and modify check_monitoring_data.csh to vary the verbosity of the output.<br />
<br />
==Data Versions==<br />
<br />
To document the conditions of the monitoring data that is created, for the sake of reproducability and further analysis we save several pieces of information. The format is intended to be comprehensive enough to document not just monitoring data, but versions of raw and reconstructed data, so that this database table can be used for the event database as well.<br />
<br />
We store one record per pass through one run period, with the following structure:<br />
<br />
{| class="wikitable"<br />
! Field !! Description<br />
|-<br />
| data_type || The level of data we are processing. For the purposes of monitoring, "rawdata" is the online monitoring, "recon" is the offline monitoring<br />
|- <br />
| run_period || The run period of the data<br />
|- <br />
| revision || An integer specifying which pass through the run period this data corresponds to<br />
|- <br />
| software_version || The name of the XML file that specifies the different software versions used<br />
|-<br />
| jana_config || The name of the text file that specifies which JANA options were passed to the reconstruction program<br />
|-<br />
| ccdb_context || The value of JANA_CALIB_CONTEXT, which specifies the version of calibration constants that were used<br />
|-<br />
| production_time || The data at which monitoring/reconstruction began<br />
|-<br />
| dataVersionString || A convenient string for identifying this version of the data<br />
|}<br />
<br />
<br />
An example file used as as input to ./register_new_version.py is:<br />
<syntaxhighlight><br />
data_type = recon<br />
run_period = RunPeriod-2014-10<br />
revision = 1<br />
software_version = soft_comm_2014_11_06.xml<br />
jana_config = jana_rawdata_comm_2014_11_06.conf<br />
ccdb_context = calibtime=2014-11-10<br />
production_time = 2014-11-10<br />
dataVersionString = recon_RunPeriod-2014-10_20141110_ver01<br />
</syntaxhighlight></div>Kmoriyahttps://halldweb.jlab.org/wiki/index.php?title=Data_Monitoring_Procedures&diff=70848Data Monitoring Procedures2015-10-22T19:34:17Z<p>Kmoriya: /* Preparing the software for the launch */</p>
<hr />
<div>__TOC__<br />
<br />
== Master List of File / Database / Webpage Locations ==<br />
=== Run Conditions ===<br />
* Online Run-by-run condition files (B-field, current, etc.): /work/halld/online_monitoring/conditions/<br />
* Offline monitoring run conditions (software versions, jana config): /group/halld/data_monitoring/run_conditions/<br />
*[http://www.jlab.org/Hall-D/test/RunInfo/ Run Info vers. 1]<br />
*[https://halldweb.jlab.org/cgi-bin/data_monitoring/run_conditions.pl Run Info vers. 2]<br />
<br />
=== Monitoring Output Files ===<br />
* Run Periods 201Y-MM is for example 2015-03, launch ver verVV is for example ver15<br />
* Online monitoring histograms: /work/halld/online_monitoring/root/<br />
* Offline monitoring histogram ROOT files: /work/halld/data_monitoring/RunPeriod-201Y-MM/verVV/rootfiles<br />
* REST files (most recent launch only): /work/halld/data_monitoring/RunPeriod-201Y-MM/REST/verVV<br />
* individual files for each job (ROOT, log, etc.): /volatile/halld/offline_monitoring/RunPeriod-201Y-MM/verVV/<br />
<br />
=== Monitoring Database ===<br />
* Accessing monitoring database (on ifarm): mysql -u datmon -h hallddb.jlab.org data_monitoring<br />
<br />
=== Monitoring Webpages ===<br />
*[https://halldweb.jlab.org/cgi-bin/data_monitoring/monitoring/plotBrowser.py Plot Browser]<br />
*[https://halldweb.jlab.org/cgi-bin/data_monitoring/monitoring/runBrowser.py Run Browser]<br />
*[https://halldweb.jlab.org/cgi-bin/data_monitoring/monitoring/timeSeries.py Time Series]<br />
*[https://halldweb.jlab.org/data_monitoring/launch_analysis/ Launch Analysis]<br />
<br />
== Job Monitoring Links ==<br />
* [http://scicomp.jlab.org/farm2/job.html Completed Job History]<br />
* [http://scicomp.jlab.org/farm2/project.html Job Stats By Project]<br />
* [http://scicomp.jlab.org/farm2/trackOrg.html Job Stats By Track]<br />
* [http://scicomp.jlab.org/farm2/report.html Cluster Report]<br />
* [http://scicomp.jlab.org/farm2/walltime.html Walltime Distribution]<br />
<br />
== Saving Online Monitoring Data ==<br />
<br />
The procedure for writing the data out is given in, e.g.,<br />
[https://halldweb.jlab.org/wiki/index.php/Raid-to-Silo_Transfer_Strategy Raid-to-Silo Transfer Strategy].<br />
<br />
Once the DAQ writes out the data to the raid disk, cron jobs will copy the file to tape,<br />
and within ~20 min., we will have access to the file on tape at<br />
/mss/halld/$RUN_PERIOD/rawdata/RunXXXXXX.<br />
<br />
All online monitoring plugins will be run as data is taken.<br />
They will be accessible within the counting house via RootSpy, and<br />
for each run and file, a ROOT file containing the histograms will be saved<br />
within a subdirectory for each run.<br />
<br />
For immediate access to these files, the raid disk files may be accessed directly<br />
from the counting house, or the tape files will be available within ~20 min. of the<br />
file being written out.<br />
<br />
== Offline Monitoring: Running Over Archived Data ==<br />
<br />
Once files are written to tape we run the online plugins on these files to confirm what we were seeing in the online monitoring, and also to update the results from the latest calibration and software. <br />
Manual scripts and cron jobs are set up to look for new data and run the plugins over a sample of files.<br />
<br />
Every other Friday (usually the Friday before the offline meetings) jobs will be started to run the newest software on all previous runs,and allows everybody to see improvements in each detector.<br />
For each launch, independent builds of hdds, sim-recon, the monitoring plugins, and an sqlite file will be generated.<br />
<br />
Below the procedures are described for<br />
# Preparing the software for the launch<br />
# Starting the launch (using hdswif)<br />
# Post-analysis of statistics of the launch<br />
<br />
Processing the results and making them available to the collaboration<br />
is handled in the section [[#Post-Processing_Procedures | Post-Processing Procedures]] below.<br />
<br />
<!--<br />
==== Generating an offline plugin job ====<br />
<br />
The user gxproj1 should be used for official offline monitoring jobs.<br />
Within /home/gxproj1/halld/monitoring/batch/ there will be scripts to run the online monitoring plugins over tape files.<br />
The main script is generatejobs_plugins_rawdata.sh, which can be used as<br />
generatejobs_plugins_rawdata.sh [minrun] [maxrun] (minfile) (maxfile)<br />
where minrun, maxrun specify the range of the run #, and minfile, maxfile (optional) specify<br />
the file range for that run.<br />
<br />
This will generate a script run_rawdata_XXXXXX.sh, where the run # has now been formatted to be 6 digits.<br />
Executing this script will send the monitoring plugins job to the Auger batch system.<br />
<br />
There is also a script clean.sh which can be used as<br />
clean.sh XXX<br />
This will clean up all associated files created in association with run XXX.<br />
<br />
Internally, the xml file used to submit the job will be created, and the job to run<br />
will be given within script.sh. All run parameters should be specified in at the beginning<br />
of generatejobs_plugins_rawdata.sh<br />
Since we are running on tape, the tape file will first be copied over to the cache disk, and the job will run<br />
over this cached file.<br />
<br />
==== Using cron to run automatically ====<br />
Within /home/gluex/halld/monitoring/cron/ there is a file cron_plugins<br />
that can be executed via<br />
crontab cron_plugins<br />
This will set up a cron job to call the script scan_for_jobs.sh, which will<br />
check in the rawdata directory and call generatejobs_plugins_rawdata.sh for<br />
any run that is more than 5 min old. The cron job is set up to run every 10 min.<br />
<br />
==== Magnetic Field Settings ====<br />
For tracking, it is necessary to set the correct magnetic field settings.<br />
The below table is now obsolete. Please refer to the [https://halldweb.jlab.org/cgi-bin/data_monitoring/run_conditions.pl GlueX Run Conditions] webpage.<br />
<br />
The actual field settings for each run have not been documented well.<br />
Below are values based on going through entries in the [https://logbooks.jlab.org/book/halld halld log].<br />
<br />
{| border="1" cellpadding="1" style="text-align: center;"<br />
!width="100"| Run #s<br />
!width="150"| Solenoid Current (A)<br />
!width="300"| JANA option<br />
!width="300"| notes<br />
|-<br />
| 940 - 996 || 1000 || -PBFIELD_MAP=Magnets/Solenoid/solenoid_1000A_poisson_20141104 || At run 997, the solenoid started ramping down<br />
|-<br />
| 998 - 1448 || 0 || -PBFIELD_TYPE=NoField -PDEFTAG:DTrackCandidate=StraightLine || Solenoid started ramping up around run 1431. Runs 1432 - 1448 should not be used for tracking. See [https://logbooks.jlab.org/entry/3309039 this] and [https://logbooks.jlab.org/entry/3309092 this] entry.<br />
|-<br />
| 1449 - 1620 || 1200 || -PBFIELD_MAP=Magnets/Solenoid/solenoid_1200A_poisson_20140520 || <br />
|-<br />
|}<br />
<br />
'''Note:''' The following run ranges were taken with a 300A field, and could be analyzed with the magnetic field map Magnets/Solenoid/solenoid_300A_poisson_20140819 , more study is needed to see if the straight line track fitter is needed or not: 1036 - 1053, 1065 - 1121, 1212? - 1254, 1309 - 1318 [1307,1308,1319-1329 were taken while the magnet was ramping].<br />
--><br />
<br />
=== General Information on Procedures ===<br />
This section explains how the offline monitoring should be run.<br />
Since we may want to simultaneously run offline monitoring for different run periods that require<br />
different environment variables, the scripts are set up so that a generic user can download the<br />
scripts and run them from anywhere. Most output directories for offline monitoring are created<br />
with group read/write permissions so that any Hall D group user has access to the contents,<br />
but there are some cases where use of the account that created the launch is necessary.<br />
<br />
The accounts used for offline monitoring are the gxprojN accounts created and maintained by<br />
Mark Ito (see [https://halldweb.jlab.org/wiki/index.php/GlueX-related_shared_accounts_on_the_JLab_CUE here] for how each account is used).<br />
As of October 2015, the following are used:<br />
* gxproj1 for running over Fall 2014 data (deprecated since June 2015)<br />
* gxproj5 for running over Spring 2015 data<br />
<br />
Since the summer of 2015 we have transitioned from a system using Mark Ito's jproj scripts<br />
to integrating the swif system that Chris Larrieu (SciComp) has been developing. For offline monitoring,<br />
the hdswif system that Kei developed is used for launching the jobs, and the jproj system is used<br />
for meta-analysis of launch statistics.<br />
<br />
Both hdswif and jproj are maintained in svn:<br />
* hdswif: https://halldsvn.jlab.org/repos/trunk/scripts/monitoring/hdswif<br />
* jproj : https://halldsvn.jlab.org/repos/trunk/scripts/monitoring/jproj<br />
<br />
To run the offline monitoring each package should be checked out and all necessary<br />
scripts are included.<br />
<br />
=== Preparing the software for the launch ===<br />
<br />
To begin a new launch the software must be built to the latest versions.<br />
For the gxprojN user accounts used, all software builds are contained in the directory ~/builds<br />
(which are soft links to /work/halld/home/gxprojN/builds). When logging into these accounts<br />
the setup files ~/setup_jlab-2015-03.csh or similar files should be sourced.<br />
<br />
<b>Note that Mark Ito does <u>not</u> want you to change the contents of each .cshrc file.</b><br />
You should consult him if you feel the need.<br />
<br />
<span style="color:red">NOTE: FOR BUILDING SOFTWARE IT IS A GOOD IDEA TO DO A COMPLETE WIPEOUT/CHECKOUT EACH TIME TO AVOID STALE HEADER FILES.</span><br />
<br />
# Building hdds: Go to ~/builds/hdds. The directory hdds is the one from git. Delete the contents, then download the newest version from git and build: <pre>cd ~/builds/hdds/</pre> <pre>rm -frv hdds</pre> <pre>git clone https://github.com/JeffersonLab/hdds</pre> <pre>scons install</pre><br />
# Building sim-recon: Go to ~/builds/sim-recon. The directory sim-recon is the one from git. Delete the contents, then download the newest version from git and build: <pre>cd ~/builds/sim-recon</pre> <pre>rm -frv sim-recon</pre> <pre>git clone https://github.com/JeffersonLab/sim-recon</pre> <pre>scons install -j8</pre><br />
# Prepare the latest sqlite file: The sqlite is set in the ~/setup_jlab-2015-03.csh script as <b>sqlite:////home/gxproj5/ccdb.sqlite</b> through the environment variables <b>JANA_CALIB_URL</b> and <b>CCDB_CONNECTION</b>. Therefore, go to this directory and create a new sqlite file. We create the sqlite file in a temporary directory since creating the sqlite file in a directory where the output file exists causes errors. Original documentation on creating sqlite files are [https://halldweb.jlab.org/wiki/index.php/SQLite-form_of_the_CCDB_database here]. <br><span style="color:red">NOTE: SQLITE FILES DO NOT WORK ON THE NEW /work DISK INSTALLED IN OCTOBER 2015 </span> <pre>cd ~/tmp</pre> <pre>$CCDB_HOME/scripts/mysql2sqlite/mysql2sqlite.sh -hhallddb.jlab.org -uccdb_user ccdb | sqlite3 ccdb.sqlite</pre> <pre>mv ccdb.sqlite ../</pre><br />
# Note that the above steps must be done <b>BEFORE</b> launch project creation. This is because we will track the revisions of the libraries used, and this is done by extracting the svn information in each directory. Also, note that the system assumes that we have the topmost build directory (usually called GLUEX_TOP) to be $HOME/builds . Such an assumption is necessary to be able to extract information about the library locations automatically.<br />
<br />
Create the appropriate project(s) and submit the jobs using hdswif, as detailed in the section below.<br />
<br />
=== Starting the Launch and Submitting Jobs ===<br />
<br />
Until the summer of 2015 we relied solely on Mark Ito's jproj system for submitting and keeping track of jobs. We have since moved to the swif system and use the hdswif wrapper for this. Below are instructions for how to use these.<br />
<br />
# Downloading hdswif: Download the hdswif directory from svn. For the gxprojN accounts, use the directory ~/halld/hdswif. <pre>cd ~/halld </pre> <pre> svn co https://halldsvn.jlab.org/repos/trunk/scripts/monitoring/hdswif</pre> <pre>cd hdswif</pre><br />
# Creating the workflow: Within SWIF jobs are registered into workflows. First create the workflow. For offline monitoring, the workflow names are of the form <b>offline_monitoring_RunPeriod201Y_MM_verVV_hd_rawdata</b> with suitable replacements for the run period and version number. The command "swif list" will list all existing workflows. Also, for most simple SWIF commands hdswif also provides a wrapper. <pre>swif list</pre> For creation of workflows for offline monitoring the command <pre>hdswif.py create [workflow] -c [config file] </pre> should be used. When a config file is passed in, hdswif will automatically create files that record the configuration of the current launch. These files are stored as for example<br />
#* /group/halld/data_monitoring/run_conditions/jana_rawdata_comm_2015_03_ver15.conf<br />
#* /group/halld/data_monitoring/run_conditions/soft_comm_2015_03_ver15.xml<br />
#* The software packages stored in git (sim-recon and hdds) can have git tags applied to them, which makes it easier to find versions of the software than a SHA-1 hash. hdswif will ask if you would like to create a tag, and execute the following sequence: <pre>git tag -a offmon-201Y_MM-verVV -m "Used for offline monitoring 201Y-MM verVV started on 201y/mm/dd"</pre> <pre>git push offmon-201Y_MM-verVV</pre> This will only be invoked when the user name is gxprojN, and for the configuration files, the output directory will be /group/halld/data_monitoring/run_conditions/ for gxprojN accounts while it will be the current directory for other users.<br />
#* To use the git-tagged software versions do for example <pre>cd $HALLD_HOME</pre> <pre>git checkout offmon-2015_03-ver15</pre><br />
# Registering jobs in the workflow: To register jobs within the workflow, hdswif provides the use of config files. Jobs can be registered by specifying the workflow, config file (-c), run (-r) and file (-f) numbers if necessary. A typical config file will look this:<br />
PROJECT gluex<br />
TRACK reconstruction<br />
OS centos65<br />
NCORES 6<br />
DISK 40<br />
RAM 8<br />
TIMELIMIT 8<br />
JOBNAMEBASE offmon_<br />
RUNPERIOD 2015-03<br />
VERSION 15<br />
OUTPUT_TOPDIR /volatile/halld/offline_monitoring/RunPeriod-[RUNPERIOD]/ver[VERSION] # Example of other variables included in variable<br />
SCRIPTFILE /home/gxproj5/halld/hdswif/script.sh # Must specify full path<br />
ENVFILE /home/gxproj5/halld/hdswif/setup_jlab-2015-03.csh # Must specify full path<br />
The config file contains configuration parameters for each of the jobs.<br><br />
<span style="color:red">Note: Job configuration parameters can be set differently for jobs within the same workflow if necessary.</span><br />
Edit the config file and save as a new file if necessary. Once the configuration is set, jobs can be added via <pre>hdswif.py add [workflow] -c input.config</pre><br />
By default, hdswif will add all files found within the directory /mss/halld/RunPeriod-201Y-MM/rawdata/ where 201Y-MM is specified by the RUNPERIOD parameter in the config file. If only some of the runs or files are needed, these can be specified for example with<br />
<pre>hdswif.py add [workflow] -c input.config -r 3180 -f '00[0-4]'</pre><br />
to specify to register running only over run 3180 files 000 - 004 (Unix-style brackets and wildcards can be used).<br />
<br />
Running the workflow: To run the workflow, simply use swif run: <pre>swif run </pre> or equivalently, using the hdswif wrapper, <pre>hdsswif.py run [workflow]</pre><br><br />
<b>It is recommended that some jobs be tested to make sure that everything is working rather than fail thousands of jobs.</b><br> For this purpose, hdswif will take an additional parameter to run which limits the number of jobs to submit: <pre>hdswif.py run [workflow] 10</pre><br />
in which case only 10 jobs will be submitted. To submit all jobs after checking the results, do <pre>hdswif.py run [workflow]</pre><br />
<br />
=== Post-analysis of statistics of the launch ===<br />
<br />
# After jobs have been submitted, it will usually take a few days for all of the jobs to be processed.<br />
#* Status of Auger: http://scicomp.jlab.org/scicomp/#/auger/jobs (see also links above)<br />
#* Status of user jobs: <pre>jobstat -u [user name]</pre><br />
# The status and results of jobs are saved within the SWIF internal server, and are available via the command <pre>swif status [workflow] -summary -runs</pre> where the arguments -summary and -runs show summary statistics and statistics for individual jobs, respectively. hdswif has a command that takes this output in XML output and creates an HTML webpage showing results of the launch. To do this, do <pre>hdswif.py summary [workflow]</pre> This will create an XML file swif_output_[workflow].xml that contains all information from SWIF. If the file already exists, hdswif will ask whether to overwrite the existing file.<br />
# At this stage the html output and figure files are created and ready to be put online. For this step and other steps involving analysis of the statistics of the launch results, it is convenient to change to the jproj system.<br />
# The jproj scripts for offline monitoring are maintained in the svn directory https://halldsvn.jlab.org/repos/trunk/scripts/monitoring/jproj Do <pre> svn co https://halldsvn.jlab.org/repos/trunk/scripts/monitoring/jproj </pre> For the gxprojN accounts used for offline monitoring the directory should be ~/halld/jproj<br />
# The jproj directory contains two subdirectories, scripts and projects. The scripts directory contains useful scripts for processing the jobs registered in the jproj system, and each of the offline monitoring launches will be handled in the projects directory.<br />
# Go to the projects directory<pre>cd ~/halld/jproj/projects</pre> and use the script create_project.sh to create a new directory that contains the processing scripts for the current launch <pre>./create_project.sh [workflow]</pre> This should create a directory such as offline_monitoring_RunPeriod2015_03_ver15_hd_rawdata (same as the workflow name). The script uses the template files in the directory templates and by substitution creates script files for the current launch. Now go to the newly created analysis directory: <pre>cd [workflow]/analysis</pre><br />
which for the gxprojN accounts should have the full path /home/gxprojN/halld/jproj/projects/[workflow]/analysis.<br><br><br />
<span style="color:red">All of the analysis commands including arguments are contained in run_analysis.sh, but it is <b>strongly</b> recommended that all commands are run manually to check for errors.</span><br><br><br />
# The first thing to do is to make the html output from hdswif public. Copy the html file and related figures that were created from hdswif to the appropriate space within /group/halld/www/halldweb/html/data_monitoring/ : <pre>python publish_offmon_results.py [run period] [version]</pre> Note that the command with appropriate substitutions for arguments can be found within run_analysis.sh (same for all commands below).<br />
# Next, we need to create a few MySQL tables for the current launch. The MySQL tables are useful for comparing run/file combinations for different launches. For the SWIF launches, there are two tables needed, and will be named<br />
#* [workflow]Job<br />
#* [workflow]_aux<br />
# This naming scheme of tables and their roles are the same as from the jproj only launches. The [workflow]Job table will contain information gathered from SWIF about each job (which node it went to, start time of each stage, memory usage, etc.). The [workflow]_aux table will contain information gathered from the output stdout files from each job. First, create the Job table using <pre>python create_jproj_job_table.py [run period] [version]</pre> Note that this script uses the XML output from hdswif summary and inserts the contents into the MySQL table, so the XML output file must exist.<br />
# To check the contents of this MySQL table, do <pre>mysql -hhalldb -ufarmer farming</pre> <pre>mysql> describe offline_monitoring_RunPeriod2015_03_ver11_hd_rawdataJob;</pre><br />
<br />
== Hall D Job Management System ==<br />
<br />
This section details instructions on how to create and launch a set of jobs using the Hall-D Job Management System developed by Mark Ito. These instructions are generic: this system can be used for the weekly monitoring jobs, but can also be used for other sets of job launches as well. <br />
<br />
=== Database Table Overview ===<br />
<br />
* Job management database table (<project_name>): For each input file, keeps track of whether or not a job for it has been submitted, along with other optional fields. <br />
<br />
* Job status database table (<project_name>Job (no space)): For each job, keeps track of the job-id, the job status, memory used, cpu & wall time, time taken to complete various stages (e.g. pending, dependency, active), and others. <br />
<br />
* Job metrics database table (<project_name>_aux (no space)): For each job, keeps track of the job-id, how many events were processed, the time it took to copy the cache file, and the time it took to run the plugin. This information is culled from the log files of each job, and is done within the analysis directory of each launch.<br />
<br />
=== Initialize Project Management ===<br />
<br />
* Log into the ifarm machine with one of the gxproj accounts. For this example we will use gxproj1.<br />
<pre><br />
ssh gxproj1@ifarm -Y<br />
</pre><br />
<br />
* Go to a directory to do the launch. In principle, any directory will work, but for gxproj1 this is usually done in /home/gxproj1/halld/<br />
<br />
* Check out the necessary scripts <pre>svn co https://halldsvn.jlab.org/repos/trunk/scripts/monitoring/jproj</pre> This will get all necessary scripts for launching. Once checked out, <pre>cd projects</pre><br />
<br />
* The script create_project.sh can be used to create a new project. It will take a single argument, the project name. It is assumed that the project name is of the form offline_monitoring_RunPeriod20YY_MM_verVV_hd_rawdata where this string will be parsed to give the run period 20YY_MM and version number VV. One thing to do BEFORE creation of a new project is to editthe conditions of the launch (plugins to run over, memory requested, disk space requested) within templates/template.jsub . This<br />
information is saved automatically at project creation time into files in /group/halld/data_monitoring/run_conditions .<br />
To create a project do <pre>./create_project.sh offline_monitoring_RunPeriod20YY_MM_verVV_hd_rawdata</pre><br />
The name has been chosen to be as consistent as possible with other directory structures.<br />
However, mysql requires that "-" be escaped in table names, so unfortunately run periods will be<br />
given as 2YYY_MM instead of 2YYY-MM.<br />
<br />
* A new directory with the project name (offline_monitoring_RunPeriod20YY_MM_verVV_hd_rawdata) and files will be copied and modified from the template directory to reflect the run period, the user, the directory that it was created in, project name, etc.<br />
<br />
* For each project, cd into the new directory<br />
* The script clear.sh will remove any existing tables for the current project name, then recreate it. Do<br />
<pre>./clear.sh</pre><br />
* To use the jproj.pl script that was checked in, add the directory to your path with<br />
<pre>source ../../scripts/setup.csh</pre><br />
or always specify the full path<br />
<pre>../../scripts/jproj.pl</pre><br />
* Now update the table of runs with<br />
<pre>jproj.pl <project name> update</pre><br />
This will fill the table with all files within /mss that are of the same form as what is in <project name>.jproj . If you want to register only on a subset of all such files, you can edit this file directly.<br />
* Once you have registered all of the files you would like to run over, do<br />
<pre>jproj.pl <project name> submit [max # of jobs] [run number]</pre><br />
where the additional options specify how many jobs to submit and which run number to run on. Without these options all files that are registered and have not been submitted yet will be submitted.<br />
<br />
At this stage you are ready to submit all files. It is a good idea to submit a few test jobs<br />
at first to check that all scripts are working and that the plugins do not crash. Once you are<br />
sure that this does not happen, you can send all jobs in. The remaining jobs are then the monitoring<br />
which will (among other things) put the results on the online webpage for the collaboration to view, and<br />
the analysis of the launch.<br />
<br />
=== Project File Overview ===<br />
<br />
An overview of each project file:<br />
* '''clear.sh:''' For the current project, deletes the job status and management database tables (if any), and creates new, empty ones. <br />
* '''<project_name>.jproj:''' Contains the path and file name format for the input files for the jobs. <br />
* '''<project_name>.jsub:''' The xml job submission script. The run number and file number variables are set during job submission for each input file. <br />
* '''script.sh:''' The script that is executed during the job. If output job directories are not pre-created manually, they should be created in this script with the proper permissions:<br />
<pre><br />
mkdir -p -m 775 my_directory<br />
</pre><br />
* '''setup_jlab-[run period].csh:''' The environment that is sourced at the beginning of the job execution. <br />
* '''status.sh:''' Updates the job status database table, and prints some of its columns to screen.<br />
<br />
=== Project Management ===<br />
<br />
* Delete (if any) and create the database table(s) for the current set of job submissions:<br />
<pre><br />
./clear.sh<br />
</pre><br />
Also, if testing was done with jobs, it is best to delete the output directory and the configuration files:<br />
<pre><br />
rm -frv /volatile/halld/offline_monitoring/RunPeriod-20YY-MM/verVV /group/halld/data_monitoring/run_conditions/soft_comm_20YY_MM_verVV.xml /group/halld/data_monitoring/run_conditions/jana_rawdata_comm_20YY_MM_verVV.conf<br />
</pre><br />
<br />
* Search for input files matching the string in the .jproj file, and create a row for each in the job management database table (called <project_name>). You can test by adding an optional argument at the end, which only selects files with a specific file number:<br />
<pre><br />
jproj.pl <project_name> update <optional_file_number><br />
</pre><br />
<br />
* Confirm that the job management database is accurate by printing it's contents to screen:<br />
<pre><br />
mysql -hhallddb -ufarmer farming -e "select * from <project_name>"<br />
</pre><br />
<br />
* ONLY if a mistake was made, to delete the tables from the database and recreate new, empty ones, run: <br />
<pre><br />
./clear.sh<br />
</pre><br />
<br />
* To look at the status of the submitted jobs, first query auger and update the job status database:<br />
<pre><br />
fill_in_job_details.pl <project_name><br />
</pre><br />
<br />
* The job status can then be viewed by submitting a query to the job status database (called <project_name>Job (no space in between)):<br />
<pre><br />
mysql -hhallddb -ufarmer farming -e "select id,run,file,jobId,hostname,status,timeSubmitted,timeActive,walltime,cput,timeComplete,result,error from <project_name>Job"<br />
</pre><br />
<br />
* These last two commands can instead be executed simultaneously by running:<br />
<pre><br />
./status.sh<br />
</pre><br />
<br />
=== Handy mysql Instructions ===<br />
<br />
* Handy mysql instructions:<br />
<pre><br />
mysql -hhallddb -ufarmer farming # Enter the "farming" mysql database on "hallddb" as user "farmer"<br />
quit; # Exit mysql<br />
show tables; # Show a list of the tables in the current database<br />
show columns from <project_name>; # show all of the columns for the given table<br />
select * from <project_name>; # show the contents of all rows from the given table<br />
</pre><br />
<br />
=== Backing Up Offline Monitoring Tables ===<br />
Tables created for offline monitoring can be backed up using the script backup_tables.sh which<br />
can be checked out with the other files from https://halldsvn.jlab.org/repos/trunk/scripts/monitoring/jproj/projects<br />
<br />
The script uses the command mysqldump to print out a file that can be executed to recreate the tables.<br />
Since executing this output file will drop the table when it exists, caution is advised.<br />
Example usage to backup all three tables created for run period 2014_10 ver 17:<br />
<pre>backup_tables.sh 2014_10 17</pre><br />
<br />
== Running Over Data As It Comes In ==<br />
<br />
A special user gxproj1 will have a cron job set up to run the plugins as new data appears on /mss.<br />
During the week, gxproj1 will submit offline plugin jobs with the same setup as the weekly jobs<br />
run the previous Friday. The procedure for this is shown below.<br />
<br />
<!--<br />
=== Setting up the environment ===<br />
The file<br />
/home/gxproj1/setup_jlab.csh<br />
is sourced through .tcshrc.<br />
This file is the same as what is linked to by<br />
/home/gluex/setup_jlab_commissioning.csh,<br />
except HALLD_HOME, HDDS_HOME, and JANA_CALIB_URL are set separately so that this<br />
user can have a separate build.<br />
<br />
To obtain the builds from the previous Friday's runs,<br />
execute<br />
/home/gxproj1/halld/monitoring/newruns/setup_previous.sh [year] [month] [day]<br />
The build revisions from the previous Friday are archived in files<br />
/work/halld/data_monitoring/run_conditions/soft_comm_[year]_[month]_[day].xml<br />
and the script will build libraries based on those stored revision numbers.<br />
--><br />
<br />
=== Running the cron job ===<br />
<br />
'''IMPORTANT:''' The cron job should not be running while you are manually submitting jobs using the jproj.pl script for the same project, or else you will probably multiply-submit a job. <br />
<br />
* Go to the cron job directory: <br />
<pre><br />
cd /u/home/gxproj1/halld/monitoring/newruns<br />
</pre><br />
<br />
* The cron_plugins file is the cronjob that will be executed. During execution, it runs the exec.sh command in the same folder. This command takes two arguments: the project name, and the maximum file number for each run. These fields should be updated in the cron_plugins file before running. <br />
<br />
* The exec.sh command updates the job management database table with any data that has arrived on tape since it was last updated, ignoring file numbers greater than the maximum file number. It then submits jobs for these files. <br />
<br />
* To start the cron job, run:<br />
<pre><br />
crontab cron_plugins<br />
</pre><br />
<br />
* To check whether the cron job is running, do<br />
<pre><br />
crontab -l<br />
</pre><br />
<br />
* To remove the cron job do<br />
<pre><br />
crontab -r<br />
</pre><br />
<br />
<!--<br />
The cron job will run the script scan_for_jobs.sh,<br />
which runs generatejobs_plugins_rawdata.sh for any<br />
new runs that it had not seen before. All previous<br />
runs are recorded in the file filelists/files_current.txt<br />
so clear this to run over runs, or set the parameters<br />
MINRUN and MAXRUN which will set the range of runs submitted.<br />
--><br />
<br />
==Post-Processing Procedures==<br />
<br />
To visualize the monitoring data, we save images of selected histograms and store time series of selected quantities in a database, which are then displayed on a web page. This section describes how to generate the monitoring images and database information.<br />
<br />
The scripts used to generate this summary data are primarily run from /home/gxprojN/halld/monitoring/process . If you want a new copy of the scripts, e.g., for a new monitoring run, you should check the scripts out from SVN:<br />
<syntaxhighlight><br />
svn co https://halldsvn.jlab.org/repos/trunk/scripts/monitoring/process<br />
</syntaxhighlight><br />
Note that these scripts currently have some parameters which must be periodically set by hand.<br />
<br />
The default python version on most JLab machine does not have the modules to allow these scripts to connect to the MySQL database. To run these scripts, load the environment with the following command<br />
<syntaxhighlight><br />
source /home/gxproj1/halld/monitoring/process/monitoring_env.csh<br />
</syntaxhighlight><br />
<br />
===Online Monitoring===<br />
<br />
There are two primary scripts for running over the monitoring data generated by the online system and offline reconstruction. The online script can be run with either of the following commands:<br />
<syntaxhighlight><br />
/home/gluex/halld/monitoring/process/check_new_runs.py<br />
<br />
OR <br />
<br />
/home/gluex/halld/monitoring/process/check_new_runs.csh<br />
</syntaxhighlight><br />
The shell script sets up the environment properly to run the python script. To connect to the monitoring database on the JLab CUE, modules included in the local installation of python >= 2.7 are needed. The shell script is appropriate to use in a cron job. The cronjob is currently run under the "gluex" account.<br />
<br />
The online monitoring system copies a ROOT file containing the results of the online monitoring, and other configuration files into a directory accessible outside the counting house. This python script automatically checks for new ROOT files, which it will then automatically process. It contains several configuration variables that must be correctly set, which contains the location of input/output directories, etc...<br />
<br />
===Offline Monitoring===<br />
<br />
After the data is run over, the results should be processed, so that summary data is entered into the monitoring database and plots are made for the monitoring webpages. Currently, this processing is controlled by a cronjob that runs the following script:<br />
<syntaxhighlight><br />
/home/gxproj1/halld/monitoring/process/check_monitoring_data.csh <br />
</syntaxhighlight><br />
This script checks for new ROOT files, and only runs over those it hasn't processed yet. Since one monitoring ROOT file is produced for each EVIO file, whenever a new file is produced, the plots for the corresponding run are recreated and all the ROOT files for a run are combined into one file. Information is stored in the database on a per-file basis. <br />
<br />
Plots for the monitoring web page can be made from single histograms or multiple histograms using RootSpy macros. If you want to change the list of plots made, you must modify one of the following files:<br />
* histograms_to_monitor - specify either the name of the histogram or its the full ROOT path <br />
* macros_to_monitor - specify the full path to the RootSpy macro .C file<br />
<br />
When a new monitoring run is started, or the conditions are changed, the following steps should be taken to process the new files:<br />
# Add a new data version, as described below:<br />
# Change the following parameters in check_monitoring_data.csh:<br />
## JOBDATE should correspond to the ouptut date used by the job submission script<br />
## OUTPUTDIR should correspond to the directory corresponding to the run period and revision corresponding to the new version you just submitted. Presumably, this directory will be empty at the beginning.<br />
## Once you create a new data version as defined below, you should pass the needed information as a command line option. Currently this is done by the ARGS variable. For example, the argument "-v RunPeriod-2014-10,8" tells the monitoring scripts to look up the version corresponding to revision 8 of RunPeriod-2014-10 in the monitoring DB and to use to store the results.<br />
<br />
<syntaxhighlight><br />
Example configuration parameters:<br />
set JOBDATE=2015-01-09<br />
set INPUTDIR=/volatile/halld/RunPeriod-2014-10/offline_monitoring<br />
set OUTPUTDIR=/w/halld-scifs1a/data_monitoring/RunPeriod-2014-10/ver08<br />
set ARGS=" -v RunPeriod-2014-10,8 "<br />
</syntaxhighlight><br />
If you want to process the results manually, the data is processed using the following script:<br />
<syntaxhighlight><br />
./process_new_offline_data.py <input directory> <output directory><br />
<br />
EXAMPLE:<br />
<br />
./process_new_offline_data.py 2014-11-14 /volatile/halld/RunPeriod-2014-10/offline_monitoring/ /w/halld-scifs1a/data_monitoring/RunPeriod-2014-10/ver02<br />
</syntaxhighlight><br />
The python script takes several options to enable/disable various steps in the processing. Of interest is the "--force" option, which will run over all monitoring ROOT files, whether or not they've been previously identified.<br />
<br />
Every time a new reconstruction pass is performed, a new version number must be generated. To do this, prepare a version file as described below. Then run the register_new_version.py script to store the information in the database. The script will return a version number, which then should be set by hand in process_new_offline_data.py - future versions of the script will streamline this part of the procedure. An example of how to generate a new version is:<br />
<syntaxhighlight><br />
./register_new_version.py add /home/gxproj1/halld/monitoring/process/versions/vers_RunPeriod-2014-10_pass1.txt<br />
</syntaxhighlight><br />
<br />
<b>If you are running the offline monitoring by checking out the files in trunk/scripts/monitoring/jproj/projects/<br />
of the svn repository</b>, and created a project with<br />
create_project.sh [project name] hd_rawdata<br />
Then go to the directory [project name]/processing/<br />
and execute<br />
./run_processing.sh<br />
which will run register_new_version.py as well as check_monitoring_data.csh for that project.<br />
<br />
===Step-by-Step Instructions For Processing a New Monitoring Run===<br />
<br />
The monitoring runs are current run out of the gxproj1 and gxproj5 accounts. After an offline monitoring run has been successfully started on the batch farm, the following steps should be followed to setup the post-processing for these runs.<br />
<br />
# The post-processing scripts are stored in $HOME/halld/monitoring/process and are automatically run by cron.<br />
# Run "svn update" to bring any changes in. Be sure that the list of histograms and macros to plot are current.<br />
# Edit check_monitoring_data.csh to point to the current revisions/directories<br />
#* VERSION<br />
#* ARGS<br />
#* Note that the environment depends on a standard script - $HOME/setup_jlab.csh<br />
# Update files in the web directory, so that the results are displayed on the web pages: /group/halld/www/halldweb/html/data_monitoring/textdata<br />
# Copy the REST files to more permanent locations:<br />
#* cp -a /volatile/halld/offline_monitoring/RunPeriod-YYYY-MM/verVV/REST /work/halld/data_monitoring/RunPeriod-YYYY-MM/REST/verVV<br />
#* cp -a /volatile/halld/offline_monitoring/RunPeriod-YYYY-MM/verVV/REST /cache/halld/RunPeriod-YYYY-MM/REST/verVV [under testing]<br />
<br />
Check log files in $HOME/halld/monitoring/process/log for more information on how each run went. If there are problems, check log files, and modify check_monitoring_data.csh to vary the verbosity of the output.<br />
<br />
==Data Versions==<br />
<br />
To document the conditions of the monitoring data that is created, for the sake of reproducability and further analysis we save several pieces of information. The format is intended to be comprehensive enough to document not just monitoring data, but versions of raw and reconstructed data, so that this database table can be used for the event database as well.<br />
<br />
We store one record per pass through one run period, with the following structure:<br />
<br />
{| class="wikitable"<br />
! Field !! Description<br />
|-<br />
| data_type || The level of data we are processing. For the purposes of monitoring, "rawdata" is the online monitoring, "recon" is the offline monitoring<br />
|- <br />
| run_period || The run period of the data<br />
|- <br />
| revision || An integer specifying which pass through the run period this data corresponds to<br />
|- <br />
| software_version || The name of the XML file that specifies the different software versions used<br />
|-<br />
| jana_config || The name of the text file that specifies which JANA options were passed to the reconstruction program<br />
|-<br />
| ccdb_context || The value of JANA_CALIB_CONTEXT, which specifies the version of calibration constants that were used<br />
|-<br />
| production_time || The data at which monitoring/reconstruction began<br />
|-<br />
| dataVersionString || A convenient string for identifying this version of the data<br />
|}<br />
<br />
<br />
An example file used as as input to ./register_new_version.py is:<br />
<syntaxhighlight><br />
data_type = recon<br />
run_period = RunPeriod-2014-10<br />
revision = 1<br />
software_version = soft_comm_2014_11_06.xml<br />
jana_config = jana_rawdata_comm_2014_11_06.conf<br />
ccdb_context = calibtime=2014-11-10<br />
production_time = 2014-11-10<br />
dataVersionString = recon_RunPeriod-2014-10_20141110_ver01<br />
</syntaxhighlight></div>Kmoriyahttps://halldweb.jlab.org/wiki/index.php?title=Data_Monitoring_Procedures&diff=70815Data Monitoring Procedures2015-10-21T18:34:07Z<p>Kmoriya: /* Preparing the software for the launch */</p>
<hr />
<div>__TOC__<br />
<br />
== Master List of File / Database / Webpage Locations ==<br />
=== Run Conditions ===<br />
* Online Run-by-run condition files (B-field, current, etc.): /work/halld/online_monitoring/conditions/<br />
* Offline monitoring run conditions (software versions, jana config): /group/halld/data_monitoring/run_conditions/<br />
*[http://www.jlab.org/Hall-D/test/RunInfo/ Run Info vers. 1]<br />
*[https://halldweb.jlab.org/cgi-bin/data_monitoring/run_conditions.pl Run Info vers. 2]<br />
<br />
=== Monitoring Output Files ===<br />
* Run Periods 201Y-MM is for example 2015-03, launch ver verVV is for example ver15<br />
* Online monitoring histograms: /work/halld/online_monitoring/root/<br />
* Offline monitoring histogram ROOT files: /work/halld/data_monitoring/RunPeriod-201Y-MM/verVV/rootfiles<br />
* REST files (most recent launch only): /work/halld/data_monitoring/RunPeriod-201Y-MM/REST/verVV<br />
* individual files for each job (ROOT, log, etc.): /volatile/halld/offline_monitoring/RunPeriod-201Y-MM/verVV/<br />
<br />
=== Monitoring Database ===<br />
* Accessing monitoring database (on ifarm): mysql -u datmon -h hallddb.jlab.org data_monitoring<br />
<br />
=== Monitoring Webpages ===<br />
*[https://halldweb.jlab.org/cgi-bin/data_monitoring/monitoring/plotBrowser.py Plot Browser]<br />
*[https://halldweb.jlab.org/cgi-bin/data_monitoring/monitoring/runBrowser.py Run Browser]<br />
*[https://halldweb.jlab.org/cgi-bin/data_monitoring/monitoring/timeSeries.py Time Series]<br />
*[https://halldweb.jlab.org/data_monitoring/launch_analysis/ Launch Analysis]<br />
<br />
== Job Monitoring Links ==<br />
* [http://scicomp.jlab.org/farm2/job.html Completed Job History]<br />
* [http://scicomp.jlab.org/farm2/project.html Job Stats By Project]<br />
* [http://scicomp.jlab.org/farm2/trackOrg.html Job Stats By Track]<br />
* [http://scicomp.jlab.org/farm2/report.html Cluster Report]<br />
* [http://scicomp.jlab.org/farm2/walltime.html Walltime Distribution]<br />
<br />
== Saving Online Monitoring Data ==<br />
<br />
The procedure for writing the data out is given in, e.g.,<br />
[https://halldweb.jlab.org/wiki/index.php/Raid-to-Silo_Transfer_Strategy Raid-to-Silo Transfer Strategy].<br />
<br />
Once the DAQ writes out the data to the raid disk, cron jobs will copy the file to tape,<br />
and within ~20 min., we will have access to the file on tape at<br />
/mss/halld/$RUN_PERIOD/rawdata/RunXXXXXX.<br />
<br />
All online monitoring plugins will be run as data is taken.<br />
They will be accessible within the counting house via RootSpy, and<br />
for each run and file, a ROOT file containing the histograms will be saved<br />
within a subdirectory for each run.<br />
<br />
For immediate access to these files, the raid disk files may be accessed directly<br />
from the counting house, or the tape files will be available within ~20 min. of the<br />
file being written out.<br />
<br />
== Offline Monitoring: Running Over Archived Data ==<br />
<br />
Once files are written to tape we run the online plugins on these files to confirm what we were seeing in the online monitoring, and also to update the results from the latest calibration and software. <br />
Manual scripts and cron jobs are set up to look for new data and run the plugins over a sample of files.<br />
<br />
Every other Friday (usually the Friday before the offline meetings) jobs will be started to run the newest software on all previous runs,and allows everybody to see improvements in each detector.<br />
For each launch, independent builds of hdds, sim-recon, the monitoring plugins, and an sqlite file will be generated.<br />
<br />
Below the procedures are described for<br />
# Preparing the software for the launch<br />
# Starting the launch (using hdswif)<br />
# Post-analysis of statistics of the launch<br />
<br />
Processing the results and making them available to the collaboration<br />
is handled in the section [[#Post-Processing_Procedures | Post-Processing Procedures]] below.<br />
<br />
<!--<br />
==== Generating an offline plugin job ====<br />
<br />
The user gxproj1 should be used for official offline monitoring jobs.<br />
Within /home/gxproj1/halld/monitoring/batch/ there will be scripts to run the online monitoring plugins over tape files.<br />
The main script is generatejobs_plugins_rawdata.sh, which can be used as<br />
generatejobs_plugins_rawdata.sh [minrun] [maxrun] (minfile) (maxfile)<br />
where minrun, maxrun specify the range of the run #, and minfile, maxfile (optional) specify<br />
the file range for that run.<br />
<br />
This will generate a script run_rawdata_XXXXXX.sh, where the run # has now been formatted to be 6 digits.<br />
Executing this script will send the monitoring plugins job to the Auger batch system.<br />
<br />
There is also a script clean.sh which can be used as<br />
clean.sh XXX<br />
This will clean up all associated files created in association with run XXX.<br />
<br />
Internally, the xml file used to submit the job will be created, and the job to run<br />
will be given within script.sh. All run parameters should be specified in at the beginning<br />
of generatejobs_plugins_rawdata.sh<br />
Since we are running on tape, the tape file will first be copied over to the cache disk, and the job will run<br />
over this cached file.<br />
<br />
==== Using cron to run automatically ====<br />
Within /home/gluex/halld/monitoring/cron/ there is a file cron_plugins<br />
that can be executed via<br />
crontab cron_plugins<br />
This will set up a cron job to call the script scan_for_jobs.sh, which will<br />
check in the rawdata directory and call generatejobs_plugins_rawdata.sh for<br />
any run that is more than 5 min old. The cron job is set up to run every 10 min.<br />
<br />
==== Magnetic Field Settings ====<br />
For tracking, it is necessary to set the correct magnetic field settings.<br />
The below table is now obsolete. Please refer to the [https://halldweb.jlab.org/cgi-bin/data_monitoring/run_conditions.pl GlueX Run Conditions] webpage.<br />
<br />
The actual field settings for each run have not been documented well.<br />
Below are values based on going through entries in the [https://logbooks.jlab.org/book/halld halld log].<br />
<br />
{| border="1" cellpadding="1" style="text-align: center;"<br />
!width="100"| Run #s<br />
!width="150"| Solenoid Current (A)<br />
!width="300"| JANA option<br />
!width="300"| notes<br />
|-<br />
| 940 - 996 || 1000 || -PBFIELD_MAP=Magnets/Solenoid/solenoid_1000A_poisson_20141104 || At run 997, the solenoid started ramping down<br />
|-<br />
| 998 - 1448 || 0 || -PBFIELD_TYPE=NoField -PDEFTAG:DTrackCandidate=StraightLine || Solenoid started ramping up around run 1431. Runs 1432 - 1448 should not be used for tracking. See [https://logbooks.jlab.org/entry/3309039 this] and [https://logbooks.jlab.org/entry/3309092 this] entry.<br />
|-<br />
| 1449 - 1620 || 1200 || -PBFIELD_MAP=Magnets/Solenoid/solenoid_1200A_poisson_20140520 || <br />
|-<br />
|}<br />
<br />
'''Note:''' The following run ranges were taken with a 300A field, and could be analyzed with the magnetic field map Magnets/Solenoid/solenoid_300A_poisson_20140819 , more study is needed to see if the straight line track fitter is needed or not: 1036 - 1053, 1065 - 1121, 1212? - 1254, 1309 - 1318 [1307,1308,1319-1329 were taken while the magnet was ramping].<br />
--><br />
<br />
=== General Information on Procedures ===<br />
This section explains how the offline monitoring should be run.<br />
Since we may want to simultaneously run offline monitoring for different run periods that require<br />
different environment variables, the scripts are set up so that a generic user can download the<br />
scripts and run them from anywhere. Most output directories for offline monitoring are created<br />
with group read/write permissions so that any Hall D group user has access to the contents,<br />
but there are some cases where use of the account that created the launch is necessary.<br />
<br />
The accounts used for offline monitoring are the gxprojN accounts created and maintained by<br />
Mark Ito (see [https://halldweb.jlab.org/wiki/index.php/GlueX-related_shared_accounts_on_the_JLab_CUE here] for how each account is used).<br />
As of October 2015, the following are used:<br />
* gxproj1 for running over Fall 2014 data (deprecated since June 2015)<br />
* gxproj5 for running over Spring 2015 data<br />
<br />
Since the summer of 2015 we have transitioned from a system using Mark Ito's jproj scripts<br />
to integrating the swif system that Chris Larrieu (SciComp) has been developing. For offline monitoring,<br />
the hdswif system that Kei developed is used for launching the jobs, and the jproj system is used<br />
for meta-analysis of launch statistics.<br />
<br />
Both hdswif and jproj are maintained in svn:<br />
* hdswif: https://halldsvn.jlab.org/repos/trunk/scripts/monitoring/hdswif<br />
* jproj : https://halldsvn.jlab.org/repos/trunk/scripts/monitoring/jproj<br />
<br />
To run the offline monitoring each package should be checked out and all necessary<br />
scripts are included.<br />
<br />
=== Preparing the software for the launch ===<br />
<br />
To begin a new launch the software must be built to the latest versions.<br />
For the gxprojN user accounts used, all software builds are contained in the directory ~/builds<br />
(which are soft links to /work/halld/home/gxprojN/builds). When logging into these accounts<br />
the setup files ~/setup_jlab-2015-03.csh or similar files should be sourced.<br />
<br />
<b>Note that Mark Ito does <u>not</u> want you to change the contents of each .cshrc file.</b><br />
You should consult him if you feel the need.<br />
<br />
<span style="color:red">NOTE: FOR BUILDING SOFTWARE IT IS A GOOD IDEA TO DO A COMPLETE WIPEOUT/CHECKOUT EACH TIME TO AVOID STALE HEADER FILES.</span><br />
<br />
# Building hdds: Go to ~/builds/hdds. The directory hdds is the one from git. Delete the contents, then download the newest version from git and build: <pre>cd ~/builds/hdds/</pre> <pre>rm -frv hdds</pre> <pre>git clone https://github.com/JeffersonLab/hdds</pre> <pre>scons install</pre><br />
# Building sim-recon: Go to ~/builds/sim-recon. The directory sim-recon is the one from git. Delete the contents, then download the newest version from git and build: <pre>cd ~/builds/sim-recon</pre> <pre>rm -frv sim-recon</pre> <pre>git clone https://github.com/JeffersonLab/sim-recon</pre> <pre>scons install -j8</pre><br />
# Prepare the latest sqlite file: The sqlite is set in the ~/setup_jlab-2015-03.csh script as <b>sqlite:////group/halld/Users/gxproj5/builds/ccdb.sqlite</b> through the environment variables <b>JANA_CALIB_URL</b> and <b>CCDB_CONNECTION</b>. Therefore, go to this directory and create a new sqlite file. We create the sqlite file in a temporary directory since creating the sqlite file in a directory where the output file exists causes errors. Original documentation on creating sqlite files are [https://halldweb.jlab.org/wiki/index.php/SQLite-form_of_the_CCDB_database here]. <br><span style="color:red">NOTE: SQLITE FILES DO NOT WORK ON THE NEW /work DISK INSTALLED IN OCTOBER 2015, USE THE /group DISK</span> <pre>cd /group/halld/Users/gxproj5/builds/tmp</pre> <pre>$CCDB_HOME/scripts/mysql2sqlite/mysql2sqlite.sh -hhallddb.jlab.org -uccdb_user ccdb | sqlite3 ccdb.sqlite</pre> <pre>mv ccdb.sqlite ../</pre><br />
# Note that the above steps must be done <b>BEFORE</b> launch project creation. This is because we will track the revisions of the libraries used, and this is done by extracting the svn information in each directory. Also, note that the system assumes that we have the topmost build directory (usually called GLUEX_TOP) to be $HOME/builds . Such an assumption is necessary to be able to extract information about the library locations automatically.<br />
<br />
Create the appropriate project(s) and submit the jobs using hdswif, as detailed in the section below.<br />
<br />
=== Starting the Launch and Submitting Jobs ===<br />
<br />
Until the summer of 2015 we relied solely on Mark Ito's jproj system for submitting and keeping track of jobs. We have since moved to the swif system and use the hdswif wrapper for this. Below are instructions for how to use these.<br />
<br />
# Downloading hdswif: Download the hdswif directory from svn. For the gxprojN accounts, use the directory ~/halld/hdswif. <pre>cd ~/halld </pre> <pre> svn co https://halldsvn.jlab.org/repos/trunk/scripts/monitoring/hdswif</pre> <pre>cd hdswif</pre><br />
# Creating the workflow: Within SWIF jobs are registered into workflows. First create the workflow. For offline monitoring, the workflow names are of the form <b>offline_monitoring_RunPeriod201Y_MM_verVV_hd_rawdata</b> with suitable replacements for the run period and version number. The command "swif list" will list all existing workflows. Also, for most simple SWIF commands hdswif also provides a wrapper. <pre>swif list</pre> For creation of workflows for offline monitoring the command <pre>hdswif.py create [workflow] -c [config file] </pre> should be used. When a config file is passed in, hdswif will automatically create files that record the configuration of the current launch. These files are stored as for example<br />
#* /group/halld/data_monitoring/run_conditions/jana_rawdata_comm_2015_03_ver15.conf<br />
#* /group/halld/data_monitoring/run_conditions/soft_comm_2015_03_ver15.xml<br />
#* The software packages stored in git (sim-recon and hdds) can have git tags applied to them, which makes it easier to find versions of the software than a SHA-1 hash. hdswif will ask if you would like to create a tag, and execute the following sequence: <pre>git tag -a offmon-201Y_MM-verVV -m "Used for offline monitoring 201Y-MM verVV started on 201y/mm/dd"</pre> <pre>git push offmon-201Y_MM-verVV</pre> This will only be invoked when the user name is gxprojN, and for the configuration files, the output directory will be /group/halld/data_monitoring/run_conditions/ for gxprojN accounts while it will be the current directory for other users.<br />
#* To use the git-tagged software versions do for example <pre>cd $HALLD_HOME</pre> <pre>git checkout offmon-2015_03-ver15</pre><br />
# Registering jobs in the workflow: To register jobs within the workflow, hdswif provides the use of config files. Jobs can be registered by specifying the workflow, config file (-c), run (-r) and file (-f) numbers if necessary. A typical config file will look this:<br />
PROJECT gluex<br />
TRACK reconstruction<br />
OS centos65<br />
NCORES 6<br />
DISK 40<br />
RAM 8<br />
TIMELIMIT 8<br />
JOBNAMEBASE offmon_<br />
RUNPERIOD 2015-03<br />
VERSION 15<br />
OUTPUT_TOPDIR /volatile/halld/offline_monitoring/RunPeriod-[RUNPERIOD]/ver[VERSION] # Example of other variables included in variable<br />
SCRIPTFILE /home/gxproj5/halld/hdswif/script.sh # Must specify full path<br />
ENVFILE /home/gxproj5/halld/hdswif/setup_jlab-2015-03.csh # Must specify full path<br />
The config file contains configuration parameters for each of the jobs.<br><br />
<span style="color:red">Note: Job configuration parameters can be set differently for jobs within the same workflow if necessary.</span><br />
Edit the config file and save as a new file if necessary. Once the configuration is set, jobs can be added via <pre>hdswif.py add [workflow] -c input.config</pre><br />
By default, hdswif will add all files found within the directory /mss/halld/RunPeriod-201Y-MM/rawdata/ where 201Y-MM is specified by the RUNPERIOD parameter in the config file. If only some of the runs or files are needed, these can be specified for example with<br />
<pre>hdswif.py add [workflow] -c input.config -r 3180 -f '00[0-4]'</pre><br />
to specify to register running only over run 3180 files 000 - 004 (Unix-style brackets and wildcards can be used).<br />
<br />
Running the workflow: To run the workflow, simply use swif run: <pre>swif run </pre> or equivalently, using the hdswif wrapper, <pre>hdsswif.py run [workflow]</pre><br><br />
<b>It is recommended that some jobs be tested to make sure that everything is working rather than fail thousands of jobs.</b><br> For this purpose, hdswif will take an additional parameter to run which limits the number of jobs to submit: <pre>hdswif.py run [workflow] 10</pre><br />
in which case only 10 jobs will be submitted. To submit all jobs after checking the results, do <pre>hdswif.py run [workflow]</pre><br />
<br />
=== Post-analysis of statistics of the launch ===<br />
<br />
# After jobs have been submitted, it will usually take a few days for all of the jobs to be processed.<br />
#* Status of Auger: http://scicomp.jlab.org/scicomp/#/auger/jobs (see also links above)<br />
#* Status of user jobs: <pre>jobstat -u [user name]</pre><br />
# The status and results of jobs are saved within the SWIF internal server, and are available via the command <pre>swif status [workflow] -summary -runs</pre> where the arguments -summary and -runs show summary statistics and statistics for individual jobs, respectively. hdswif has a command that takes this output in XML output and creates an HTML webpage showing results of the launch. To do this, do <pre>hdswif.py summary [workflow]</pre> This will create an XML file swif_output_[workflow].xml that contains all information from SWIF. If the file already exists, hdswif will ask whether to overwrite the existing file.<br />
# At this stage the html output and figure files are created and ready to be put online. For this step and other steps involving analysis of the statistics of the launch results, it is convenient to change to the jproj system.<br />
# The jproj scripts for offline monitoring are maintained in the svn directory https://halldsvn.jlab.org/repos/trunk/scripts/monitoring/jproj Do <pre> svn co https://halldsvn.jlab.org/repos/trunk/scripts/monitoring/jproj </pre> For the gxprojN accounts used for offline monitoring the directory should be ~/halld/jproj<br />
# The jproj directory contains two subdirectories, scripts and projects. The scripts directory contains useful scripts for processing the jobs registered in the jproj system, and each of the offline monitoring launches will be handled in the projects directory.<br />
# Go to the projects directory<pre>cd ~/halld/jproj/projects</pre> and use the script create_project.sh to create a new directory that contains the processing scripts for the current launch <pre>./create_project.sh [workflow]</pre> This should create a directory such as offline_monitoring_RunPeriod2015_03_ver15_hd_rawdata (same as the workflow name). The script uses the template files in the directory templates and by substitution creates script files for the current launch. Now go to the newly created analysis directory: <pre>cd [workflow]/analysis</pre><br />
which for the gxprojN accounts should have the full path /home/gxprojN/halld/jproj/projects/[workflow]/analysis.<br><br><br />
<span style="color:red">All of the analysis commands including arguments are contained in run_analysis.sh, but it is <b>strongly</b> recommended that all commands are run manually to check for errors.</span><br><br><br />
# The first thing to do is to make the html output from hdswif public. Copy the html file and related figures that were created from hdswif to the appropriate space within /group/halld/www/halldweb/html/data_monitoring/ : <pre>python publish_offmon_results.py [run period] [version]</pre> Note that the command with appropriate substitutions for arguments can be found within run_analysis.sh (same for all commands below).<br />
# Next, we need to create a few MySQL tables for the current launch. The MySQL tables are useful for comparing run/file combinations for different launches. For the SWIF launches, there are two tables needed, and will be named<br />
#* [workflow]Job<br />
#* [workflow]_aux<br />
# This naming scheme of tables and their roles are the same as from the jproj only launches. The [workflow]Job table will contain information gathered from SWIF about each job (which node it went to, start time of each stage, memory usage, etc.). The [workflow]_aux table will contain information gathered from the output stdout files from each job. First, create the Job table using <pre>python create_jproj_job_table.py [run period] [version]</pre> Note that this script uses the XML output from hdswif summary and inserts the contents into the MySQL table, so the XML output file must exist.<br />
# To check the contents of this MySQL table, do <pre>mysql -hhalldb -ufarmer farming</pre> <pre>mysql> describe offline_monitoring_RunPeriod2015_03_ver11_hd_rawdataJob;</pre><br />
<br />
== Hall D Job Management System ==<br />
<br />
This section details instructions on how to create and launch a set of jobs using the Hall-D Job Management System developed by Mark Ito. These instructions are generic: this system can be used for the weekly monitoring jobs, but can also be used for other sets of job launches as well. <br />
<br />
=== Database Table Overview ===<br />
<br />
* Job management database table (<project_name>): For each input file, keeps track of whether or not a job for it has been submitted, along with other optional fields. <br />
<br />
* Job status database table (<project_name>Job (no space)): For each job, keeps track of the job-id, the job status, memory used, cpu & wall time, time taken to complete various stages (e.g. pending, dependency, active), and others. <br />
<br />
* Job metrics database table (<project_name>_aux (no space)): For each job, keeps track of the job-id, how many events were processed, the time it took to copy the cache file, and the time it took to run the plugin. This information is culled from the log files of each job, and is done within the analysis directory of each launch.<br />
<br />
=== Initialize Project Management ===<br />
<br />
* Log into the ifarm machine with one of the gxproj accounts. For this example we will use gxproj1.<br />
<pre><br />
ssh gxproj1@ifarm -Y<br />
</pre><br />
<br />
* Go to a directory to do the launch. In principle, any directory will work, but for gxproj1 this is usually done in /home/gxproj1/halld/<br />
<br />
* Check out the necessary scripts <pre>svn co https://halldsvn.jlab.org/repos/trunk/scripts/monitoring/jproj</pre> This will get all necessary scripts for launching. Once checked out, <pre>cd projects</pre><br />
<br />
* The script create_project.sh can be used to create a new project. It will take a single argument, the project name. It is assumed that the project name is of the form offline_monitoring_RunPeriod20YY_MM_verVV_hd_rawdata where this string will be parsed to give the run period 20YY_MM and version number VV. One thing to do BEFORE creation of a new project is to editthe conditions of the launch (plugins to run over, memory requested, disk space requested) within templates/template.jsub . This<br />
information is saved automatically at project creation time into files in /group/halld/data_monitoring/run_conditions .<br />
To create a project do <pre>./create_project.sh offline_monitoring_RunPeriod20YY_MM_verVV_hd_rawdata</pre><br />
The name has been chosen to be as consistent as possible with other directory structures.<br />
However, mysql requires that "-" be escaped in table names, so unfortunately run periods will be<br />
given as 2YYY_MM instead of 2YYY-MM.<br />
<br />
* A new directory with the project name (offline_monitoring_RunPeriod20YY_MM_verVV_hd_rawdata) and files will be copied and modified from the template directory to reflect the run period, the user, the directory that it was created in, project name, etc.<br />
<br />
* For each project, cd into the new directory<br />
* The script clear.sh will remove any existing tables for the current project name, then recreate it. Do<br />
<pre>./clear.sh</pre><br />
* To use the jproj.pl script that was checked in, add the directory to your path with<br />
<pre>source ../../scripts/setup.csh</pre><br />
or always specify the full path<br />
<pre>../../scripts/jproj.pl</pre><br />
* Now update the table of runs with<br />
<pre>jproj.pl <project name> update</pre><br />
This will fill the table with all files within /mss that are of the same form as what is in <project name>.jproj . If you want to register only on a subset of all such files, you can edit this file directly.<br />
* Once you have registered all of the files you would like to run over, do<br />
<pre>jproj.pl <project name> submit [max # of jobs] [run number]</pre><br />
where the additional options specify how many jobs to submit and which run number to run on. Without these options all files that are registered and have not been submitted yet will be submitted.<br />
<br />
At this stage you are ready to submit all files. It is a good idea to submit a few test jobs<br />
at first to check that all scripts are working and that the plugins do not crash. Once you are<br />
sure that this does not happen, you can send all jobs in. The remaining jobs are then the monitoring<br />
which will (among other things) put the results on the online webpage for the collaboration to view, and<br />
the analysis of the launch.<br />
<br />
=== Project File Overview ===<br />
<br />
An overview of each project file:<br />
* '''clear.sh:''' For the current project, deletes the job status and management database tables (if any), and creates new, empty ones. <br />
* '''<project_name>.jproj:''' Contains the path and file name format for the input files for the jobs. <br />
* '''<project_name>.jsub:''' The xml job submission script. The run number and file number variables are set during job submission for each input file. <br />
* '''script.sh:''' The script that is executed during the job. If output job directories are not pre-created manually, they should be created in this script with the proper permissions:<br />
<pre><br />
mkdir -p -m 775 my_directory<br />
</pre><br />
* '''setup_jlab-[run period].csh:''' The environment that is sourced at the beginning of the job execution. <br />
* '''status.sh:''' Updates the job status database table, and prints some of its columns to screen.<br />
<br />
=== Project Management ===<br />
<br />
* Delete (if any) and create the database table(s) for the current set of job submissions:<br />
<pre><br />
./clear.sh<br />
</pre><br />
Also, if testing was done with jobs, it is best to delete the output directory and the configuration files:<br />
<pre><br />
rm -frv /volatile/halld/offline_monitoring/RunPeriod-20YY-MM/verVV /group/halld/data_monitoring/run_conditions/soft_comm_20YY_MM_verVV.xml /group/halld/data_monitoring/run_conditions/jana_rawdata_comm_20YY_MM_verVV.conf<br />
</pre><br />
<br />
* Search for input files matching the string in the .jproj file, and create a row for each in the job management database table (called <project_name>). You can test by adding an optional argument at the end, which only selects files with a specific file number:<br />
<pre><br />
jproj.pl <project_name> update <optional_file_number><br />
</pre><br />
<br />
* Confirm that the job management database is accurate by printing it's contents to screen:<br />
<pre><br />
mysql -hhallddb -ufarmer farming -e "select * from <project_name>"<br />
</pre><br />
<br />
* ONLY if a mistake was made, to delete the tables from the database and recreate new, empty ones, run: <br />
<pre><br />
./clear.sh<br />
</pre><br />
<br />
* To look at the status of the submitted jobs, first query auger and update the job status database:<br />
<pre><br />
fill_in_job_details.pl <project_name><br />
</pre><br />
<br />
* The job status can then be viewed by submitting a query to the job status database (called <project_name>Job (no space in between)):<br />
<pre><br />
mysql -hhallddb -ufarmer farming -e "select id,run,file,jobId,hostname,status,timeSubmitted,timeActive,walltime,cput,timeComplete,result,error from <project_name>Job"<br />
</pre><br />
<br />
* These last two commands can instead be executed simultaneously by running:<br />
<pre><br />
./status.sh<br />
</pre><br />
<br />
=== Handy mysql Instructions ===<br />
<br />
* Handy mysql instructions:<br />
<pre><br />
mysql -hhallddb -ufarmer farming # Enter the "farming" mysql database on "hallddb" as user "farmer"<br />
quit; # Exit mysql<br />
show tables; # Show a list of the tables in the current database<br />
show columns from <project_name>; # show all of the columns for the given table<br />
select * from <project_name>; # show the contents of all rows from the given table<br />
</pre><br />
<br />
=== Backing Up Offline Monitoring Tables ===<br />
Tables created for offline monitoring can be backed up using the script backup_tables.sh which<br />
can be checked out with the other files from https://halldsvn.jlab.org/repos/trunk/scripts/monitoring/jproj/projects<br />
<br />
The script uses the command mysqldump to print out a file that can be executed to recreate the tables.<br />
Since executing this output file will drop the table when it exists, caution is advised.<br />
Example usage to backup all three tables created for run period 2014_10 ver 17:<br />
<pre>backup_tables.sh 2014_10 17</pre><br />
<br />
== Running Over Data As It Comes In ==<br />
<br />
A special user gxproj1 will have a cron job set up to run the plugins as new data appears on /mss.<br />
During the week, gxproj1 will submit offline plugin jobs with the same setup as the weekly jobs<br />
run the previous Friday. The procedure for this is shown below.<br />
<br />
<!--<br />
=== Setting up the environment ===<br />
The file<br />
/home/gxproj1/setup_jlab.csh<br />
is sourced through .tcshrc.<br />
This file is the same as what is linked to by<br />
/home/gluex/setup_jlab_commissioning.csh,<br />
except HALLD_HOME, HDDS_HOME, and JANA_CALIB_URL are set separately so that this<br />
user can have a separate build.<br />
<br />
To obtain the builds from the previous Friday's runs,<br />
execute<br />
/home/gxproj1/halld/monitoring/newruns/setup_previous.sh [year] [month] [day]<br />
The build revisions from the previous Friday are archived in files<br />
/work/halld/data_monitoring/run_conditions/soft_comm_[year]_[month]_[day].xml<br />
and the script will build libraries based on those stored revision numbers.<br />
--><br />
<br />
=== Running the cron job ===<br />
<br />
'''IMPORTANT:''' The cron job should not be running while you are manually submitting jobs using the jproj.pl script for the same project, or else you will probably multiply-submit a job. <br />
<br />
* Go to the cron job directory: <br />
<pre><br />
cd /u/home/gxproj1/halld/monitoring/newruns<br />
</pre><br />
<br />
* The cron_plugins file is the cronjob that will be executed. During execution, it runs the exec.sh command in the same folder. This command takes two arguments: the project name, and the maximum file number for each run. These fields should be updated in the cron_plugins file before running. <br />
<br />
* The exec.sh command updates the job management database table with any data that has arrived on tape since it was last updated, ignoring file numbers greater than the maximum file number. It then submits jobs for these files. <br />
<br />
* To start the cron job, run:<br />
<pre><br />
crontab cron_plugins<br />
</pre><br />
<br />
* To check whether the cron job is running, do<br />
<pre><br />
crontab -l<br />
</pre><br />
<br />
* To remove the cron job do<br />
<pre><br />
crontab -r<br />
</pre><br />
<br />
<!--<br />
The cron job will run the script scan_for_jobs.sh,<br />
which runs generatejobs_plugins_rawdata.sh for any<br />
new runs that it had not seen before. All previous<br />
runs are recorded in the file filelists/files_current.txt<br />
so clear this to run over runs, or set the parameters<br />
MINRUN and MAXRUN which will set the range of runs submitted.<br />
--><br />
<br />
==Post-Processing Procedures==<br />
<br />
To visualize the monitoring data, we save images of selected histograms and store time series of selected quantities in a database, which are then displayed on a web page. This section describes how to generate the monitoring images and database information.<br />
<br />
The scripts used to generate this summary data are primarily run from /home/gxprojN/halld/monitoring/process . If you want a new copy of the scripts, e.g., for a new monitoring run, you should check the scripts out from SVN:<br />
<syntaxhighlight><br />
svn co https://halldsvn.jlab.org/repos/trunk/scripts/monitoring/process<br />
</syntaxhighlight><br />
Note that these scripts currently have some parameters which must be periodically set by hand.<br />
<br />
The default python version on most JLab machine does not have the modules to allow these scripts to connect to the MySQL database. To run these scripts, load the environment with the following command<br />
<syntaxhighlight><br />
source /home/gxproj1/halld/monitoring/process/monitoring_env.csh<br />
</syntaxhighlight><br />
<br />
===Online Monitoring===<br />
<br />
There are two primary scripts for running over the monitoring data generated by the online system and offline reconstruction. The online script can be run with either of the following commands:<br />
<syntaxhighlight><br />
/home/gluex/halld/monitoring/process/check_new_runs.py<br />
<br />
OR <br />
<br />
/home/gluex/halld/monitoring/process/check_new_runs.csh<br />
</syntaxhighlight><br />
The shell script sets up the environment properly to run the python script. To connect to the monitoring database on the JLab CUE, modules included in the local installation of python >= 2.7 are needed. The shell script is appropriate to use in a cron job. The cronjob is currently run under the "gluex" account.<br />
<br />
The online monitoring system copies a ROOT file containing the results of the online monitoring, and other configuration files into a directory accessible outside the counting house. This python script automatically checks for new ROOT files, which it will then automatically process. It contains several configuration variables that must be correctly set, which contains the location of input/output directories, etc...<br />
<br />
===Offline Monitoring===<br />
<br />
After the data is run over, the results should be processed, so that summary data is entered into the monitoring database and plots are made for the monitoring webpages. Currently, this processing is controlled by a cronjob that runs the following script:<br />
<syntaxhighlight><br />
/home/gxproj1/halld/monitoring/process/check_monitoring_data.csh <br />
</syntaxhighlight><br />
This script checks for new ROOT files, and only runs over those it hasn't processed yet. Since one monitoring ROOT file is produced for each EVIO file, whenever a new file is produced, the plots for the corresponding run are recreated and all the ROOT files for a run are combined into one file. Information is stored in the database on a per-file basis. <br />
<br />
Plots for the monitoring web page can be made from single histograms or multiple histograms using RootSpy macros. If you want to change the list of plots made, you must modify one of the following files:<br />
* histograms_to_monitor - specify either the name of the histogram or its the full ROOT path <br />
* macros_to_monitor - specify the full path to the RootSpy macro .C file<br />
<br />
When a new monitoring run is started, or the conditions are changed, the following steps should be taken to process the new files:<br />
# Add a new data version, as described below:<br />
# Change the following parameters in check_monitoring_data.csh:<br />
## JOBDATE should correspond to the ouptut date used by the job submission script<br />
## OUTPUTDIR should correspond to the directory corresponding to the run period and revision corresponding to the new version you just submitted. Presumably, this directory will be empty at the beginning.<br />
## Once you create a new data version as defined below, you should pass the needed information as a command line option. Currently this is done by the ARGS variable. For example, the argument "-v RunPeriod-2014-10,8" tells the monitoring scripts to look up the version corresponding to revision 8 of RunPeriod-2014-10 in the monitoring DB and to use to store the results.<br />
<br />
<syntaxhighlight><br />
Example configuration parameters:<br />
set JOBDATE=2015-01-09<br />
set INPUTDIR=/volatile/halld/RunPeriod-2014-10/offline_monitoring<br />
set OUTPUTDIR=/w/halld-scifs1a/data_monitoring/RunPeriod-2014-10/ver08<br />
set ARGS=" -v RunPeriod-2014-10,8 "<br />
</syntaxhighlight><br />
If you want to process the results manually, the data is processed using the following script:<br />
<syntaxhighlight><br />
./process_new_offline_data.py <input directory> <output directory><br />
<br />
EXAMPLE:<br />
<br />
./process_new_offline_data.py 2014-11-14 /volatile/halld/RunPeriod-2014-10/offline_monitoring/ /w/halld-scifs1a/data_monitoring/RunPeriod-2014-10/ver02<br />
</syntaxhighlight><br />
The python script takes several options to enable/disable various steps in the processing. Of interest is the "--force" option, which will run over all monitoring ROOT files, whether or not they've been previously identified.<br />
<br />
Every time a new reconstruction pass is performed, a new version number must be generated. To do this, prepare a version file as described below. Then run the register_new_version.py script to store the information in the database. The script will return a version number, which then should be set by hand in process_new_offline_data.py - future versions of the script will streamline this part of the procedure. An example of how to generate a new version is:<br />
<syntaxhighlight><br />
./register_new_version.py add /home/gxproj1/halld/monitoring/process/versions/vers_RunPeriod-2014-10_pass1.txt<br />
</syntaxhighlight><br />
<br />
<b>If you are running the offline monitoring by checking out the files in trunk/scripts/monitoring/jproj/projects/<br />
of the svn repository</b>, and created a project with<br />
create_project.sh [project name] hd_rawdata<br />
Then go to the directory [project name]/processing/<br />
and execute<br />
./run_processing.sh<br />
which will run register_new_version.py as well as check_monitoring_data.csh for that project.<br />
<br />
===Step-by-Step Instructions For Processing a New Monitoring Run===<br />
<br />
The monitoring runs are current run out of the gxproj1 and gxproj5 accounts. After an offline monitoring run has been successfully started on the batch farm, the following steps should be followed to setup the post-processing for these runs.<br />
<br />
# The post-processing scripts are stored in $HOME/halld/monitoring/process and are automatically run by cron.<br />
# Run "svn update" to bring any changes in. Be sure that the list of histograms and macros to plot are current.<br />
# Edit check_monitoring_data.csh to point to the current revisions/directories<br />
#* VERSION<br />
#* ARGS<br />
#* Note that the environment depends on a standard script - $HOME/setup_jlab.csh<br />
# Update files in the web directory, so that the results are displayed on the web pages: /group/halld/www/halldweb/html/data_monitoring/textdata<br />
# Copy the REST files to more permanent locations:<br />
#* cp -a /volatile/halld/offline_monitoring/RunPeriod-YYYY-MM/verVV/REST /work/halld/data_monitoring/RunPeriod-YYYY-MM/REST/verVV<br />
#* cp -a /volatile/halld/offline_monitoring/RunPeriod-YYYY-MM/verVV/REST /cache/halld/RunPeriod-YYYY-MM/REST/verVV [under testing]<br />
<br />
Check log files in $HOME/halld/monitoring/process/log for more information on how each run went. If there are problems, check log files, and modify check_monitoring_data.csh to vary the verbosity of the output.<br />
<br />
==Data Versions==<br />
<br />
To document the conditions of the monitoring data that is created, for the sake of reproducability and further analysis we save several pieces of information. The format is intended to be comprehensive enough to document not just monitoring data, but versions of raw and reconstructed data, so that this database table can be used for the event database as well.<br />
<br />
We store one record per pass through one run period, with the following structure:<br />
<br />
{| class="wikitable"<br />
! Field !! Description<br />
|-<br />
| data_type || The level of data we are processing. For the purposes of monitoring, "rawdata" is the online monitoring, "recon" is the offline monitoring<br />
|- <br />
| run_period || The run period of the data<br />
|- <br />
| revision || An integer specifying which pass through the run period this data corresponds to<br />
|- <br />
| software_version || The name of the XML file that specifies the different software versions used<br />
|-<br />
| jana_config || The name of the text file that specifies which JANA options were passed to the reconstruction program<br />
|-<br />
| ccdb_context || The value of JANA_CALIB_CONTEXT, which specifies the version of calibration constants that were used<br />
|-<br />
| production_time || The data at which monitoring/reconstruction began<br />
|-<br />
| dataVersionString || A convenient string for identifying this version of the data<br />
|}<br />
<br />
<br />
An example file used as as input to ./register_new_version.py is:<br />
<syntaxhighlight><br />
data_type = recon<br />
run_period = RunPeriod-2014-10<br />
revision = 1<br />
software_version = soft_comm_2014_11_06.xml<br />
jana_config = jana_rawdata_comm_2014_11_06.conf<br />
ccdb_context = calibtime=2014-11-10<br />
production_time = 2014-11-10<br />
dataVersionString = recon_RunPeriod-2014-10_20141110_ver01<br />
</syntaxhighlight></div>Kmoriyahttps://halldweb.jlab.org/wiki/index.php?title=Data_Monitoring_Procedures&diff=70784Data Monitoring Procedures2015-10-19T14:29:52Z<p>Kmoriya: /* Starting the Launch and Submitting Jobs */</p>
<hr />
<div>__TOC__<br />
<br />
== Master List of File / Database / Webpage Locations ==<br />
=== Run Conditions ===<br />
* Online Run-by-run condition files (B-field, current, etc.): /work/halld/online_monitoring/conditions/<br />
* Offline monitoring run conditions (software versions, jana config): /group/halld/data_monitoring/run_conditions/<br />
*[http://www.jlab.org/Hall-D/test/RunInfo/ Run Info vers. 1]<br />
*[https://halldweb.jlab.org/cgi-bin/data_monitoring/run_conditions.pl Run Info vers. 2]<br />
<br />
=== Monitoring Output Files ===<br />
* Run Periods 201Y-MM is for example 2015-03, launch ver verVV is for example ver15<br />
* Online monitoring histograms: /work/halld/online_monitoring/root/<br />
* Offline monitoring histogram ROOT files: /work/halld/data_monitoring/RunPeriod-201Y-MM/verVV/rootfiles<br />
* REST files (most recent launch only): /work/halld/data_monitoring/RunPeriod-201Y-MM/REST/verVV<br />
* individual files for each job (ROOT, log, etc.): /volatile/halld/offline_monitoring/RunPeriod-201Y-MM/verVV/<br />
<br />
=== Monitoring Database ===<br />
* Accessing monitoring database (on ifarm): mysql -u datmon -h hallddb.jlab.org data_monitoring<br />
<br />
=== Monitoring Webpages ===<br />
*[https://halldweb.jlab.org/cgi-bin/data_monitoring/monitoring/plotBrowser.py Plot Browser]<br />
*[https://halldweb.jlab.org/cgi-bin/data_monitoring/monitoring/runBrowser.py Run Browser]<br />
*[https://halldweb.jlab.org/cgi-bin/data_monitoring/monitoring/timeSeries.py Time Series]<br />
*[https://halldweb.jlab.org/data_monitoring/launch_analysis/ Launch Analysis]<br />
<br />
== Job Monitoring Links ==<br />
* [http://scicomp.jlab.org/farm2/job.html Completed Job History]<br />
* [http://scicomp.jlab.org/farm2/project.html Job Stats By Project]<br />
* [http://scicomp.jlab.org/farm2/trackOrg.html Job Stats By Track]<br />
* [http://scicomp.jlab.org/farm2/report.html Cluster Report]<br />
* [http://scicomp.jlab.org/farm2/walltime.html Walltime Distribution]<br />
<br />
== Saving Online Monitoring Data ==<br />
<br />
The procedure for writing the data out is given in, e.g.,<br />
[https://halldweb.jlab.org/wiki/index.php/Raid-to-Silo_Transfer_Strategy Raid-to-Silo Transfer Strategy].<br />
<br />
Once the DAQ writes out the data to the raid disk, cron jobs will copy the file to tape,<br />
and within ~20 min., we will have access to the file on tape at<br />
/mss/halld/$RUN_PERIOD/rawdata/RunXXXXXX.<br />
<br />
All online monitoring plugins will be run as data is taken.<br />
They will be accessible within the counting house via RootSpy, and<br />
for each run and file, a ROOT file containing the histograms will be saved<br />
within a subdirectory for each run.<br />
<br />
For immediate access to these files, the raid disk files may be accessed directly<br />
from the counting house, or the tape files will be available within ~20 min. of the<br />
file being written out.<br />
<br />
== Offline Monitoring: Running Over Archived Data ==<br />
<br />
Once files are written to tape we run the online plugins on these files to confirm what we were seeing in the online monitoring, and also to update the results from the latest calibration and software. <br />
Manual scripts and cron jobs are set up to look for new data and run the plugins over a sample of files.<br />
<br />
Every other Friday (usually the Friday before the offline meetings) jobs will be started to run the newest software on all previous runs,and allows everybody to see improvements in each detector.<br />
For each launch, independent builds of hdds, sim-recon, the monitoring plugins, and an sqlite file will be generated.<br />
<br />
Below the procedures are described for<br />
# Preparing the software for the launch<br />
# Starting the launch (using hdswif)<br />
# Post-analysis of statistics of the launch<br />
<br />
Processing the results and making them available to the collaboration<br />
is handled in the section [[#Post-Processing_Procedures | Post-Processing Procedures]] below.<br />
<br />
<!--<br />
==== Generating an offline plugin job ====<br />
<br />
The user gxproj1 should be used for official offline monitoring jobs.<br />
Within /home/gxproj1/halld/monitoring/batch/ there will be scripts to run the online monitoring plugins over tape files.<br />
The main script is generatejobs_plugins_rawdata.sh, which can be used as<br />
generatejobs_plugins_rawdata.sh [minrun] [maxrun] (minfile) (maxfile)<br />
where minrun, maxrun specify the range of the run #, and minfile, maxfile (optional) specify<br />
the file range for that run.<br />
<br />
This will generate a script run_rawdata_XXXXXX.sh, where the run # has now been formatted to be 6 digits.<br />
Executing this script will send the monitoring plugins job to the Auger batch system.<br />
<br />
There is also a script clean.sh which can be used as<br />
clean.sh XXX<br />
This will clean up all associated files created in association with run XXX.<br />
<br />
Internally, the xml file used to submit the job will be created, and the job to run<br />
will be given within script.sh. All run parameters should be specified in at the beginning<br />
of generatejobs_plugins_rawdata.sh<br />
Since we are running on tape, the tape file will first be copied over to the cache disk, and the job will run<br />
over this cached file.<br />
<br />
==== Using cron to run automatically ====<br />
Within /home/gluex/halld/monitoring/cron/ there is a file cron_plugins<br />
that can be executed via<br />
crontab cron_plugins<br />
This will set up a cron job to call the script scan_for_jobs.sh, which will<br />
check in the rawdata directory and call generatejobs_plugins_rawdata.sh for<br />
any run that is more than 5 min old. The cron job is set up to run every 10 min.<br />
<br />
==== Magnetic Field Settings ====<br />
For tracking, it is necessary to set the correct magnetic field settings.<br />
The below table is now obsolete. Please refer to the [https://halldweb.jlab.org/cgi-bin/data_monitoring/run_conditions.pl GlueX Run Conditions] webpage.<br />
<br />
The actual field settings for each run have not been documented well.<br />
Below are values based on going through entries in the [https://logbooks.jlab.org/book/halld halld log].<br />
<br />
{| border="1" cellpadding="1" style="text-align: center;"<br />
!width="100"| Run #s<br />
!width="150"| Solenoid Current (A)<br />
!width="300"| JANA option<br />
!width="300"| notes<br />
|-<br />
| 940 - 996 || 1000 || -PBFIELD_MAP=Magnets/Solenoid/solenoid_1000A_poisson_20141104 || At run 997, the solenoid started ramping down<br />
|-<br />
| 998 - 1448 || 0 || -PBFIELD_TYPE=NoField -PDEFTAG:DTrackCandidate=StraightLine || Solenoid started ramping up around run 1431. Runs 1432 - 1448 should not be used for tracking. See [https://logbooks.jlab.org/entry/3309039 this] and [https://logbooks.jlab.org/entry/3309092 this] entry.<br />
|-<br />
| 1449 - 1620 || 1200 || -PBFIELD_MAP=Magnets/Solenoid/solenoid_1200A_poisson_20140520 || <br />
|-<br />
|}<br />
<br />
'''Note:''' The following run ranges were taken with a 300A field, and could be analyzed with the magnetic field map Magnets/Solenoid/solenoid_300A_poisson_20140819 , more study is needed to see if the straight line track fitter is needed or not: 1036 - 1053, 1065 - 1121, 1212? - 1254, 1309 - 1318 [1307,1308,1319-1329 were taken while the magnet was ramping].<br />
--><br />
<br />
=== General Information on Procedures ===<br />
This section explains how the offline monitoring should be run.<br />
Since we may want to simultaneously run offline monitoring for different run periods that require<br />
different environment variables, the scripts are set up so that a generic user can download the<br />
scripts and run them from anywhere. Most output directories for offline monitoring are created<br />
with group read/write permissions so that any Hall D group user has access to the contents,<br />
but there are some cases where use of the account that created the launch is necessary.<br />
<br />
The accounts used for offline monitoring are the gxprojN accounts created and maintained by<br />
Mark Ito (see [https://halldweb.jlab.org/wiki/index.php/GlueX-related_shared_accounts_on_the_JLab_CUE here] for how each account is used).<br />
As of October 2015, the following are used:<br />
* gxproj1 for running over Fall 2014 data (deprecated since June 2015)<br />
* gxproj5 for running over Spring 2015 data<br />
<br />
Since the summer of 2015 we have transitioned from a system using Mark Ito's jproj scripts<br />
to integrating the swif system that Chris Larrieu (SciComp) has been developing. For offline monitoring,<br />
the hdswif system that Kei developed is used for launching the jobs, and the jproj system is used<br />
for meta-analysis of launch statistics.<br />
<br />
Both hdswif and jproj are maintained in svn:<br />
* hdswif: https://halldsvn.jlab.org/repos/trunk/scripts/monitoring/hdswif<br />
* jproj : https://halldsvn.jlab.org/repos/trunk/scripts/monitoring/jproj<br />
<br />
To run the offline monitoring each package should be checked out and all necessary<br />
scripts are included.<br />
<br />
=== Preparing the software for the launch ===<br />
<br />
To begin a new launch the software must be built to the latest versions.<br />
For the gxprojN user accounts used, all software builds are contained in the directory ~/builds<br />
(which are soft links to /work/halld/home/gxprojN/builds). When logging into these accounts<br />
the setup files ~/setup_jlab-2015-03.csh or similar files should be sourced.<br />
<br />
<b>Note that Mark Ito does <u>not</u> want you to change the contents of each .cshrc file.</b><br />
You should consult him if you feel the need.<br />
<br />
<span style="color:red">NOTE: FOR BUILDING SOFTWARE IT IS A GOOD IDEA TO DO A COMPLETE WIPEOUT/CHECKOUT EACH TIME TO AVOID STALE HEADER FILES.</span><br />
<br />
# Building hdds: Go to ~/builds/hdds. The directory hdds is the one from git. Delete the contents, then download the newest version from git and build: <pre>cd ~/builds/hdds/</pre> <pre>rm -frv hdds</pre> <pre>git clone https://github.com/JeffersonLab/hdds</pre> <pre>scons install</pre><br />
# Building sim-recon: Go to ~/builds/sim-recon. The directory sim-recon is the one from git. Delete the contents, then download the newest version from git and build: <pre>cd ~/builds/sim-recon</pre> <pre>rm -frv sim-recon</pre> <pre>git clone https://github.com/JeffersonLab/sim-recon</pre> <pre>scons install -j8</pre><br />
# Prepare the latest sqlite file: The sqlite is set in the ~/setup_jlab-2015-03.csh script as sqlite:////home/gxproj5/builds/ccdb.sqlite through the environment variables <b>JANA_CALIB_URL</b> and <b>CCDB_CONNECTION</b>. Therefore, go to ~/builds,and create a new sqlite file. We create the sqlite file in a temporary directory since creating the sqlite file in a directory where the output file exists causes errors. Original documentation on creating sqlite files are [https://halldweb.jlab.org/wiki/index.php/SQLite-form_of_the_CCDB_database here]. <pre>cd ~/builds/tmp</pre> <pre>$CCDB_HOME/scripts/mysql2sqlite/mysql2sqlite.sh -hhallddb.jlab.org -uccdb_user ccdb | sqlite3 ccdb.sqlite</pre> <pre>mv ccdb.sqlite ../</pre><br />
# Note that the above steps must be done <b>BEFORE</b> launch project creation. This is because we will track the revisions of the libraries used, and this is done by extracting the svn information in each directory. Also, note that the system assumes that we have the topmost build directory (usually called GLUEX_TOP) to be $HOME/builds . Such an assumption is necessary to be able to extract information about the library locations automatically.<br />
<br />
Create the appropriate project(s) and submit the jobs using hdswif, as detailed in the section below.<br />
<br />
=== Starting the Launch and Submitting Jobs ===<br />
<br />
Until the summer of 2015 we relied solely on Mark Ito's jproj system for submitting and keeping track of jobs. We have since moved to the swif system and use the hdswif wrapper for this. Below are instructions for how to use these.<br />
<br />
# Downloading hdswif: Download the hdswif directory from svn. For the gxprojN accounts, use the directory ~/halld/hdswif. <pre>cd ~/halld </pre> <pre> svn co https://halldsvn.jlab.org/repos/trunk/scripts/monitoring/hdswif</pre> <pre>cd hdswif</pre><br />
# Creating the workflow: Within SWIF jobs are registered into workflows. First create the workflow. For offline monitoring, the workflow names are of the form <b>offline_monitoring_RunPeriod201Y_MM_verVV_hd_rawdata</b> with suitable replacements for the run period and version number. The command "swif list" will list all existing workflows. Also, for most simple SWIF commands hdswif also provides a wrapper. <pre>swif list</pre> For creation of workflows for offline monitoring the command <pre>hdswif.py create [workflow] -c [config file] </pre> should be used. When a config file is passed in, hdswif will automatically create files that record the configuration of the current launch. These files are stored as for example<br />
#* /group/halld/data_monitoring/run_conditions/jana_rawdata_comm_2015_03_ver15.conf<br />
#* /group/halld/data_monitoring/run_conditions/soft_comm_2015_03_ver15.xml<br />
#* The software packages stored in git (sim-recon and hdds) can have git tags applied to them, which makes it easier to find versions of the software than a SHA-1 hash. hdswif will ask if you would like to create a tag, and execute the following sequence: <pre>git tag -a offmon-201Y_MM-verVV -m "Used for offline monitoring 201Y-MM verVV started on 201y/mm/dd"</pre> <pre>git push offmon-201Y_MM-verVV</pre> This will only be invoked when the user name is gxprojN, and for the configuration files, the output directory will be /group/halld/data_monitoring/run_conditions/ for gxprojN accounts while it will be the current directory for other users.<br />
#* To use the git-tagged software versions do for example <pre>cd $HALLD_HOME</pre> <pre>git checkout offmon-2015_03-ver15</pre><br />
# Registering jobs in the workflow: To register jobs within the workflow, hdswif provides the use of config files. Jobs can be registered by specifying the workflow, config file (-c), run (-r) and file (-f) numbers if necessary. A typical config file will look this:<br />
PROJECT gluex<br />
TRACK reconstruction<br />
OS centos65<br />
NCORES 6<br />
DISK 40<br />
RAM 8<br />
TIMELIMIT 8<br />
JOBNAMEBASE offmon_<br />
RUNPERIOD 2015-03<br />
VERSION 15<br />
OUTPUT_TOPDIR /volatile/halld/offline_monitoring/RunPeriod-[RUNPERIOD]/ver[VERSION] # Example of other variables included in variable<br />
SCRIPTFILE /home/gxproj5/halld/hdswif/script.sh # Must specify full path<br />
ENVFILE /home/gxproj5/halld/hdswif/setup_jlab-2015-03.csh # Must specify full path<br />
The config file contains configuration parameters for each of the jobs.<br><br />
<span style="color:red">Note: Job configuration parameters can be set differently for jobs within the same workflow if necessary.</span><br />
Edit the config file and save as a new file if necessary. Once the configuration is set, jobs can be added via <pre>hdswif.py add [workflow] -c input.config</pre><br />
By default, hdswif will add all files found within the directory /mss/halld/RunPeriod-201Y-MM/rawdata/ where 201Y-MM is specified by the RUNPERIOD parameter in the config file. If only some of the runs or files are needed, these can be specified for example with<br />
<pre>hdswif.py add [workflow] -c input.config -r 3180 -f '00[0-4]'</pre><br />
to specify to register running only over run 3180 files 000 - 004 (Unix-style brackets and wildcards can be used).<br />
<br />
Running the workflow: To run the workflow, simply use swif run: <pre>swif run </pre> or equivalently, using the hdswif wrapper, <pre>hdsswif.py run [workflow]</pre><br><br />
<b>It is recommended that some jobs be tested to make sure that everything is working rather than fail thousands of jobs.</b><br> For this purpose, hdswif will take an additional parameter to run which limits the number of jobs to submit: <pre>hdswif.py run [workflow] 10</pre><br />
in which case only 10 jobs will be submitted. To submit all jobs after checking the results, do <pre>hdswif.py run [workflow]</pre><br />
<br />
=== Post-analysis of statistics of the launch ===<br />
<br />
# After jobs have been submitted, it will usually take a few days for all of the jobs to be processed.<br />
#* Status of Auger: http://scicomp.jlab.org/scicomp/#/auger/jobs (see also links above)<br />
#* Status of user jobs: <pre>jobstat -u [user name]</pre><br />
# The status and results of jobs are saved within the SWIF internal server, and are available via the command <pre>swif status [workflow] -summary -runs</pre> where the arguments -summary and -runs show summary statistics and statistics for individual jobs, respectively. hdswif has a command that takes this output in XML output and creates an HTML webpage showing results of the launch. To do this, do <pre>hdswif.py summary [workflow]</pre> This will create an XML file swif_output_[workflow].xml that contains all information from SWIF. If the file already exists, hdswif will ask whether to overwrite the existing file.<br />
# At this stage the html output and figure files are created and ready to be put online. For this step and other steps involving analysis of the statistics of the launch results, it is convenient to change to the jproj system.<br />
# The jproj scripts for offline monitoring are maintained in the svn directory https://halldsvn.jlab.org/repos/trunk/scripts/monitoring/jproj Do <pre> svn co https://halldsvn.jlab.org/repos/trunk/scripts/monitoring/jproj </pre> For the gxprojN accounts used for offline monitoring the directory should be ~/halld/jproj<br />
# The jproj directory contains two subdirectories, scripts and projects. The scripts directory contains useful scripts for processing the jobs registered in the jproj system, and each of the offline monitoring launches will be handled in the projects directory.<br />
# Go to the projects directory<pre>cd ~/halld/jproj/projects</pre> and use the script create_project.sh to create a new directory that contains the processing scripts for the current launch <pre>./create_project.sh [workflow]</pre> This should create a directory such as offline_monitoring_RunPeriod2015_03_ver15_hd_rawdata (same as the workflow name). The script uses the template files in the directory templates and by substitution creates script files for the current launch. Now go to the newly created analysis directory: <pre>cd [workflow]/analysis</pre><br />
which for the gxprojN accounts should have the full path /home/gxprojN/halld/jproj/projects/[workflow]/analysis.<br><br><br />
<span style="color:red">All of the analysis commands including arguments are contained in run_analysis.sh, but it is <b>strongly</b> recommended that all commands are run manually to check for errors.</span><br><br><br />
# The first thing to do is to make the html output from hdswif public. Copy the html file and related figures that were created from hdswif to the appropriate space within /group/halld/www/halldweb/html/data_monitoring/ : <pre>python publish_offmon_results.py [run period] [version]</pre> Note that the command with appropriate substitutions for arguments can be found within run_analysis.sh (same for all commands below).<br />
# Next, we need to create a few MySQL tables for the current launch. The MySQL tables are useful for comparing run/file combinations for different launches. For the SWIF launches, there are two tables needed, and will be named<br />
#* [workflow]Job<br />
#* [workflow]_aux<br />
# This naming scheme of tables and their roles are the same as from the jproj only launches. The [workflow]Job table will contain information gathered from SWIF about each job (which node it went to, start time of each stage, memory usage, etc.). The [workflow]_aux table will contain information gathered from the output stdout files from each job. First, create the Job table using <pre>python create_jproj_job_table.py [run period] [version]</pre> Note that this script uses the XML output from hdswif summary and inserts the contents into the MySQL table, so the XML output file must exist.<br />
# To check the contents of this MySQL table, do <pre>mysql -hhalldb -ufarmer farming</pre> <pre>mysql> describe offline_monitoring_RunPeriod2015_03_ver11_hd_rawdataJob;</pre><br />
<br />
== Hall D Job Management System ==<br />
<br />
This section details instructions on how to create and launch a set of jobs using the Hall-D Job Management System developed by Mark Ito. These instructions are generic: this system can be used for the weekly monitoring jobs, but can also be used for other sets of job launches as well. <br />
<br />
=== Database Table Overview ===<br />
<br />
* Job management database table (<project_name>): For each input file, keeps track of whether or not a job for it has been submitted, along with other optional fields. <br />
<br />
* Job status database table (<project_name>Job (no space)): For each job, keeps track of the job-id, the job status, memory used, cpu & wall time, time taken to complete various stages (e.g. pending, dependency, active), and others. <br />
<br />
* Job metrics database table (<project_name>_aux (no space)): For each job, keeps track of the job-id, how many events were processed, the time it took to copy the cache file, and the time it took to run the plugin. This information is culled from the log files of each job, and is done within the analysis directory of each launch.<br />
<br />
=== Initialize Project Management ===<br />
<br />
* Log into the ifarm machine with one of the gxproj accounts. For this example we will use gxproj1.<br />
<pre><br />
ssh gxproj1@ifarm -Y<br />
</pre><br />
<br />
* Go to a directory to do the launch. In principle, any directory will work, but for gxproj1 this is usually done in /home/gxproj1/halld/<br />
<br />
* Check out the necessary scripts <pre>svn co https://halldsvn.jlab.org/repos/trunk/scripts/monitoring/jproj</pre> This will get all necessary scripts for launching. Once checked out, <pre>cd projects</pre><br />
<br />
* The script create_project.sh can be used to create a new project. It will take a single argument, the project name. It is assumed that the project name is of the form offline_monitoring_RunPeriod20YY_MM_verVV_hd_rawdata where this string will be parsed to give the run period 20YY_MM and version number VV. One thing to do BEFORE creation of a new project is to editthe conditions of the launch (plugins to run over, memory requested, disk space requested) within templates/template.jsub . This<br />
information is saved automatically at project creation time into files in /group/halld/data_monitoring/run_conditions .<br />
To create a project do <pre>./create_project.sh offline_monitoring_RunPeriod20YY_MM_verVV_hd_rawdata</pre><br />
The name has been chosen to be as consistent as possible with other directory structures.<br />
However, mysql requires that "-" be escaped in table names, so unfortunately run periods will be<br />
given as 2YYY_MM instead of 2YYY-MM.<br />
<br />
* A new directory with the project name (offline_monitoring_RunPeriod20YY_MM_verVV_hd_rawdata) and files will be copied and modified from the template directory to reflect the run period, the user, the directory that it was created in, project name, etc.<br />
<br />
* For each project, cd into the new directory<br />
* The script clear.sh will remove any existing tables for the current project name, then recreate it. Do<br />
<pre>./clear.sh</pre><br />
* To use the jproj.pl script that was checked in, add the directory to your path with<br />
<pre>source ../../scripts/setup.csh</pre><br />
or always specify the full path<br />
<pre>../../scripts/jproj.pl</pre><br />
* Now update the table of runs with<br />
<pre>jproj.pl <project name> update</pre><br />
This will fill the table with all files within /mss that are of the same form as what is in <project name>.jproj . If you want to register only on a subset of all such files, you can edit this file directly.<br />
* Once you have registered all of the files you would like to run over, do<br />
<pre>jproj.pl <project name> submit [max # of jobs] [run number]</pre><br />
where the additional options specify how many jobs to submit and which run number to run on. Without these options all files that are registered and have not been submitted yet will be submitted.<br />
<br />
At this stage you are ready to submit all files. It is a good idea to submit a few test jobs<br />
at first to check that all scripts are working and that the plugins do not crash. Once you are<br />
sure that this does not happen, you can send all jobs in. The remaining jobs are then the monitoring<br />
which will (among other things) put the results on the online webpage for the collaboration to view, and<br />
the analysis of the launch.<br />
<br />
=== Project File Overview ===<br />
<br />
An overview of each project file:<br />
* '''clear.sh:''' For the current project, deletes the job status and management database tables (if any), and creates new, empty ones. <br />
* '''<project_name>.jproj:''' Contains the path and file name format for the input files for the jobs. <br />
* '''<project_name>.jsub:''' The xml job submission script. The run number and file number variables are set during job submission for each input file. <br />
* '''script.sh:''' The script that is executed during the job. If output job directories are not pre-created manually, they should be created in this script with the proper permissions:<br />
<pre><br />
mkdir -p -m 775 my_directory<br />
</pre><br />
* '''setup_jlab-[run period].csh:''' The environment that is sourced at the beginning of the job execution. <br />
* '''status.sh:''' Updates the job status database table, and prints some of its columns to screen.<br />
<br />
=== Project Management ===<br />
<br />
* Delete (if any) and create the database table(s) for the current set of job submissions:<br />
<pre><br />
./clear.sh<br />
</pre><br />
Also, if testing was done with jobs, it is best to delete the output directory and the configuration files:<br />
<pre><br />
rm -frv /volatile/halld/offline_monitoring/RunPeriod-20YY-MM/verVV /group/halld/data_monitoring/run_conditions/soft_comm_20YY_MM_verVV.xml /group/halld/data_monitoring/run_conditions/jana_rawdata_comm_20YY_MM_verVV.conf<br />
</pre><br />
<br />
* Search for input files matching the string in the .jproj file, and create a row for each in the job management database table (called <project_name>). You can test by adding an optional argument at the end, which only selects files with a specific file number:<br />
<pre><br />
jproj.pl <project_name> update <optional_file_number><br />
</pre><br />
<br />
* Confirm that the job management database is accurate by printing it's contents to screen:<br />
<pre><br />
mysql -hhallddb -ufarmer farming -e "select * from <project_name>"<br />
</pre><br />
<br />
* ONLY if a mistake was made, to delete the tables from the database and recreate new, empty ones, run: <br />
<pre><br />
./clear.sh<br />
</pre><br />
<br />
* To look at the status of the submitted jobs, first query auger and update the job status database:<br />
<pre><br />
fill_in_job_details.pl <project_name><br />
</pre><br />
<br />
* The job status can then be viewed by submitting a query to the job status database (called <project_name>Job (no space in between)):<br />
<pre><br />
mysql -hhallddb -ufarmer farming -e "select id,run,file,jobId,hostname,status,timeSubmitted,timeActive,walltime,cput,timeComplete,result,error from <project_name>Job"<br />
</pre><br />
<br />
* These last two commands can instead be executed simultaneously by running:<br />
<pre><br />
./status.sh<br />
</pre><br />
<br />
=== Handy mysql Instructions ===<br />
<br />
* Handy mysql instructions:<br />
<pre><br />
mysql -hhallddb -ufarmer farming # Enter the "farming" mysql database on "hallddb" as user "farmer"<br />
quit; # Exit mysql<br />
show tables; # Show a list of the tables in the current database<br />
show columns from <project_name>; # show all of the columns for the given table<br />
select * from <project_name>; # show the contents of all rows from the given table<br />
</pre><br />
<br />
=== Backing Up Offline Monitoring Tables ===<br />
Tables created for offline monitoring can be backed up using the script backup_tables.sh which<br />
can be checked out with the other files from https://halldsvn.jlab.org/repos/trunk/scripts/monitoring/jproj/projects<br />
<br />
The script uses the command mysqldump to print out a file that can be executed to recreate the tables.<br />
Since executing this output file will drop the table when it exists, caution is advised.<br />
Example usage to backup all three tables created for run period 2014_10 ver 17:<br />
<pre>backup_tables.sh 2014_10 17</pre><br />
<br />
== Running Over Data As It Comes In ==<br />
<br />
A special user gxproj1 will have a cron job set up to run the plugins as new data appears on /mss.<br />
During the week, gxproj1 will submit offline plugin jobs with the same setup as the weekly jobs<br />
run the previous Friday. The procedure for this is shown below.<br />
<br />
<!--<br />
=== Setting up the environment ===<br />
The file<br />
/home/gxproj1/setup_jlab.csh<br />
is sourced through .tcshrc.<br />
This file is the same as what is linked to by<br />
/home/gluex/setup_jlab_commissioning.csh,<br />
except HALLD_HOME, HDDS_HOME, and JANA_CALIB_URL are set separately so that this<br />
user can have a separate build.<br />
<br />
To obtain the builds from the previous Friday's runs,<br />
execute<br />
/home/gxproj1/halld/monitoring/newruns/setup_previous.sh [year] [month] [day]<br />
The build revisions from the previous Friday are archived in files<br />
/work/halld/data_monitoring/run_conditions/soft_comm_[year]_[month]_[day].xml<br />
and the script will build libraries based on those stored revision numbers.<br />
--><br />
<br />
=== Running the cron job ===<br />
<br />
'''IMPORTANT:''' The cron job should not be running while you are manually submitting jobs using the jproj.pl script for the same project, or else you will probably multiply-submit a job. <br />
<br />
* Go to the cron job directory: <br />
<pre><br />
cd /u/home/gxproj1/halld/monitoring/newruns<br />
</pre><br />
<br />
* The cron_plugins file is the cronjob that will be executed. During execution, it runs the exec.sh command in the same folder. This command takes two arguments: the project name, and the maximum file number for each run. These fields should be updated in the cron_plugins file before running. <br />
<br />
* The exec.sh command updates the job management database table with any data that has arrived on tape since it was last updated, ignoring file numbers greater than the maximum file number. It then submits jobs for these files. <br />
<br />
* To start the cron job, run:<br />
<pre><br />
crontab cron_plugins<br />
</pre><br />
<br />
* To check whether the cron job is running, do<br />
<pre><br />
crontab -l<br />
</pre><br />
<br />
* To remove the cron job do<br />
<pre><br />
crontab -r<br />
</pre><br />
<br />
<!--<br />
The cron job will run the script scan_for_jobs.sh,<br />
which runs generatejobs_plugins_rawdata.sh for any<br />
new runs that it had not seen before. All previous<br />
runs are recorded in the file filelists/files_current.txt<br />
so clear this to run over runs, or set the parameters<br />
MINRUN and MAXRUN which will set the range of runs submitted.<br />
--><br />
<br />
==Post-Processing Procedures==<br />
<br />
To visualize the monitoring data, we save images of selected histograms and store time series of selected quantities in a database, which are then displayed on a web page. This section describes how to generate the monitoring images and database information.<br />
<br />
The scripts used to generate this summary data are primarily run from /home/gxprojN/halld/monitoring/process . If you want a new copy of the scripts, e.g., for a new monitoring run, you should check the scripts out from SVN:<br />
<syntaxhighlight><br />
svn co https://halldsvn.jlab.org/repos/trunk/scripts/monitoring/process<br />
</syntaxhighlight><br />
Note that these scripts currently have some parameters which must be periodically set by hand.<br />
<br />
The default python version on most JLab machine does not have the modules to allow these scripts to connect to the MySQL database. To run these scripts, load the environment with the following command<br />
<syntaxhighlight><br />
source /home/gxproj1/halld/monitoring/process/monitoring_env.csh<br />
</syntaxhighlight><br />
<br />
===Online Monitoring===<br />
<br />
There are two primary scripts for running over the monitoring data generated by the online system and offline reconstruction. The online script can be run with either of the following commands:<br />
<syntaxhighlight><br />
/home/gluex/halld/monitoring/process/check_new_runs.py<br />
<br />
OR <br />
<br />
/home/gluex/halld/monitoring/process/check_new_runs.csh<br />
</syntaxhighlight><br />
The shell script sets up the environment properly to run the python script. To connect to the monitoring database on the JLab CUE, modules included in the local installation of python >= 2.7 are needed. The shell script is appropriate to use in a cron job. The cronjob is currently run under the "gluex" account.<br />
<br />
The online monitoring system copies a ROOT file containing the results of the online monitoring, and other configuration files into a directory accessible outside the counting house. This python script automatically checks for new ROOT files, which it will then automatically process. It contains several configuration variables that must be correctly set, which contains the location of input/output directories, etc...<br />
<br />
===Offline Monitoring===<br />
<br />
After the data is run over, the results should be processed, so that summary data is entered into the monitoring database and plots are made for the monitoring webpages. Currently, this processing is controlled by a cronjob that runs the following script:<br />
<syntaxhighlight><br />
/home/gxproj1/halld/monitoring/process/check_monitoring_data.csh <br />
</syntaxhighlight><br />
This script checks for new ROOT files, and only runs over those it hasn't processed yet. Since one monitoring ROOT file is produced for each EVIO file, whenever a new file is produced, the plots for the corresponding run are recreated and all the ROOT files for a run are combined into one file. Information is stored in the database on a per-file basis. <br />
<br />
Plots for the monitoring web page can be made from single histograms or multiple histograms using RootSpy macros. If you want to change the list of plots made, you must modify one of the following files:<br />
* histograms_to_monitor - specify either the name of the histogram or its the full ROOT path <br />
* macros_to_monitor - specify the full path to the RootSpy macro .C file<br />
<br />
When a new monitoring run is started, or the conditions are changed, the following steps should be taken to process the new files:<br />
# Add a new data version, as described below:<br />
# Change the following parameters in check_monitoring_data.csh:<br />
## JOBDATE should correspond to the ouptut date used by the job submission script<br />
## OUTPUTDIR should correspond to the directory corresponding to the run period and revision corresponding to the new version you just submitted. Presumably, this directory will be empty at the beginning.<br />
## Once you create a new data version as defined below, you should pass the needed information as a command line option. Currently this is done by the ARGS variable. For example, the argument "-v RunPeriod-2014-10,8" tells the monitoring scripts to look up the version corresponding to revision 8 of RunPeriod-2014-10 in the monitoring DB and to use to store the results.<br />
<br />
<syntaxhighlight><br />
Example configuration parameters:<br />
set JOBDATE=2015-01-09<br />
set INPUTDIR=/volatile/halld/RunPeriod-2014-10/offline_monitoring<br />
set OUTPUTDIR=/w/halld-scifs1a/data_monitoring/RunPeriod-2014-10/ver08<br />
set ARGS=" -v RunPeriod-2014-10,8 "<br />
</syntaxhighlight><br />
If you want to process the results manually, the data is processed using the following script:<br />
<syntaxhighlight><br />
./process_new_offline_data.py <input directory> <output directory><br />
<br />
EXAMPLE:<br />
<br />
./process_new_offline_data.py 2014-11-14 /volatile/halld/RunPeriod-2014-10/offline_monitoring/ /w/halld-scifs1a/data_monitoring/RunPeriod-2014-10/ver02<br />
</syntaxhighlight><br />
The python script takes several options to enable/disable various steps in the processing. Of interest is the "--force" option, which will run over all monitoring ROOT files, whether or not they've been previously identified.<br />
<br />
Every time a new reconstruction pass is performed, a new version number must be generated. To do this, prepare a version file as described below. Then run the register_new_version.py script to store the information in the database. The script will return a version number, which then should be set by hand in process_new_offline_data.py - future versions of the script will streamline this part of the procedure. An example of how to generate a new version is:<br />
<syntaxhighlight><br />
./register_new_version.py add /home/gxproj1/halld/monitoring/process/versions/vers_RunPeriod-2014-10_pass1.txt<br />
</syntaxhighlight><br />
<br />
<b>If you are running the offline monitoring by checking out the files in trunk/scripts/monitoring/jproj/projects/<br />
of the svn repository</b>, and created a project with<br />
create_project.sh [project name] hd_rawdata<br />
Then go to the directory [project name]/processing/<br />
and execute<br />
./run_processing.sh<br />
which will run register_new_version.py as well as check_monitoring_data.csh for that project.<br />
<br />
===Step-by-Step Instructions For Processing a New Monitoring Run===<br />
<br />
The monitoring runs are current run out of the gxproj1 and gxproj5 accounts. After an offline monitoring run has been successfully started on the batch farm, the following steps should be followed to setup the post-processing for these runs.<br />
<br />
# The post-processing scripts are stored in $HOME/halld/monitoring/process and are automatically run by cron.<br />
# Run "svn update" to bring any changes in. Be sure that the list of histograms and macros to plot are current.<br />
# Edit check_monitoring_data.csh to point to the current revisions/directories<br />
#* VERSION<br />
#* ARGS<br />
#* Note that the environment depends on a standard script - $HOME/setup_jlab.csh<br />
# Update files in the web directory, so that the results are displayed on the web pages: /group/halld/www/halldweb/html/data_monitoring/textdata<br />
# Copy the REST files to more permanent locations:<br />
#* cp -a /volatile/halld/offline_monitoring/RunPeriod-YYYY-MM/verVV/REST /work/halld/data_monitoring/RunPeriod-YYYY-MM/REST/verVV<br />
#* cp -a /volatile/halld/offline_monitoring/RunPeriod-YYYY-MM/verVV/REST /cache/halld/RunPeriod-YYYY-MM/REST/verVV [under testing]<br />
<br />
Check log files in $HOME/halld/monitoring/process/log for more information on how each run went. If there are problems, check log files, and modify check_monitoring_data.csh to vary the verbosity of the output.<br />
<br />
==Data Versions==<br />
<br />
To document the conditions of the monitoring data that is created, for the sake of reproducability and further analysis we save several pieces of information. The format is intended to be comprehensive enough to document not just monitoring data, but versions of raw and reconstructed data, so that this database table can be used for the event database as well.<br />
<br />
We store one record per pass through one run period, with the following structure:<br />
<br />
{| class="wikitable"<br />
! Field !! Description<br />
|-<br />
| data_type || The level of data we are processing. For the purposes of monitoring, "rawdata" is the online monitoring, "recon" is the offline monitoring<br />
|- <br />
| run_period || The run period of the data<br />
|- <br />
| revision || An integer specifying which pass through the run period this data corresponds to<br />
|- <br />
| software_version || The name of the XML file that specifies the different software versions used<br />
|-<br />
| jana_config || The name of the text file that specifies which JANA options were passed to the reconstruction program<br />
|-<br />
| ccdb_context || The value of JANA_CALIB_CONTEXT, which specifies the version of calibration constants that were used<br />
|-<br />
| production_time || The data at which monitoring/reconstruction began<br />
|-<br />
| dataVersionString || A convenient string for identifying this version of the data<br />
|}<br />
<br />
<br />
An example file used as as input to ./register_new_version.py is:<br />
<syntaxhighlight><br />
data_type = recon<br />
run_period = RunPeriod-2014-10<br />
revision = 1<br />
software_version = soft_comm_2014_11_06.xml<br />
jana_config = jana_rawdata_comm_2014_11_06.conf<br />
ccdb_context = calibtime=2014-11-10<br />
production_time = 2014-11-10<br />
dataVersionString = recon_RunPeriod-2014-10_20141110_ver01<br />
</syntaxhighlight></div>Kmoriyahttps://halldweb.jlab.org/wiki/index.php?title=Data_Monitoring_Procedures&diff=70783Data Monitoring Procedures2015-10-19T14:27:24Z<p>Kmoriya: /* Starting the Launch and Submitting Jobs */</p>
<hr />
<div>__TOC__<br />
<br />
== Master List of File / Database / Webpage Locations ==<br />
=== Run Conditions ===<br />
* Online Run-by-run condition files (B-field, current, etc.): /work/halld/online_monitoring/conditions/<br />
* Offline monitoring run conditions (software versions, jana config): /group/halld/data_monitoring/run_conditions/<br />
*[http://www.jlab.org/Hall-D/test/RunInfo/ Run Info vers. 1]<br />
*[https://halldweb.jlab.org/cgi-bin/data_monitoring/run_conditions.pl Run Info vers. 2]<br />
<br />
=== Monitoring Output Files ===<br />
* Run Periods 201Y-MM is for example 2015-03, launch ver verVV is for example ver15<br />
* Online monitoring histograms: /work/halld/online_monitoring/root/<br />
* Offline monitoring histogram ROOT files: /work/halld/data_monitoring/RunPeriod-201Y-MM/verVV/rootfiles<br />
* REST files (most recent launch only): /work/halld/data_monitoring/RunPeriod-201Y-MM/REST/verVV<br />
* individual files for each job (ROOT, log, etc.): /volatile/halld/offline_monitoring/RunPeriod-201Y-MM/verVV/<br />
<br />
=== Monitoring Database ===<br />
* Accessing monitoring database (on ifarm): mysql -u datmon -h hallddb.jlab.org data_monitoring<br />
<br />
=== Monitoring Webpages ===<br />
*[https://halldweb.jlab.org/cgi-bin/data_monitoring/monitoring/plotBrowser.py Plot Browser]<br />
*[https://halldweb.jlab.org/cgi-bin/data_monitoring/monitoring/runBrowser.py Run Browser]<br />
*[https://halldweb.jlab.org/cgi-bin/data_monitoring/monitoring/timeSeries.py Time Series]<br />
*[https://halldweb.jlab.org/data_monitoring/launch_analysis/ Launch Analysis]<br />
<br />
== Job Monitoring Links ==<br />
* [http://scicomp.jlab.org/farm2/job.html Completed Job History]<br />
* [http://scicomp.jlab.org/farm2/project.html Job Stats By Project]<br />
* [http://scicomp.jlab.org/farm2/trackOrg.html Job Stats By Track]<br />
* [http://scicomp.jlab.org/farm2/report.html Cluster Report]<br />
* [http://scicomp.jlab.org/farm2/walltime.html Walltime Distribution]<br />
<br />
== Saving Online Monitoring Data ==<br />
<br />
The procedure for writing the data out is given in, e.g.,<br />
[https://halldweb.jlab.org/wiki/index.php/Raid-to-Silo_Transfer_Strategy Raid-to-Silo Transfer Strategy].<br />
<br />
Once the DAQ writes out the data to the raid disk, cron jobs will copy the file to tape,<br />
and within ~20 min., we will have access to the file on tape at<br />
/mss/halld/$RUN_PERIOD/rawdata/RunXXXXXX.<br />
<br />
All online monitoring plugins will be run as data is taken.<br />
They will be accessible within the counting house via RootSpy, and<br />
for each run and file, a ROOT file containing the histograms will be saved<br />
within a subdirectory for each run.<br />
<br />
For immediate access to these files, the raid disk files may be accessed directly<br />
from the counting house, or the tape files will be available within ~20 min. of the<br />
file being written out.<br />
<br />
== Offline Monitoring: Running Over Archived Data ==<br />
<br />
Once files are written to tape we run the online plugins on these files to confirm what we were seeing in the online monitoring, and also to update the results from the latest calibration and software. <br />
Manual scripts and cron jobs are set up to look for new data and run the plugins over a sample of files.<br />
<br />
Every other Friday (usually the Friday before the offline meetings) jobs will be started to run the newest software on all previous runs,and allows everybody to see improvements in each detector.<br />
For each launch, independent builds of hdds, sim-recon, the monitoring plugins, and an sqlite file will be generated.<br />
<br />
Below the procedures are described for<br />
# Preparing the software for the launch<br />
# Starting the launch (using hdswif)<br />
# Post-analysis of statistics of the launch<br />
<br />
Processing the results and making them available to the collaboration<br />
is handled in the section [[#Post-Processing_Procedures | Post-Processing Procedures]] below.<br />
<br />
<!--<br />
==== Generating an offline plugin job ====<br />
<br />
The user gxproj1 should be used for official offline monitoring jobs.<br />
Within /home/gxproj1/halld/monitoring/batch/ there will be scripts to run the online monitoring plugins over tape files.<br />
The main script is generatejobs_plugins_rawdata.sh, which can be used as<br />
generatejobs_plugins_rawdata.sh [minrun] [maxrun] (minfile) (maxfile)<br />
where minrun, maxrun specify the range of the run #, and minfile, maxfile (optional) specify<br />
the file range for that run.<br />
<br />
This will generate a script run_rawdata_XXXXXX.sh, where the run # has now been formatted to be 6 digits.<br />
Executing this script will send the monitoring plugins job to the Auger batch system.<br />
<br />
There is also a script clean.sh which can be used as<br />
clean.sh XXX<br />
This will clean up all associated files created in association with run XXX.<br />
<br />
Internally, the xml file used to submit the job will be created, and the job to run<br />
will be given within script.sh. All run parameters should be specified in at the beginning<br />
of generatejobs_plugins_rawdata.sh<br />
Since we are running on tape, the tape file will first be copied over to the cache disk, and the job will run<br />
over this cached file.<br />
<br />
==== Using cron to run automatically ====<br />
Within /home/gluex/halld/monitoring/cron/ there is a file cron_plugins<br />
that can be executed via<br />
crontab cron_plugins<br />
This will set up a cron job to call the script scan_for_jobs.sh, which will<br />
check in the rawdata directory and call generatejobs_plugins_rawdata.sh for<br />
any run that is more than 5 min old. The cron job is set up to run every 10 min.<br />
<br />
==== Magnetic Field Settings ====<br />
For tracking, it is necessary to set the correct magnetic field settings.<br />
The below table is now obsolete. Please refer to the [https://halldweb.jlab.org/cgi-bin/data_monitoring/run_conditions.pl GlueX Run Conditions] webpage.<br />
<br />
The actual field settings for each run have not been documented well.<br />
Below are values based on going through entries in the [https://logbooks.jlab.org/book/halld halld log].<br />
<br />
{| border="1" cellpadding="1" style="text-align: center;"<br />
!width="100"| Run #s<br />
!width="150"| Solenoid Current (A)<br />
!width="300"| JANA option<br />
!width="300"| notes<br />
|-<br />
| 940 - 996 || 1000 || -PBFIELD_MAP=Magnets/Solenoid/solenoid_1000A_poisson_20141104 || At run 997, the solenoid started ramping down<br />
|-<br />
| 998 - 1448 || 0 || -PBFIELD_TYPE=NoField -PDEFTAG:DTrackCandidate=StraightLine || Solenoid started ramping up around run 1431. Runs 1432 - 1448 should not be used for tracking. See [https://logbooks.jlab.org/entry/3309039 this] and [https://logbooks.jlab.org/entry/3309092 this] entry.<br />
|-<br />
| 1449 - 1620 || 1200 || -PBFIELD_MAP=Magnets/Solenoid/solenoid_1200A_poisson_20140520 || <br />
|-<br />
|}<br />
<br />
'''Note:''' The following run ranges were taken with a 300A field, and could be analyzed with the magnetic field map Magnets/Solenoid/solenoid_300A_poisson_20140819 , more study is needed to see if the straight line track fitter is needed or not: 1036 - 1053, 1065 - 1121, 1212? - 1254, 1309 - 1318 [1307,1308,1319-1329 were taken while the magnet was ramping].<br />
--><br />
<br />
=== General Information on Procedures ===<br />
This section explains how the offline monitoring should be run.<br />
Since we may want to simultaneously run offline monitoring for different run periods that require<br />
different environment variables, the scripts are set up so that a generic user can download the<br />
scripts and run them from anywhere. Most output directories for offline monitoring are created<br />
with group read/write permissions so that any Hall D group user has access to the contents,<br />
but there are some cases where use of the account that created the launch is necessary.<br />
<br />
The accounts used for offline monitoring are the gxprojN accounts created and maintained by<br />
Mark Ito (see [https://halldweb.jlab.org/wiki/index.php/GlueX-related_shared_accounts_on_the_JLab_CUE here] for how each account is used).<br />
As of October 2015, the following are used:<br />
* gxproj1 for running over Fall 2014 data (deprecated since June 2015)<br />
* gxproj5 for running over Spring 2015 data<br />
<br />
Since the summer of 2015 we have transitioned from a system using Mark Ito's jproj scripts<br />
to integrating the swif system that Chris Larrieu (SciComp) has been developing. For offline monitoring,<br />
the hdswif system that Kei developed is used for launching the jobs, and the jproj system is used<br />
for meta-analysis of launch statistics.<br />
<br />
Both hdswif and jproj are maintained in svn:<br />
* hdswif: https://halldsvn.jlab.org/repos/trunk/scripts/monitoring/hdswif<br />
* jproj : https://halldsvn.jlab.org/repos/trunk/scripts/monitoring/jproj<br />
<br />
To run the offline monitoring each package should be checked out and all necessary<br />
scripts are included.<br />
<br />
=== Preparing the software for the launch ===<br />
<br />
To begin a new launch the software must be built to the latest versions.<br />
For the gxprojN user accounts used, all software builds are contained in the directory ~/builds<br />
(which are soft links to /work/halld/home/gxprojN/builds). When logging into these accounts<br />
the setup files ~/setup_jlab-2015-03.csh or similar files should be sourced.<br />
<br />
<b>Note that Mark Ito does <u>not</u> want you to change the contents of each .cshrc file.</b><br />
You should consult him if you feel the need.<br />
<br />
<span style="color:red">NOTE: FOR BUILDING SOFTWARE IT IS A GOOD IDEA TO DO A COMPLETE WIPEOUT/CHECKOUT EACH TIME TO AVOID STALE HEADER FILES.</span><br />
<br />
# Building hdds: Go to ~/builds/hdds. The directory hdds is the one from git. Delete the contents, then download the newest version from git and build: <pre>cd ~/builds/hdds/</pre> <pre>rm -frv hdds</pre> <pre>git clone https://github.com/JeffersonLab/hdds</pre> <pre>scons install</pre><br />
# Building sim-recon: Go to ~/builds/sim-recon. The directory sim-recon is the one from git. Delete the contents, then download the newest version from git and build: <pre>cd ~/builds/sim-recon</pre> <pre>rm -frv sim-recon</pre> <pre>git clone https://github.com/JeffersonLab/sim-recon</pre> <pre>scons install -j8</pre><br />
# Prepare the latest sqlite file: The sqlite is set in the ~/setup_jlab-2015-03.csh script as sqlite:////home/gxproj5/builds/ccdb.sqlite through the environment variables <b>JANA_CALIB_URL</b> and <b>CCDB_CONNECTION</b>. Therefore, go to ~/builds,and create a new sqlite file. We create the sqlite file in a temporary directory since creating the sqlite file in a directory where the output file exists causes errors. Original documentation on creating sqlite files are [https://halldweb.jlab.org/wiki/index.php/SQLite-form_of_the_CCDB_database here]. <pre>cd ~/builds/tmp</pre> <pre>$CCDB_HOME/scripts/mysql2sqlite/mysql2sqlite.sh -hhallddb.jlab.org -uccdb_user ccdb | sqlite3 ccdb.sqlite</pre> <pre>mv ccdb.sqlite ../</pre><br />
# Note that the above steps must be done <b>BEFORE</b> launch project creation. This is because we will track the revisions of the libraries used, and this is done by extracting the svn information in each directory. Also, note that the system assumes that we have the topmost build directory (usually called GLUEX_TOP) to be $HOME/builds . Such an assumption is necessary to be able to extract information about the library locations automatically.<br />
<br />
Create the appropriate project(s) and submit the jobs using hdswif, as detailed in the section below.<br />
<br />
=== Starting the Launch and Submitting Jobs ===<br />
<br />
Until the summer of 2015 we relied solely on Mark Ito's jproj system for submitting and keeping track of jobs. We have since moved to the swif system and use the hdswif wrapper for this. Below are instructions for how to use these.<br />
<br />
# Downloading hdswif: Download the hdswif directory from svn. For the gxprojN accounts, use the directory ~/halld/hdswif. <pre>cd ~/halld </pre> <pre> svn co https://halldsvn.jlab.org/repos/trunk/scripts/monitoring/hdswif</pre> <pre>cd hdswif</pre><br />
# Creating the workflow: Within SWIF jobs are registered into workflows. First create the workflow. For offline monitoring, the workflow names are of the form <b>offline_monitoring_RunPeriod201Y_MM_verVV_hd_rawdata</b> with suitable replacements for the run period and version number. The command "swif list" will list all existing workflows. Also, for most simple SWIF commands hdswif also provides a wrapper. <pre>swif list</pre> For creation of workflows for offline monitoring the command <pre>hdswif.py create [workflow] -c [config file] </pre> should be used. When a config file is passed in, hdswif will automatically create files that record the configuration of the current launch. These files are stored as for example<br />
#* /group/halld/data_monitoring/run_conditions/jana_rawdata_comm_2015_03_ver15.conf<br />
#* /group/halld/data_monitoring/run_conditions/soft_comm_2015_03_ver15.xml<br />
#The software packages stored in git (sim-recon and hdds) can have git tags applied to them, which makes it easier to find versions of the software than a SHA-1 hash. hdswif will ask if you would like to create a tag, and execute the following sequence: <pre>git tag -a offmon-201Y_MM-verVV -m "Used for offline monitoring 201Y-MM verVV started on 201y/mm/dd"</pre> <pre>git push offmon-201Y_MM-verVV</pre> This will only be invoked when the user name is gxprojN, and for the configuration files, the output directory will be /group/halld/data_monitoring/run_conditions/ for gxprojN accounts while it will be the current directory for other users.<br />
# Registering jobs in the workflow: To register jobs within the workflow, hdswif provides the use of config files. Jobs can be registered by specifying the workflow, config file (-c), run (-r) and file (-f) numbers if necessary. A typical config file will look this:<br />
PROJECT gluex<br />
TRACK reconstruction<br />
OS centos65<br />
NCORES 6<br />
DISK 40<br />
RAM 8<br />
TIMELIMIT 8<br />
JOBNAMEBASE offmon_<br />
RUNPERIOD 2015-03<br />
VERSION 15<br />
OUTPUT_TOPDIR /volatile/halld/offline_monitoring/RunPeriod-[RUNPERIOD]/ver[VERSION] # Example of other variables included in variable<br />
SCRIPTFILE /home/gxproj5/halld/hdswif/script.sh # Must specify full path<br />
ENVFILE /home/gxproj5/halld/hdswif/setup_jlab-2015-03.csh # Must specify full path<br />
The config file contains configuration parameters for each of the jobs.<br><br />
<span style="color:red">Note: Job configuration parameters can be set differently for jobs within the same workflow if necessary.</span><br />
Edit the config file and save as a new file if necessary. Once the configuration is set, jobs can be added via <pre>hdswif.py add [workflow] -c input.config</pre><br />
By default, hdswif will add all files found within the directory /mss/halld/RunPeriod-201Y-MM/rawdata/ where 201Y-MM is specified by the RUNPERIOD parameter in the config file. If only some of the runs or files are needed, these can be specified for example with<br />
<pre>hdswif.py add [workflow] -c input.config -r 3180 -f '00[0-4]'</pre><br />
to specify to register running only over run 3180 files 000 - 004 (Unix-style brackets and wildcards can be used).<br />
<br />
Running the workflow: To run the workflow, simply use swif run: <pre>swif run </pre> or equivalently, using the hdswif wrapper, <pre>hdsswif.py run [workflow]</pre><br><br />
<b>It is recommended that some jobs be tested to make sure that everything is working rather than fail thousands of jobs.</b><br> For this purpose, hdswif will take an additional parameter to run which limits the number of jobs to submit: <pre>hdswif.py run [workflow] 10</pre><br />
in which case only 10 jobs will be submitted. To submit all jobs after checking the results, do <pre>hdswif.py run [workflow]</pre><br />
<br />
=== Post-analysis of statistics of the launch ===<br />
<br />
# After jobs have been submitted, it will usually take a few days for all of the jobs to be processed.<br />
#* Status of Auger: http://scicomp.jlab.org/scicomp/#/auger/jobs (see also links above)<br />
#* Status of user jobs: <pre>jobstat -u [user name]</pre><br />
# The status and results of jobs are saved within the SWIF internal server, and are available via the command <pre>swif status [workflow] -summary -runs</pre> where the arguments -summary and -runs show summary statistics and statistics for individual jobs, respectively. hdswif has a command that takes this output in XML output and creates an HTML webpage showing results of the launch. To do this, do <pre>hdswif.py summary [workflow]</pre> This will create an XML file swif_output_[workflow].xml that contains all information from SWIF. If the file already exists, hdswif will ask whether to overwrite the existing file.<br />
# At this stage the html output and figure files are created and ready to be put online. For this step and other steps involving analysis of the statistics of the launch results, it is convenient to change to the jproj system.<br />
# The jproj scripts for offline monitoring are maintained in the svn directory https://halldsvn.jlab.org/repos/trunk/scripts/monitoring/jproj Do <pre> svn co https://halldsvn.jlab.org/repos/trunk/scripts/monitoring/jproj </pre> For the gxprojN accounts used for offline monitoring the directory should be ~/halld/jproj<br />
# The jproj directory contains two subdirectories, scripts and projects. The scripts directory contains useful scripts for processing the jobs registered in the jproj system, and each of the offline monitoring launches will be handled in the projects directory.<br />
# Go to the projects directory<pre>cd ~/halld/jproj/projects</pre> and use the script create_project.sh to create a new directory that contains the processing scripts for the current launch <pre>./create_project.sh [workflow]</pre> This should create a directory such as offline_monitoring_RunPeriod2015_03_ver15_hd_rawdata (same as the workflow name). The script uses the template files in the directory templates and by substitution creates script files for the current launch. Now go to the newly created analysis directory: <pre>cd [workflow]/analysis</pre><br />
which for the gxprojN accounts should have the full path /home/gxprojN/halld/jproj/projects/[workflow]/analysis.<br><br><br />
<span style="color:red">All of the analysis commands including arguments are contained in run_analysis.sh, but it is <b>strongly</b> recommended that all commands are run manually to check for errors.</span><br><br><br />
# The first thing to do is to make the html output from hdswif public. Copy the html file and related figures that were created from hdswif to the appropriate space within /group/halld/www/halldweb/html/data_monitoring/ : <pre>python publish_offmon_results.py [run period] [version]</pre> Note that the command with appropriate substitutions for arguments can be found within run_analysis.sh (same for all commands below).<br />
# Next, we need to create a few MySQL tables for the current launch. The MySQL tables are useful for comparing run/file combinations for different launches. For the SWIF launches, there are two tables needed, and will be named<br />
#* [workflow]Job<br />
#* [workflow]_aux<br />
# This naming scheme of tables and their roles are the same as from the jproj only launches. The [workflow]Job table will contain information gathered from SWIF about each job (which node it went to, start time of each stage, memory usage, etc.). The [workflow]_aux table will contain information gathered from the output stdout files from each job. First, create the Job table using <pre>python create_jproj_job_table.py [run period] [version]</pre> Note that this script uses the XML output from hdswif summary and inserts the contents into the MySQL table, so the XML output file must exist.<br />
# To check the contents of this MySQL table, do <pre>mysql -hhalldb -ufarmer farming</pre> <pre>mysql> describe offline_monitoring_RunPeriod2015_03_ver11_hd_rawdataJob;</pre><br />
<br />
== Hall D Job Management System ==<br />
<br />
This section details instructions on how to create and launch a set of jobs using the Hall-D Job Management System developed by Mark Ito. These instructions are generic: this system can be used for the weekly monitoring jobs, but can also be used for other sets of job launches as well. <br />
<br />
=== Database Table Overview ===<br />
<br />
* Job management database table (<project_name>): For each input file, keeps track of whether or not a job for it has been submitted, along with other optional fields. <br />
<br />
* Job status database table (<project_name>Job (no space)): For each job, keeps track of the job-id, the job status, memory used, cpu & wall time, time taken to complete various stages (e.g. pending, dependency, active), and others. <br />
<br />
* Job metrics database table (<project_name>_aux (no space)): For each job, keeps track of the job-id, how many events were processed, the time it took to copy the cache file, and the time it took to run the plugin. This information is culled from the log files of each job, and is done within the analysis directory of each launch.<br />
<br />
=== Initialize Project Management ===<br />
<br />
* Log into the ifarm machine with one of the gxproj accounts. For this example we will use gxproj1.<br />
<pre><br />
ssh gxproj1@ifarm -Y<br />
</pre><br />
<br />
* Go to a directory to do the launch. In principle, any directory will work, but for gxproj1 this is usually done in /home/gxproj1/halld/<br />
<br />
* Check out the necessary scripts <pre>svn co https://halldsvn.jlab.org/repos/trunk/scripts/monitoring/jproj</pre> This will get all necessary scripts for launching. Once checked out, <pre>cd projects</pre><br />
<br />
* The script create_project.sh can be used to create a new project. It will take a single argument, the project name. It is assumed that the project name is of the form offline_monitoring_RunPeriod20YY_MM_verVV_hd_rawdata where this string will be parsed to give the run period 20YY_MM and version number VV. One thing to do BEFORE creation of a new project is to editthe conditions of the launch (plugins to run over, memory requested, disk space requested) within templates/template.jsub . This<br />
information is saved automatically at project creation time into files in /group/halld/data_monitoring/run_conditions .<br />
To create a project do <pre>./create_project.sh offline_monitoring_RunPeriod20YY_MM_verVV_hd_rawdata</pre><br />
The name has been chosen to be as consistent as possible with other directory structures.<br />
However, mysql requires that "-" be escaped in table names, so unfortunately run periods will be<br />
given as 2YYY_MM instead of 2YYY-MM.<br />
<br />
* A new directory with the project name (offline_monitoring_RunPeriod20YY_MM_verVV_hd_rawdata) and files will be copied and modified from the template directory to reflect the run period, the user, the directory that it was created in, project name, etc.<br />
<br />
* For each project, cd into the new directory<br />
* The script clear.sh will remove any existing tables for the current project name, then recreate it. Do<br />
<pre>./clear.sh</pre><br />
* To use the jproj.pl script that was checked in, add the directory to your path with<br />
<pre>source ../../scripts/setup.csh</pre><br />
or always specify the full path<br />
<pre>../../scripts/jproj.pl</pre><br />
* Now update the table of runs with<br />
<pre>jproj.pl <project name> update</pre><br />
This will fill the table with all files within /mss that are of the same form as what is in <project name>.jproj . If you want to register only on a subset of all such files, you can edit this file directly.<br />
* Once you have registered all of the files you would like to run over, do<br />
<pre>jproj.pl <project name> submit [max # of jobs] [run number]</pre><br />
where the additional options specify how many jobs to submit and which run number to run on. Without these options all files that are registered and have not been submitted yet will be submitted.<br />
<br />
At this stage you are ready to submit all files. It is a good idea to submit a few test jobs<br />
at first to check that all scripts are working and that the plugins do not crash. Once you are<br />
sure that this does not happen, you can send all jobs in. The remaining jobs are then the monitoring<br />
which will (among other things) put the results on the online webpage for the collaboration to view, and<br />
the analysis of the launch.<br />
<br />
=== Project File Overview ===<br />
<br />
An overview of each project file:<br />
* '''clear.sh:''' For the current project, deletes the job status and management database tables (if any), and creates new, empty ones. <br />
* '''<project_name>.jproj:''' Contains the path and file name format for the input files for the jobs. <br />
* '''<project_name>.jsub:''' The xml job submission script. The run number and file number variables are set during job submission for each input file. <br />
* '''script.sh:''' The script that is executed during the job. If output job directories are not pre-created manually, they should be created in this script with the proper permissions:<br />
<pre><br />
mkdir -p -m 775 my_directory<br />
</pre><br />
* '''setup_jlab-[run period].csh:''' The environment that is sourced at the beginning of the job execution. <br />
* '''status.sh:''' Updates the job status database table, and prints some of its columns to screen.<br />
<br />
=== Project Management ===<br />
<br />
* Delete (if any) and create the database table(s) for the current set of job submissions:<br />
<pre><br />
./clear.sh<br />
</pre><br />
Also, if testing was done with jobs, it is best to delete the output directory and the configuration files:<br />
<pre><br />
rm -frv /volatile/halld/offline_monitoring/RunPeriod-20YY-MM/verVV /group/halld/data_monitoring/run_conditions/soft_comm_20YY_MM_verVV.xml /group/halld/data_monitoring/run_conditions/jana_rawdata_comm_20YY_MM_verVV.conf<br />
</pre><br />
<br />
* Search for input files matching the string in the .jproj file, and create a row for each in the job management database table (called <project_name>). You can test by adding an optional argument at the end, which only selects files with a specific file number:<br />
<pre><br />
jproj.pl <project_name> update <optional_file_number><br />
</pre><br />
<br />
* Confirm that the job management database is accurate by printing it's contents to screen:<br />
<pre><br />
mysql -hhallddb -ufarmer farming -e "select * from <project_name>"<br />
</pre><br />
<br />
* ONLY if a mistake was made, to delete the tables from the database and recreate new, empty ones, run: <br />
<pre><br />
./clear.sh<br />
</pre><br />
<br />
* To look at the status of the submitted jobs, first query auger and update the job status database:<br />
<pre><br />
fill_in_job_details.pl <project_name><br />
</pre><br />
<br />
* The job status can then be viewed by submitting a query to the job status database (called <project_name>Job (no space in between)):<br />
<pre><br />
mysql -hhallddb -ufarmer farming -e "select id,run,file,jobId,hostname,status,timeSubmitted,timeActive,walltime,cput,timeComplete,result,error from <project_name>Job"<br />
</pre><br />
<br />
* These last two commands can instead be executed simultaneously by running:<br />
<pre><br />
./status.sh<br />
</pre><br />
<br />
=== Handy mysql Instructions ===<br />
<br />
* Handy mysql instructions:<br />
<pre><br />
mysql -hhallddb -ufarmer farming # Enter the "farming" mysql database on "hallddb" as user "farmer"<br />
quit; # Exit mysql<br />
show tables; # Show a list of the tables in the current database<br />
show columns from <project_name>; # show all of the columns for the given table<br />
select * from <project_name>; # show the contents of all rows from the given table<br />
</pre><br />
<br />
=== Backing Up Offline Monitoring Tables ===<br />
Tables created for offline monitoring can be backed up using the script backup_tables.sh which<br />
can be checked out with the other files from https://halldsvn.jlab.org/repos/trunk/scripts/monitoring/jproj/projects<br />
<br />
The script uses the command mysqldump to print out a file that can be executed to recreate the tables.<br />
Since executing this output file will drop the table when it exists, caution is advised.<br />
Example usage to backup all three tables created for run period 2014_10 ver 17:<br />
<pre>backup_tables.sh 2014_10 17</pre><br />
<br />
== Running Over Data As It Comes In ==<br />
<br />
A special user gxproj1 will have a cron job set up to run the plugins as new data appears on /mss.<br />
During the week, gxproj1 will submit offline plugin jobs with the same setup as the weekly jobs<br />
run the previous Friday. The procedure for this is shown below.<br />
<br />
<!--<br />
=== Setting up the environment ===<br />
The file<br />
/home/gxproj1/setup_jlab.csh<br />
is sourced through .tcshrc.<br />
This file is the same as what is linked to by<br />
/home/gluex/setup_jlab_commissioning.csh,<br />
except HALLD_HOME, HDDS_HOME, and JANA_CALIB_URL are set separately so that this<br />
user can have a separate build.<br />
<br />
To obtain the builds from the previous Friday's runs,<br />
execute<br />
/home/gxproj1/halld/monitoring/newruns/setup_previous.sh [year] [month] [day]<br />
The build revisions from the previous Friday are archived in files<br />
/work/halld/data_monitoring/run_conditions/soft_comm_[year]_[month]_[day].xml<br />
and the script will build libraries based on those stored revision numbers.<br />
--><br />
<br />
=== Running the cron job ===<br />
<br />
'''IMPORTANT:''' The cron job should not be running while you are manually submitting jobs using the jproj.pl script for the same project, or else you will probably multiply-submit a job. <br />
<br />
* Go to the cron job directory: <br />
<pre><br />
cd /u/home/gxproj1/halld/monitoring/newruns<br />
</pre><br />
<br />
* The cron_plugins file is the cronjob that will be executed. During execution, it runs the exec.sh command in the same folder. This command takes two arguments: the project name, and the maximum file number for each run. These fields should be updated in the cron_plugins file before running. <br />
<br />
* The exec.sh command updates the job management database table with any data that has arrived on tape since it was last updated, ignoring file numbers greater than the maximum file number. It then submits jobs for these files. <br />
<br />
* To start the cron job, run:<br />
<pre><br />
crontab cron_plugins<br />
</pre><br />
<br />
* To check whether the cron job is running, do<br />
<pre><br />
crontab -l<br />
</pre><br />
<br />
* To remove the cron job do<br />
<pre><br />
crontab -r<br />
</pre><br />
<br />
<!--<br />
The cron job will run the script scan_for_jobs.sh,<br />
which runs generatejobs_plugins_rawdata.sh for any<br />
new runs that it had not seen before. All previous<br />
runs are recorded in the file filelists/files_current.txt<br />
so clear this to run over runs, or set the parameters<br />
MINRUN and MAXRUN which will set the range of runs submitted.<br />
--><br />
<br />
==Post-Processing Procedures==<br />
<br />
To visualize the monitoring data, we save images of selected histograms and store time series of selected quantities in a database, which are then displayed on a web page. This section describes how to generate the monitoring images and database information.<br />
<br />
The scripts used to generate this summary data are primarily run from /home/gxprojN/halld/monitoring/process . If you want a new copy of the scripts, e.g., for a new monitoring run, you should check the scripts out from SVN:<br />
<syntaxhighlight><br />
svn co https://halldsvn.jlab.org/repos/trunk/scripts/monitoring/process<br />
</syntaxhighlight><br />
Note that these scripts currently have some parameters which must be periodically set by hand.<br />
<br />
The default python version on most JLab machine does not have the modules to allow these scripts to connect to the MySQL database. To run these scripts, load the environment with the following command<br />
<syntaxhighlight><br />
source /home/gxproj1/halld/monitoring/process/monitoring_env.csh<br />
</syntaxhighlight><br />
<br />
===Online Monitoring===<br />
<br />
There are two primary scripts for running over the monitoring data generated by the online system and offline reconstruction. The online script can be run with either of the following commands:<br />
<syntaxhighlight><br />
/home/gluex/halld/monitoring/process/check_new_runs.py<br />
<br />
OR <br />
<br />
/home/gluex/halld/monitoring/process/check_new_runs.csh<br />
</syntaxhighlight><br />
The shell script sets up the environment properly to run the python script. To connect to the monitoring database on the JLab CUE, modules included in the local installation of python >= 2.7 are needed. The shell script is appropriate to use in a cron job. The cronjob is currently run under the "gluex" account.<br />
<br />
The online monitoring system copies a ROOT file containing the results of the online monitoring, and other configuration files into a directory accessible outside the counting house. This python script automatically checks for new ROOT files, which it will then automatically process. It contains several configuration variables that must be correctly set, which contains the location of input/output directories, etc...<br />
<br />
===Offline Monitoring===<br />
<br />
After the data is run over, the results should be processed, so that summary data is entered into the monitoring database and plots are made for the monitoring webpages. Currently, this processing is controlled by a cronjob that runs the following script:<br />
<syntaxhighlight><br />
/home/gxproj1/halld/monitoring/process/check_monitoring_data.csh <br />
</syntaxhighlight><br />
This script checks for new ROOT files, and only runs over those it hasn't processed yet. Since one monitoring ROOT file is produced for each EVIO file, whenever a new file is produced, the plots for the corresponding run are recreated and all the ROOT files for a run are combined into one file. Information is stored in the database on a per-file basis. <br />
<br />
Plots for the monitoring web page can be made from single histograms or multiple histograms using RootSpy macros. If you want to change the list of plots made, you must modify one of the following files:<br />
* histograms_to_monitor - specify either the name of the histogram or its the full ROOT path <br />
* macros_to_monitor - specify the full path to the RootSpy macro .C file<br />
<br />
When a new monitoring run is started, or the conditions are changed, the following steps should be taken to process the new files:<br />
# Add a new data version, as described below:<br />
# Change the following parameters in check_monitoring_data.csh:<br />
## JOBDATE should correspond to the ouptut date used by the job submission script<br />
## OUTPUTDIR should correspond to the directory corresponding to the run period and revision corresponding to the new version you just submitted. Presumably, this directory will be empty at the beginning.<br />
## Once you create a new data version as defined below, you should pass the needed information as a command line option. Currently this is done by the ARGS variable. For example, the argument "-v RunPeriod-2014-10,8" tells the monitoring scripts to look up the version corresponding to revision 8 of RunPeriod-2014-10 in the monitoring DB and to use to store the results.<br />
<br />
<syntaxhighlight><br />
Example configuration parameters:<br />
set JOBDATE=2015-01-09<br />
set INPUTDIR=/volatile/halld/RunPeriod-2014-10/offline_monitoring<br />
set OUTPUTDIR=/w/halld-scifs1a/data_monitoring/RunPeriod-2014-10/ver08<br />
set ARGS=" -v RunPeriod-2014-10,8 "<br />
</syntaxhighlight><br />
If you want to process the results manually, the data is processed using the following script:<br />
<syntaxhighlight><br />
./process_new_offline_data.py <input directory> <output directory><br />
<br />
EXAMPLE:<br />
<br />
./process_new_offline_data.py 2014-11-14 /volatile/halld/RunPeriod-2014-10/offline_monitoring/ /w/halld-scifs1a/data_monitoring/RunPeriod-2014-10/ver02<br />
</syntaxhighlight><br />
The python script takes several options to enable/disable various steps in the processing. Of interest is the "--force" option, which will run over all monitoring ROOT files, whether or not they've been previously identified.<br />
<br />
Every time a new reconstruction pass is performed, a new version number must be generated. To do this, prepare a version file as described below. Then run the register_new_version.py script to store the information in the database. The script will return a version number, which then should be set by hand in process_new_offline_data.py - future versions of the script will streamline this part of the procedure. An example of how to generate a new version is:<br />
<syntaxhighlight><br />
./register_new_version.py add /home/gxproj1/halld/monitoring/process/versions/vers_RunPeriod-2014-10_pass1.txt<br />
</syntaxhighlight><br />
<br />
<b>If you are running the offline monitoring by checking out the files in trunk/scripts/monitoring/jproj/projects/<br />
of the svn repository</b>, and created a project with<br />
create_project.sh [project name] hd_rawdata<br />
Then go to the directory [project name]/processing/<br />
and execute<br />
./run_processing.sh<br />
which will run register_new_version.py as well as check_monitoring_data.csh for that project.<br />
<br />
===Step-by-Step Instructions For Processing a New Monitoring Run===<br />
<br />
The monitoring runs are current run out of the gxproj1 and gxproj5 accounts. After an offline monitoring run has been successfully started on the batch farm, the following steps should be followed to setup the post-processing for these runs.<br />
<br />
# The post-processing scripts are stored in $HOME/halld/monitoring/process and are automatically run by cron.<br />
# Run "svn update" to bring any changes in. Be sure that the list of histograms and macros to plot are current.<br />
# Edit check_monitoring_data.csh to point to the current revisions/directories<br />
#* VERSION<br />
#* ARGS<br />
#* Note that the environment depends on a standard script - $HOME/setup_jlab.csh<br />
# Update files in the web directory, so that the results are displayed on the web pages: /group/halld/www/halldweb/html/data_monitoring/textdata<br />
# Copy the REST files to more permanent locations:<br />
#* cp -a /volatile/halld/offline_monitoring/RunPeriod-YYYY-MM/verVV/REST /work/halld/data_monitoring/RunPeriod-YYYY-MM/REST/verVV<br />
#* cp -a /volatile/halld/offline_monitoring/RunPeriod-YYYY-MM/verVV/REST /cache/halld/RunPeriod-YYYY-MM/REST/verVV [under testing]<br />
<br />
Check log files in $HOME/halld/monitoring/process/log for more information on how each run went. If there are problems, check log files, and modify check_monitoring_data.csh to vary the verbosity of the output.<br />
<br />
==Data Versions==<br />
<br />
To document the conditions of the monitoring data that is created, for the sake of reproducability and further analysis we save several pieces of information. The format is intended to be comprehensive enough to document not just monitoring data, but versions of raw and reconstructed data, so that this database table can be used for the event database as well.<br />
<br />
We store one record per pass through one run period, with the following structure:<br />
<br />
{| class="wikitable"<br />
! Field !! Description<br />
|-<br />
| data_type || The level of data we are processing. For the purposes of monitoring, "rawdata" is the online monitoring, "recon" is the offline monitoring<br />
|- <br />
| run_period || The run period of the data<br />
|- <br />
| revision || An integer specifying which pass through the run period this data corresponds to<br />
|- <br />
| software_version || The name of the XML file that specifies the different software versions used<br />
|-<br />
| jana_config || The name of the text file that specifies which JANA options were passed to the reconstruction program<br />
|-<br />
| ccdb_context || The value of JANA_CALIB_CONTEXT, which specifies the version of calibration constants that were used<br />
|-<br />
| production_time || The data at which monitoring/reconstruction began<br />
|-<br />
| dataVersionString || A convenient string for identifying this version of the data<br />
|}<br />
<br />
<br />
An example file used as as input to ./register_new_version.py is:<br />
<syntaxhighlight><br />
data_type = recon<br />
run_period = RunPeriod-2014-10<br />
revision = 1<br />
software_version = soft_comm_2014_11_06.xml<br />
jana_config = jana_rawdata_comm_2014_11_06.conf<br />
ccdb_context = calibtime=2014-11-10<br />
production_time = 2014-11-10<br />
dataVersionString = recon_RunPeriod-2014-10_20141110_ver01<br />
</syntaxhighlight></div>Kmoriyahttps://halldweb.jlab.org/wiki/index.php?title=Data_Monitoring_Procedures&diff=70742Data Monitoring Procedures2015-10-15T14:46:08Z<p>Kmoriya: /* Offline Monitoring: Running Over Archived Data */</p>
<hr />
<div>__TOC__<br />
<br />
== Master List of File / Database / Webpage Locations ==<br />
=== Run Conditions ===<br />
* Online Run-by-run condition files (B-field, current, etc.): /work/halld/online_monitoring/conditions/<br />
* Offline monitoring run conditions (software versions, jana config): /group/halld/data_monitoring/run_conditions/<br />
*[http://www.jlab.org/Hall-D/test/RunInfo/ Run Info vers. 1]<br />
*[https://halldweb.jlab.org/cgi-bin/data_monitoring/run_conditions.pl Run Info vers. 2]<br />
<br />
=== Monitoring Output Files ===<br />
* Run Periods 201Y-MM is for example 2015-03, launch ver verVV is for example ver15<br />
* Online monitoring histograms: /work/halld/online_monitoring/root/<br />
* Offline monitoring histogram ROOT files: /work/halld/data_monitoring/RunPeriod-201Y-MM/verVV/rootfiles<br />
* REST files (most recent launch only): /work/halld/data_monitoring/RunPeriod-201Y-MM/REST/verVV<br />
* individual files for each job (ROOT, log, etc.): /volatile/halld/offline_monitoring/RunPeriod-201Y-MM/verVV/<br />
<br />
=== Monitoring Database ===<br />
* Accessing monitoring database (on ifarm): mysql -u datmon -h hallddb.jlab.org data_monitoring<br />
<br />
=== Monitoring Webpages ===<br />
*[https://halldweb.jlab.org/cgi-bin/data_monitoring/monitoring/plotBrowser.py Plot Browser]<br />
*[https://halldweb.jlab.org/cgi-bin/data_monitoring/monitoring/runBrowser.py Run Browser]<br />
*[https://halldweb.jlab.org/cgi-bin/data_monitoring/monitoring/timeSeries.py Time Series]<br />
*[https://halldweb.jlab.org/data_monitoring/launch_analysis/ Launch Analysis]<br />
<br />
== Job Monitoring Links ==<br />
* [http://scicomp.jlab.org/farm2/job.html Completed Job History]<br />
* [http://scicomp.jlab.org/farm2/project.html Job Stats By Project]<br />
* [http://scicomp.jlab.org/farm2/trackOrg.html Job Stats By Track]<br />
* [http://scicomp.jlab.org/farm2/report.html Cluster Report]<br />
* [http://scicomp.jlab.org/farm2/walltime.html Walltime Distribution]<br />
<br />
== Saving Online Monitoring Data ==<br />
<br />
The procedure for writing the data out is given in, e.g.,<br />
[https://halldweb.jlab.org/wiki/index.php/Raid-to-Silo_Transfer_Strategy Raid-to-Silo Transfer Strategy].<br />
<br />
Once the DAQ writes out the data to the raid disk, cron jobs will copy the file to tape,<br />
and within ~20 min., we will have access to the file on tape at<br />
/mss/halld/$RUN_PERIOD/rawdata/RunXXXXXX.<br />
<br />
All online monitoring plugins will be run as data is taken.<br />
They will be accessible within the counting house via RootSpy, and<br />
for each run and file, a ROOT file containing the histograms will be saved<br />
within a subdirectory for each run.<br />
<br />
For immediate access to these files, the raid disk files may be accessed directly<br />
from the counting house, or the tape files will be available within ~20 min. of the<br />
file being written out.<br />
<br />
== Offline Monitoring: Running Over Archived Data ==<br />
<br />
Once files are written to tape we run the online plugins on these files to confirm what we were seeing in the online monitoring, and also to update the results from the latest calibration and software. <br />
Manual scripts and cron jobs are set up to look for new data and run the plugins over a sample of files.<br />
<br />
Every other Friday (usually the Friday before the offline meetings) jobs will be started to run the newest software on all previous runs,and allows everybody to see improvements in each detector.<br />
For each launch, independent builds of hdds, sim-recon, the monitoring plugins, and an sqlite file will be generated.<br />
<br />
Below the procedures are described for<br />
# Preparing the software for the launch<br />
# Starting the launch (using hdswif)<br />
# Post-analysis of statistics of the launch<br />
<br />
Processing the results and making them available to the collaboration<br />
is handled in the section [[#Post-Processing_Procedures | Post-Processing Procedures]] below.<br />
<br />
<!--<br />
==== Generating an offline plugin job ====<br />
<br />
The user gxproj1 should be used for official offline monitoring jobs.<br />
Within /home/gxproj1/halld/monitoring/batch/ there will be scripts to run the online monitoring plugins over tape files.<br />
The main script is generatejobs_plugins_rawdata.sh, which can be used as<br />
generatejobs_plugins_rawdata.sh [minrun] [maxrun] (minfile) (maxfile)<br />
where minrun, maxrun specify the range of the run #, and minfile, maxfile (optional) specify<br />
the file range for that run.<br />
<br />
This will generate a script run_rawdata_XXXXXX.sh, where the run # has now been formatted to be 6 digits.<br />
Executing this script will send the monitoring plugins job to the Auger batch system.<br />
<br />
There is also a script clean.sh which can be used as<br />
clean.sh XXX<br />
This will clean up all associated files created in association with run XXX.<br />
<br />
Internally, the xml file used to submit the job will be created, and the job to run<br />
will be given within script.sh. All run parameters should be specified in at the beginning<br />
of generatejobs_plugins_rawdata.sh<br />
Since we are running on tape, the tape file will first be copied over to the cache disk, and the job will run<br />
over this cached file.<br />
<br />
==== Using cron to run automatically ====<br />
Within /home/gluex/halld/monitoring/cron/ there is a file cron_plugins<br />
that can be executed via<br />
crontab cron_plugins<br />
This will set up a cron job to call the script scan_for_jobs.sh, which will<br />
check in the rawdata directory and call generatejobs_plugins_rawdata.sh for<br />
any run that is more than 5 min old. The cron job is set up to run every 10 min.<br />
<br />
==== Magnetic Field Settings ====<br />
For tracking, it is necessary to set the correct magnetic field settings.<br />
The below table is now obsolete. Please refer to the [https://halldweb.jlab.org/cgi-bin/data_monitoring/run_conditions.pl GlueX Run Conditions] webpage.<br />
<br />
The actual field settings for each run have not been documented well.<br />
Below are values based on going through entries in the [https://logbooks.jlab.org/book/halld halld log].<br />
<br />
{| border="1" cellpadding="1" style="text-align: center;"<br />
!width="100"| Run #s<br />
!width="150"| Solenoid Current (A)<br />
!width="300"| JANA option<br />
!width="300"| notes<br />
|-<br />
| 940 - 996 || 1000 || -PBFIELD_MAP=Magnets/Solenoid/solenoid_1000A_poisson_20141104 || At run 997, the solenoid started ramping down<br />
|-<br />
| 998 - 1448 || 0 || -PBFIELD_TYPE=NoField -PDEFTAG:DTrackCandidate=StraightLine || Solenoid started ramping up around run 1431. Runs 1432 - 1448 should not be used for tracking. See [https://logbooks.jlab.org/entry/3309039 this] and [https://logbooks.jlab.org/entry/3309092 this] entry.<br />
|-<br />
| 1449 - 1620 || 1200 || -PBFIELD_MAP=Magnets/Solenoid/solenoid_1200A_poisson_20140520 || <br />
|-<br />
|}<br />
<br />
'''Note:''' The following run ranges were taken with a 300A field, and could be analyzed with the magnetic field map Magnets/Solenoid/solenoid_300A_poisson_20140819 , more study is needed to see if the straight line track fitter is needed or not: 1036 - 1053, 1065 - 1121, 1212? - 1254, 1309 - 1318 [1307,1308,1319-1329 were taken while the magnet was ramping].<br />
--><br />
<br />
=== General Information on Procedures ===<br />
This section explains how the offline monitoring should be run.<br />
Since we may want to simultaneously run offline monitoring for different run periods that require<br />
different environment variables, the scripts are set up so that a generic user can download the<br />
scripts and run them from anywhere. Most output directories for offline monitoring are created<br />
with group read/write permissions so that any Hall D group user has access to the contents,<br />
but there are some cases where use of the account that created the launch is necessary.<br />
<br />
The accounts used for offline monitoring are the gxprojN accounts created and maintained by<br />
Mark Ito (see [https://halldweb.jlab.org/wiki/index.php/GlueX-related_shared_accounts_on_the_JLab_CUE here] for how each account is used).<br />
As of October 2015, the following are used:<br />
* gxproj1 for running over Fall 2014 data (deprecated since June 2015)<br />
* gxproj5 for running over Spring 2015 data<br />
<br />
Since the summer of 2015 we have transitioned from a system using Mark Ito's jproj scripts<br />
to integrating the swif system that Chris Larrieu (SciComp) has been developing. For offline monitoring,<br />
the hdswif system that Kei developed is used for launching the jobs, and the jproj system is used<br />
for meta-analysis of launch statistics.<br />
<br />
Both hdswif and jproj are maintained in svn:<br />
* hdswif: https://halldsvn.jlab.org/repos/trunk/scripts/monitoring/hdswif<br />
* jproj : https://halldsvn.jlab.org/repos/trunk/scripts/monitoring/jproj<br />
<br />
To run the offline monitoring each package should be checked out and all necessary<br />
scripts are included.<br />
<br />
=== Preparing the software for the launch ===<br />
<br />
To begin a new launch the software must be built to the latest versions.<br />
For the gxprojN user accounts used, all software builds are contained in the directory ~/builds<br />
(which are soft links to /work/halld/home/gxprojN/builds). When logging into these accounts<br />
the setup files ~/setup_jlab-2015-03.csh or similar files should be sourced.<br />
<br />
<b>Note that Mark Ito does <u>not</u> want you to change the contents of each .cshrc file.</b><br />
You should consult him if you feel the need.<br />
<br />
<span style="color:red">NOTE: FOR BUILDING SOFTWARE IT IS A GOOD IDEA TO DO A COMPLETE WIPEOUT/CHECKOUT EACH TIME TO AVOID STALE HEADER FILES.</span><br />
<br />
# Building hdds: Go to ~/builds/hdds. The directory hdds is the one from git. Delete the contents, then download the newest version from git and build: <pre>cd ~/builds/hdds/</pre> <pre>rm -frv hdds</pre> <pre>git clone https://github.com/JeffersonLab/hdds</pre> <pre>scons install</pre><br />
# Building sim-recon: Go to ~/builds/sim-recon. The directory sim-recon is the one from git. Delete the contents, then download the newest version from git and build: <pre>cd ~/builds/sim-recon</pre> <pre>rm -frv sim-recon</pre> <pre>git clone https://github.com/JeffersonLab/sim-recon</pre> <pre>scons install -j8</pre><br />
# Prepare the latest sqlite file: The sqlite is set in the ~/setup_jlab-2015-03.csh script as sqlite:////home/gxproj5/builds/ccdb.sqlite through the environment variables <b>JANA_CALIB_URL</b> and <b>CCDB_CONNECTION</b>. Therefore, go to ~/builds,and create a new sqlite file. We create the sqlite file in a temporary directory since creating the sqlite file in a directory where the output file exists causes errors. Original documentation on creating sqlite files are [https://halldweb.jlab.org/wiki/index.php/SQLite-form_of_the_CCDB_database here]. <pre>cd ~/builds/tmp</pre> <pre>$CCDB_HOME/scripts/mysql2sqlite/mysql2sqlite.sh -hhallddb.jlab.org -uccdb_user ccdb | sqlite3 ccdb.sqlite</pre> <pre>mv ccdb.sqlite ../</pre><br />
# Note that the above steps must be done <b>BEFORE</b> launch project creation. This is because we will track the revisions of the libraries used, and this is done by extracting the svn information in each directory. Also, note that the system assumes that we have the topmost build directory (usually called GLUEX_TOP) to be $HOME/builds . Such an assumption is necessary to be able to extract information about the library locations automatically.<br />
<br />
Create the appropriate project(s) and submit the jobs using hdswif, as detailed in the section below.<br />
<br />
=== Starting the Launch and Submitting Jobs ===<br />
<br />
Until the summer of 2015 we relied solely on Mark Ito's jproj system for submitting and keeping track of jobs. We have since moved to the swif system and use the hdswif wrapper for this. Below are instructions for how to use these.<br />
<br />
# Downloading hdswif: Download the hdswif directory from svn. For the gxprojN accounts, use the directory ~/halld/hdswif. <pre>cd ~/halld </pre> <pre> svn co https://halldsvn.jlab.org/repos/trunk/scripts/monitoring/hdswif</pre> <pre>cd hdswif</pre><br />
# Creating the workflow: Within SWIF jobs are registered into workflows. First create the workflow. For offline monitoring, the workflow names are of the form <b>offline_monitoring_RunPeriod201Y_MM_verVV_hd_rawdata</b> with suitable replacements for the run period and version number. The command "swif list" will list all existing workflows. Also, for most simple SWIF commands hdswif also provides a wrapper. <pre>swif list</pre> For creation of workflows for offline monitoring the command <pre>hdswif.py create [workflow] -c [config file] </pre> should be used. When a config file is passed in, hdswif will automatically create files that record the configuration of the current launch. These files are stored as for example<br />
#* /group/halld/data_monitoring/run_conditions/jana_rawdata_comm_2015_03_ver15.conf<br />
#* /group/halld/data_monitoring/run_conditions/soft_comm_2015_03_ver15.xml<br />
# Registering jobs in the workflow: To register jobs within the workflow, hdswif provides the use of config files. Jobs can be registered by specifying the workflow, config file (-c), run (-r) and file (-f) numbers if necessary. A typical config file will look this:<br />
PROJECT gluex<br />
TRACK reconstruction<br />
OS centos65<br />
NCORES 6<br />
DISK 40<br />
RAM 8<br />
TIMELIMIT 8<br />
JOBNAMEBASE offmon_<br />
RUNPERIOD 2015-03<br />
VERSION 15<br />
OUTPUT_TOPDIR /volatile/halld/offline_monitoring/RunPeriod-[RUNPERIOD]/ver[VERSION] # Example of other variables included in variable<br />
SCRIPTFILE /home/gxproj5/halld/hdswif/script.sh # Must specify full path<br />
ENVFILE /home/gxproj5/halld/hdswif/setup_jlab-2015-03.csh # Must specify full path<br />
The config file contains configuration parameters for each of the jobs.<br><br />
<span style="color:red">Note: Job configuration parameters can be set differently for jobs within the same workflow if necessary.</span><br />
Edit the config file and save as a new file if necessary. Once the configuration is set, jobs can be added via <pre>hdswif.py add [workflow] -c input.config</pre><br />
By default, hdswif will add all files found within the directory /mss/halld/RunPeriod-201Y-MM/rawdata/ where 201Y-MM is specified by the RUNPERIOD parameter in the config file. If only some of the runs or files are needed, these can be specified for example with<br />
<pre>hdswif.py add [workflow] -c input.config -r 3180 -f '00[0-4]'</pre><br />
to specify to register running only over run 3180 files 000 - 004 (Unix-style brackets and wildcards can be used).<br />
<br />
Running the workflow: To run the workflow, simply use swif run: <pre>swif run </pre> or equivalently, using the hdswif wrapper, <pre>hdsswif.py run [workflow]</pre><br><br />
<b>It is recommended that some jobs be tested to make sure that everything is working rather than fail thousands of jobs.</b><br> For this purpose, hdswif will take an additional parameter to run which limits the number of jobs to submit: <pre>hdswif.py run [workflow] 10</pre><br />
in which case only 10 jobs will be submitted. To submit all jobs after checking the results, do <pre>hdswif.py run [workflow]</pre><br />
<br />
=== Post-analysis of statistics of the launch ===<br />
<br />
# After jobs have been submitted, it will usually take a few days for all of the jobs to be processed.<br />
#* Status of Auger: http://scicomp.jlab.org/scicomp/#/auger/jobs (see also links above)<br />
#* Status of user jobs: <pre>jobstat -u [user name]</pre><br />
# The status and results of jobs are saved within the SWIF internal server, and are available via the command <pre>swif status [workflow] -summary -runs</pre> where the arguments -summary and -runs show summary statistics and statistics for individual jobs, respectively. hdswif has a command that takes this output in XML output and creates an HTML webpage showing results of the launch. To do this, do <pre>hdswif.py summary [workflow]</pre> This will create an XML file swif_output_[workflow].xml that contains all information from SWIF. If the file already exists, hdswif will ask whether to overwrite the existing file.<br />
# At this stage the html output and figure files are created and ready to be put online. For this step and other steps involving analysis of the statistics of the launch results, it is convenient to change to the jproj system.<br />
# The jproj scripts for offline monitoring are maintained in the svn directory https://halldsvn.jlab.org/repos/trunk/scripts/monitoring/jproj Do <pre> svn co https://halldsvn.jlab.org/repos/trunk/scripts/monitoring/jproj </pre> For the gxprojN accounts used for offline monitoring the directory should be ~/halld/jproj<br />
# The jproj directory contains two subdirectories, scripts and projects. The scripts directory contains useful scripts for processing the jobs registered in the jproj system, and each of the offline monitoring launches will be handled in the projects directory.<br />
# Go to the projects directory<pre>cd ~/halld/jproj/projects</pre> and use the script create_project.sh to create a new directory that contains the processing scripts for the current launch <pre>./create_project.sh [workflow]</pre> This should create a directory such as offline_monitoring_RunPeriod2015_03_ver15_hd_rawdata (same as the workflow name). The script uses the template files in the directory templates and by substitution creates script files for the current launch. Now go to the newly created analysis directory: <pre>cd [workflow]/analysis</pre><br />
which for the gxprojN accounts should have the full path /home/gxprojN/halld/jproj/projects/[workflow]/analysis.<br><br><br />
<span style="color:red">All of the analysis commands including arguments are contained in run_analysis.sh, but it is <b>strongly</b> recommended that all commands are run manually to check for errors.</span><br><br><br />
# The first thing to do is to make the html output from hdswif public. Copy the html file and related figures that were created from hdswif to the appropriate space within /group/halld/www/halldweb/html/data_monitoring/ : <pre>python publish_offmon_results.py [run period] [version]</pre> Note that the command with appropriate substitutions for arguments can be found within run_analysis.sh (same for all commands below).<br />
# Next, we need to create a few MySQL tables for the current launch. The MySQL tables are useful for comparing run/file combinations for different launches. For the SWIF launches, there are two tables needed, and will be named<br />
#* [workflow]Job<br />
#* [workflow]_aux<br />
# This naming scheme of tables and their roles are the same as from the jproj only launches. The [workflow]Job table will contain information gathered from SWIF about each job (which node it went to, start time of each stage, memory usage, etc.). The [workflow]_aux table will contain information gathered from the output stdout files from each job. First, create the Job table using <pre>python create_jproj_job_table.py [run period] [version]</pre> Note that this script uses the XML output from hdswif summary and inserts the contents into the MySQL table, so the XML output file must exist.<br />
# To check the contents of this MySQL table, do <pre>mysql -hhalldb -ufarmer farming</pre> <pre>mysql> describe offline_monitoring_RunPeriod2015_03_ver11_hd_rawdataJob;</pre><br />
<br />
== Hall D Job Management System ==<br />
<br />
This section details instructions on how to create and launch a set of jobs using the Hall-D Job Management System developed by Mark Ito. These instructions are generic: this system can be used for the weekly monitoring jobs, but can also be used for other sets of job launches as well. <br />
<br />
=== Database Table Overview ===<br />
<br />
* Job management database table (<project_name>): For each input file, keeps track of whether or not a job for it has been submitted, along with other optional fields. <br />
<br />
* Job status database table (<project_name>Job (no space)): For each job, keeps track of the job-id, the job status, memory used, cpu & wall time, time taken to complete various stages (e.g. pending, dependency, active), and others. <br />
<br />
* Job metrics database table (<project_name>_aux (no space)): For each job, keeps track of the job-id, how many events were processed, the time it took to copy the cache file, and the time it took to run the plugin. This information is culled from the log files of each job, and is done within the analysis directory of each launch.<br />
<br />
=== Initialize Project Management ===<br />
<br />
* Log into the ifarm machine with one of the gxproj accounts. For this example we will use gxproj1.<br />
<pre><br />
ssh gxproj1@ifarm -Y<br />
</pre><br />
<br />
* Go to a directory to do the launch. In principle, any directory will work, but for gxproj1 this is usually done in /home/gxproj1/halld/<br />
<br />
* Check out the necessary scripts <pre>svn co https://halldsvn.jlab.org/repos/trunk/scripts/monitoring/jproj</pre> This will get all necessary scripts for launching. Once checked out, <pre>cd projects</pre><br />
<br />
* The script create_project.sh can be used to create a new project. It will take a single argument, the project name. It is assumed that the project name is of the form offline_monitoring_RunPeriod20YY_MM_verVV_hd_rawdata where this string will be parsed to give the run period 20YY_MM and version number VV. One thing to do BEFORE creation of a new project is to editthe conditions of the launch (plugins to run over, memory requested, disk space requested) within templates/template.jsub . This<br />
information is saved automatically at project creation time into files in /group/halld/data_monitoring/run_conditions .<br />
To create a project do <pre>./create_project.sh offline_monitoring_RunPeriod20YY_MM_verVV_hd_rawdata</pre><br />
The name has been chosen to be as consistent as possible with other directory structures.<br />
However, mysql requires that "-" be escaped in table names, so unfortunately run periods will be<br />
given as 2YYY_MM instead of 2YYY-MM.<br />
<br />
* A new directory with the project name (offline_monitoring_RunPeriod20YY_MM_verVV_hd_rawdata) and files will be copied and modified from the template directory to reflect the run period, the user, the directory that it was created in, project name, etc.<br />
<br />
* For each project, cd into the new directory<br />
* The script clear.sh will remove any existing tables for the current project name, then recreate it. Do<br />
<pre>./clear.sh</pre><br />
* To use the jproj.pl script that was checked in, add the directory to your path with<br />
<pre>source ../../scripts/setup.csh</pre><br />
or always specify the full path<br />
<pre>../../scripts/jproj.pl</pre><br />
* Now update the table of runs with<br />
<pre>jproj.pl <project name> update</pre><br />
This will fill the table with all files within /mss that are of the same form as what is in <project name>.jproj . If you want to register only on a subset of all such files, you can edit this file directly.<br />
* Once you have registered all of the files you would like to run over, do<br />
<pre>jproj.pl <project name> submit [max # of jobs] [run number]</pre><br />
where the additional options specify how many jobs to submit and which run number to run on. Without these options all files that are registered and have not been submitted yet will be submitted.<br />
<br />
At this stage you are ready to submit all files. It is a good idea to submit a few test jobs<br />
at first to check that all scripts are working and that the plugins do not crash. Once you are<br />
sure that this does not happen, you can send all jobs in. The remaining jobs are then the monitoring<br />
which will (among other things) put the results on the online webpage for the collaboration to view, and<br />
the analysis of the launch.<br />
<br />
=== Project File Overview ===<br />
<br />
An overview of each project file:<br />
* '''clear.sh:''' For the current project, deletes the job status and management database tables (if any), and creates new, empty ones. <br />
* '''<project_name>.jproj:''' Contains the path and file name format for the input files for the jobs. <br />
* '''<project_name>.jsub:''' The xml job submission script. The run number and file number variables are set during job submission for each input file. <br />
* '''script.sh:''' The script that is executed during the job. If output job directories are not pre-created manually, they should be created in this script with the proper permissions:<br />
<pre><br />
mkdir -p -m 775 my_directory<br />
</pre><br />
* '''setup_jlab-[run period].csh:''' The environment that is sourced at the beginning of the job execution. <br />
* '''status.sh:''' Updates the job status database table, and prints some of its columns to screen.<br />
<br />
=== Project Management ===<br />
<br />
* Delete (if any) and create the database table(s) for the current set of job submissions:<br />
<pre><br />
./clear.sh<br />
</pre><br />
Also, if testing was done with jobs, it is best to delete the output directory and the configuration files:<br />
<pre><br />
rm -frv /volatile/halld/offline_monitoring/RunPeriod-20YY-MM/verVV /group/halld/data_monitoring/run_conditions/soft_comm_20YY_MM_verVV.xml /group/halld/data_monitoring/run_conditions/jana_rawdata_comm_20YY_MM_verVV.conf<br />
</pre><br />
<br />
* Search for input files matching the string in the .jproj file, and create a row for each in the job management database table (called <project_name>). You can test by adding an optional argument at the end, which only selects files with a specific file number:<br />
<pre><br />
jproj.pl <project_name> update <optional_file_number><br />
</pre><br />
<br />
* Confirm that the job management database is accurate by printing it's contents to screen:<br />
<pre><br />
mysql -hhallddb -ufarmer farming -e "select * from <project_name>"<br />
</pre><br />
<br />
* ONLY if a mistake was made, to delete the tables from the database and recreate new, empty ones, run: <br />
<pre><br />
./clear.sh<br />
</pre><br />
<br />
* To look at the status of the submitted jobs, first query auger and update the job status database:<br />
<pre><br />
fill_in_job_details.pl <project_name><br />
</pre><br />
<br />
* The job status can then be viewed by submitting a query to the job status database (called <project_name>Job (no space in between)):<br />
<pre><br />
mysql -hhallddb -ufarmer farming -e "select id,run,file,jobId,hostname,status,timeSubmitted,timeActive,walltime,cput,timeComplete,result,error from <project_name>Job"<br />
</pre><br />
<br />
* These last two commands can instead be executed simultaneously by running:<br />
<pre><br />
./status.sh<br />
</pre><br />
<br />
=== Handy mysql Instructions ===<br />
<br />
* Handy mysql instructions:<br />
<pre><br />
mysql -hhallddb -ufarmer farming # Enter the "farming" mysql database on "hallddb" as user "farmer"<br />
quit; # Exit mysql<br />
show tables; # Show a list of the tables in the current database<br />
show columns from <project_name>; # show all of the columns for the given table<br />
select * from <project_name>; # show the contents of all rows from the given table<br />
</pre><br />
<br />
=== Backing Up Offline Monitoring Tables ===<br />
Tables created for offline monitoring can be backed up using the script backup_tables.sh which<br />
can be checked out with the other files from https://halldsvn.jlab.org/repos/trunk/scripts/monitoring/jproj/projects<br />
<br />
The script uses the command mysqldump to print out a file that can be executed to recreate the tables.<br />
Since executing this output file will drop the table when it exists, caution is advised.<br />
Example usage to backup all three tables created for run period 2014_10 ver 17:<br />
<pre>backup_tables.sh 2014_10 17</pre><br />
<br />
== Running Over Data As It Comes In ==<br />
<br />
A special user gxproj1 will have a cron job set up to run the plugins as new data appears on /mss.<br />
During the week, gxproj1 will submit offline plugin jobs with the same setup as the weekly jobs<br />
run the previous Friday. The procedure for this is shown below.<br />
<br />
<!--<br />
=== Setting up the environment ===<br />
The file<br />
/home/gxproj1/setup_jlab.csh<br />
is sourced through .tcshrc.<br />
This file is the same as what is linked to by<br />
/home/gluex/setup_jlab_commissioning.csh,<br />
except HALLD_HOME, HDDS_HOME, and JANA_CALIB_URL are set separately so that this<br />
user can have a separate build.<br />
<br />
To obtain the builds from the previous Friday's runs,<br />
execute<br />
/home/gxproj1/halld/monitoring/newruns/setup_previous.sh [year] [month] [day]<br />
The build revisions from the previous Friday are archived in files<br />
/work/halld/data_monitoring/run_conditions/soft_comm_[year]_[month]_[day].xml<br />
and the script will build libraries based on those stored revision numbers.<br />
--><br />
<br />
=== Running the cron job ===<br />
<br />
'''IMPORTANT:''' The cron job should not be running while you are manually submitting jobs using the jproj.pl script for the same project, or else you will probably multiply-submit a job. <br />
<br />
* Go to the cron job directory: <br />
<pre><br />
cd /u/home/gxproj1/halld/monitoring/newruns<br />
</pre><br />
<br />
* The cron_plugins file is the cronjob that will be executed. During execution, it runs the exec.sh command in the same folder. This command takes two arguments: the project name, and the maximum file number for each run. These fields should be updated in the cron_plugins file before running. <br />
<br />
* The exec.sh command updates the job management database table with any data that has arrived on tape since it was last updated, ignoring file numbers greater than the maximum file number. It then submits jobs for these files. <br />
<br />
* To start the cron job, run:<br />
<pre><br />
crontab cron_plugins<br />
</pre><br />
<br />
* To check whether the cron job is running, do<br />
<pre><br />
crontab -l<br />
</pre><br />
<br />
* To remove the cron job do<br />
<pre><br />
crontab -r<br />
</pre><br />
<br />
<!--<br />
The cron job will run the script scan_for_jobs.sh,<br />
which runs generatejobs_plugins_rawdata.sh for any<br />
new runs that it had not seen before. All previous<br />
runs are recorded in the file filelists/files_current.txt<br />
so clear this to run over runs, or set the parameters<br />
MINRUN and MAXRUN which will set the range of runs submitted.<br />
--><br />
<br />
==Post-Processing Procedures==<br />
<br />
To visualize the monitoring data, we save images of selected histograms and store time series of selected quantities in a database, which are then displayed on a web page. This section describes how to generate the monitoring images and database information.<br />
<br />
The scripts used to generate this summary data are primarily run from /home/gxprojN/halld/monitoring/process . If you want a new copy of the scripts, e.g., for a new monitoring run, you should check the scripts out from SVN:<br />
<syntaxhighlight><br />
svn co https://halldsvn.jlab.org/repos/trunk/scripts/monitoring/process<br />
</syntaxhighlight><br />
Note that these scripts currently have some parameters which must be periodically set by hand.<br />
<br />
The default python version on most JLab machine does not have the modules to allow these scripts to connect to the MySQL database. To run these scripts, load the environment with the following command<br />
<syntaxhighlight><br />
source /home/gxproj1/halld/monitoring/process/monitoring_env.csh<br />
</syntaxhighlight><br />
<br />
===Online Monitoring===<br />
<br />
There are two primary scripts for running over the monitoring data generated by the online system and offline reconstruction. The online script can be run with either of the following commands:<br />
<syntaxhighlight><br />
/home/gluex/halld/monitoring/process/check_new_runs.py<br />
<br />
OR <br />
<br />
/home/gluex/halld/monitoring/process/check_new_runs.csh<br />
</syntaxhighlight><br />
The shell script sets up the environment properly to run the python script. To connect to the monitoring database on the JLab CUE, modules included in the local installation of python >= 2.7 are needed. The shell script is appropriate to use in a cron job. The cronjob is currently run under the "gluex" account.<br />
<br />
The online monitoring system copies a ROOT file containing the results of the online monitoring, and other configuration files into a directory accessible outside the counting house. This python script automatically checks for new ROOT files, which it will then automatically process. It contains several configuration variables that must be correctly set, which contains the location of input/output directories, etc...<br />
<br />
===Offline Monitoring===<br />
<br />
After the data is run over, the results should be processed, so that summary data is entered into the monitoring database and plots are made for the monitoring webpages. Currently, this processing is controlled by a cronjob that runs the following script:<br />
<syntaxhighlight><br />
/home/gxproj1/halld/monitoring/process/check_monitoring_data.csh <br />
</syntaxhighlight><br />
This script checks for new ROOT files, and only runs over those it hasn't processed yet. Since one monitoring ROOT file is produced for each EVIO file, whenever a new file is produced, the plots for the corresponding run are recreated and all the ROOT files for a run are combined into one file. Information is stored in the database on a per-file basis. <br />
<br />
Plots for the monitoring web page can be made from single histograms or multiple histograms using RootSpy macros. If you want to change the list of plots made, you must modify one of the following files:<br />
* histograms_to_monitor - specify either the name of the histogram or its the full ROOT path <br />
* macros_to_monitor - specify the full path to the RootSpy macro .C file<br />
<br />
When a new monitoring run is started, or the conditions are changed, the following steps should be taken to process the new files:<br />
# Add a new data version, as described below:<br />
# Change the following parameters in check_monitoring_data.csh:<br />
## JOBDATE should correspond to the ouptut date used by the job submission script<br />
## OUTPUTDIR should correspond to the directory corresponding to the run period and revision corresponding to the new version you just submitted. Presumably, this directory will be empty at the beginning.<br />
## Once you create a new data version as defined below, you should pass the needed information as a command line option. Currently this is done by the ARGS variable. For example, the argument "-v RunPeriod-2014-10,8" tells the monitoring scripts to look up the version corresponding to revision 8 of RunPeriod-2014-10 in the monitoring DB and to use to store the results.<br />
<br />
<syntaxhighlight><br />
Example configuration parameters:<br />
set JOBDATE=2015-01-09<br />
set INPUTDIR=/volatile/halld/RunPeriod-2014-10/offline_monitoring<br />
set OUTPUTDIR=/w/halld-scifs1a/data_monitoring/RunPeriod-2014-10/ver08<br />
set ARGS=" -v RunPeriod-2014-10,8 "<br />
</syntaxhighlight><br />
If you want to process the results manually, the data is processed using the following script:<br />
<syntaxhighlight><br />
./process_new_offline_data.py <input directory> <output directory><br />
<br />
EXAMPLE:<br />
<br />
./process_new_offline_data.py 2014-11-14 /volatile/halld/RunPeriod-2014-10/offline_monitoring/ /w/halld-scifs1a/data_monitoring/RunPeriod-2014-10/ver02<br />
</syntaxhighlight><br />
The python script takes several options to enable/disable various steps in the processing. Of interest is the "--force" option, which will run over all monitoring ROOT files, whether or not they've been previously identified.<br />
<br />
Every time a new reconstruction pass is performed, a new version number must be generated. To do this, prepare a version file as described below. Then run the register_new_version.py script to store the information in the database. The script will return a version number, which then should be set by hand in process_new_offline_data.py - future versions of the script will streamline this part of the procedure. An example of how to generate a new version is:<br />
<syntaxhighlight><br />
./register_new_version.py add /home/gxproj1/halld/monitoring/process/versions/vers_RunPeriod-2014-10_pass1.txt<br />
</syntaxhighlight><br />
<br />
<b>If you are running the offline monitoring by checking out the files in trunk/scripts/monitoring/jproj/projects/<br />
of the svn repository</b>, and created a project with<br />
create_project.sh [project name] hd_rawdata<br />
Then go to the directory [project name]/processing/<br />
and execute<br />
./run_processing.sh<br />
which will run register_new_version.py as well as check_monitoring_data.csh for that project.<br />
<br />
===Step-by-Step Instructions For Processing a New Monitoring Run===<br />
<br />
The monitoring runs are current run out of the gxproj1 and gxproj5 accounts. After an offline monitoring run has been successfully started on the batch farm, the following steps should be followed to setup the post-processing for these runs.<br />
<br />
# The post-processing scripts are stored in $HOME/halld/monitoring/process and are automatically run by cron.<br />
# Run "svn update" to bring any changes in. Be sure that the list of histograms and macros to plot are current.<br />
# Edit check_monitoring_data.csh to point to the current revisions/directories<br />
#* VERSION<br />
#* ARGS<br />
#* Note that the environment depends on a standard script - $HOME/setup_jlab.csh<br />
# Update files in the web directory, so that the results are displayed on the web pages: /group/halld/www/halldweb/html/data_monitoring/textdata<br />
# Copy the REST files to more permanent locations:<br />
#* cp -a /volatile/halld/offline_monitoring/RunPeriod-YYYY-MM/verVV/REST /work/halld/data_monitoring/RunPeriod-YYYY-MM/REST/verVV<br />
#* cp -a /volatile/halld/offline_monitoring/RunPeriod-YYYY-MM/verVV/REST /cache/halld/RunPeriod-YYYY-MM/REST/verVV [under testing]<br />
<br />
Check log files in $HOME/halld/monitoring/process/log for more information on how each run went. If there are problems, check log files, and modify check_monitoring_data.csh to vary the verbosity of the output.<br />
<br />
==Data Versions==<br />
<br />
To document the conditions of the monitoring data that is created, for the sake of reproducability and further analysis we save several pieces of information. The format is intended to be comprehensive enough to document not just monitoring data, but versions of raw and reconstructed data, so that this database table can be used for the event database as well.<br />
<br />
We store one record per pass through one run period, with the following structure:<br />
<br />
{| class="wikitable"<br />
! Field !! Description<br />
|-<br />
| data_type || The level of data we are processing. For the purposes of monitoring, "rawdata" is the online monitoring, "recon" is the offline monitoring<br />
|- <br />
| run_period || The run period of the data<br />
|- <br />
| revision || An integer specifying which pass through the run period this data corresponds to<br />
|- <br />
| software_version || The name of the XML file that specifies the different software versions used<br />
|-<br />
| jana_config || The name of the text file that specifies which JANA options were passed to the reconstruction program<br />
|-<br />
| ccdb_context || The value of JANA_CALIB_CONTEXT, which specifies the version of calibration constants that were used<br />
|-<br />
| production_time || The data at which monitoring/reconstruction began<br />
|-<br />
| dataVersionString || A convenient string for identifying this version of the data<br />
|}<br />
<br />
<br />
An example file used as as input to ./register_new_version.py is:<br />
<syntaxhighlight><br />
data_type = recon<br />
run_period = RunPeriod-2014-10<br />
revision = 1<br />
software_version = soft_comm_2014_11_06.xml<br />
jana_config = jana_rawdata_comm_2014_11_06.conf<br />
ccdb_context = calibtime=2014-11-10<br />
production_time = 2014-11-10<br />
dataVersionString = recon_RunPeriod-2014-10_20141110_ver01<br />
</syntaxhighlight></div>Kmoriyahttps://halldweb.jlab.org/wiki/index.php?title=Data_Monitoring_Procedures&diff=70741Data Monitoring Procedures2015-10-15T02:02:51Z<p>Kmoriya: /* Starting the Launch and Submitting Jobs */</p>
<hr />
<div>__TOC__<br />
<br />
== Master List of File / Database / Webpage Locations ==<br />
=== Run Conditions ===<br />
* Online Run-by-run condition files (B-field, current, etc.): /work/halld/online_monitoring/conditions/<br />
* Offline monitoring run conditions (software versions, jana config): /group/halld/data_monitoring/run_conditions/<br />
*[http://www.jlab.org/Hall-D/test/RunInfo/ Run Info vers. 1]<br />
*[https://halldweb.jlab.org/cgi-bin/data_monitoring/run_conditions.pl Run Info vers. 2]<br />
<br />
=== Monitoring Output Files ===<br />
* Run Periods 201Y-MM is for example 2015-03, launch ver verVV is for example ver15<br />
* Online monitoring histograms: /work/halld/online_monitoring/root/<br />
* Offline monitoring histogram ROOT files: /work/halld/data_monitoring/RunPeriod-201Y-MM/verVV/rootfiles<br />
* REST files (most recent launch only): /work/halld/data_monitoring/RunPeriod-201Y-MM/REST/verVV<br />
* individual files for each job (ROOT, log, etc.): /volatile/halld/offline_monitoring/RunPeriod-201Y-MM/verVV/<br />
<br />
=== Monitoring Database ===<br />
* Accessing monitoring database (on ifarm): mysql -u datmon -h hallddb.jlab.org data_monitoring<br />
<br />
=== Monitoring Webpages ===<br />
*[https://halldweb.jlab.org/cgi-bin/data_monitoring/monitoring/plotBrowser.py Plot Browser]<br />
*[https://halldweb.jlab.org/cgi-bin/data_monitoring/monitoring/runBrowser.py Run Browser]<br />
*[https://halldweb.jlab.org/cgi-bin/data_monitoring/monitoring/timeSeries.py Time Series]<br />
*[https://halldweb.jlab.org/data_monitoring/launch_analysis/ Launch Analysis]<br />
<br />
== Job Monitoring Links ==<br />
* [http://scicomp.jlab.org/farm2/job.html Completed Job History]<br />
* [http://scicomp.jlab.org/farm2/project.html Job Stats By Project]<br />
* [http://scicomp.jlab.org/farm2/trackOrg.html Job Stats By Track]<br />
* [http://scicomp.jlab.org/farm2/report.html Cluster Report]<br />
* [http://scicomp.jlab.org/farm2/walltime.html Walltime Distribution]<br />
<br />
== Saving Online Monitoring Data ==<br />
<br />
The procedure for writing the data out is given in, e.g.,<br />
[https://halldweb.jlab.org/wiki/index.php/Raid-to-Silo_Transfer_Strategy Raid-to-Silo Transfer Strategy].<br />
<br />
Once the DAQ writes out the data to the raid disk, cron jobs will copy the file to tape,<br />
and within ~20 min., we will have access to the file on tape at<br />
/mss/halld/$RUN_PERIOD/rawdata/RunXXXXXX.<br />
<br />
All online monitoring plugins will be run as data is taken.<br />
They will be accessible within the counting house via RootSpy, and<br />
for each run and file, a ROOT file containing the histograms will be saved<br />
within a subdirectory for each run.<br />
<br />
For immediate access to these files, the raid disk files may be accessed directly<br />
from the counting house, or the tape files will be available within ~20 min. of the<br />
file being written out.<br />
<br />
== Offline Monitoring: Running Over Archived Data ==<br />
<br />
Once files are written to tape we run the online plugins on these files to confirm what we were seeing in the online monitoring, and also to update the results from the latest calibration and software. <br />
Manual scripts and cron jobs are set up to look for new data and run the plugins over a sample of files.<br />
<br />
Every other Friday (usually the Friday before the offline meetings) jobs will be started to run the newest software on all previous runs,and allows everybody to see improvements in each detector.<br />
For each launch, independent builds of hdds, sim-recon, the monitoring plugins, and an sqlite file will be generated.<br />
<br />
Below the procedures are described for<br />
# Preparing the software for the launch<br />
# Starting the launch (using hdswif)<br />
# Post-analysis of statistics of the launch<br />
<br />
Processing the results and making them available to the collaboration<br />
is handled in the section [[#Post-Processing_Procedures | Post-Processing_Procedures]] below.<br />
<br />
<!--<br />
==== Generating an offline plugin job ====<br />
<br />
The user gxproj1 should be used for official offline monitoring jobs.<br />
Within /home/gxproj1/halld/monitoring/batch/ there will be scripts to run the online monitoring plugins over tape files.<br />
The main script is generatejobs_plugins_rawdata.sh, which can be used as<br />
generatejobs_plugins_rawdata.sh [minrun] [maxrun] (minfile) (maxfile)<br />
where minrun, maxrun specify the range of the run #, and minfile, maxfile (optional) specify<br />
the file range for that run.<br />
<br />
This will generate a script run_rawdata_XXXXXX.sh, where the run # has now been formatted to be 6 digits.<br />
Executing this script will send the monitoring plugins job to the Auger batch system.<br />
<br />
There is also a script clean.sh which can be used as<br />
clean.sh XXX<br />
This will clean up all associated files created in association with run XXX.<br />
<br />
Internally, the xml file used to submit the job will be created, and the job to run<br />
will be given within script.sh. All run parameters should be specified in at the beginning<br />
of generatejobs_plugins_rawdata.sh<br />
Since we are running on tape, the tape file will first be copied over to the cache disk, and the job will run<br />
over this cached file.<br />
<br />
==== Using cron to run automatically ====<br />
Within /home/gluex/halld/monitoring/cron/ there is a file cron_plugins<br />
that can be executed via<br />
crontab cron_plugins<br />
This will set up a cron job to call the script scan_for_jobs.sh, which will<br />
check in the rawdata directory and call generatejobs_plugins_rawdata.sh for<br />
any run that is more than 5 min old. The cron job is set up to run every 10 min.<br />
<br />
==== Magnetic Field Settings ====<br />
For tracking, it is necessary to set the correct magnetic field settings.<br />
The below table is now obsolete. Please refer to the [https://halldweb.jlab.org/cgi-bin/data_monitoring/run_conditions.pl GlueX Run Conditions] webpage.<br />
<br />
The actual field settings for each run have not been documented well.<br />
Below are values based on going through entries in the [https://logbooks.jlab.org/book/halld halld log].<br />
<br />
{| border="1" cellpadding="1" style="text-align: center;"<br />
!width="100"| Run #s<br />
!width="150"| Solenoid Current (A)<br />
!width="300"| JANA option<br />
!width="300"| notes<br />
|-<br />
| 940 - 996 || 1000 || -PBFIELD_MAP=Magnets/Solenoid/solenoid_1000A_poisson_20141104 || At run 997, the solenoid started ramping down<br />
|-<br />
| 998 - 1448 || 0 || -PBFIELD_TYPE=NoField -PDEFTAG:DTrackCandidate=StraightLine || Solenoid started ramping up around run 1431. Runs 1432 - 1448 should not be used for tracking. See [https://logbooks.jlab.org/entry/3309039 this] and [https://logbooks.jlab.org/entry/3309092 this] entry.<br />
|-<br />
| 1449 - 1620 || 1200 || -PBFIELD_MAP=Magnets/Solenoid/solenoid_1200A_poisson_20140520 || <br />
|-<br />
|}<br />
<br />
'''Note:''' The following run ranges were taken with a 300A field, and could be analyzed with the magnetic field map Magnets/Solenoid/solenoid_300A_poisson_20140819 , more study is needed to see if the straight line track fitter is needed or not: 1036 - 1053, 1065 - 1121, 1212? - 1254, 1309 - 1318 [1307,1308,1319-1329 were taken while the magnet was ramping].<br />
--><br />
<br />
=== General Information on Procedures ===<br />
This section explains how the offline monitoring should be run.<br />
Since we may want to simultaneously run offline monitoring for different run periods that require<br />
different environment variables, the scripts are set up so that a generic user can download the<br />
scripts and run them from anywhere. Most output directories for offline monitoring are created<br />
with group read/write permissions so that any Hall D group user has access to the contents,<br />
but there are some cases where use of the account that created the launch is necessary.<br />
<br />
The accounts used for offline monitoring are the gxprojN accounts created and maintained by<br />
Mark Ito (see [https://halldweb.jlab.org/wiki/index.php/GlueX-related_shared_accounts_on_the_JLab_CUE here]).<br />
As of October 2015, we have been using<br />
* gxproj1 for running over Fall 2014 data (deprecated since June 2015)<br />
* gxproj5 for running over Spring 2015 data<br />
<br />
Since the summer of 2015 we have transitioned from a system using Mark Ito's jproj scripts<br />
to integrating the swif system that Chris Larrieu (SciComp) has been developing. For offline monitoring,<br />
the hdswif system that Kei developed is used for launching the jobs, and the jproj system is used<br />
for meta-analysis of launch statistics.<br />
<br />
Both hdswif and jproj are maintained in svn:<br />
* hdswif: https://halldsvn.jlab.org/repos/trunk/scripts/monitoring/hdswif<br />
* jproj : https://halldsvn.jlab.org/repos/trunk/scripts/monitoring/jproj<br />
<br />
To run each package the directories will need to be checked out and the necessary<br />
scripts will need to be run.<br />
<br />
=== Preparing the software for the launch ===<br />
<br />
To begin a new launch the software must be built to the latest versions.<br />
For the gxprojN user accounts used, all software builds are contained in the directory ~/builds<br />
(which are soft links to /work/halld/home/gxproj5/builds). When logging into these accounts<br />
the setup files ~/setup_jlab-2015-03.csh or similar files should be sourced.<br />
<br />
<b>Note that Mark Ito does <u>not</u> want you to change the contents of each .cshrc file.</b><br />
You should consult him if you feel the need.<br />
<br />
<span style="color:red">NOTE: IT IS A GOOD IDEA TO DO A COMPLETE WIPEOUT/CHECKOUT OF A NEW VERSION OF EACH SOFTWARE TO AVOID STALE HEADER FILES.</span><br />
<br />
# Building hdds: Go to ~/builds/hdds. The directory hdds is the one from git. Delete the contents, then download the newest version from git and build: <pre>cd ~/builds/hdds/</pre> <pre>rm -frv hdds</pre> <pre>git clone https://github.com/JeffersonLab/hdds</pre> <pre>scons install</pre><br />
# Building sim-recon: Go to ~/builds/sim-recon. The directory sim-recon is the one from git. Delete the contents, then download the newest version from git and build: <pre>cd ~/builds/sim-recon</pre> <pre>rm -frv sim-recon</pre> <pre>git clone https://github.com/JeffersonLab/sim-recon</pre> <pre>scons install -j8</pre><br />
# Prepare the latest sqlite file: The sqlite is set in the ~/setup_jlab-2015-03.csh script as sqlite:////home/gxproj5/builds/ccdb.sqlite through the environment variables <b>JANA_CALIB_URL</b> and <b>CCDB_CONNECTION</b>. Therefore, go to ~/builds,and create a new sqlite file. We create the sqlite file in a temporary directory since creating the sqlite file in a directory where the output file exists causes errors. Original documentation on creating sqlite files are [https://halldweb.jlab.org/wiki/index.php/SQLite-form_of_the_CCDB_database here]. <pre>cd ~/builds/tmp</pre> <pre>$CCDB_HOME/scripts/mysql2sqlite/mysql2sqlite.sh -hhallddb.jlab.org -uccdb_user ccdb | sqlite3 ccdb.sqlite</pre><br />
# Note that the above steps must be done <b>BEFORE</b> launch project creation. This is because we will track the revisions of the libraries used, and this is done by extracting the svn information in each directory. Also, note that the system assumes that we have the topmost build directory (usually called GLUEX_TOP) to be $HOME/builds . Such an assumption is necessary to be able to extract information about the library locations automatically.<br />
<br />
Create the appropriate project(s) and submit the jobs using hdswif, as detailed in the section below.<br />
<br />
=== Starting the Launch and Submitting Jobs ===<br />
<br />
Until the summer of 2015 we relied solely on Mark Ito's jproj system for submitting and keeping track of jobs. We have since moved to the swif system and use the hdswif wrapper for this. Below are instructions for how to use these.<br />
<br />
# Downloading hdswif: Download the hdswif directory from svn. For the gxprojN accounts, use the directory ~/halld/hdswif. <pre>cd ~/halld </pre> <pre> svn co https://halldsvn.jlab.org/repos/trunk/scripts/monitoring/hdswif</pre> <pre>cd hdswif</pre><br />
# Creating the workflow: Within SWIF jobs are registered into workflows. First create the workflow. For offline monitoring, the workflow names are of the form <b>offline_monitoring_RunPeriod201Y_MM_verVV_hd_rawdata</b> with suitable replacements for the run period and version number. The command "swif list" will list all existing workflows. Also, or most simple SWIF commands hdswif also provides a wrapper. <pre>swif list</pre> <pre>swif create [workflow]</pre> Or equivalently, <pre>hdswif.py create [workflow] </pre><br />
# Registering jobs in the workflow: To register jobs within the workflow, hdswif provides the use of config files. Jobs can be registered by specifying the workflow, config file (-c), run (-r) and file (-f) numbers if necessary. A typical config file will look this:<br />
PROJECT gluex<br />
TRACK reconstruction<br />
OS centos65<br />
NCORES 6<br />
DISK 40<br />
RAM 8<br />
TIMELIMIT 8<br />
JOBNAMEBASE offmon_<br />
RUNPERIOD 2015-03<br />
VERSION 15<br />
OUTPUT_TOPDIR /volatile/halld/offline_monitoring/RunPeriod-[RUNPERIOD]/ver[VERSION] # Example of other variables included in variable<br />
SCRIPTFILE /home/gxproj5/halld/hdswif/script.sh # Must specify full path<br />
ENVFILE /home/gxproj5/halld/hdswif/setup_jlab-2015-03.csh # Must specify full path<br />
The config file contains configuration parameters for each of the jobs.<br><br />
<span style="color:red">Note: Job configuration parameters can be set differently for jobs within the same workflow if necessary.</span><br />
Edit the config file and save as a new file if necessary. Once the configuration is set, jobs can be added via <pre>hdswif.py add [workflow] -c input.config</pre><br />
By default, hdswif will add all files found within the directory /mss/halld/RunPeriod-201Y-MM/rawdata/ where 201Y-MM is specified by the RUNPERIOD parameter in the config file. If only some of the runs or files are needed, these can be specified for example with<br />
<pre>hdswif.py add [workflow] -c input.config -r 3180 -f '00[0-4]'</pre><br />
to specify to register running only over run 3180 files 000 - 004 (Unix-style brackets and wildcards can be used).<br />
# Running the workflow: To run the workflow, simply use swif run: <pre>swif run </pre> or equivalently, using the hdswif wrapper, <pre>hdsswif.py run [workflow]</pre><br><br />
<b>It is recommended that some jobs be tested to make sure that everything is working rather than fail thousands of jobs.</b><br> For this purpose, hdswif will take an additional parameter to run which limits the number of jobs to submit: <pre>hdswif.py run [workflow] 10</pre><br />
in which case only 10 jobs will be submitted. To submit all jobs after checking the results, do <pre>hdswif.py run [workflow]</pre><br />
<br />
=== Post-analysis of statistics of the launch ===<br />
<br />
# After jobs have been submitted, it will usually take a few days for all of the jobs to be processed.<br />
#* Status of Auger: http://scicomp.jlab.org/scicomp/#/auger/jobs (see also links above)<br />
#* Status of user jobs: <pre>jobstat -u [user name]</pre><br />
# The status and results of jobs are saved within the SWIF internal server, and are available via the command <pre>swif status [workflow] -summary -runs</pre> where the arguments -summary and -runs show summary statistics and statistics for individual jobs, respectively. hdswif has a command that takes this output in XML output and creates an HTML webpage showing results of the launch. To do this, do <pre>hdswif.py summary [workflow]</pre> This will create an XML file swif_output_[workflow].xml that contains all information from SWIF. If the file already exists, hdswif will ask whether to overwrite the existing file.<br />
# At this stage the html output and figure files are created and ready to be put online. For this step and other steps involving analysis of the statistics of the launch results, it is convenient to change to the jproj system.<br />
<br />
# The jproj scripts for offline monitoring are maintained in the svn directory https://halldsvn.jlab.org/repos/trunk/scripts/monitoring/jproj Do <pre> svn co https://halldsvn.jlab.org/repos/trunk/scripts/monitoring/jproj </pre><br />
For the gxprojN accounts used for offline monitoring the directory should be ~/halld/jproj<br />
# The jproj directory contains two subdirectories, scripts and projects. The scripts directory contains useful scripts for processing the jobs registered in the jproj system, and each of the offline monitoring launches will be handled in the projects directory.<br />
# Go to the projects directory<pre>cd ~/halld/jproj/projects</pre> and use the script create_project.sh to create a new directory that contains the processing scripts for the current launch <pre>./create_project.sh [workflow]</pre> This should create a directory such as offline_monitoring_RunPeriod2015_03_ver15_hd_rawdata (same as the workflow name). The script uses the template files in the directory templates and by substitution creates script files for the current launch. Now go to the newly created analysis directory: <pre>cd [workflow]/analysis</pre><br />
which for the gxprojN accounts should have the full path /home/gxprojN/halld/jproj/projects/[workflow]/analysis.<br><br><br />
<span style="color:red">All of the analysis commands including arguments are contained in run_analysis.sh, but it is <b>strongly</b> recommended that all commands are run manually to check for errors.</span><br><br><br />
# The first thing to do is to make the html output from hdswif public. Copy the html file and related figures that were created from hdswif to the appropriate space within /group/halld/www/halldweb/html/data_monitoring/ : <pre>python publish_offmon_results.py [run period] [version]</pre> Note that the command with appropriate substitutions for arguments can be found within run_analysis.sh (same for all commands below).<br />
# Next, we need to create a few MySQL tables for the current launch. The MySQL tables are useful for comparing run/file combinations for different launches. For the SWIF launches, there are two tables needed, and will be named<br />
#* [workflow]Job<br />
#* [workflow]_aux<br />
# This naming scheme of tables and their roles are the same as from the jproj only launches. The [workflow]Job table will contain information gathered from SWIF about each job (which node it went to, start time of each stage, memory usage, etc.). The [workflow]_aux table will contain information gathered from the output stdout files from each job. First, create the Job table using <pre>python create_jproj_job_table.py [run period] [version]</pre> Note that this script uses the XML output from hdswif summary and inserts the contents into the MySQL table, so the XML output file must exist.<br />
# To check the contents of this MySQL table, do <pre>mysql -hhalldb -ufarmer farming</pre> <pre>mysql> describe offline_monitoring_RunPeriod2015_03_ver11_hd_rawdataJob;</pre><br />
<br />
== Hall D Job Management System ==<br />
<br />
This section details instructions on how to create and launch a set of jobs using the Hall-D Job Management System developed by Mark Ito. These instructions are generic: this system can be used for the weekly monitoring jobs, but can also be used for other sets of job launches as well. <br />
<br />
=== Database Table Overview ===<br />
<br />
* Job management database table (<project_name>): For each input file, keeps track of whether or not a job for it has been submitted, along with other optional fields. <br />
<br />
* Job status database table (<project_name>Job (no space)): For each job, keeps track of the job-id, the job status, memory used, cpu & wall time, time taken to complete various stages (e.g. pending, dependency, active), and others. <br />
<br />
* Job metrics database table (<project_name>_aux (no space)): For each job, keeps track of the job-id, how many events were processed, the time it took to copy the cache file, and the time it took to run the plugin. This information is culled from the log files of each job, and is done within the analysis directory of each launch.<br />
<br />
=== Initialize Project Management ===<br />
<br />
* Log into the ifarm machine with one of the gxproj accounts. For this example we will use gxproj1.<br />
<pre><br />
ssh gxproj1@ifarm -Y<br />
</pre><br />
<br />
* Go to a directory to do the launch. In principle, any directory will work, but for gxproj1 this is usually done in /home/gxproj1/halld/<br />
<br />
* Check out the necessary scripts <pre>svn co https://halldsvn.jlab.org/repos/trunk/scripts/monitoring/jproj</pre> This will get all necessary scripts for launching. Once checked out, <pre>cd projects</pre><br />
<br />
* The script create_project.sh can be used to create a new project. It will take a single argument, the project name. It is assumed that the project name is of the form offline_monitoring_RunPeriod20YY_MM_verVV_hd_rawdata where this string will be parsed to give the run period 20YY_MM and version number VV. One thing to do BEFORE creation of a new project is to editthe conditions of the launch (plugins to run over, memory requested, disk space requested) within templates/template.jsub . This<br />
information is saved automatically at project creation time into files in /group/halld/data_monitoring/run_conditions .<br />
To create a project do <pre>./create_project.sh offline_monitoring_RunPeriod20YY_MM_verVV_hd_rawdata</pre><br />
The name has been chosen to be as consistent as possible with other directory structures.<br />
However, mysql requires that "-" be escaped in table names, so unfortunately run periods will be<br />
given as 2YYY_MM instead of 2YYY-MM.<br />
<br />
* A new directory with the project name (offline_monitoring_RunPeriod20YY_MM_verVV_hd_rawdata) and files will be copied and modified from the template directory to reflect the run period, the user, the directory that it was created in, project name, etc.<br />
<br />
* For each project, cd into the new directory<br />
* The script clear.sh will remove any existing tables for the current project name, then recreate it. Do<br />
<pre>./clear.sh</pre><br />
* To use the jproj.pl script that was checked in, add the directory to your path with<br />
<pre>source ../../scripts/setup.csh</pre><br />
or always specify the full path<br />
<pre>../../scripts/jproj.pl</pre><br />
* Now update the table of runs with<br />
<pre>jproj.pl <project name> update</pre><br />
This will fill the table with all files within /mss that are of the same form as what is in <project name>.jproj . If you want to register only on a subset of all such files, you can edit this file directly.<br />
* Once you have registered all of the files you would like to run over, do<br />
<pre>jproj.pl <project name> submit [max # of jobs] [run number]</pre><br />
where the additional options specify how many jobs to submit and which run number to run on. Without these options all files that are registered and have not been submitted yet will be submitted.<br />
<br />
At this stage you are ready to submit all files. It is a good idea to submit a few test jobs<br />
at first to check that all scripts are working and that the plugins do not crash. Once you are<br />
sure that this does not happen, you can send all jobs in. The remaining jobs are then the monitoring<br />
which will (among other things) put the results on the online webpage for the collaboration to view, and<br />
the analysis of the launch.<br />
<br />
=== Project File Overview ===<br />
<br />
An overview of each project file:<br />
* '''clear.sh:''' For the current project, deletes the job status and management database tables (if any), and creates new, empty ones. <br />
* '''<project_name>.jproj:''' Contains the path and file name format for the input files for the jobs. <br />
* '''<project_name>.jsub:''' The xml job submission script. The run number and file number variables are set during job submission for each input file. <br />
* '''script.sh:''' The script that is executed during the job. If output job directories are not pre-created manually, they should be created in this script with the proper permissions:<br />
<pre><br />
mkdir -p -m 775 my_directory<br />
</pre><br />
* '''setup_jlab-[run period].csh:''' The environment that is sourced at the beginning of the job execution. <br />
* '''status.sh:''' Updates the job status database table, and prints some of its columns to screen.<br />
<br />
=== Project Management ===<br />
<br />
* Delete (if any) and create the database table(s) for the current set of job submissions:<br />
<pre><br />
./clear.sh<br />
</pre><br />
Also, if testing was done with jobs, it is best to delete the output directory and the configuration files:<br />
<pre><br />
rm -frv /volatile/halld/offline_monitoring/RunPeriod-20YY-MM/verVV /group/halld/data_monitoring/run_conditions/soft_comm_20YY_MM_verVV.xml /group/halld/data_monitoring/run_conditions/jana_rawdata_comm_20YY_MM_verVV.conf<br />
</pre><br />
<br />
* Search for input files matching the string in the .jproj file, and create a row for each in the job management database table (called <project_name>). You can test by adding an optional argument at the end, which only selects files with a specific file number:<br />
<pre><br />
jproj.pl <project_name> update <optional_file_number><br />
</pre><br />
<br />
* Confirm that the job management database is accurate by printing it's contents to screen:<br />
<pre><br />
mysql -hhallddb -ufarmer farming -e "select * from <project_name>"<br />
</pre><br />
<br />
* ONLY if a mistake was made, to delete the tables from the database and recreate new, empty ones, run: <br />
<pre><br />
./clear.sh<br />
</pre><br />
<br />
* To look at the status of the submitted jobs, first query auger and update the job status database:<br />
<pre><br />
fill_in_job_details.pl <project_name><br />
</pre><br />
<br />
* The job status can then be viewed by submitting a query to the job status database (called <project_name>Job (no space in between)):<br />
<pre><br />
mysql -hhallddb -ufarmer farming -e "select id,run,file,jobId,hostname,status,timeSubmitted,timeActive,walltime,cput,timeComplete,result,error from <project_name>Job"<br />
</pre><br />
<br />
* These last two commands can instead be executed simultaneously by running:<br />
<pre><br />
./status.sh<br />
</pre><br />
<br />
=== Handy mysql Instructions ===<br />
<br />
* Handy mysql instructions:<br />
<pre><br />
mysql -hhallddb -ufarmer farming # Enter the "farming" mysql database on "hallddb" as user "farmer"<br />
quit; # Exit mysql<br />
show tables; # Show a list of the tables in the current database<br />
show columns from <project_name>; # show all of the columns for the given table<br />
select * from <project_name>; # show the contents of all rows from the given table<br />
</pre><br />
<br />
=== Backing Up Offline Monitoring Tables ===<br />
Tables created for offline monitoring can be backed up using the script backup_tables.sh which<br />
can be checked out with the other files from https://halldsvn.jlab.org/repos/trunk/scripts/monitoring/jproj/projects<br />
<br />
The script uses the command mysqldump to print out a file that can be executed to recreate the tables.<br />
Since executing this output file will drop the table when it exists, caution is advised.<br />
Example usage to backup all three tables created for run period 2014_10 ver 17:<br />
<pre>backup_tables.sh 2014_10 17</pre><br />
<br />
== Running Over Data As It Comes In ==<br />
<br />
A special user gxproj1 will have a cron job set up to run the plugins as new data appears on /mss.<br />
During the week, gxproj1 will submit offline plugin jobs with the same setup as the weekly jobs<br />
run the previous Friday. The procedure for this is shown below.<br />
<br />
<!--<br />
=== Setting up the environment ===<br />
The file<br />
/home/gxproj1/setup_jlab.csh<br />
is sourced through .tcshrc.<br />
This file is the same as what is linked to by<br />
/home/gluex/setup_jlab_commissioning.csh,<br />
except HALLD_HOME, HDDS_HOME, and JANA_CALIB_URL are set separately so that this<br />
user can have a separate build.<br />
<br />
To obtain the builds from the previous Friday's runs,<br />
execute<br />
/home/gxproj1/halld/monitoring/newruns/setup_previous.sh [year] [month] [day]<br />
The build revisions from the previous Friday are archived in files<br />
/work/halld/data_monitoring/run_conditions/soft_comm_[year]_[month]_[day].xml<br />
and the script will build libraries based on those stored revision numbers.<br />
--><br />
<br />
=== Running the cron job ===<br />
<br />
'''IMPORTANT:''' The cron job should not be running while you are manually submitting jobs using the jproj.pl script for the same project, or else you will probably multiply-submit a job. <br />
<br />
* Go to the cron job directory: <br />
<pre><br />
cd /u/home/gxproj1/halld/monitoring/newruns<br />
</pre><br />
<br />
* The cron_plugins file is the cronjob that will be executed. During execution, it runs the exec.sh command in the same folder. This command takes two arguments: the project name, and the maximum file number for each run. These fields should be updated in the cron_plugins file before running. <br />
<br />
* The exec.sh command updates the job management database table with any data that has arrived on tape since it was last updated, ignoring file numbers greater than the maximum file number. It then submits jobs for these files. <br />
<br />
* To start the cron job, run:<br />
<pre><br />
crontab cron_plugins<br />
</pre><br />
<br />
* To check whether the cron job is running, do<br />
<pre><br />
crontab -l<br />
</pre><br />
<br />
* To remove the cron job do<br />
<pre><br />
crontab -r<br />
</pre><br />
<br />
<!--<br />
The cron job will run the script scan_for_jobs.sh,<br />
which runs generatejobs_plugins_rawdata.sh for any<br />
new runs that it had not seen before. All previous<br />
runs are recorded in the file filelists/files_current.txt<br />
so clear this to run over runs, or set the parameters<br />
MINRUN and MAXRUN which will set the range of runs submitted.<br />
--><br />
<br />
==Post-Processing Procedures==<br />
<br />
To visualize the monitoring data, we save images of selected histograms and store time series of selected quantities in a database, which are then displayed on a web page. This section describes how to generate the monitoring images and database information.<br />
<br />
The scripts used to generate this summary data are primarily run from /home/gxprojN/halld/monitoring/process . If you want a new copy of the scripts, e.g., for a new monitoring run, you should check the scripts out from SVN:<br />
<syntaxhighlight><br />
svn co https://halldsvn.jlab.org/repos/trunk/scripts/monitoring/process<br />
</syntaxhighlight><br />
Note that these scripts currently have some parameters which must be periodically set by hand.<br />
<br />
The default python version on most JLab machine does not have the modules to allow these scripts to connect to the MySQL database. To run these scripts, load the environment with the following command<br />
<syntaxhighlight><br />
source /home/gxproj1/halld/monitoring/process/monitoring_env.csh<br />
</syntaxhighlight><br />
<br />
===Online Monitoring===<br />
<br />
There are two primary scripts for running over the monitoring data generated by the online system and offline reconstruction. The online script can be run with either of the following commands:<br />
<syntaxhighlight><br />
/home/gluex/halld/monitoring/process/check_new_runs.py<br />
<br />
OR <br />
<br />
/home/gluex/halld/monitoring/process/check_new_runs.csh<br />
</syntaxhighlight><br />
The shell script sets up the environment properly to run the python script. To connect to the monitoring database on the JLab CUE, modules included in the local installation of python >= 2.7 are needed. The shell script is appropriate to use in a cron job. The cronjob is currently run under the "gluex" account.<br />
<br />
The online monitoring system copies a ROOT file containing the results of the online monitoring, and other configuration files into a directory accessible outside the counting house. This python script automatically checks for new ROOT files, which it will then automatically process. It contains several configuration variables that must be correctly set, which contains the location of input/output directories, etc...<br />
<br />
===Offline Monitoring===<br />
<br />
After the data is run over, the results should be processed, so that summary data is entered into the monitoring database and plots are made for the monitoring webpages. Currently, this processing is controlled by a cronjob that runs the following script:<br />
<syntaxhighlight><br />
/home/gxproj1/halld/monitoring/process/check_monitoring_data.csh <br />
</syntaxhighlight><br />
This script checks for new ROOT files, and only runs over those it hasn't processed yet. Since one monitoring ROOT file is produced for each EVIO file, whenever a new file is produced, the plots for the corresponding run are recreated and all the ROOT files for a run are combined into one file. Information is stored in the database on a per-file basis. <br />
<br />
Plots for the monitoring web page can be made from single histograms or multiple histograms using RootSpy macros. If you want to change the list of plots made, you must modify one of the following files:<br />
* histograms_to_monitor - specify either the name of the histogram or its the full ROOT path <br />
* macros_to_monitor - specify the full path to the RootSpy macro .C file<br />
<br />
When a new monitoring run is started, or the conditions are changed, the following steps should be taken to process the new files:<br />
# Add a new data version, as described below:<br />
# Change the following parameters in check_monitoring_data.csh:<br />
## JOBDATE should correspond to the ouptut date used by the job submission script<br />
## OUTPUTDIR should correspond to the directory corresponding to the run period and revision corresponding to the new version you just submitted. Presumably, this directory will be empty at the beginning.<br />
## Once you create a new data version as defined below, you should pass the needed information as a command line option. Currently this is done by the ARGS variable. For example, the argument "-v RunPeriod-2014-10,8" tells the monitoring scripts to look up the version corresponding to revision 8 of RunPeriod-2014-10 in the monitoring DB and to use to store the results.<br />
<br />
<syntaxhighlight><br />
Example configuration parameters:<br />
set JOBDATE=2015-01-09<br />
set INPUTDIR=/volatile/halld/RunPeriod-2014-10/offline_monitoring<br />
set OUTPUTDIR=/w/halld-scifs1a/data_monitoring/RunPeriod-2014-10/ver08<br />
set ARGS=" -v RunPeriod-2014-10,8 "<br />
</syntaxhighlight><br />
If you want to process the results manually, the data is processed using the following script:<br />
<syntaxhighlight><br />
./process_new_offline_data.py <input directory> <output directory><br />
<br />
EXAMPLE:<br />
<br />
./process_new_offline_data.py 2014-11-14 /volatile/halld/RunPeriod-2014-10/offline_monitoring/ /w/halld-scifs1a/data_monitoring/RunPeriod-2014-10/ver02<br />
</syntaxhighlight><br />
The python script takes several options to enable/disable various steps in the processing. Of interest is the "--force" option, which will run over all monitoring ROOT files, whether or not they've been previously identified.<br />
<br />
Every time a new reconstruction pass is performed, a new version number must be generated. To do this, prepare a version file as described below. Then run the register_new_version.py script to store the information in the database. The script will return a version number, which then should be set by hand in process_new_offline_data.py - future versions of the script will streamline this part of the procedure. An example of how to generate a new version is:<br />
<syntaxhighlight><br />
./register_new_version.py add /home/gxproj1/halld/monitoring/process/versions/vers_RunPeriod-2014-10_pass1.txt<br />
</syntaxhighlight><br />
<br />
<b>If you are running the offline monitoring by checking out the files in trunk/scripts/monitoring/jproj/projects/<br />
of the svn repository</b>, and created a project with<br />
create_project.sh [project name] hd_rawdata<br />
Then go to the directory [project name]/processing/<br />
and execute<br />
./run_processing.sh<br />
which will run register_new_version.py as well as check_monitoring_data.csh for that project.<br />
<br />
===Step-by-Step Instructions For Processing a New Monitoring Run===<br />
<br />
The monitoring runs are current run out of the gxproj1 and gxproj5 accounts. After an offline monitoring run has been successfully started on the batch farm, the following steps should be followed to setup the post-processing for these runs.<br />
<br />
# The post-processing scripts are stored in $HOME/halld/monitoring/process and are automatically run by cron.<br />
# Run "svn update" to bring any changes in. Be sure that the list of histograms and macros to plot are current.<br />
# Edit check_monitoring_data.csh to point to the current revisions/directories<br />
#* VERSION<br />
#* ARGS<br />
#* Note that the environment depends on a standard script - $HOME/setup_jlab.csh<br />
# Update files in the web directory, so that the results are displayed on the web pages: /group/halld/www/halldweb/html/data_monitoring/textdata<br />
# Copy the REST files to more permanent locations:<br />
#* cp -a /volatile/halld/offline_monitoring/RunPeriod-YYYY-MM/verVV/REST /work/halld/data_monitoring/RunPeriod-YYYY-MM/REST/verVV<br />
#* cp -a /volatile/halld/offline_monitoring/RunPeriod-YYYY-MM/verVV/REST /cache/halld/RunPeriod-YYYY-MM/REST/verVV [under testing]<br />
<br />
Check log files in $HOME/halld/monitoring/process/log for more information on how each run went. If there are problems, check log files, and modify check_monitoring_data.csh to vary the verbosity of the output.<br />
<br />
==Data Versions==<br />
<br />
To document the conditions of the monitoring data that is created, for the sake of reproducability and further analysis we save several pieces of information. The format is intended to be comprehensive enough to document not just monitoring data, but versions of raw and reconstructed data, so that this database table can be used for the event database as well.<br />
<br />
We store one record per pass through one run period, with the following structure:<br />
<br />
{| class="wikitable"<br />
! Field !! Description<br />
|-<br />
| data_type || The level of data we are processing. For the purposes of monitoring, "rawdata" is the online monitoring, "recon" is the offline monitoring<br />
|- <br />
| run_period || The run period of the data<br />
|- <br />
| revision || An integer specifying which pass through the run period this data corresponds to<br />
|- <br />
| software_version || The name of the XML file that specifies the different software versions used<br />
|-<br />
| jana_config || The name of the text file that specifies which JANA options were passed to the reconstruction program<br />
|-<br />
| ccdb_context || The value of JANA_CALIB_CONTEXT, which specifies the version of calibration constants that were used<br />
|-<br />
| production_time || The data at which monitoring/reconstruction began<br />
|-<br />
| dataVersionString || A convenient string for identifying this version of the data<br />
|}<br />
<br />
<br />
An example file used as as input to ./register_new_version.py is:<br />
<syntaxhighlight><br />
data_type = recon<br />
run_period = RunPeriod-2014-10<br />
revision = 1<br />
software_version = soft_comm_2014_11_06.xml<br />
jana_config = jana_rawdata_comm_2014_11_06.conf<br />
ccdb_context = calibtime=2014-11-10<br />
production_time = 2014-11-10<br />
dataVersionString = recon_RunPeriod-2014-10_20141110_ver01<br />
</syntaxhighlight></div>Kmoriyahttps://halldweb.jlab.org/wiki/index.php?title=Data_Monitoring_Procedures&diff=70740Data Monitoring Procedures2015-10-15T01:40:40Z<p>Kmoriya: /* Post-analysis of statistics of the launch */</p>
<hr />
<div>__TOC__<br />
<br />
== Master List of File / Database / Webpage Locations ==<br />
=== Run Conditions ===<br />
* Online Run-by-run condition files (B-field, current, etc.): /work/halld/online_monitoring/conditions/<br />
* Offline monitoring run conditions (software versions, jana config): /group/halld/data_monitoring/run_conditions/<br />
*[http://www.jlab.org/Hall-D/test/RunInfo/ Run Info vers. 1]<br />
*[https://halldweb.jlab.org/cgi-bin/data_monitoring/run_conditions.pl Run Info vers. 2]<br />
<br />
=== Monitoring Output Files ===<br />
* Run Periods 201Y-MM is for example 2015-03, launch ver verVV is for example ver15<br />
* Online monitoring histograms: /work/halld/online_monitoring/root/<br />
* Offline monitoring histogram ROOT files: /work/halld/data_monitoring/RunPeriod-201Y-MM/verVV/rootfiles<br />
* REST files (most recent launch only): /work/halld/data_monitoring/RunPeriod-201Y-MM/REST/verVV<br />
* individual files for each job (ROOT, log, etc.): /volatile/halld/offline_monitoring/RunPeriod-201Y-MM/verVV/<br />
<br />
=== Monitoring Database ===<br />
* Accessing monitoring database (on ifarm): mysql -u datmon -h hallddb.jlab.org data_monitoring<br />
<br />
=== Monitoring Webpages ===<br />
*[https://halldweb.jlab.org/cgi-bin/data_monitoring/monitoring/plotBrowser.py Plot Browser]<br />
*[https://halldweb.jlab.org/cgi-bin/data_monitoring/monitoring/runBrowser.py Run Browser]<br />
*[https://halldweb.jlab.org/cgi-bin/data_monitoring/monitoring/timeSeries.py Time Series]<br />
*[https://halldweb.jlab.org/data_monitoring/launch_analysis/ Launch Analysis]<br />
<br />
== Job Monitoring Links ==<br />
* [http://scicomp.jlab.org/farm2/job.html Completed Job History]<br />
* [http://scicomp.jlab.org/farm2/project.html Job Stats By Project]<br />
* [http://scicomp.jlab.org/farm2/trackOrg.html Job Stats By Track]<br />
* [http://scicomp.jlab.org/farm2/report.html Cluster Report]<br />
* [http://scicomp.jlab.org/farm2/walltime.html Walltime Distribution]<br />
<br />
== Saving Online Monitoring Data ==<br />
<br />
The procedure for writing the data out is given in, e.g.,<br />
[https://halldweb.jlab.org/wiki/index.php/Raid-to-Silo_Transfer_Strategy Raid-to-Silo Transfer Strategy].<br />
<br />
Once the DAQ writes out the data to the raid disk, cron jobs will copy the file to tape,<br />
and within ~20 min., we will have access to the file on tape at<br />
/mss/halld/$RUN_PERIOD/rawdata/RunXXXXXX.<br />
<br />
All online monitoring plugins will be run as data is taken.<br />
They will be accessible within the counting house via RootSpy, and<br />
for each run and file, a ROOT file containing the histograms will be saved<br />
within a subdirectory for each run.<br />
<br />
For immediate access to these files, the raid disk files may be accessed directly<br />
from the counting house, or the tape files will be available within ~20 min. of the<br />
file being written out.<br />
<br />
== Offline Monitoring: Running Over Archived Data ==<br />
<br />
Once files are written to tape we run the online plugins on these files to confirm what we were seeing in the online monitoring, and also to update the results from the latest calibration and software. <br />
Manual scripts and cron jobs are set up to look for new data and run the plugins over a sample of files.<br />
<br />
Every other Friday (usually the Friday before the offline meetings) jobs will be started to run the newest software on all previous runs,and allows everybody to see improvements in each detector.<br />
For each launch, independent builds of hdds, sim-recon, the monitoring plugins, and an sqlite file will be generated.<br />
<br />
Below the procedures are described for<br />
# Preparing the software for the launch<br />
# Starting the launch (using hdswif)<br />
# Post-analysis of statistics of the launch<br />
<br />
Processing the results and making them available to the collaboration<br />
is handled in the section [[#Post-Processing_Procedures | Post-Processing_Procedures]] below.<br />
<br />
<!--<br />
==== Generating an offline plugin job ====<br />
<br />
The user gxproj1 should be used for official offline monitoring jobs.<br />
Within /home/gxproj1/halld/monitoring/batch/ there will be scripts to run the online monitoring plugins over tape files.<br />
The main script is generatejobs_plugins_rawdata.sh, which can be used as<br />
generatejobs_plugins_rawdata.sh [minrun] [maxrun] (minfile) (maxfile)<br />
where minrun, maxrun specify the range of the run #, and minfile, maxfile (optional) specify<br />
the file range for that run.<br />
<br />
This will generate a script run_rawdata_XXXXXX.sh, where the run # has now been formatted to be 6 digits.<br />
Executing this script will send the monitoring plugins job to the Auger batch system.<br />
<br />
There is also a script clean.sh which can be used as<br />
clean.sh XXX<br />
This will clean up all associated files created in association with run XXX.<br />
<br />
Internally, the xml file used to submit the job will be created, and the job to run<br />
will be given within script.sh. All run parameters should be specified in at the beginning<br />
of generatejobs_plugins_rawdata.sh<br />
Since we are running on tape, the tape file will first be copied over to the cache disk, and the job will run<br />
over this cached file.<br />
<br />
==== Using cron to run automatically ====<br />
Within /home/gluex/halld/monitoring/cron/ there is a file cron_plugins<br />
that can be executed via<br />
crontab cron_plugins<br />
This will set up a cron job to call the script scan_for_jobs.sh, which will<br />
check in the rawdata directory and call generatejobs_plugins_rawdata.sh for<br />
any run that is more than 5 min old. The cron job is set up to run every 10 min.<br />
<br />
==== Magnetic Field Settings ====<br />
For tracking, it is necessary to set the correct magnetic field settings.<br />
The below table is now obsolete. Please refer to the [https://halldweb.jlab.org/cgi-bin/data_monitoring/run_conditions.pl GlueX Run Conditions] webpage.<br />
<br />
The actual field settings for each run have not been documented well.<br />
Below are values based on going through entries in the [https://logbooks.jlab.org/book/halld halld log].<br />
<br />
{| border="1" cellpadding="1" style="text-align: center;"<br />
!width="100"| Run #s<br />
!width="150"| Solenoid Current (A)<br />
!width="300"| JANA option<br />
!width="300"| notes<br />
|-<br />
| 940 - 996 || 1000 || -PBFIELD_MAP=Magnets/Solenoid/solenoid_1000A_poisson_20141104 || At run 997, the solenoid started ramping down<br />
|-<br />
| 998 - 1448 || 0 || -PBFIELD_TYPE=NoField -PDEFTAG:DTrackCandidate=StraightLine || Solenoid started ramping up around run 1431. Runs 1432 - 1448 should not be used for tracking. See [https://logbooks.jlab.org/entry/3309039 this] and [https://logbooks.jlab.org/entry/3309092 this] entry.<br />
|-<br />
| 1449 - 1620 || 1200 || -PBFIELD_MAP=Magnets/Solenoid/solenoid_1200A_poisson_20140520 || <br />
|-<br />
|}<br />
<br />
'''Note:''' The following run ranges were taken with a 300A field, and could be analyzed with the magnetic field map Magnets/Solenoid/solenoid_300A_poisson_20140819 , more study is needed to see if the straight line track fitter is needed or not: 1036 - 1053, 1065 - 1121, 1212? - 1254, 1309 - 1318 [1307,1308,1319-1329 were taken while the magnet was ramping].<br />
--><br />
<br />
=== General Information on Procedures ===<br />
This section explains how the offline monitoring should be run.<br />
Since we may want to simultaneously run offline monitoring for different run periods that require<br />
different environment variables, the scripts are set up so that a generic user can download the<br />
scripts and run them from anywhere. Most output directories for offline monitoring are created<br />
with group read/write permissions so that any Hall D group user has access to the contents,<br />
but there are some cases where use of the account that created the launch is necessary.<br />
<br />
The accounts used for offline monitoring are the gxprojN accounts created and maintained by<br />
Mark Ito (see [https://halldweb.jlab.org/wiki/index.php/GlueX-related_shared_accounts_on_the_JLab_CUE here]).<br />
As of October 2015, we have been using<br />
* gxproj1 for running over Fall 2014 data (deprecated since June 2015)<br />
* gxproj5 for running over Spring 2015 data<br />
<br />
Since the summer of 2015 we have transitioned from a system using Mark Ito's jproj scripts<br />
to integrating the swif system that Chris Larrieu (SciComp) has been developing. For offline monitoring,<br />
the hdswif system that Kei developed is used for launching the jobs, and the jproj system is used<br />
for meta-analysis of launch statistics.<br />
<br />
Both hdswif and jproj are maintained in svn:<br />
* hdswif: https://halldsvn.jlab.org/repos/trunk/scripts/monitoring/hdswif<br />
* jproj : https://halldsvn.jlab.org/repos/trunk/scripts/monitoring/jproj<br />
<br />
To run each package the directories will need to be checked out and the necessary<br />
scripts will need to be run.<br />
<br />
=== Preparing the software for the launch ===<br />
<br />
To begin a new launch the software must be built to the latest versions.<br />
For the gxprojN user accounts used, all software builds are contained in the directory ~/builds<br />
(which are soft links to /work/halld/home/gxproj5/builds). When logging into these accounts<br />
the setup files ~/setup_jlab-2015-03.csh or similar files should be sourced.<br />
<br />
<b>Note that Mark Ito does <u>not</u> want you to change the contents of each .cshrc file.</b><br />
You should consult him if you feel the need.<br />
<br />
<span style="color:red">NOTE: IT IS A GOOD IDEA TO DO A COMPLETE WIPEOUT/CHECKOUT OF A NEW VERSION OF EACH SOFTWARE TO AVOID STALE HEADER FILES.</span><br />
<br />
# Building hdds: Go to ~/builds/hdds. The directory hdds is the one from git. Delete the contents, then download the newest version from git and build: <pre>cd ~/builds/hdds/</pre> <pre>rm -frv hdds</pre> <pre>git clone https://github.com/JeffersonLab/hdds</pre> <pre>scons install</pre><br />
# Building sim-recon: Go to ~/builds/sim-recon. The directory sim-recon is the one from git. Delete the contents, then download the newest version from git and build: <pre>cd ~/builds/sim-recon</pre> <pre>rm -frv sim-recon</pre> <pre>git clone https://github.com/JeffersonLab/sim-recon</pre> <pre>scons install -j8</pre><br />
# Prepare the latest sqlite file: The sqlite is set in the ~/setup_jlab-2015-03.csh script as sqlite:////home/gxproj5/builds/ccdb.sqlite through the environment variables <b>JANA_CALIB_URL</b> and <b>CCDB_CONNECTION</b>. Therefore, go to ~/builds,and create a new sqlite file. We create the sqlite file in a temporary directory since creating the sqlite file in a directory where the output file exists causes errors. Original documentation on creating sqlite files are [https://halldweb.jlab.org/wiki/index.php/SQLite-form_of_the_CCDB_database here]. <pre>cd ~/builds/tmp</pre> <pre>$CCDB_HOME/scripts/mysql2sqlite/mysql2sqlite.sh -hhallddb.jlab.org -uccdb_user ccdb | sqlite3 ccdb.sqlite</pre><br />
# Note that the above steps must be done <b>BEFORE</b> launch project creation. This is because we will track the revisions of the libraries used, and this is done by extracting the svn information in each directory. Also, note that the system assumes that we have the topmost build directory (usually called GLUEX_TOP) to be $HOME/builds . Such an assumption is necessary to be able to extract information about the library locations automatically.<br />
<br />
Create the appropriate project(s) and submit the jobs using hdswif, as detailed in the section below.<br />
<br />
=== Starting the Launch and Submitting Jobs ===<br />
<br />
Until the summer of 2015 we relied solely on Mark Ito's jproj system for submitting and keeping track of jobs. We have since moved to the swif system and use the hdswif wrapper for this. Below are instructions for how to use these.<br />
<br />
# Downloading hdswif: Download the hdswif directory from svn. For the gxprojN accounts, use the directory ~/halld/hdswif. <pre>cd ~/halld </pre> <pre> svn co https://halldsvn.jlab.org/repos/trunk/scripts/monitoring/hdswif</pre> <pre>cd hdswif</pre><br />
# Creating the workflow: Within SWIF jobs are registered into workflows. First create the workflow. For offline monitoring, the workflow names are of the form <b>offline_monitoring_RunPeriod201Y_MM_verVV_hd_rawdata</b> with suitable replacements for the run period and version number. The command "swif list" will list all existing workflows. Also, or most simple SWIF commands hdswif also provides a wrapper. <pre>swif list</pre> <pre>swif create [workflow]</pre> Or equivalently, <pre>hdswif.py create [workflow] </pre><br />
# Registering jobs in the workflow: To register jobs within the workflow, hdswif provides the use of config files. Jobs can be registered by specifying the workflow, config file (-c), run (-r) and file (-f) numbers if necessary. A typical config file will look this:<br />
<pre><br />
PROJECT gluex<br />
TRACK reconstruction<br />
OS centos65<br />
NCORES 4<br />
DISK 40<br />
RAM 8<br />
TIMELIMIT 16<br />
JOBNAMEBASE offmon_<br />
RUNPERIOD 2015-03<br />
VERSION 92<br />
OUTPUT_TOPDIR /volatile/halld/offline_monitoring/RunPeriod-[RUNPERIOD]/ver[VERSION] # Example of other variables included in variable<br />
SCRIPTFILE /home/gxproj5/halld/hdswif/script.sh # Must specify full path<br />
ENVFILE /home/gxproj5/halld/hdswif/setup_jlab-2015-03.csh # Must specify full path<br />
</pre><br />
The config file contains configuration parameters for each of the jobs.<br />
<span style="color:red">Note: Job configuration parameters can be set differently for jobs within the same workflow if necessary.</span><br />
Edit the config file and save as a new file if necessary. Once the configuration is set, jobs can be added via <pre>hdswif.py add [workflow] -c input.config</pre><br />
By default, hdswif will add all files found within the directory /mss/halld/RunPeriod-201Y-MM/rawdata/ where 201Y-MM is specified by the RUNPERIOD parameter in the config file. If only some of the runs or files are needed, these can be specified for example with<br />
<pre>hdswif.py add [workflow] -c input.config -r 3180 -f '00[0-4]'</pre><br />
to specify to register running only over run 3180 files 000 - 004 (Unix-style brackets and wildcards can be used).<br />
# Running the workflow: To run the workflow, simply use swif run: <pre>swif run </pre> or equivalently, using the hdswif wrapper, <pre>hdsswif.py run [workflow]</pre><br />
<b>It is recommended that some jobs be tested over to make sure that everything is working rather than fail thousands of jobs.</b> For this purpose, hdswif will take an additional parameter to run which limits the number of jobs to submit: <pre>hdswif.py run [workflow] 10</pre><br />
in which case only 10 jobs will be submitted. To submit all jobs after checking the results, do <pre>hdswif.py run [workflow]</pre><br />
<br />
=== Post-analysis of statistics of the launch ===<br />
<br />
# After jobs have been submitted, it will usually take a few days for all of the jobs to be processed.<br />
#* Status of Auger: http://scicomp.jlab.org/scicomp/#/auger/jobs (see also links above)<br />
#* Status of user jobs: <pre>jobstat -u [user name]</pre><br />
# The status and results of jobs are saved within the SWIF internal server, and are available via the command <pre>swif status [workflow] -summary -runs</pre> where the arguments -summary and -runs show summary statistics and statistics for individual jobs, respectively. hdswif has a command that takes this output in XML output and creates an HTML webpage showing results of the launch. To do this, do <pre>hdswif.py summary [workflow]</pre> This will create an XML file swif_output_[workflow].xml that contains all information from SWIF. If the file already exists, hdswif will ask whether to overwrite the existing file.<br />
# At this stage the html output and figure files are created and ready to be put online. For this step and other steps involving analysis of the statistics of the launch results, it is convenient to change to the jproj system.<br />
<br />
# The jproj scripts for offline monitoring are maintained in the svn directory https://halldsvn.jlab.org/repos/trunk/scripts/monitoring/jproj Do <pre> svn co https://halldsvn.jlab.org/repos/trunk/scripts/monitoring/jproj </pre><br />
For the gxprojN accounts used for offline monitoring the directory should be ~/halld/jproj<br />
# The jproj directory contains two subdirectories, scripts and projects. The scripts directory contains useful scripts for processing the jobs registered in the jproj system, and each of the offline monitoring launches will be handled in the projects directory.<br />
# Go to the projects directory<pre>cd ~/halld/jproj/projects</pre> and use the script create_project.sh to create a new directory that contains the processing scripts for the current launch <pre>./create_project.sh [workflow]</pre> This should create a directory such as offline_monitoring_RunPeriod2015_03_ver15_hd_rawdata (same as the workflow name). The script uses the template files in the directory templates and by substitution creates script files for the current launch. Now go to the newly created analysis directory: <pre>cd [workflow]/analysis</pre><br />
which for the gxprojN accounts should have the full path /home/gxprojN/halld/jproj/projects/[workflow]/analysis.<br><br><br />
<span style="color:red">All of the analysis commands including arguments are contained in run_analysis.sh, but it is <b>strongly</b> recommended that all commands are run manually to check for errors.</span><br><br><br />
# The first thing to do is to make the html output from hdswif public. Copy the html file and related figures that were created from hdswif to the appropriate space within /group/halld/www/halldweb/html/data_monitoring/ : <pre>python publish_offmon_results.py [run period] [version]</pre> Note that the command with appropriate substitutions for arguments can be found within run_analysis.sh (same for all commands below).<br />
# Next, we need to create a few MySQL tables for the current launch. The MySQL tables are useful for comparing run/file combinations for different launches. For the SWIF launches, there are two tables needed, and will be named<br />
#* [workflow]Job<br />
#* [workflow]_aux<br />
# This naming scheme of tables and their roles are the same as from the jproj only launches. The [workflow]Job table will contain information gathered from SWIF about each job (which node it went to, start time of each stage, memory usage, etc.). The [workflow]_aux table will contain information gathered from the output stdout files from each job. First, create the Job table using <pre>python create_jproj_job_table.py [run period] [version]</pre> Note that this script uses the XML output from hdswif summary and inserts the contents into the MySQL table, so the XML output file must exist.<br />
# To check the contents of this MySQL table, do <pre>mysql -hhalldb -ufarmer farming</pre> <pre>mysql> describe offline_monitoring_RunPeriod2015_03_ver11_hd_rawdataJob;</pre><br />
<br />
== Hall D Job Management System ==<br />
<br />
This section details instructions on how to create and launch a set of jobs using the Hall-D Job Management System developed by Mark Ito. These instructions are generic: this system can be used for the weekly monitoring jobs, but can also be used for other sets of job launches as well. <br />
<br />
=== Database Table Overview ===<br />
<br />
* Job management database table (<project_name>): For each input file, keeps track of whether or not a job for it has been submitted, along with other optional fields. <br />
<br />
* Job status database table (<project_name>Job (no space)): For each job, keeps track of the job-id, the job status, memory used, cpu & wall time, time taken to complete various stages (e.g. pending, dependency, active), and others. <br />
<br />
* Job metrics database table (<project_name>_aux (no space)): For each job, keeps track of the job-id, how many events were processed, the time it took to copy the cache file, and the time it took to run the plugin. This information is culled from the log files of each job, and is done within the analysis directory of each launch.<br />
<br />
=== Initialize Project Management ===<br />
<br />
* Log into the ifarm machine with one of the gxproj accounts. For this example we will use gxproj1.<br />
<pre><br />
ssh gxproj1@ifarm -Y<br />
</pre><br />
<br />
* Go to a directory to do the launch. In principle, any directory will work, but for gxproj1 this is usually done in /home/gxproj1/halld/<br />
<br />
* Check out the necessary scripts <pre>svn co https://halldsvn.jlab.org/repos/trunk/scripts/monitoring/jproj</pre> This will get all necessary scripts for launching. Once checked out, <pre>cd projects</pre><br />
<br />
* The script create_project.sh can be used to create a new project. It will take a single argument, the project name. It is assumed that the project name is of the form offline_monitoring_RunPeriod20YY_MM_verVV_hd_rawdata where this string will be parsed to give the run period 20YY_MM and version number VV. One thing to do BEFORE creation of a new project is to editthe conditions of the launch (plugins to run over, memory requested, disk space requested) within templates/template.jsub . This<br />
information is saved automatically at project creation time into files in /group/halld/data_monitoring/run_conditions .<br />
To create a project do <pre>./create_project.sh offline_monitoring_RunPeriod20YY_MM_verVV_hd_rawdata</pre><br />
The name has been chosen to be as consistent as possible with other directory structures.<br />
However, mysql requires that "-" be escaped in table names, so unfortunately run periods will be<br />
given as 2YYY_MM instead of 2YYY-MM.<br />
<br />
* A new directory with the project name (offline_monitoring_RunPeriod20YY_MM_verVV_hd_rawdata) and files will be copied and modified from the template directory to reflect the run period, the user, the directory that it was created in, project name, etc.<br />
<br />
* For each project, cd into the new directory<br />
* The script clear.sh will remove any existing tables for the current project name, then recreate it. Do<br />
<pre>./clear.sh</pre><br />
* To use the jproj.pl script that was checked in, add the directory to your path with<br />
<pre>source ../../scripts/setup.csh</pre><br />
or always specify the full path<br />
<pre>../../scripts/jproj.pl</pre><br />
* Now update the table of runs with<br />
<pre>jproj.pl <project name> update</pre><br />
This will fill the table with all files within /mss that are of the same form as what is in <project name>.jproj . If you want to register only on a subset of all such files, you can edit this file directly.<br />
* Once you have registered all of the files you would like to run over, do<br />
<pre>jproj.pl <project name> submit [max # of jobs] [run number]</pre><br />
where the additional options specify how many jobs to submit and which run number to run on. Without these options all files that are registered and have not been submitted yet will be submitted.<br />
<br />
At this stage you are ready to submit all files. It is a good idea to submit a few test jobs<br />
at first to check that all scripts are working and that the plugins do not crash. Once you are<br />
sure that this does not happen, you can send all jobs in. The remaining jobs are then the monitoring<br />
which will (among other things) put the results on the online webpage for the collaboration to view, and<br />
the analysis of the launch.<br />
<br />
=== Project File Overview ===<br />
<br />
An overview of each project file:<br />
* '''clear.sh:''' For the current project, deletes the job status and management database tables (if any), and creates new, empty ones. <br />
* '''<project_name>.jproj:''' Contains the path and file name format for the input files for the jobs. <br />
* '''<project_name>.jsub:''' The xml job submission script. The run number and file number variables are set during job submission for each input file. <br />
* '''script.sh:''' The script that is executed during the job. If output job directories are not pre-created manually, they should be created in this script with the proper permissions:<br />
<pre><br />
mkdir -p -m 775 my_directory<br />
</pre><br />
* '''setup_jlab-[run period].csh:''' The environment that is sourced at the beginning of the job execution. <br />
* '''status.sh:''' Updates the job status database table, and prints some of its columns to screen.<br />
<br />
=== Project Management ===<br />
<br />
* Delete (if any) and create the database table(s) for the current set of job submissions:<br />
<pre><br />
./clear.sh<br />
</pre><br />
Also, if testing was done with jobs, it is best to delete the output directory and the configuration files:<br />
<pre><br />
rm -frv /volatile/halld/offline_monitoring/RunPeriod-20YY-MM/verVV /group/halld/data_monitoring/run_conditions/soft_comm_20YY_MM_verVV.xml /group/halld/data_monitoring/run_conditions/jana_rawdata_comm_20YY_MM_verVV.conf<br />
</pre><br />
<br />
* Search for input files matching the string in the .jproj file, and create a row for each in the job management database table (called <project_name>). You can test by adding an optional argument at the end, which only selects files with a specific file number:<br />
<pre><br />
jproj.pl <project_name> update <optional_file_number><br />
</pre><br />
<br />
* Confirm that the job management database is accurate by printing it's contents to screen:<br />
<pre><br />
mysql -hhallddb -ufarmer farming -e "select * from <project_name>"<br />
</pre><br />
<br />
* ONLY if a mistake was made, to delete the tables from the database and recreate new, empty ones, run: <br />
<pre><br />
./clear.sh<br />
</pre><br />
<br />
* To look at the status of the submitted jobs, first query auger and update the job status database:<br />
<pre><br />
fill_in_job_details.pl <project_name><br />
</pre><br />
<br />
* The job status can then be viewed by submitting a query to the job status database (called <project_name>Job (no space in between)):<br />
<pre><br />
mysql -hhallddb -ufarmer farming -e "select id,run,file,jobId,hostname,status,timeSubmitted,timeActive,walltime,cput,timeComplete,result,error from <project_name>Job"<br />
</pre><br />
<br />
* These last two commands can instead be executed simultaneously by running:<br />
<pre><br />
./status.sh<br />
</pre><br />
<br />
=== Handy mysql Instructions ===<br />
<br />
* Handy mysql instructions:<br />
<pre><br />
mysql -hhallddb -ufarmer farming # Enter the "farming" mysql database on "hallddb" as user "farmer"<br />
quit; # Exit mysql<br />
show tables; # Show a list of the tables in the current database<br />
show columns from <project_name>; # show all of the columns for the given table<br />
select * from <project_name>; # show the contents of all rows from the given table<br />
</pre><br />
<br />
=== Backing Up Offline Monitoring Tables ===<br />
Tables created for offline monitoring can be backed up using the script backup_tables.sh which<br />
can be checked out with the other files from https://halldsvn.jlab.org/repos/trunk/scripts/monitoring/jproj/projects<br />
<br />
The script uses the command mysqldump to print out a file that can be executed to recreate the tables.<br />
Since executing this output file will drop the table when it exists, caution is advised.<br />
Example usage to backup all three tables created for run period 2014_10 ver 17:<br />
<pre>backup_tables.sh 2014_10 17</pre><br />
<br />
== Running Over Data As It Comes In ==<br />
<br />
A special user gxproj1 will have a cron job set up to run the plugins as new data appears on /mss.<br />
During the week, gxproj1 will submit offline plugin jobs with the same setup as the weekly jobs<br />
run the previous Friday. The procedure for this is shown below.<br />
<br />
<!--<br />
=== Setting up the environment ===<br />
The file<br />
/home/gxproj1/setup_jlab.csh<br />
is sourced through .tcshrc.<br />
This file is the same as what is linked to by<br />
/home/gluex/setup_jlab_commissioning.csh,<br />
except HALLD_HOME, HDDS_HOME, and JANA_CALIB_URL are set separately so that this<br />
user can have a separate build.<br />
<br />
To obtain the builds from the previous Friday's runs,<br />
execute<br />
/home/gxproj1/halld/monitoring/newruns/setup_previous.sh [year] [month] [day]<br />
The build revisions from the previous Friday are archived in files<br />
/work/halld/data_monitoring/run_conditions/soft_comm_[year]_[month]_[day].xml<br />
and the script will build libraries based on those stored revision numbers.<br />
--><br />
<br />
=== Running the cron job ===<br />
<br />
'''IMPORTANT:''' The cron job should not be running while you are manually submitting jobs using the jproj.pl script for the same project, or else you will probably multiply-submit a job. <br />
<br />
* Go to the cron job directory: <br />
<pre><br />
cd /u/home/gxproj1/halld/monitoring/newruns<br />
</pre><br />
<br />
* The cron_plugins file is the cronjob that will be executed. During execution, it runs the exec.sh command in the same folder. This command takes two arguments: the project name, and the maximum file number for each run. These fields should be updated in the cron_plugins file before running. <br />
<br />
* The exec.sh command updates the job management database table with any data that has arrived on tape since it was last updated, ignoring file numbers greater than the maximum file number. It then submits jobs for these files. <br />
<br />
* To start the cron job, run:<br />
<pre><br />
crontab cron_plugins<br />
</pre><br />
<br />
* To check whether the cron job is running, do<br />
<pre><br />
crontab -l<br />
</pre><br />
<br />
* To remove the cron job do<br />
<pre><br />
crontab -r<br />
</pre><br />
<br />
<!--<br />
The cron job will run the script scan_for_jobs.sh,<br />
which runs generatejobs_plugins_rawdata.sh for any<br />
new runs that it had not seen before. All previous<br />
runs are recorded in the file filelists/files_current.txt<br />
so clear this to run over runs, or set the parameters<br />
MINRUN and MAXRUN which will set the range of runs submitted.<br />
--><br />
<br />
==Post-Processing Procedures==<br />
<br />
To visualize the monitoring data, we save images of selected histograms and store time series of selected quantities in a database, which are then displayed on a web page. This section describes how to generate the monitoring images and database information.<br />
<br />
The scripts used to generate this summary data are primarily run from /home/gxprojN/halld/monitoring/process . If you want a new copy of the scripts, e.g., for a new monitoring run, you should check the scripts out from SVN:<br />
<syntaxhighlight><br />
svn co https://halldsvn.jlab.org/repos/trunk/scripts/monitoring/process<br />
</syntaxhighlight><br />
Note that these scripts currently have some parameters which must be periodically set by hand.<br />
<br />
The default python version on most JLab machine does not have the modules to allow these scripts to connect to the MySQL database. To run these scripts, load the environment with the following command<br />
<syntaxhighlight><br />
source /home/gxproj1/halld/monitoring/process/monitoring_env.csh<br />
</syntaxhighlight><br />
<br />
===Online Monitoring===<br />
<br />
There are two primary scripts for running over the monitoring data generated by the online system and offline reconstruction. The online script can be run with either of the following commands:<br />
<syntaxhighlight><br />
/home/gluex/halld/monitoring/process/check_new_runs.py<br />
<br />
OR <br />
<br />
/home/gluex/halld/monitoring/process/check_new_runs.csh<br />
</syntaxhighlight><br />
The shell script sets up the environment properly to run the python script. To connect to the monitoring database on the JLab CUE, modules included in the local installation of python >= 2.7 are needed. The shell script is appropriate to use in a cron job. The cronjob is currently run under the "gluex" account.<br />
<br />
The online monitoring system copies a ROOT file containing the results of the online monitoring, and other configuration files into a directory accessible outside the counting house. This python script automatically checks for new ROOT files, which it will then automatically process. It contains several configuration variables that must be correctly set, which contains the location of input/output directories, etc...<br />
<br />
===Offline Monitoring===<br />
<br />
After the data is run over, the results should be processed, so that summary data is entered into the monitoring database and plots are made for the monitoring webpages. Currently, this processing is controlled by a cronjob that runs the following script:<br />
<syntaxhighlight><br />
/home/gxproj1/halld/monitoring/process/check_monitoring_data.csh <br />
</syntaxhighlight><br />
This script checks for new ROOT files, and only runs over those it hasn't processed yet. Since one monitoring ROOT file is produced for each EVIO file, whenever a new file is produced, the plots for the corresponding run are recreated and all the ROOT files for a run are combined into one file. Information is stored in the database on a per-file basis. <br />
<br />
Plots for the monitoring web page can be made from single histograms or multiple histograms using RootSpy macros. If you want to change the list of plots made, you must modify one of the following files:<br />
* histograms_to_monitor - specify either the name of the histogram or its the full ROOT path <br />
* macros_to_monitor - specify the full path to the RootSpy macro .C file<br />
<br />
When a new monitoring run is started, or the conditions are changed, the following steps should be taken to process the new files:<br />
# Add a new data version, as described below:<br />
# Change the following parameters in check_monitoring_data.csh:<br />
## JOBDATE should correspond to the ouptut date used by the job submission script<br />
## OUTPUTDIR should correspond to the directory corresponding to the run period and revision corresponding to the new version you just submitted. Presumably, this directory will be empty at the beginning.<br />
## Once you create a new data version as defined below, you should pass the needed information as a command line option. Currently this is done by the ARGS variable. For example, the argument "-v RunPeriod-2014-10,8" tells the monitoring scripts to look up the version corresponding to revision 8 of RunPeriod-2014-10 in the monitoring DB and to use to store the results.<br />
<br />
<syntaxhighlight><br />
Example configuration parameters:<br />
set JOBDATE=2015-01-09<br />
set INPUTDIR=/volatile/halld/RunPeriod-2014-10/offline_monitoring<br />
set OUTPUTDIR=/w/halld-scifs1a/data_monitoring/RunPeriod-2014-10/ver08<br />
set ARGS=" -v RunPeriod-2014-10,8 "<br />
</syntaxhighlight><br />
If you want to process the results manually, the data is processed using the following script:<br />
<syntaxhighlight><br />
./process_new_offline_data.py <input directory> <output directory><br />
<br />
EXAMPLE:<br />
<br />
./process_new_offline_data.py 2014-11-14 /volatile/halld/RunPeriod-2014-10/offline_monitoring/ /w/halld-scifs1a/data_monitoring/RunPeriod-2014-10/ver02<br />
</syntaxhighlight><br />
The python script takes several options to enable/disable various steps in the processing. Of interest is the "--force" option, which will run over all monitoring ROOT files, whether or not they've been previously identified.<br />
<br />
Every time a new reconstruction pass is performed, a new version number must be generated. To do this, prepare a version file as described below. Then run the register_new_version.py script to store the information in the database. The script will return a version number, which then should be set by hand in process_new_offline_data.py - future versions of the script will streamline this part of the procedure. An example of how to generate a new version is:<br />
<syntaxhighlight><br />
./register_new_version.py add /home/gxproj1/halld/monitoring/process/versions/vers_RunPeriod-2014-10_pass1.txt<br />
</syntaxhighlight><br />
<br />
<b>If you are running the offline monitoring by checking out the files in trunk/scripts/monitoring/jproj/projects/<br />
of the svn repository</b>, and created a project with<br />
create_project.sh [project name] hd_rawdata<br />
Then go to the directory [project name]/processing/<br />
and execute<br />
./run_processing.sh<br />
which will run register_new_version.py as well as check_monitoring_data.csh for that project.<br />
<br />
===Step-by-Step Instructions For Processing a New Monitoring Run===<br />
<br />
The monitoring runs are current run out of the gxproj1 and gxproj5 accounts. After an offline monitoring run has been successfully started on the batch farm, the following steps should be followed to setup the post-processing for these runs.<br />
<br />
# The post-processing scripts are stored in $HOME/halld/monitoring/process and are automatically run by cron.<br />
# Run "svn update" to bring any changes in. Be sure that the list of histograms and macros to plot are current.<br />
# Edit check_monitoring_data.csh to point to the current revisions/directories<br />
#* VERSION<br />
#* ARGS<br />
#* Note that the environment depends on a standard script - $HOME/setup_jlab.csh<br />
# Update files in the web directory, so that the results are displayed on the web pages: /group/halld/www/halldweb/html/data_monitoring/textdata<br />
# Copy the REST files to more permanent locations:<br />
#* cp -a /volatile/halld/offline_monitoring/RunPeriod-YYYY-MM/verVV/REST /work/halld/data_monitoring/RunPeriod-YYYY-MM/REST/verVV<br />
#* cp -a /volatile/halld/offline_monitoring/RunPeriod-YYYY-MM/verVV/REST /cache/halld/RunPeriod-YYYY-MM/REST/verVV [under testing]<br />
<br />
Check log files in $HOME/halld/monitoring/process/log for more information on how each run went. If there are problems, check log files, and modify check_monitoring_data.csh to vary the verbosity of the output.<br />
<br />
==Data Versions==<br />
<br />
To document the conditions of the monitoring data that is created, for the sake of reproducability and further analysis we save several pieces of information. The format is intended to be comprehensive enough to document not just monitoring data, but versions of raw and reconstructed data, so that this database table can be used for the event database as well.<br />
<br />
We store one record per pass through one run period, with the following structure:<br />
<br />
{| class="wikitable"<br />
! Field !! Description<br />
|-<br />
| data_type || The level of data we are processing. For the purposes of monitoring, "rawdata" is the online monitoring, "recon" is the offline monitoring<br />
|- <br />
| run_period || The run period of the data<br />
|- <br />
| revision || An integer specifying which pass through the run period this data corresponds to<br />
|- <br />
| software_version || The name of the XML file that specifies the different software versions used<br />
|-<br />
| jana_config || The name of the text file that specifies which JANA options were passed to the reconstruction program<br />
|-<br />
| ccdb_context || The value of JANA_CALIB_CONTEXT, which specifies the version of calibration constants that were used<br />
|-<br />
| production_time || The data at which monitoring/reconstruction began<br />
|-<br />
| dataVersionString || A convenient string for identifying this version of the data<br />
|}<br />
<br />
<br />
An example file used as as input to ./register_new_version.py is:<br />
<syntaxhighlight><br />
data_type = recon<br />
run_period = RunPeriod-2014-10<br />
revision = 1<br />
software_version = soft_comm_2014_11_06.xml<br />
jana_config = jana_rawdata_comm_2014_11_06.conf<br />
ccdb_context = calibtime=2014-11-10<br />
production_time = 2014-11-10<br />
dataVersionString = recon_RunPeriod-2014-10_20141110_ver01<br />
</syntaxhighlight></div>Kmoriyahttps://halldweb.jlab.org/wiki/index.php?title=Data_Monitoring_Procedures&diff=70739Data Monitoring Procedures2015-10-15T01:38:35Z<p>Kmoriya: /* Post-analysis of statistics of the launch */</p>
<hr />
<div>__TOC__<br />
<br />
== Master List of File / Database / Webpage Locations ==<br />
=== Run Conditions ===<br />
* Online Run-by-run condition files (B-field, current, etc.): /work/halld/online_monitoring/conditions/<br />
* Offline monitoring run conditions (software versions, jana config): /group/halld/data_monitoring/run_conditions/<br />
*[http://www.jlab.org/Hall-D/test/RunInfo/ Run Info vers. 1]<br />
*[https://halldweb.jlab.org/cgi-bin/data_monitoring/run_conditions.pl Run Info vers. 2]<br />
<br />
=== Monitoring Output Files ===<br />
* Run Periods 201Y-MM is for example 2015-03, launch ver verVV is for example ver15<br />
* Online monitoring histograms: /work/halld/online_monitoring/root/<br />
* Offline monitoring histogram ROOT files: /work/halld/data_monitoring/RunPeriod-201Y-MM/verVV/rootfiles<br />
* REST files (most recent launch only): /work/halld/data_monitoring/RunPeriod-201Y-MM/REST/verVV<br />
* individual files for each job (ROOT, log, etc.): /volatile/halld/offline_monitoring/RunPeriod-201Y-MM/verVV/<br />
<br />
=== Monitoring Database ===<br />
* Accessing monitoring database (on ifarm): mysql -u datmon -h hallddb.jlab.org data_monitoring<br />
<br />
=== Monitoring Webpages ===<br />
*[https://halldweb.jlab.org/cgi-bin/data_monitoring/monitoring/plotBrowser.py Plot Browser]<br />
*[https://halldweb.jlab.org/cgi-bin/data_monitoring/monitoring/runBrowser.py Run Browser]<br />
*[https://halldweb.jlab.org/cgi-bin/data_monitoring/monitoring/timeSeries.py Time Series]<br />
*[https://halldweb.jlab.org/data_monitoring/launch_analysis/ Launch Analysis]<br />
<br />
== Job Monitoring Links ==<br />
* [http://scicomp.jlab.org/farm2/job.html Completed Job History]<br />
* [http://scicomp.jlab.org/farm2/project.html Job Stats By Project]<br />
* [http://scicomp.jlab.org/farm2/trackOrg.html Job Stats By Track]<br />
* [http://scicomp.jlab.org/farm2/report.html Cluster Report]<br />
* [http://scicomp.jlab.org/farm2/walltime.html Walltime Distribution]<br />
<br />
== Saving Online Monitoring Data ==<br />
<br />
The procedure for writing the data out is given in, e.g.,<br />
[https://halldweb.jlab.org/wiki/index.php/Raid-to-Silo_Transfer_Strategy Raid-to-Silo Transfer Strategy].<br />
<br />
Once the DAQ writes out the data to the raid disk, cron jobs will copy the file to tape,<br />
and within ~20 min., we will have access to the file on tape at<br />
/mss/halld/$RUN_PERIOD/rawdata/RunXXXXXX.<br />
<br />
All online monitoring plugins will be run as data is taken.<br />
They will be accessible within the counting house via RootSpy, and<br />
for each run and file, a ROOT file containing the histograms will be saved<br />
within a subdirectory for each run.<br />
<br />
For immediate access to these files, the raid disk files may be accessed directly<br />
from the counting house, or the tape files will be available within ~20 min. of the<br />
file being written out.<br />
<br />
== Offline Monitoring: Running Over Archived Data ==<br />
<br />
Once files are written to tape we run the online plugins on these files to confirm what we were seeing in the online monitoring, and also to update the results from the latest calibration and software. <br />
Manual scripts and cron jobs are set up to look for new data and run the plugins over a sample of files.<br />
<br />
Every other Friday (usually the Friday before the offline meetings) jobs will be started to run the newest software on all previous runs,and allows everybody to see improvements in each detector.<br />
For each launch, independent builds of hdds, sim-recon, the monitoring plugins, and an sqlite file will be generated.<br />
<br />
Below the procedures are described for<br />
# Preparing the software for the launch<br />
# Starting the launch (using hdswif)<br />
# Post-analysis of statistics of the launch<br />
<br />
Processing the results and making them available to the collaboration<br />
is handled in the section [[#Post-Processing_Procedures | Post-Processing_Procedures]] below.<br />
<br />
<!--<br />
==== Generating an offline plugin job ====<br />
<br />
The user gxproj1 should be used for official offline monitoring jobs.<br />
Within /home/gxproj1/halld/monitoring/batch/ there will be scripts to run the online monitoring plugins over tape files.<br />
The main script is generatejobs_plugins_rawdata.sh, which can be used as<br />
generatejobs_plugins_rawdata.sh [minrun] [maxrun] (minfile) (maxfile)<br />
where minrun, maxrun specify the range of the run #, and minfile, maxfile (optional) specify<br />
the file range for that run.<br />
<br />
This will generate a script run_rawdata_XXXXXX.sh, where the run # has now been formatted to be 6 digits.<br />
Executing this script will send the monitoring plugins job to the Auger batch system.<br />
<br />
There is also a script clean.sh which can be used as<br />
clean.sh XXX<br />
This will clean up all associated files created in association with run XXX.<br />
<br />
Internally, the xml file used to submit the job will be created, and the job to run<br />
will be given within script.sh. All run parameters should be specified in at the beginning<br />
of generatejobs_plugins_rawdata.sh<br />
Since we are running on tape, the tape file will first be copied over to the cache disk, and the job will run<br />
over this cached file.<br />
<br />
==== Using cron to run automatically ====<br />
Within /home/gluex/halld/monitoring/cron/ there is a file cron_plugins<br />
that can be executed via<br />
crontab cron_plugins<br />
This will set up a cron job to call the script scan_for_jobs.sh, which will<br />
check in the rawdata directory and call generatejobs_plugins_rawdata.sh for<br />
any run that is more than 5 min old. The cron job is set up to run every 10 min.<br />
<br />
==== Magnetic Field Settings ====<br />
For tracking, it is necessary to set the correct magnetic field settings.<br />
The below table is now obsolete. Please refer to the [https://halldweb.jlab.org/cgi-bin/data_monitoring/run_conditions.pl GlueX Run Conditions] webpage.<br />
<br />
The actual field settings for each run have not been documented well.<br />
Below are values based on going through entries in the [https://logbooks.jlab.org/book/halld halld log].<br />
<br />
{| border="1" cellpadding="1" style="text-align: center;"<br />
!width="100"| Run #s<br />
!width="150"| Solenoid Current (A)<br />
!width="300"| JANA option<br />
!width="300"| notes<br />
|-<br />
| 940 - 996 || 1000 || -PBFIELD_MAP=Magnets/Solenoid/solenoid_1000A_poisson_20141104 || At run 997, the solenoid started ramping down<br />
|-<br />
| 998 - 1448 || 0 || -PBFIELD_TYPE=NoField -PDEFTAG:DTrackCandidate=StraightLine || Solenoid started ramping up around run 1431. Runs 1432 - 1448 should not be used for tracking. See [https://logbooks.jlab.org/entry/3309039 this] and [https://logbooks.jlab.org/entry/3309092 this] entry.<br />
|-<br />
| 1449 - 1620 || 1200 || -PBFIELD_MAP=Magnets/Solenoid/solenoid_1200A_poisson_20140520 || <br />
|-<br />
|}<br />
<br />
'''Note:''' The following run ranges were taken with a 300A field, and could be analyzed with the magnetic field map Magnets/Solenoid/solenoid_300A_poisson_20140819 , more study is needed to see if the straight line track fitter is needed or not: 1036 - 1053, 1065 - 1121, 1212? - 1254, 1309 - 1318 [1307,1308,1319-1329 were taken while the magnet was ramping].<br />
--><br />
<br />
=== General Information on Procedures ===<br />
This section explains how the offline monitoring should be run.<br />
Since we may want to simultaneously run offline monitoring for different run periods that require<br />
different environment variables, the scripts are set up so that a generic user can download the<br />
scripts and run them from anywhere. Most output directories for offline monitoring are created<br />
with group read/write permissions so that any Hall D group user has access to the contents,<br />
but there are some cases where use of the account that created the launch is necessary.<br />
<br />
The accounts used for offline monitoring are the gxprojN accounts created and maintained by<br />
Mark Ito (see [https://halldweb.jlab.org/wiki/index.php/GlueX-related_shared_accounts_on_the_JLab_CUE here]).<br />
As of October 2015, we have been using<br />
* gxproj1 for running over Fall 2014 data (deprecated since June 2015)<br />
* gxproj5 for running over Spring 2015 data<br />
<br />
Since the summer of 2015 we have transitioned from a system using Mark Ito's jproj scripts<br />
to integrating the swif system that Chris Larrieu (SciComp) has been developing. For offline monitoring,<br />
the hdswif system that Kei developed is used for launching the jobs, and the jproj system is used<br />
for meta-analysis of launch statistics.<br />
<br />
Both hdswif and jproj are maintained in svn:<br />
* hdswif: https://halldsvn.jlab.org/repos/trunk/scripts/monitoring/hdswif<br />
* jproj : https://halldsvn.jlab.org/repos/trunk/scripts/monitoring/jproj<br />
<br />
To run each package the directories will need to be checked out and the necessary<br />
scripts will need to be run.<br />
<br />
=== Preparing the software for the launch ===<br />
<br />
To begin a new launch the software must be built to the latest versions.<br />
For the gxprojN user accounts used, all software builds are contained in the directory ~/builds<br />
(which are soft links to /work/halld/home/gxproj5/builds). When logging into these accounts<br />
the setup files ~/setup_jlab-2015-03.csh or similar files should be sourced.<br />
<br />
<b>Note that Mark Ito does <u>not</u> want you to change the contents of each .cshrc file.</b><br />
You should consult him if you feel the need.<br />
<br />
<span style="color:red">NOTE: IT IS A GOOD IDEA TO DO A COMPLETE WIPEOUT/CHECKOUT OF A NEW VERSION OF EACH SOFTWARE TO AVOID STALE HEADER FILES.</span><br />
<br />
# Building hdds: Go to ~/builds/hdds. The directory hdds is the one from git. Delete the contents, then download the newest version from git and build: <pre>cd ~/builds/hdds/</pre> <pre>rm -frv hdds</pre> <pre>git clone https://github.com/JeffersonLab/hdds</pre> <pre>scons install</pre><br />
# Building sim-recon: Go to ~/builds/sim-recon. The directory sim-recon is the one from git. Delete the contents, then download the newest version from git and build: <pre>cd ~/builds/sim-recon</pre> <pre>rm -frv sim-recon</pre> <pre>git clone https://github.com/JeffersonLab/sim-recon</pre> <pre>scons install -j8</pre><br />
# Prepare the latest sqlite file: The sqlite is set in the ~/setup_jlab-2015-03.csh script as sqlite:////home/gxproj5/builds/ccdb.sqlite through the environment variables <b>JANA_CALIB_URL</b> and <b>CCDB_CONNECTION</b>. Therefore, go to ~/builds,and create a new sqlite file. We create the sqlite file in a temporary directory since creating the sqlite file in a directory where the output file exists causes errors. Original documentation on creating sqlite files are [https://halldweb.jlab.org/wiki/index.php/SQLite-form_of_the_CCDB_database here]. <pre>cd ~/builds/tmp</pre> <pre>$CCDB_HOME/scripts/mysql2sqlite/mysql2sqlite.sh -hhallddb.jlab.org -uccdb_user ccdb | sqlite3 ccdb.sqlite</pre><br />
# Note that the above steps must be done <b>BEFORE</b> launch project creation. This is because we will track the revisions of the libraries used, and this is done by extracting the svn information in each directory. Also, note that the system assumes that we have the topmost build directory (usually called GLUEX_TOP) to be $HOME/builds . Such an assumption is necessary to be able to extract information about the library locations automatically.<br />
<br />
Create the appropriate project(s) and submit the jobs using hdswif, as detailed in the section below.<br />
<br />
=== Starting the Launch and Submitting Jobs ===<br />
<br />
Until the summer of 2015 we relied solely on Mark Ito's jproj system for submitting and keeping track of jobs. We have since moved to the swif system and use the hdswif wrapper for this. Below are instructions for how to use these.<br />
<br />
# Downloading hdswif: Download the hdswif directory from svn. For the gxprojN accounts, use the directory ~/halld/hdswif. <pre>cd ~/halld </pre> <pre> svn co https://halldsvn.jlab.org/repos/trunk/scripts/monitoring/hdswif</pre> <pre>cd hdswif</pre><br />
# Creating the workflow: Within SWIF jobs are registered into workflows. First create the workflow. For offline monitoring, the workflow names are of the form <b>offline_monitoring_RunPeriod201Y_MM_verVV_hd_rawdata</b> with suitable replacements for the run period and version number. The command "swif list" will list all existing workflows. Also, or most simple SWIF commands hdswif also provides a wrapper. <pre>swif list</pre> <pre>swif create [workflow]</pre> Or equivalently, <pre>hdswif.py create [workflow] </pre><br />
# Registering jobs in the workflow: To register jobs within the workflow, hdswif provides the use of config files. Jobs can be registered by specifying the workflow, config file (-c), run (-r) and file (-f) numbers if necessary. A typical config file will look this:<br />
<pre><br />
PROJECT gluex<br />
TRACK reconstruction<br />
OS centos65<br />
NCORES 4<br />
DISK 40<br />
RAM 8<br />
TIMELIMIT 16<br />
JOBNAMEBASE offmon_<br />
RUNPERIOD 2015-03<br />
VERSION 92<br />
OUTPUT_TOPDIR /volatile/halld/offline_monitoring/RunPeriod-[RUNPERIOD]/ver[VERSION] # Example of other variables included in variable<br />
SCRIPTFILE /home/gxproj5/halld/hdswif/script.sh # Must specify full path<br />
ENVFILE /home/gxproj5/halld/hdswif/setup_jlab-2015-03.csh # Must specify full path<br />
</pre><br />
The config file contains configuration parameters for each of the jobs.<br />
<span style="color:red">Note: Job configuration parameters can be set differently for jobs within the same workflow if necessary.</span><br />
Edit the config file and save as a new file if necessary. Once the configuration is set, jobs can be added via <pre>hdswif.py add [workflow] -c input.config</pre><br />
By default, hdswif will add all files found within the directory /mss/halld/RunPeriod-201Y-MM/rawdata/ where 201Y-MM is specified by the RUNPERIOD parameter in the config file. If only some of the runs or files are needed, these can be specified for example with<br />
<pre>hdswif.py add [workflow] -c input.config -r 3180 -f '00[0-4]'</pre><br />
to specify to register running only over run 3180 files 000 - 004 (Unix-style brackets and wildcards can be used).<br />
# Running the workflow: To run the workflow, simply use swif run: <pre>swif run </pre> or equivalently, using the hdswif wrapper, <pre>hdsswif.py run [workflow]</pre><br />
<b>It is recommended that some jobs be tested over to make sure that everything is working rather than fail thousands of jobs.</b> For this purpose, hdswif will take an additional parameter to run which limits the number of jobs to submit: <pre>hdswif.py run [workflow] 10</pre><br />
in which case only 10 jobs will be submitted. To submit all jobs after checking the results, do <pre>hdswif.py run [workflow]</pre><br />
<br />
=== Post-analysis of statistics of the launch ===<br />
<br />
# After jobs have been submitted, it will usually take a few days for all of the jobs to be processed.<br />
#* Status of Auger: http://scicomp.jlab.org/scicomp/#/auger/jobs (see also links above)<br />
#* Status of user jobs: <pre>jobstat -u [user name]</pre><br />
# The status and results of jobs are saved within the SWIF internal server, and are available via the command <pre>swif status [workflow] -summary -runs</pre> where the arguments -summary and -runs show summary statistics and statistics for individual jobs, respectively. hdswif has a command that takes this output in XML output and creates an HTML webpage showing results of the launch. To do this, do <pre>hdswif.py summary [workflow]</pre> This will create an XML file swif_output_[workflow].xml that contains all information from SWIF. If the file already exists, hdswif will ask whether to overwrite the existing file.<br />
# At this stage the html output and figure files are created and ready to be put online. For this step and other steps involving analysis of the statistics of the launch results, it is convenient to change to the jproj system.<br />
<br />
# The jproj scripts for offline monitoring are maintained in the svn directory https://halldsvn.jlab.org/repos/trunk/scripts/monitoring/jproj Do <pre> svn co https://halldsvn.jlab.org/repos/trunk/scripts/monitoring/jproj </pre><br />
For the gxprojN accounts used for offline monitoring the directory should be ~/halld/jproj<br />
# The jproj directory contains two subdirectories, scripts and projects. The scripts directory contains useful scripts for processing the jobs registered in the jproj system, and each of the offline monitoring launches will be handled in the projects directory.<br />
# Go to the projects directory<pre>cd ~/halld/jproj/projects</pre> and use the script create_project.sh to create a new directory that contains the processing scripts for the current launch <pre>./create_project.sh [workflow]</pre> This should create a directory such as offline_monitoring_RunPeriod2015_03_ver15_hd_rawdata (same as the workflow name). The script uses the template files in the directory templates and by substitution creates script files for the current launch. Now go to the newly created analysis directory: <pre>cd [workflow]/analysis</pre><br />
which for the gxprojN accounts should have the full path /home/gxprojN/halld/jproj/projects/[workflow]/analysis.<br><br><br />
<span style="color:red">All of the analysis commands including arguments are contained in run_analysis.sh, but it is <b>strongly</b> recommended that all commands are run manually to check for errors.</span><br><br><br />
# The first thing to do is to make the html output from hdswif public. Copy the html file and related figures that were created from hdswif to the appropriate space within /group/halld/www/halldweb/html/data_monitoring/ : <pre>python publish_offmon_results.py [run period] [version]</pre> Note that the command with appropriate substitutions for arguments can be found within run_analysis.sh (same for all commands below).<br />
# Next, we need to create a few MySQL tables for the current launch. The MySQL tables are useful for comparing run/file combinations for different launches. For the SWIF launches, there are two tables needed, and will be named<br />
#* [workflow]Job<br />
#* [workflow]_aux<br />
# This naming scheme of tables and their roles are the same as from the jproj only launches. The [workflow]Job table will contain information gathered from SWIF about each job (which node it went to, start time of each stage, memory usage, etc.). The [workflow]_aux table will contain information gathered from the output stdout files from each job. First, create the Job table using <pre>python create_jproj_job_table.py [run period] [version]</pre><br />
# To check the contents of this MySQL table, do <pre>mysql -hhalldb -ufarmer farming</pre> <pre>mysql> describe offline_monitoring_RunPeriod2015_03_ver11_hd_rawdataJob;</pre><br />
<br />
== Hall D Job Management System ==<br />
<br />
This section details instructions on how to create and launch a set of jobs using the Hall-D Job Management System developed by Mark Ito. These instructions are generic: this system can be used for the weekly monitoring jobs, but can also be used for other sets of job launches as well. <br />
<br />
=== Database Table Overview ===<br />
<br />
* Job management database table (<project_name>): For each input file, keeps track of whether or not a job for it has been submitted, along with other optional fields. <br />
<br />
* Job status database table (<project_name>Job (no space)): For each job, keeps track of the job-id, the job status, memory used, cpu & wall time, time taken to complete various stages (e.g. pending, dependency, active), and others. <br />
<br />
* Job metrics database table (<project_name>_aux (no space)): For each job, keeps track of the job-id, how many events were processed, the time it took to copy the cache file, and the time it took to run the plugin. This information is culled from the log files of each job, and is done within the analysis directory of each launch.<br />
<br />
=== Initialize Project Management ===<br />
<br />
* Log into the ifarm machine with one of the gxproj accounts. For this example we will use gxproj1.<br />
<pre><br />
ssh gxproj1@ifarm -Y<br />
</pre><br />
<br />
* Go to a directory to do the launch. In principle, any directory will work, but for gxproj1 this is usually done in /home/gxproj1/halld/<br />
<br />
* Check out the necessary scripts <pre>svn co https://halldsvn.jlab.org/repos/trunk/scripts/monitoring/jproj</pre> This will get all necessary scripts for launching. Once checked out, <pre>cd projects</pre><br />
<br />
* The script create_project.sh can be used to create a new project. It will take a single argument, the project name. It is assumed that the project name is of the form offline_monitoring_RunPeriod20YY_MM_verVV_hd_rawdata where this string will be parsed to give the run period 20YY_MM and version number VV. One thing to do BEFORE creation of a new project is to editthe conditions of the launch (plugins to run over, memory requested, disk space requested) within templates/template.jsub . This<br />
information is saved automatically at project creation time into files in /group/halld/data_monitoring/run_conditions .<br />
To create a project do <pre>./create_project.sh offline_monitoring_RunPeriod20YY_MM_verVV_hd_rawdata</pre><br />
The name has been chosen to be as consistent as possible with other directory structures.<br />
However, mysql requires that "-" be escaped in table names, so unfortunately run periods will be<br />
given as 2YYY_MM instead of 2YYY-MM.<br />
<br />
* A new directory with the project name (offline_monitoring_RunPeriod20YY_MM_verVV_hd_rawdata) and files will be copied and modified from the template directory to reflect the run period, the user, the directory that it was created in, project name, etc.<br />
<br />
* For each project, cd into the new directory<br />
* The script clear.sh will remove any existing tables for the current project name, then recreate it. Do<br />
<pre>./clear.sh</pre><br />
* To use the jproj.pl script that was checked in, add the directory to your path with<br />
<pre>source ../../scripts/setup.csh</pre><br />
or always specify the full path<br />
<pre>../../scripts/jproj.pl</pre><br />
* Now update the table of runs with<br />
<pre>jproj.pl <project name> update</pre><br />
This will fill the table with all files within /mss that are of the same form as what is in <project name>.jproj . If you want to register only on a subset of all such files, you can edit this file directly.<br />
* Once you have registered all of the files you would like to run over, do<br />
<pre>jproj.pl <project name> submit [max # of jobs] [run number]</pre><br />
where the additional options specify how many jobs to submit and which run number to run on. Without these options all files that are registered and have not been submitted yet will be submitted.<br />
<br />
At this stage you are ready to submit all files. It is a good idea to submit a few test jobs<br />
at first to check that all scripts are working and that the plugins do not crash. Once you are<br />
sure that this does not happen, you can send all jobs in. The remaining jobs are then the monitoring<br />
which will (among other things) put the results on the online webpage for the collaboration to view, and<br />
the analysis of the launch.<br />
<br />
=== Project File Overview ===<br />
<br />
An overview of each project file:<br />
* '''clear.sh:''' For the current project, deletes the job status and management database tables (if any), and creates new, empty ones. <br />
* '''<project_name>.jproj:''' Contains the path and file name format for the input files for the jobs. <br />
* '''<project_name>.jsub:''' The xml job submission script. The run number and file number variables are set during job submission for each input file. <br />
* '''script.sh:''' The script that is executed during the job. If output job directories are not pre-created manually, they should be created in this script with the proper permissions:<br />
<pre><br />
mkdir -p -m 775 my_directory<br />
</pre><br />
* '''setup_jlab-[run period].csh:''' The environment that is sourced at the beginning of the job execution. <br />
* '''status.sh:''' Updates the job status database table, and prints some of its columns to screen.<br />
<br />
=== Project Management ===<br />
<br />
* Delete (if any) and create the database table(s) for the current set of job submissions:<br />
<pre><br />
./clear.sh<br />
</pre><br />
Also, if testing was done with jobs, it is best to delete the output directory and the configuration files:<br />
<pre><br />
rm -frv /volatile/halld/offline_monitoring/RunPeriod-20YY-MM/verVV /group/halld/data_monitoring/run_conditions/soft_comm_20YY_MM_verVV.xml /group/halld/data_monitoring/run_conditions/jana_rawdata_comm_20YY_MM_verVV.conf<br />
</pre><br />
<br />
* Search for input files matching the string in the .jproj file, and create a row for each in the job management database table (called <project_name>). You can test by adding an optional argument at the end, which only selects files with a specific file number:<br />
<pre><br />
jproj.pl <project_name> update <optional_file_number><br />
</pre><br />
<br />
* Confirm that the job management database is accurate by printing it's contents to screen:<br />
<pre><br />
mysql -hhallddb -ufarmer farming -e "select * from <project_name>"<br />
</pre><br />
<br />
* ONLY if a mistake was made, to delete the tables from the database and recreate new, empty ones, run: <br />
<pre><br />
./clear.sh<br />
</pre><br />
<br />
* To look at the status of the submitted jobs, first query auger and update the job status database:<br />
<pre><br />
fill_in_job_details.pl <project_name><br />
</pre><br />
<br />
* The job status can then be viewed by submitting a query to the job status database (called <project_name>Job (no space in between)):<br />
<pre><br />
mysql -hhallddb -ufarmer farming -e "select id,run,file,jobId,hostname,status,timeSubmitted,timeActive,walltime,cput,timeComplete,result,error from <project_name>Job"<br />
</pre><br />
<br />
* These last two commands can instead be executed simultaneously by running:<br />
<pre><br />
./status.sh<br />
</pre><br />
<br />
=== Handy mysql Instructions ===<br />
<br />
* Handy mysql instructions:<br />
<pre><br />
mysql -hhallddb -ufarmer farming # Enter the "farming" mysql database on "hallddb" as user "farmer"<br />
quit; # Exit mysql<br />
show tables; # Show a list of the tables in the current database<br />
show columns from <project_name>; # show all of the columns for the given table<br />
select * from <project_name>; # show the contents of all rows from the given table<br />
</pre><br />
<br />
=== Backing Up Offline Monitoring Tables ===<br />
Tables created for offline monitoring can be backed up using the script backup_tables.sh which<br />
can be checked out with the other files from https://halldsvn.jlab.org/repos/trunk/scripts/monitoring/jproj/projects<br />
<br />
The script uses the command mysqldump to print out a file that can be executed to recreate the tables.<br />
Since executing this output file will drop the table when it exists, caution is advised.<br />
Example usage to backup all three tables created for run period 2014_10 ver 17:<br />
<pre>backup_tables.sh 2014_10 17</pre><br />
<br />
== Running Over Data As It Comes In ==<br />
<br />
A special user gxproj1 will have a cron job set up to run the plugins as new data appears on /mss.<br />
During the week, gxproj1 will submit offline plugin jobs with the same setup as the weekly jobs<br />
run the previous Friday. The procedure for this is shown below.<br />
<br />
<!--<br />
=== Setting up the environment ===<br />
The file<br />
/home/gxproj1/setup_jlab.csh<br />
is sourced through .tcshrc.<br />
This file is the same as what is linked to by<br />
/home/gluex/setup_jlab_commissioning.csh,<br />
except HALLD_HOME, HDDS_HOME, and JANA_CALIB_URL are set separately so that this<br />
user can have a separate build.<br />
<br />
To obtain the builds from the previous Friday's runs,<br />
execute<br />
/home/gxproj1/halld/monitoring/newruns/setup_previous.sh [year] [month] [day]<br />
The build revisions from the previous Friday are archived in files<br />
/work/halld/data_monitoring/run_conditions/soft_comm_[year]_[month]_[day].xml<br />
and the script will build libraries based on those stored revision numbers.<br />
--><br />
<br />
=== Running the cron job ===<br />
<br />
'''IMPORTANT:''' The cron job should not be running while you are manually submitting jobs using the jproj.pl script for the same project, or else you will probably multiply-submit a job. <br />
<br />
* Go to the cron job directory: <br />
<pre><br />
cd /u/home/gxproj1/halld/monitoring/newruns<br />
</pre><br />
<br />
* The cron_plugins file is the cronjob that will be executed. During execution, it runs the exec.sh command in the same folder. This command takes two arguments: the project name, and the maximum file number for each run. These fields should be updated in the cron_plugins file before running. <br />
<br />
* The exec.sh command updates the job management database table with any data that has arrived on tape since it was last updated, ignoring file numbers greater than the maximum file number. It then submits jobs for these files. <br />
<br />
* To start the cron job, run:<br />
<pre><br />
crontab cron_plugins<br />
</pre><br />
<br />
* To check whether the cron job is running, do<br />
<pre><br />
crontab -l<br />
</pre><br />
<br />
* To remove the cron job do<br />
<pre><br />
crontab -r<br />
</pre><br />
<br />
<!--<br />
The cron job will run the script scan_for_jobs.sh,<br />
which runs generatejobs_plugins_rawdata.sh for any<br />
new runs that it had not seen before. All previous<br />
runs are recorded in the file filelists/files_current.txt<br />
so clear this to run over runs, or set the parameters<br />
MINRUN and MAXRUN which will set the range of runs submitted.<br />
--><br />
<br />
==Post-Processing Procedures==<br />
<br />
To visualize the monitoring data, we save images of selected histograms and store time series of selected quantities in a database, which are then displayed on a web page. This section describes how to generate the monitoring images and database information.<br />
<br />
The scripts used to generate this summary data are primarily run from /home/gxprojN/halld/monitoring/process . If you want a new copy of the scripts, e.g., for a new monitoring run, you should check the scripts out from SVN:<br />
<syntaxhighlight><br />
svn co https://halldsvn.jlab.org/repos/trunk/scripts/monitoring/process<br />
</syntaxhighlight><br />
Note that these scripts currently have some parameters which must be periodically set by hand.<br />
<br />
The default python version on most JLab machine does not have the modules to allow these scripts to connect to the MySQL database. To run these scripts, load the environment with the following command<br />
<syntaxhighlight><br />
source /home/gxproj1/halld/monitoring/process/monitoring_env.csh<br />
</syntaxhighlight><br />
<br />
===Online Monitoring===<br />
<br />
There are two primary scripts for running over the monitoring data generated by the online system and offline reconstruction. The online script can be run with either of the following commands:<br />
<syntaxhighlight><br />
/home/gluex/halld/monitoring/process/check_new_runs.py<br />
<br />
OR <br />
<br />
/home/gluex/halld/monitoring/process/check_new_runs.csh<br />
</syntaxhighlight><br />
The shell script sets up the environment properly to run the python script. To connect to the monitoring database on the JLab CUE, modules included in the local installation of python >= 2.7 are needed. The shell script is appropriate to use in a cron job. The cronjob is currently run under the "gluex" account.<br />
<br />
The online monitoring system copies a ROOT file containing the results of the online monitoring, and other configuration files into a directory accessible outside the counting house. This python script automatically checks for new ROOT files, which it will then automatically process. It contains several configuration variables that must be correctly set, which contains the location of input/output directories, etc...<br />
<br />
===Offline Monitoring===<br />
<br />
After the data is run over, the results should be processed, so that summary data is entered into the monitoring database and plots are made for the monitoring webpages. Currently, this processing is controlled by a cronjob that runs the following script:<br />
<syntaxhighlight><br />
/home/gxproj1/halld/monitoring/process/check_monitoring_data.csh <br />
</syntaxhighlight><br />
This script checks for new ROOT files, and only runs over those it hasn't processed yet. Since one monitoring ROOT file is produced for each EVIO file, whenever a new file is produced, the plots for the corresponding run are recreated and all the ROOT files for a run are combined into one file. Information is stored in the database on a per-file basis. <br />
<br />
Plots for the monitoring web page can be made from single histograms or multiple histograms using RootSpy macros. If you want to change the list of plots made, you must modify one of the following files:<br />
* histograms_to_monitor - specify either the name of the histogram or its the full ROOT path <br />
* macros_to_monitor - specify the full path to the RootSpy macro .C file<br />
<br />
When a new monitoring run is started, or the conditions are changed, the following steps should be taken to process the new files:<br />
# Add a new data version, as described below:<br />
# Change the following parameters in check_monitoring_data.csh:<br />
## JOBDATE should correspond to the ouptut date used by the job submission script<br />
## OUTPUTDIR should correspond to the directory corresponding to the run period and revision corresponding to the new version you just submitted. Presumably, this directory will be empty at the beginning.<br />
## Once you create a new data version as defined below, you should pass the needed information as a command line option. Currently this is done by the ARGS variable. For example, the argument "-v RunPeriod-2014-10,8" tells the monitoring scripts to look up the version corresponding to revision 8 of RunPeriod-2014-10 in the monitoring DB and to use to store the results.<br />
<br />
<syntaxhighlight><br />
Example configuration parameters:<br />
set JOBDATE=2015-01-09<br />
set INPUTDIR=/volatile/halld/RunPeriod-2014-10/offline_monitoring<br />
set OUTPUTDIR=/w/halld-scifs1a/data_monitoring/RunPeriod-2014-10/ver08<br />
set ARGS=" -v RunPeriod-2014-10,8 "<br />
</syntaxhighlight><br />
If you want to process the results manually, the data is processed using the following script:<br />
<syntaxhighlight><br />
./process_new_offline_data.py <input directory> <output directory><br />
<br />
EXAMPLE:<br />
<br />
./process_new_offline_data.py 2014-11-14 /volatile/halld/RunPeriod-2014-10/offline_monitoring/ /w/halld-scifs1a/data_monitoring/RunPeriod-2014-10/ver02<br />
</syntaxhighlight><br />
The python script takes several options to enable/disable various steps in the processing. Of interest is the "--force" option, which will run over all monitoring ROOT files, whether or not they've been previously identified.<br />
<br />
Every time a new reconstruction pass is performed, a new version number must be generated. To do this, prepare a version file as described below. Then run the register_new_version.py script to store the information in the database. The script will return a version number, which then should be set by hand in process_new_offline_data.py - future versions of the script will streamline this part of the procedure. An example of how to generate a new version is:<br />
<syntaxhighlight><br />
./register_new_version.py add /home/gxproj1/halld/monitoring/process/versions/vers_RunPeriod-2014-10_pass1.txt<br />
</syntaxhighlight><br />
<br />
<b>If you are running the offline monitoring by checking out the files in trunk/scripts/monitoring/jproj/projects/<br />
of the svn repository</b>, and created a project with<br />
create_project.sh [project name] hd_rawdata<br />
Then go to the directory [project name]/processing/<br />
and execute<br />
./run_processing.sh<br />
which will run register_new_version.py as well as check_monitoring_data.csh for that project.<br />
<br />
===Step-by-Step Instructions For Processing a New Monitoring Run===<br />
<br />
The monitoring runs are current run out of the gxproj1 and gxproj5 accounts. After an offline monitoring run has been successfully started on the batch farm, the following steps should be followed to setup the post-processing for these runs.<br />
<br />
# The post-processing scripts are stored in $HOME/halld/monitoring/process and are automatically run by cron.<br />
# Run "svn update" to bring any changes in. Be sure that the list of histograms and macros to plot are current.<br />
# Edit check_monitoring_data.csh to point to the current revisions/directories<br />
#* VERSION<br />
#* ARGS<br />
#* Note that the environment depends on a standard script - $HOME/setup_jlab.csh<br />
# Update files in the web directory, so that the results are displayed on the web pages: /group/halld/www/halldweb/html/data_monitoring/textdata<br />
# Copy the REST files to more permanent locations:<br />
#* cp -a /volatile/halld/offline_monitoring/RunPeriod-YYYY-MM/verVV/REST /work/halld/data_monitoring/RunPeriod-YYYY-MM/REST/verVV<br />
#* cp -a /volatile/halld/offline_monitoring/RunPeriod-YYYY-MM/verVV/REST /cache/halld/RunPeriod-YYYY-MM/REST/verVV [under testing]<br />
<br />
Check log files in $HOME/halld/monitoring/process/log for more information on how each run went. If there are problems, check log files, and modify check_monitoring_data.csh to vary the verbosity of the output.<br />
<br />
==Data Versions==<br />
<br />
To document the conditions of the monitoring data that is created, for the sake of reproducability and further analysis we save several pieces of information. The format is intended to be comprehensive enough to document not just monitoring data, but versions of raw and reconstructed data, so that this database table can be used for the event database as well.<br />
<br />
We store one record per pass through one run period, with the following structure:<br />
<br />
{| class="wikitable"<br />
! Field !! Description<br />
|-<br />
| data_type || The level of data we are processing. For the purposes of monitoring, "rawdata" is the online monitoring, "recon" is the offline monitoring<br />
|- <br />
| run_period || The run period of the data<br />
|- <br />
| revision || An integer specifying which pass through the run period this data corresponds to<br />
|- <br />
| software_version || The name of the XML file that specifies the different software versions used<br />
|-<br />
| jana_config || The name of the text file that specifies which JANA options were passed to the reconstruction program<br />
|-<br />
| ccdb_context || The value of JANA_CALIB_CONTEXT, which specifies the version of calibration constants that were used<br />
|-<br />
| production_time || The data at which monitoring/reconstruction began<br />
|-<br />
| dataVersionString || A convenient string for identifying this version of the data<br />
|}<br />
<br />
<br />
An example file used as as input to ./register_new_version.py is:<br />
<syntaxhighlight><br />
data_type = recon<br />
run_period = RunPeriod-2014-10<br />
revision = 1<br />
software_version = soft_comm_2014_11_06.xml<br />
jana_config = jana_rawdata_comm_2014_11_06.conf<br />
ccdb_context = calibtime=2014-11-10<br />
production_time = 2014-11-10<br />
dataVersionString = recon_RunPeriod-2014-10_20141110_ver01<br />
</syntaxhighlight></div>Kmoriyahttps://halldweb.jlab.org/wiki/index.php?title=Data_Monitoring_Procedures&diff=70738Data Monitoring Procedures2015-10-15T01:02:09Z<p>Kmoriya: /* Post-analysis of statistics of the launch */</p>
<hr />
<div>__TOC__<br />
<br />
== Master List of File / Database / Webpage Locations ==<br />
=== Run Conditions ===<br />
* Online Run-by-run condition files (B-field, current, etc.): /work/halld/online_monitoring/conditions/<br />
* Offline monitoring run conditions (software versions, jana config): /group/halld/data_monitoring/run_conditions/<br />
*[http://www.jlab.org/Hall-D/test/RunInfo/ Run Info vers. 1]<br />
*[https://halldweb.jlab.org/cgi-bin/data_monitoring/run_conditions.pl Run Info vers. 2]<br />
<br />
=== Monitoring Output Files ===<br />
* Run Periods 201Y-MM is for example 2015-03, launch ver verVV is for example ver15<br />
* Online monitoring histograms: /work/halld/online_monitoring/root/<br />
* Offline monitoring histogram ROOT files: /work/halld/data_monitoring/RunPeriod-201Y-MM/verVV/rootfiles<br />
* REST files (most recent launch only): /work/halld/data_monitoring/RunPeriod-201Y-MM/REST/verVV<br />
* individual files for each job (ROOT, log, etc.): /volatile/halld/offline_monitoring/RunPeriod-201Y-MM/verVV/<br />
<br />
=== Monitoring Database ===<br />
* Accessing monitoring database (on ifarm): mysql -u datmon -h hallddb.jlab.org data_monitoring<br />
<br />
=== Monitoring Webpages ===<br />
*[https://halldweb.jlab.org/cgi-bin/data_monitoring/monitoring/plotBrowser.py Plot Browser]<br />
*[https://halldweb.jlab.org/cgi-bin/data_monitoring/monitoring/runBrowser.py Run Browser]<br />
*[https://halldweb.jlab.org/cgi-bin/data_monitoring/monitoring/timeSeries.py Time Series]<br />
*[https://halldweb.jlab.org/data_monitoring/launch_analysis/ Launch Analysis]<br />
<br />
== Job Monitoring Links ==<br />
* [http://scicomp.jlab.org/farm2/job.html Completed Job History]<br />
* [http://scicomp.jlab.org/farm2/project.html Job Stats By Project]<br />
* [http://scicomp.jlab.org/farm2/trackOrg.html Job Stats By Track]<br />
* [http://scicomp.jlab.org/farm2/report.html Cluster Report]<br />
* [http://scicomp.jlab.org/farm2/walltime.html Walltime Distribution]<br />
<br />
== Saving Online Monitoring Data ==<br />
<br />
The procedure for writing the data out is given in, e.g.,<br />
[https://halldweb.jlab.org/wiki/index.php/Raid-to-Silo_Transfer_Strategy Raid-to-Silo Transfer Strategy].<br />
<br />
Once the DAQ writes out the data to the raid disk, cron jobs will copy the file to tape,<br />
and within ~20 min., we will have access to the file on tape at<br />
/mss/halld/$RUN_PERIOD/rawdata/RunXXXXXX.<br />
<br />
All online monitoring plugins will be run as data is taken.<br />
They will be accessible within the counting house via RootSpy, and<br />
for each run and file, a ROOT file containing the histograms will be saved<br />
within a subdirectory for each run.<br />
<br />
For immediate access to these files, the raid disk files may be accessed directly<br />
from the counting house, or the tape files will be available within ~20 min. of the<br />
file being written out.<br />
<br />
== Offline Monitoring: Running Over Archived Data ==<br />
<br />
Once files are written to tape we run the online plugins on these files to confirm what we were seeing in the online monitoring, and also to update the results from the latest calibration and software. <br />
Manual scripts and cron jobs are set up to look for new data and run the plugins over a sample of files.<br />
<br />
Every other Friday (usually the Friday before the offline meetings) jobs will be started to run the newest software on all previous runs,and allows everybody to see improvements in each detector.<br />
For each launch, independent builds of hdds, sim-recon, the monitoring plugins, and an sqlite file will be generated.<br />
<br />
Below the procedures are described for<br />
# Preparing the software for the launch<br />
# Starting the launch (using hdswif)<br />
# Post-analysis of statistics of the launch<br />
<br />
Processing the results and making them available to the collaboration<br />
is handled in the section [[#Post-Processing_Procedures | Post-Processing_Procedures]] below.<br />
<br />
<!--<br />
==== Generating an offline plugin job ====<br />
<br />
The user gxproj1 should be used for official offline monitoring jobs.<br />
Within /home/gxproj1/halld/monitoring/batch/ there will be scripts to run the online monitoring plugins over tape files.<br />
The main script is generatejobs_plugins_rawdata.sh, which can be used as<br />
generatejobs_plugins_rawdata.sh [minrun] [maxrun] (minfile) (maxfile)<br />
where minrun, maxrun specify the range of the run #, and minfile, maxfile (optional) specify<br />
the file range for that run.<br />
<br />
This will generate a script run_rawdata_XXXXXX.sh, where the run # has now been formatted to be 6 digits.<br />
Executing this script will send the monitoring plugins job to the Auger batch system.<br />
<br />
There is also a script clean.sh which can be used as<br />
clean.sh XXX<br />
This will clean up all associated files created in association with run XXX.<br />
<br />
Internally, the xml file used to submit the job will be created, and the job to run<br />
will be given within script.sh. All run parameters should be specified in at the beginning<br />
of generatejobs_plugins_rawdata.sh<br />
Since we are running on tape, the tape file will first be copied over to the cache disk, and the job will run<br />
over this cached file.<br />
<br />
==== Using cron to run automatically ====<br />
Within /home/gluex/halld/monitoring/cron/ there is a file cron_plugins<br />
that can be executed via<br />
crontab cron_plugins<br />
This will set up a cron job to call the script scan_for_jobs.sh, which will<br />
check in the rawdata directory and call generatejobs_plugins_rawdata.sh for<br />
any run that is more than 5 min old. The cron job is set up to run every 10 min.<br />
<br />
==== Magnetic Field Settings ====<br />
For tracking, it is necessary to set the correct magnetic field settings.<br />
The below table is now obsolete. Please refer to the [https://halldweb.jlab.org/cgi-bin/data_monitoring/run_conditions.pl GlueX Run Conditions] webpage.<br />
<br />
The actual field settings for each run have not been documented well.<br />
Below are values based on going through entries in the [https://logbooks.jlab.org/book/halld halld log].<br />
<br />
{| border="1" cellpadding="1" style="text-align: center;"<br />
!width="100"| Run #s<br />
!width="150"| Solenoid Current (A)<br />
!width="300"| JANA option<br />
!width="300"| notes<br />
|-<br />
| 940 - 996 || 1000 || -PBFIELD_MAP=Magnets/Solenoid/solenoid_1000A_poisson_20141104 || At run 997, the solenoid started ramping down<br />
|-<br />
| 998 - 1448 || 0 || -PBFIELD_TYPE=NoField -PDEFTAG:DTrackCandidate=StraightLine || Solenoid started ramping up around run 1431. Runs 1432 - 1448 should not be used for tracking. See [https://logbooks.jlab.org/entry/3309039 this] and [https://logbooks.jlab.org/entry/3309092 this] entry.<br />
|-<br />
| 1449 - 1620 || 1200 || -PBFIELD_MAP=Magnets/Solenoid/solenoid_1200A_poisson_20140520 || <br />
|-<br />
|}<br />
<br />
'''Note:''' The following run ranges were taken with a 300A field, and could be analyzed with the magnetic field map Magnets/Solenoid/solenoid_300A_poisson_20140819 , more study is needed to see if the straight line track fitter is needed or not: 1036 - 1053, 1065 - 1121, 1212? - 1254, 1309 - 1318 [1307,1308,1319-1329 were taken while the magnet was ramping].<br />
--><br />
<br />
=== General Information on Procedures ===<br />
This section explains how the offline monitoring should be run.<br />
Since we may want to simultaneously run offline monitoring for different run periods that require<br />
different environment variables, the scripts are set up so that a generic user can download the<br />
scripts and run them from anywhere. Most output directories for offline monitoring are created<br />
with group read/write permissions so that any Hall D group user has access to the contents,<br />
but there are some cases where use of the account that created the launch is necessary.<br />
<br />
The accounts used for offline monitoring are the gxprojN accounts created and maintained by<br />
Mark Ito (see [https://halldweb.jlab.org/wiki/index.php/GlueX-related_shared_accounts_on_the_JLab_CUE here]).<br />
As of October 2015, we have been using<br />
* gxproj1 for running over Fall 2014 data (deprecated since June 2015)<br />
* gxproj5 for running over Spring 2015 data<br />
<br />
Since the summer of 2015 we have transitioned from a system using Mark Ito's jproj scripts<br />
to integrating the swif system that Chris Larrieu (SciComp) has been developing. For offline monitoring,<br />
the hdswif system that Kei developed is used for launching the jobs, and the jproj system is used<br />
for meta-analysis of launch statistics.<br />
<br />
Both hdswif and jproj are maintained in svn:<br />
* hdswif: https://halldsvn.jlab.org/repos/trunk/scripts/monitoring/hdswif<br />
* jproj : https://halldsvn.jlab.org/repos/trunk/scripts/monitoring/jproj<br />
<br />
To run each package the directories will need to be checked out and the necessary<br />
scripts will need to be run.<br />
<br />
=== Preparing the software for the launch ===<br />
<br />
To begin a new launch the software must be built to the latest versions.<br />
For the gxprojN user accounts used, all software builds are contained in the directory ~/builds<br />
(which are soft links to /work/halld/home/gxproj5/builds). When logging into these accounts<br />
the setup files ~/setup_jlab-2015-03.csh or similar files should be sourced.<br />
<br />
<b>Note that Mark Ito does <u>not</u> want you to change the contents of each .cshrc file.</b><br />
You should consult him if you feel the need.<br />
<br />
<span style="color:red">NOTE: IT IS A GOOD IDEA TO DO A COMPLETE WIPEOUT/CHECKOUT OF A NEW VERSION OF EACH SOFTWARE TO AVOID STALE HEADER FILES.</span><br />
<br />
# Building hdds: Go to ~/builds/hdds. The directory hdds is the one from git. Delete the contents, then download the newest version from git and build: <pre>cd ~/builds/hdds/</pre> <pre>rm -frv hdds</pre> <pre>git clone https://github.com/JeffersonLab/hdds</pre> <pre>scons install</pre><br />
# Building sim-recon: Go to ~/builds/sim-recon. The directory sim-recon is the one from git. Delete the contents, then download the newest version from git and build: <pre>cd ~/builds/sim-recon</pre> <pre>rm -frv sim-recon</pre> <pre>git clone https://github.com/JeffersonLab/sim-recon</pre> <pre>scons install -j8</pre><br />
# Prepare the latest sqlite file: The sqlite is set in the ~/setup_jlab-2015-03.csh script as sqlite:////home/gxproj5/builds/ccdb.sqlite through the environment variables <b>JANA_CALIB_URL</b> and <b>CCDB_CONNECTION</b>. Therefore, go to ~/builds,and create a new sqlite file. We create the sqlite file in a temporary directory since creating the sqlite file in a directory where the output file exists causes errors. Original documentation on creating sqlite files are [https://halldweb.jlab.org/wiki/index.php/SQLite-form_of_the_CCDB_database here]. <pre>cd ~/builds/tmp</pre> <pre>$CCDB_HOME/scripts/mysql2sqlite/mysql2sqlite.sh -hhallddb.jlab.org -uccdb_user ccdb | sqlite3 ccdb.sqlite</pre><br />
# Note that the above steps must be done <b>BEFORE</b> launch project creation. This is because we will track the revisions of the libraries used, and this is done by extracting the svn information in each directory. Also, note that the system assumes that we have the topmost build directory (usually called GLUEX_TOP) to be $HOME/builds . Such an assumption is necessary to be able to extract information about the library locations automatically.<br />
<br />
Create the appropriate project(s) and submit the jobs using hdswif, as detailed in the section below.<br />
<br />
=== Starting the Launch and Submitting Jobs ===<br />
<br />
Until the summer of 2015 we relied solely on Mark Ito's jproj system for submitting and keeping track of jobs. We have since moved to the swif system and use the hdswif wrapper for this. Below are instructions for how to use these.<br />
<br />
# Downloading hdswif: Download the hdswif directory from svn. For the gxprojN accounts, use the directory ~/halld/hdswif. <pre>cd ~/halld </pre> <pre> svn co https://halldsvn.jlab.org/repos/trunk/scripts/monitoring/hdswif</pre> <pre>cd hdswif</pre><br />
# Creating the workflow: Within SWIF jobs are registered into workflows. First create the workflow. For offline monitoring, the workflow names are of the form <b>offline_monitoring_RunPeriod201Y_MM_verVV_hd_rawdata</b> with suitable replacements for the run period and version number. The command "swif list" will list all existing workflows. Also, or most simple SWIF commands hdswif also provides a wrapper. <pre>swif list</pre> <pre>swif create [workflow]</pre> Or equivalently, <pre>hdswif.py create [workflow] </pre><br />
# Registering jobs in the workflow: To register jobs within the workflow, hdswif provides the use of config files. Jobs can be registered by specifying the workflow, config file (-c), run (-r) and file (-f) numbers if necessary. A typical config file will look this:<br />
<pre><br />
PROJECT gluex<br />
TRACK reconstruction<br />
OS centos65<br />
NCORES 4<br />
DISK 40<br />
RAM 8<br />
TIMELIMIT 16<br />
JOBNAMEBASE offmon_<br />
RUNPERIOD 2015-03<br />
VERSION 92<br />
OUTPUT_TOPDIR /volatile/halld/offline_monitoring/RunPeriod-[RUNPERIOD]/ver[VERSION] # Example of other variables included in variable<br />
SCRIPTFILE /home/gxproj5/halld/hdswif/script.sh # Must specify full path<br />
ENVFILE /home/gxproj5/halld/hdswif/setup_jlab-2015-03.csh # Must specify full path<br />
</pre><br />
The config file contains configuration parameters for each of the jobs.<br />
<span style="color:red">Note: Job configuration parameters can be set differently for jobs within the same workflow if necessary.</span><br />
Edit the config file and save as a new file if necessary. Once the configuration is set, jobs can be added via <pre>hdswif.py add [workflow] -c input.config</pre><br />
By default, hdswif will add all files found within the directory /mss/halld/RunPeriod-201Y-MM/rawdata/ where 201Y-MM is specified by the RUNPERIOD parameter in the config file. If only some of the runs or files are needed, these can be specified for example with<br />
<pre>hdswif.py add [workflow] -c input.config -r 3180 -f '00[0-4]'</pre><br />
to specify to register running only over run 3180 files 000 - 004 (Unix-style brackets and wildcards can be used).<br />
# Running the workflow: To run the workflow, simply use swif run: <pre>swif run </pre> or equivalently, using the hdswif wrapper, <pre>hdsswif.py run [workflow]</pre><br />
<b>It is recommended that some jobs be tested over to make sure that everything is working rather than fail thousands of jobs.</b> For this purpose, hdswif will take an additional parameter to run which limits the number of jobs to submit: <pre>hdswif.py run [workflow] 10</pre><br />
in which case only 10 jobs will be submitted. To submit all jobs after checking the results, do <pre>hdswif.py run [workflow]</pre><br />
<br />
=== Post-analysis of statistics of the launch ===<br />
<br />
# After jobs have been submitted, it will usually take a few days for all of the jobs to be processed.<br />
* Status of Auger: http://scicomp.jlab.org/scicomp/#/auger/jobs (see also links above)<br />
* Status of user jobs: <pre>jobstat -u [user name]</pre><br />
# The status and results of jobs are saved within the SWIF internal server, and are available via the command <pre>swif status [workflow] -summary -runs</pre> where the arguments -summary and -runs show summary statistics and statistics for individual jobs, respectively. hdswif has a command that takes this output in XML output and creates an HTML webpage showing results of the launch. To do this, do <pre>hdswif.py summary [workflow]</pre> This will create an XML file swif_output_[workflow].xml that contains all information from SWIF. If the file already exists, hdswif will ask whether to overwrite the existing file.<br />
# At this stage the html output and figure files are created and ready to be put online. For this step and other steps involving analysis of the statistics of the launch results, it is convenient to change to the jproj system.<br />
<br />
The jproj scripts for offline monitoring are maintained in the svn directory https://halldsvn.jlab.org/repos/trunk/scripts/monitoring/jproj Do <pre> svn co https://halldsvn.jlab.org/repos/trunk/scripts/monitoring/jproj </pre><br />
For the gxprojN accounts used for offline monitoring the directory should be ~/halld/jproj<br />
<br />
The jproj directory contains two subdirectories, scripts and projects. The scripts directory contains useful scripts for processing the jobs registered in the jproj system, and each of the offline monitoring launches will be handled in the projects directory.<br />
<br />
Go to the projects directory<pre>cd ~/halld/jproj/projects</pre> and use the script create_project.sh to create a new directory that contains the processing scripts for the current launch <pre>./create_project.sh [workflow]</pre> This should create a directory such as offline_monitoring_RunPeriod2015_03_ver15_hd_rawdata (same as the workflow name). The script uses the template files in the directory templates and by substitution creates script files for the current launch. Now go to the newly created analysis directory: <pre>cd [workflow]/analysis</pre><br />
which for the gxprojN accounts should have the full path /home/gxprojN/halld/jproj/projects/[workflow]/analysis.<br />
<br />
Once the html output is created, copy the html file and related figures: <pre>cp -r summary_swif_output_[workflow].html figures/</pre><br />
<br />
== Hall D Job Management System ==<br />
<br />
This section details instructions on how to create and launch a set of jobs using the Hall-D Job Management System developed by Mark Ito. These instructions are generic: this system can be used for the weekly monitoring jobs, but can also be used for other sets of job launches as well. <br />
<br />
=== Database Table Overview ===<br />
<br />
* Job management database table (<project_name>): For each input file, keeps track of whether or not a job for it has been submitted, along with other optional fields. <br />
<br />
* Job status database table (<project_name>Job (no space)): For each job, keeps track of the job-id, the job status, memory used, cpu & wall time, time taken to complete various stages (e.g. pending, dependency, active), and others. <br />
<br />
* Job metrics database table (<project_name>_aux (no space)): For each job, keeps track of the job-id, how many events were processed, the time it took to copy the cache file, and the time it took to run the plugin. This information is culled from the log files of each job, and is done within the analysis directory of each launch.<br />
<br />
=== Initialize Project Management ===<br />
<br />
* Log into the ifarm machine with one of the gxproj accounts. For this example we will use gxproj1.<br />
<pre><br />
ssh gxproj1@ifarm -Y<br />
</pre><br />
<br />
* Go to a directory to do the launch. In principle, any directory will work, but for gxproj1 this is usually done in /home/gxproj1/halld/<br />
<br />
* Check out the necessary scripts <pre>svn co https://halldsvn.jlab.org/repos/trunk/scripts/monitoring/jproj</pre> This will get all necessary scripts for launching. Once checked out, <pre>cd projects</pre><br />
<br />
* The script create_project.sh can be used to create a new project. It will take a single argument, the project name. It is assumed that the project name is of the form offline_monitoring_RunPeriod20YY_MM_verVV_hd_rawdata where this string will be parsed to give the run period 20YY_MM and version number VV. One thing to do BEFORE creation of a new project is to editthe conditions of the launch (plugins to run over, memory requested, disk space requested) within templates/template.jsub . This<br />
information is saved automatically at project creation time into files in /group/halld/data_monitoring/run_conditions .<br />
To create a project do <pre>./create_project.sh offline_monitoring_RunPeriod20YY_MM_verVV_hd_rawdata</pre><br />
The name has been chosen to be as consistent as possible with other directory structures.<br />
However, mysql requires that "-" be escaped in table names, so unfortunately run periods will be<br />
given as 2YYY_MM instead of 2YYY-MM.<br />
<br />
* A new directory with the project name (offline_monitoring_RunPeriod20YY_MM_verVV_hd_rawdata) and files will be copied and modified from the template directory to reflect the run period, the user, the directory that it was created in, project name, etc.<br />
<br />
* For each project, cd into the new directory<br />
* The script clear.sh will remove any existing tables for the current project name, then recreate it. Do<br />
<pre>./clear.sh</pre><br />
* To use the jproj.pl script that was checked in, add the directory to your path with<br />
<pre>source ../../scripts/setup.csh</pre><br />
or always specify the full path<br />
<pre>../../scripts/jproj.pl</pre><br />
* Now update the table of runs with<br />
<pre>jproj.pl <project name> update</pre><br />
This will fill the table with all files within /mss that are of the same form as what is in <project name>.jproj . If you want to register only on a subset of all such files, you can edit this file directly.<br />
* Once you have registered all of the files you would like to run over, do<br />
<pre>jproj.pl <project name> submit [max # of jobs] [run number]</pre><br />
where the additional options specify how many jobs to submit and which run number to run on. Without these options all files that are registered and have not been submitted yet will be submitted.<br />
<br />
At this stage you are ready to submit all files. It is a good idea to submit a few test jobs<br />
at first to check that all scripts are working and that the plugins do not crash. Once you are<br />
sure that this does not happen, you can send all jobs in. The remaining jobs are then the monitoring<br />
which will (among other things) put the results on the online webpage for the collaboration to view, and<br />
the analysis of the launch.<br />
<br />
=== Project File Overview ===<br />
<br />
An overview of each project file:<br />
* '''clear.sh:''' For the current project, deletes the job status and management database tables (if any), and creates new, empty ones. <br />
* '''<project_name>.jproj:''' Contains the path and file name format for the input files for the jobs. <br />
* '''<project_name>.jsub:''' The xml job submission script. The run number and file number variables are set during job submission for each input file. <br />
* '''script.sh:''' The script that is executed during the job. If output job directories are not pre-created manually, they should be created in this script with the proper permissions:<br />
<pre><br />
mkdir -p -m 775 my_directory<br />
</pre><br />
* '''setup_jlab-[run period].csh:''' The environment that is sourced at the beginning of the job execution. <br />
* '''status.sh:''' Updates the job status database table, and prints some of its columns to screen.<br />
<br />
=== Project Management ===<br />
<br />
* Delete (if any) and create the database table(s) for the current set of job submissions:<br />
<pre><br />
./clear.sh<br />
</pre><br />
Also, if testing was done with jobs, it is best to delete the output directory and the configuration files:<br />
<pre><br />
rm -frv /volatile/halld/offline_monitoring/RunPeriod-20YY-MM/verVV /group/halld/data_monitoring/run_conditions/soft_comm_20YY_MM_verVV.xml /group/halld/data_monitoring/run_conditions/jana_rawdata_comm_20YY_MM_verVV.conf<br />
</pre><br />
<br />
* Search for input files matching the string in the .jproj file, and create a row for each in the job management database table (called <project_name>). You can test by adding an optional argument at the end, which only selects files with a specific file number:<br />
<pre><br />
jproj.pl <project_name> update <optional_file_number><br />
</pre><br />
<br />
* Confirm that the job management database is accurate by printing it's contents to screen:<br />
<pre><br />
mysql -hhallddb -ufarmer farming -e "select * from <project_name>"<br />
</pre><br />
<br />
* ONLY if a mistake was made, to delete the tables from the database and recreate new, empty ones, run: <br />
<pre><br />
./clear.sh<br />
</pre><br />
<br />
* To look at the status of the submitted jobs, first query auger and update the job status database:<br />
<pre><br />
fill_in_job_details.pl <project_name><br />
</pre><br />
<br />
* The job status can then be viewed by submitting a query to the job status database (called <project_name>Job (no space in between)):<br />
<pre><br />
mysql -hhallddb -ufarmer farming -e "select id,run,file,jobId,hostname,status,timeSubmitted,timeActive,walltime,cput,timeComplete,result,error from <project_name>Job"<br />
</pre><br />
<br />
* These last two commands can instead be executed simultaneously by running:<br />
<pre><br />
./status.sh<br />
</pre><br />
<br />
=== Handy mysql Instructions ===<br />
<br />
* Handy mysql instructions:<br />
<pre><br />
mysql -hhallddb -ufarmer farming # Enter the "farming" mysql database on "hallddb" as user "farmer"<br />
quit; # Exit mysql<br />
show tables; # Show a list of the tables in the current database<br />
show columns from <project_name>; # show all of the columns for the given table<br />
select * from <project_name>; # show the contents of all rows from the given table<br />
</pre><br />
<br />
=== Backing Up Offline Monitoring Tables ===<br />
Tables created for offline monitoring can be backed up using the script backup_tables.sh which<br />
can be checked out with the other files from https://halldsvn.jlab.org/repos/trunk/scripts/monitoring/jproj/projects<br />
<br />
The script uses the command mysqldump to print out a file that can be executed to recreate the tables.<br />
Since executing this output file will drop the table when it exists, caution is advised.<br />
Example usage to backup all three tables created for run period 2014_10 ver 17:<br />
<pre>backup_tables.sh 2014_10 17</pre><br />
<br />
== Running Over Data As It Comes In ==<br />
<br />
A special user gxproj1 will have a cron job set up to run the plugins as new data appears on /mss.<br />
During the week, gxproj1 will submit offline plugin jobs with the same setup as the weekly jobs<br />
run the previous Friday. The procedure for this is shown below.<br />
<br />
<!--<br />
=== Setting up the environment ===<br />
The file<br />
/home/gxproj1/setup_jlab.csh<br />
is sourced through .tcshrc.<br />
This file is the same as what is linked to by<br />
/home/gluex/setup_jlab_commissioning.csh,<br />
except HALLD_HOME, HDDS_HOME, and JANA_CALIB_URL are set separately so that this<br />
user can have a separate build.<br />
<br />
To obtain the builds from the previous Friday's runs,<br />
execute<br />
/home/gxproj1/halld/monitoring/newruns/setup_previous.sh [year] [month] [day]<br />
The build revisions from the previous Friday are archived in files<br />
/work/halld/data_monitoring/run_conditions/soft_comm_[year]_[month]_[day].xml<br />
and the script will build libraries based on those stored revision numbers.<br />
--><br />
<br />
=== Running the cron job ===<br />
<br />
'''IMPORTANT:''' The cron job should not be running while you are manually submitting jobs using the jproj.pl script for the same project, or else you will probably multiply-submit a job. <br />
<br />
* Go to the cron job directory: <br />
<pre><br />
cd /u/home/gxproj1/halld/monitoring/newruns<br />
</pre><br />
<br />
* The cron_plugins file is the cronjob that will be executed. During execution, it runs the exec.sh command in the same folder. This command takes two arguments: the project name, and the maximum file number for each run. These fields should be updated in the cron_plugins file before running. <br />
<br />
* The exec.sh command updates the job management database table with any data that has arrived on tape since it was last updated, ignoring file numbers greater than the maximum file number. It then submits jobs for these files. <br />
<br />
* To start the cron job, run:<br />
<pre><br />
crontab cron_plugins<br />
</pre><br />
<br />
* To check whether the cron job is running, do<br />
<pre><br />
crontab -l<br />
</pre><br />
<br />
* To remove the cron job do<br />
<pre><br />
crontab -r<br />
</pre><br />
<br />
<!--<br />
The cron job will run the script scan_for_jobs.sh,<br />
which runs generatejobs_plugins_rawdata.sh for any<br />
new runs that it had not seen before. All previous<br />
runs are recorded in the file filelists/files_current.txt<br />
so clear this to run over runs, or set the parameters<br />
MINRUN and MAXRUN which will set the range of runs submitted.<br />
--><br />
<br />
==Post-Processing Procedures==<br />
<br />
To visualize the monitoring data, we save images of selected histograms and store time series of selected quantities in a database, which are then displayed on a web page. This section describes how to generate the monitoring images and database information.<br />
<br />
The scripts used to generate this summary data are primarily run from /home/gxprojN/halld/monitoring/process . If you want a new copy of the scripts, e.g., for a new monitoring run, you should check the scripts out from SVN:<br />
<syntaxhighlight><br />
svn co https://halldsvn.jlab.org/repos/trunk/scripts/monitoring/process<br />
</syntaxhighlight><br />
Note that these scripts currently have some parameters which must be periodically set by hand.<br />
<br />
The default python version on most JLab machine does not have the modules to allow these scripts to connect to the MySQL database. To run these scripts, load the environment with the following command<br />
<syntaxhighlight><br />
source /home/gxproj1/halld/monitoring/process/monitoring_env.csh<br />
</syntaxhighlight><br />
<br />
===Online Monitoring===<br />
<br />
There are two primary scripts for running over the monitoring data generated by the online system and offline reconstruction. The online script can be run with either of the following commands:<br />
<syntaxhighlight><br />
/home/gluex/halld/monitoring/process/check_new_runs.py<br />
<br />
OR <br />
<br />
/home/gluex/halld/monitoring/process/check_new_runs.csh<br />
</syntaxhighlight><br />
The shell script sets up the environment properly to run the python script. To connect to the monitoring database on the JLab CUE, modules included in the local installation of python >= 2.7 are needed. The shell script is appropriate to use in a cron job. The cronjob is currently run under the "gluex" account.<br />
<br />
The online monitoring system copies a ROOT file containing the results of the online monitoring, and other configuration files into a directory accessible outside the counting house. This python script automatically checks for new ROOT files, which it will then automatically process. It contains several configuration variables that must be correctly set, which contains the location of input/output directories, etc...<br />
<br />
===Offline Monitoring===<br />
<br />
After the data is run over, the results should be processed, so that summary data is entered into the monitoring database and plots are made for the monitoring webpages. Currently, this processing is controlled by a cronjob that runs the following script:<br />
<syntaxhighlight><br />
/home/gxproj1/halld/monitoring/process/check_monitoring_data.csh <br />
</syntaxhighlight><br />
This script checks for new ROOT files, and only runs over those it hasn't processed yet. Since one monitoring ROOT file is produced for each EVIO file, whenever a new file is produced, the plots for the corresponding run are recreated and all the ROOT files for a run are combined into one file. Information is stored in the database on a per-file basis. <br />
<br />
Plots for the monitoring web page can be made from single histograms or multiple histograms using RootSpy macros. If you want to change the list of plots made, you must modify one of the following files:<br />
* histograms_to_monitor - specify either the name of the histogram or its the full ROOT path <br />
* macros_to_monitor - specify the full path to the RootSpy macro .C file<br />
<br />
When a new monitoring run is started, or the conditions are changed, the following steps should be taken to process the new files:<br />
# Add a new data version, as described below:<br />
# Change the following parameters in check_monitoring_data.csh:<br />
## JOBDATE should correspond to the ouptut date used by the job submission script<br />
## OUTPUTDIR should correspond to the directory corresponding to the run period and revision corresponding to the new version you just submitted. Presumably, this directory will be empty at the beginning.<br />
## Once you create a new data version as defined below, you should pass the needed information as a command line option. Currently this is done by the ARGS variable. For example, the argument "-v RunPeriod-2014-10,8" tells the monitoring scripts to look up the version corresponding to revision 8 of RunPeriod-2014-10 in the monitoring DB and to use to store the results.<br />
<br />
<syntaxhighlight><br />
Example configuration parameters:<br />
set JOBDATE=2015-01-09<br />
set INPUTDIR=/volatile/halld/RunPeriod-2014-10/offline_monitoring<br />
set OUTPUTDIR=/w/halld-scifs1a/data_monitoring/RunPeriod-2014-10/ver08<br />
set ARGS=" -v RunPeriod-2014-10,8 "<br />
</syntaxhighlight><br />
If you want to process the results manually, the data is processed using the following script:<br />
<syntaxhighlight><br />
./process_new_offline_data.py <input directory> <output directory><br />
<br />
EXAMPLE:<br />
<br />
./process_new_offline_data.py 2014-11-14 /volatile/halld/RunPeriod-2014-10/offline_monitoring/ /w/halld-scifs1a/data_monitoring/RunPeriod-2014-10/ver02<br />
</syntaxhighlight><br />
The python script takes several options to enable/disable various steps in the processing. Of interest is the "--force" option, which will run over all monitoring ROOT files, whether or not they've been previously identified.<br />
<br />
Every time a new reconstruction pass is performed, a new version number must be generated. To do this, prepare a version file as described below. Then run the register_new_version.py script to store the information in the database. The script will return a version number, which then should be set by hand in process_new_offline_data.py - future versions of the script will streamline this part of the procedure. An example of how to generate a new version is:<br />
<syntaxhighlight><br />
./register_new_version.py add /home/gxproj1/halld/monitoring/process/versions/vers_RunPeriod-2014-10_pass1.txt<br />
</syntaxhighlight><br />
<br />
<b>If you are running the offline monitoring by checking out the files in trunk/scripts/monitoring/jproj/projects/<br />
of the svn repository</b>, and created a project with<br />
create_project.sh [project name] hd_rawdata<br />
Then go to the directory [project name]/processing/<br />
and execute<br />
./run_processing.sh<br />
which will run register_new_version.py as well as check_monitoring_data.csh for that project.<br />
<br />
===Step-by-Step Instructions For Processing a New Monitoring Run===<br />
<br />
The monitoring runs are current run out of the gxproj1 and gxproj5 accounts. After an offline monitoring run has been successfully started on the batch farm, the following steps should be followed to setup the post-processing for these runs.<br />
<br />
# The post-processing scripts are stored in $HOME/halld/monitoring/process and are automatically run by cron.<br />
# Run "svn update" to bring any changes in. Be sure that the list of histograms and macros to plot are current.<br />
# Edit check_monitoring_data.csh to point to the current revisions/directories<br />
#* VERSION<br />
#* ARGS<br />
#* Note that the environment depends on a standard script - $HOME/setup_jlab.csh<br />
# Update files in the web directory, so that the results are displayed on the web pages: /group/halld/www/halldweb/html/data_monitoring/textdata<br />
# Copy the REST files to more permanent locations:<br />
#* cp -a /volatile/halld/offline_monitoring/RunPeriod-YYYY-MM/verVV/REST /work/halld/data_monitoring/RunPeriod-YYYY-MM/REST/verVV<br />
#* cp -a /volatile/halld/offline_monitoring/RunPeriod-YYYY-MM/verVV/REST /cache/halld/RunPeriod-YYYY-MM/REST/verVV [under testing]<br />
<br />
Check log files in $HOME/halld/monitoring/process/log for more information on how each run went. If there are problems, check log files, and modify check_monitoring_data.csh to vary the verbosity of the output.<br />
<br />
==Data Versions==<br />
<br />
To document the conditions of the monitoring data that is created, for the sake of reproducability and further analysis we save several pieces of information. The format is intended to be comprehensive enough to document not just monitoring data, but versions of raw and reconstructed data, so that this database table can be used for the event database as well.<br />
<br />
We store one record per pass through one run period, with the following structure:<br />
<br />
{| class="wikitable"<br />
! Field !! Description<br />
|-<br />
| data_type || The level of data we are processing. For the purposes of monitoring, "rawdata" is the online monitoring, "recon" is the offline monitoring<br />
|- <br />
| run_period || The run period of the data<br />
|- <br />
| revision || An integer specifying which pass through the run period this data corresponds to<br />
|- <br />
| software_version || The name of the XML file that specifies the different software versions used<br />
|-<br />
| jana_config || The name of the text file that specifies which JANA options were passed to the reconstruction program<br />
|-<br />
| ccdb_context || The value of JANA_CALIB_CONTEXT, which specifies the version of calibration constants that were used<br />
|-<br />
| production_time || The data at which monitoring/reconstruction began<br />
|-<br />
| dataVersionString || A convenient string for identifying this version of the data<br />
|}<br />
<br />
<br />
An example file used as as input to ./register_new_version.py is:<br />
<syntaxhighlight><br />
data_type = recon<br />
run_period = RunPeriod-2014-10<br />
revision = 1<br />
software_version = soft_comm_2014_11_06.xml<br />
jana_config = jana_rawdata_comm_2014_11_06.conf<br />
ccdb_context = calibtime=2014-11-10<br />
production_time = 2014-11-10<br />
dataVersionString = recon_RunPeriod-2014-10_20141110_ver01<br />
</syntaxhighlight></div>Kmoriyahttps://halldweb.jlab.org/wiki/index.php?title=Data_Monitoring_Procedures&diff=70735Data Monitoring Procedures2015-10-14T20:33:53Z<p>Kmoriya: /* Offline Monitoring: Running Over Archived Data */</p>
<hr />
<div>__TOC__<br />
<br />
== Master List of File / Database / Webpage Locations ==<br />
=== Run Conditions ===<br />
* Online Run-by-run condition files (B-field, current, etc.): /work/halld/online_monitoring/conditions/<br />
* Offline monitoring run conditions (software versions, jana config): /group/halld/data_monitoring/run_conditions/<br />
*[http://www.jlab.org/Hall-D/test/RunInfo/ Run Info vers. 1]<br />
*[https://halldweb.jlab.org/cgi-bin/data_monitoring/run_conditions.pl Run Info vers. 2]<br />
<br />
=== Monitoring Output Files ===<br />
* Run Periods 201Y-MM is for example 2015-03, launch ver verVV is for example ver15<br />
* Online monitoring histograms: /work/halld/online_monitoring/root/<br />
* Offline monitoring histogram ROOT files: /work/halld/data_monitoring/RunPeriod-201Y-MM/verVV/rootfiles<br />
* REST files (most recent launch only): /work/halld/data_monitoring/RunPeriod-201Y-MM/REST/verVV<br />
* individual files for each job (ROOT, log, etc.): /volatile/halld/offline_monitoring/RunPeriod-201Y-MM/verVV/<br />
<br />
=== Monitoring Database ===<br />
* Accessing monitoring database (on ifarm): mysql -u datmon -h hallddb.jlab.org data_monitoring<br />
<br />
=== Monitoring Webpages ===<br />
*[https://halldweb.jlab.org/cgi-bin/data_monitoring/monitoring/plotBrowser.py Plot Browser]<br />
*[https://halldweb.jlab.org/cgi-bin/data_monitoring/monitoring/runBrowser.py Run Browser]<br />
*[https://halldweb.jlab.org/cgi-bin/data_monitoring/monitoring/timeSeries.py Time Series]<br />
*[https://halldweb.jlab.org/data_monitoring/launch_analysis/ Launch Analysis]<br />
<br />
== Job Monitoring Links ==<br />
* [http://scicomp.jlab.org/farm2/job.html Completed Job History]<br />
* [http://scicomp.jlab.org/farm2/project.html Job Stats By Project]<br />
* [http://scicomp.jlab.org/farm2/trackOrg.html Job Stats By Track]<br />
* [http://scicomp.jlab.org/farm2/report.html Cluster Report]<br />
* [http://scicomp.jlab.org/farm2/walltime.html Walltime Distribution]<br />
<br />
== Saving Online Monitoring Data ==<br />
<br />
The procedure for writing the data out is given in, e.g.,<br />
[https://halldweb.jlab.org/wiki/index.php/Raid-to-Silo_Transfer_Strategy Raid-to-Silo Transfer Strategy].<br />
<br />
Once the DAQ writes out the data to the raid disk, cron jobs will copy the file to tape,<br />
and within ~20 min., we will have access to the file on tape at<br />
/mss/halld/$RUN_PERIOD/rawdata/RunXXXXXX.<br />
<br />
All online monitoring plugins will be run as data is taken.<br />
They will be accessible within the counting house via RootSpy, and<br />
for each run and file, a ROOT file containing the histograms will be saved<br />
within a subdirectory for each run.<br />
<br />
For immediate access to these files, the raid disk files may be accessed directly<br />
from the counting house, or the tape files will be available within ~20 min. of the<br />
file being written out.<br />
<br />
== Offline Monitoring: Running Over Archived Data ==<br />
<br />
Once files are written to tape we run the online plugins on these files to confirm what we were seeing in the online monitoring, and also to update the results from the latest calibration and software. <br />
Manual scripts and cron jobs are set up to look for new data and run the plugins over a sample of files.<br />
<br />
Every other Friday (usually the Friday before the offline meetings) jobs will be started to run the newest software on all previous runs,and allows everybody to see improvements in each detector.<br />
For each launch, independent builds of hdds, sim-recon, the monitoring plugins, and an sqlite file will be generated.<br />
<br />
Below the procedures are described for<br />
# Preparing the software for the launch<br />
# Starting the launch (using hdswif)<br />
# Post-analysis of statistics of the launch<br />
<br />
Processing the results and making them available to the collaboration<br />
is handled in the section [[#Post-Processing_Procedures | Post-Processing_Procedures]] below.<br />
<br />
<!--<br />
==== Generating an offline plugin job ====<br />
<br />
The user gxproj1 should be used for official offline monitoring jobs.<br />
Within /home/gxproj1/halld/monitoring/batch/ there will be scripts to run the online monitoring plugins over tape files.<br />
The main script is generatejobs_plugins_rawdata.sh, which can be used as<br />
generatejobs_plugins_rawdata.sh [minrun] [maxrun] (minfile) (maxfile)<br />
where minrun, maxrun specify the range of the run #, and minfile, maxfile (optional) specify<br />
the file range for that run.<br />
<br />
This will generate a script run_rawdata_XXXXXX.sh, where the run # has now been formatted to be 6 digits.<br />
Executing this script will send the monitoring plugins job to the Auger batch system.<br />
<br />
There is also a script clean.sh which can be used as<br />
clean.sh XXX<br />
This will clean up all associated files created in association with run XXX.<br />
<br />
Internally, the xml file used to submit the job will be created, and the job to run<br />
will be given within script.sh. All run parameters should be specified in at the beginning<br />
of generatejobs_plugins_rawdata.sh<br />
Since we are running on tape, the tape file will first be copied over to the cache disk, and the job will run<br />
over this cached file.<br />
<br />
==== Using cron to run automatically ====<br />
Within /home/gluex/halld/monitoring/cron/ there is a file cron_plugins<br />
that can be executed via<br />
crontab cron_plugins<br />
This will set up a cron job to call the script scan_for_jobs.sh, which will<br />
check in the rawdata directory and call generatejobs_plugins_rawdata.sh for<br />
any run that is more than 5 min old. The cron job is set up to run every 10 min.<br />
<br />
==== Magnetic Field Settings ====<br />
For tracking, it is necessary to set the correct magnetic field settings.<br />
The below table is now obsolete. Please refer to the [https://halldweb.jlab.org/cgi-bin/data_monitoring/run_conditions.pl GlueX Run Conditions] webpage.<br />
<br />
The actual field settings for each run have not been documented well.<br />
Below are values based on going through entries in the [https://logbooks.jlab.org/book/halld halld log].<br />
<br />
{| border="1" cellpadding="1" style="text-align: center;"<br />
!width="100"| Run #s<br />
!width="150"| Solenoid Current (A)<br />
!width="300"| JANA option<br />
!width="300"| notes<br />
|-<br />
| 940 - 996 || 1000 || -PBFIELD_MAP=Magnets/Solenoid/solenoid_1000A_poisson_20141104 || At run 997, the solenoid started ramping down<br />
|-<br />
| 998 - 1448 || 0 || -PBFIELD_TYPE=NoField -PDEFTAG:DTrackCandidate=StraightLine || Solenoid started ramping up around run 1431. Runs 1432 - 1448 should not be used for tracking. See [https://logbooks.jlab.org/entry/3309039 this] and [https://logbooks.jlab.org/entry/3309092 this] entry.<br />
|-<br />
| 1449 - 1620 || 1200 || -PBFIELD_MAP=Magnets/Solenoid/solenoid_1200A_poisson_20140520 || <br />
|-<br />
|}<br />
<br />
'''Note:''' The following run ranges were taken with a 300A field, and could be analyzed with the magnetic field map Magnets/Solenoid/solenoid_300A_poisson_20140819 , more study is needed to see if the straight line track fitter is needed or not: 1036 - 1053, 1065 - 1121, 1212? - 1254, 1309 - 1318 [1307,1308,1319-1329 were taken while the magnet was ramping].<br />
--><br />
<br />
=== General Information on Procedures ===<br />
This section explains how the offline monitoring should be run.<br />
Since we may want to simultaneously run offline monitoring for different run periods that require<br />
different environment variables, the scripts are set up so that a generic user can download the<br />
scripts and run them from anywhere. Most output directories for offline monitoring are created<br />
with group read/write permissions so that any Hall D group user has access to the contents,<br />
but there are some cases where use of the account that created the launch is necessary.<br />
<br />
The accounts used for offline monitoring are the gxprojN accounts created and maintained by<br />
Mark Ito (see [https://halldweb.jlab.org/wiki/index.php/GlueX-related_shared_accounts_on_the_JLab_CUE here]).<br />
As of October 2015, we have been using<br />
* gxproj1 for running over Fall 2014 data (deprecated since June 2015)<br />
* gxproj5 for running over Spring 2015 data<br />
<br />
Since the summer of 2015 we have transitioned from a system using Mark Ito's jproj scripts<br />
to integrating the swif system that Chris Larrieu (SciComp) has been developing. For offline monitoring,<br />
the hdswif system that Kei developed is used for launching the jobs, and the jproj system is used<br />
for meta-analysis of launch statistics.<br />
<br />
Both hdswif and jproj are maintained in svn:<br />
* hdswif: https://halldsvn.jlab.org/repos/trunk/scripts/monitoring/hdswif<br />
* jproj : https://halldsvn.jlab.org/repos/trunk/scripts/monitoring/jproj<br />
<br />
To run each package the directories will need to be checked out and the necessary<br />
scripts will need to be run.<br />
<br />
=== Preparing the software for the launch ===<br />
<br />
To begin a new launch the software must be built to the latest versions.<br />
For the gxprojN user accounts used, all software builds are contained in the directory ~/builds<br />
(which are soft links to /work/halld/home/gxproj5/builds). When logging into these accounts<br />
the setup files ~/setup_jlab-2015-03.csh or similar files should be sourced.<br />
<br />
<b>Note that Mark Ito does <u>not</u> want you to change the contents of each .cshrc file.</b><br />
You should consult him if you feel the need.<br />
<br />
<span style="color:red">NOTE: IT IS A GOOD IDEA TO DO A COMPLETE WIPEOUT/CHECKOUT OF A NEW VERSION OF EACH SOFTWARE TO AVOID STALE HEADER FILES.</span><br />
<br />
# Building hdds: Go to ~/builds/hdds. The directory hdds is the one from git. Delete the contents, then download the newest version from git and build: <pre>cd ~/builds/hdds/</pre> <pre>rm -frv hdds</pre> <pre>git clone https://github.com/JeffersonLab/hdds</pre> <pre>scons install</pre><br />
# Building sim-recon: Go to ~/builds/sim-recon. The directory sim-recon is the one from git. Delete the contents, then download the newest version from git and build: <pre>cd ~/builds/sim-recon</pre> <pre>rm -frv sim-recon</pre> <pre>git clone https://github.com/JeffersonLab/sim-recon</pre> <pre>scons install -j8</pre><br />
# Prepare the latest sqlite file: The sqlite is set in the ~/setup_jlab-2015-03.csh script as sqlite:////home/gxproj5/builds/ccdb.sqlite through the environment variables <b>JANA_CALIB_URL</b> and <b>CCDB_CONNECTION</b>. Therefore, go to ~/builds,and create a new sqlite file. We create the sqlite file in a temporary directory since creating the sqlite file in a directory where the output file exists causes errors. Original documentation on creating sqlite files are [https://halldweb.jlab.org/wiki/index.php/SQLite-form_of_the_CCDB_database here]. <pre>cd ~/builds/tmp</pre> <pre>$CCDB_HOME/scripts/mysql2sqlite/mysql2sqlite.sh -hhallddb.jlab.org -uccdb_user ccdb | sqlite3 ccdb.sqlite</pre><br />
# Note that the above steps must be done <b>BEFORE</b> launch project creation. This is because we will track the revisions of the libraries used, and this is done by extracting the svn information in each directory. Also, note that the system assumes that we have the topmost build directory (usually called GLUEX_TOP) to be $HOME/builds . Such an assumption is necessary to be able to extract information about the library locations automatically.<br />
<br />
Create the appropriate project(s) and submit the jobs using hdswif, as detailed in the section below.<br />
<br />
=== Starting the Launch and Submitting Jobs ===<br />
<br />
Until the summer of 2015 we relied solely on Mark Ito's jproj system for submitting and keeping track of jobs. We have since moved to the swif system and use the hdswif wrapper for this. Below are instructions for how to use these.<br />
<br />
# Downloading hdswif: Download the hdswif directory from svn. For the gxprojN accounts, use the directory ~/halld/hdswif. <pre>cd ~/halld </pre> <pre> svn co https://halldsvn.jlab.org/repos/trunk/scripts/monitoring/hdswif</pre> <pre>cd hdswif</pre><br />
# Creating the workflow: Within SWIF jobs are registered into workflows. First create the workflow. For offline monitoring, the workflow names are of the form <b>offline_monitoring_RunPeriod201Y_MM_verVV_hd_rawdata</b> with suitable replacements for the run period and version number. The command "swif list" will list all existing workflows. Also, or most simple SWIF commands hdswif also provides a wrapper. <pre>swif list</pre> <pre>swif create [workflow]</pre> Or equivalently, <pre>hdswif.py create [workflow] </pre><br />
# Registering jobs in the workflow: To register jobs within the workflow, hdswif provides the use of config files. Jobs can be registered by specifying the workflow, config file (-c), run (-r) and file (-f) numbers if necessary. A typical config file will look this:<br />
<pre><br />
PROJECT gluex<br />
TRACK reconstruction<br />
OS centos65<br />
NCORES 4<br />
DISK 40<br />
RAM 8<br />
TIMELIMIT 16<br />
JOBNAMEBASE offmon_<br />
RUNPERIOD 2015-03<br />
VERSION 92<br />
OUTPUT_TOPDIR /volatile/halld/offline_monitoring/RunPeriod-[RUNPERIOD]/ver[VERSION] # Example of other variables included in variable<br />
SCRIPTFILE /home/gxproj5/halld/hdswif/script.sh # Must specify full path<br />
ENVFILE /home/gxproj5/halld/hdswif/setup_jlab-2015-03.csh # Must specify full path<br />
</pre><br />
The config file contains configuration parameters for each of the jobs.<br />
<span style="color:red">Note: Job configuration parameters can be set differently for jobs within the same workflow if necessary.</span><br />
Edit the config file and save as a new file if necessary. Once the configuration is set, jobs can be added via <pre>hdswif.py add [workflow] -c input.config</pre><br />
By default, hdswif will add all files found within the directory /mss/halld/RunPeriod-201Y-MM/rawdata/ where 201Y-MM is specified by the RUNPERIOD parameter in the config file. If only some of the runs or files are needed, these can be specified for example with<br />
<pre>hdswif.py add [workflow] -c input.config -r 3180 -f '00[0-4]'</pre><br />
to specify to register running only over run 3180 files 000 - 004 (Unix-style brackets and wildcards can be used).<br />
# Running the workflow: To run the workflow, simply use swif run: <pre>swif run </pre> or equivalently, using the hdswif wrapper, <pre>hdsswif.py run [workflow]</pre><br />
<b>It is recommended that some jobs be tested over to make sure that everything is working rather than fail thousands of jobs.</b> For this purpose, hdswif will take an additional parameter to run which limits the number of jobs to submit: <pre>hdswif.py run [workflow] 10</pre><br />
in which case only 10 jobs will be submitted. To submit all jobs after checking the results, do <pre>hdswif.py run [workflow]</pre><br />
<br />
=== Post-analysis of statistics of the launch ===<br />
<br />
# After jobs have been submitted, it will usually take a few days for all of the jobs to be processed.<br />
* Status of Auger: http://scicomp.jlab.org/scicomp/#/auger/jobs (see also links above)<br />
* Status of user jobs: <pre>jobstat -u [user name]</pre><br />
# The status and results of jobs are saved within the SWIF internal server, and are available via the command <pre>swif status [workflow] -summary -runs</pre> where the arguments -summary and -runs show summary statistics and statistics for individual jobs, respectively. hdswif has a command that takes this output in XML output and creates an HTML webpage showing results of the launch. To do this, do <pre>hdswif.py summary [workflow]</pre> This will create an XML file swif_output_[workflow].xml that contains all information from SWIF. If the file already exists, hdswif will ask whether to overwrite the existing file.<br />
# Once the html output is created, copy the html file and related figures: <pre>cp -r summary_swif_output_[workflow].html figures/</pre><br />
<br />
== Hall D Job Management System ==<br />
<br />
This section details instructions on how to create and launch a set of jobs using the Hall-D Job Management System developed by Mark Ito. These instructions are generic: this system can be used for the weekly monitoring jobs, but can also be used for other sets of job launches as well. <br />
<br />
=== Database Table Overview ===<br />
<br />
* Job management database table (<project_name>): For each input file, keeps track of whether or not a job for it has been submitted, along with other optional fields. <br />
<br />
* Job status database table (<project_name>Job (no space)): For each job, keeps track of the job-id, the job status, memory used, cpu & wall time, time taken to complete various stages (e.g. pending, dependency, active), and others. <br />
<br />
* Job metrics database table (<project_name>_aux (no space)): For each job, keeps track of the job-id, how many events were processed, the time it took to copy the cache file, and the time it took to run the plugin. This information is culled from the log files of each job, and is done within the analysis directory of each launch.<br />
<br />
=== Initialize Project Management ===<br />
<br />
* Log into the ifarm machine with one of the gxproj accounts. For this example we will use gxproj1.<br />
<pre><br />
ssh gxproj1@ifarm -Y<br />
</pre><br />
<br />
* Go to a directory to do the launch. In principle, any directory will work, but for gxproj1 this is usually done in /home/gxproj1/halld/<br />
<br />
* Check out the necessary scripts <pre>svn co https://halldsvn.jlab.org/repos/trunk/scripts/monitoring/jproj</pre> This will get all necessary scripts for launching. Once checked out, <pre>cd projects</pre><br />
<br />
* The script create_project.sh can be used to create a new project. It will take a single argument, the project name. It is assumed that the project name is of the form offline_monitoring_RunPeriod20YY_MM_verVV_hd_rawdata where this string will be parsed to give the run period 20YY_MM and version number VV. One thing to do BEFORE creation of a new project is to editthe conditions of the launch (plugins to run over, memory requested, disk space requested) within templates/template.jsub . This<br />
information is saved automatically at project creation time into files in /group/halld/data_monitoring/run_conditions .<br />
To create a project do <pre>./create_project.sh offline_monitoring_RunPeriod20YY_MM_verVV_hd_rawdata</pre><br />
The name has been chosen to be as consistent as possible with other directory structures.<br />
However, mysql requires that "-" be escaped in table names, so unfortunately run periods will be<br />
given as 2YYY_MM instead of 2YYY-MM.<br />
<br />
* A new directory with the project name (offline_monitoring_RunPeriod20YY_MM_verVV_hd_rawdata) and files will be copied and modified from the template directory to reflect the run period, the user, the directory that it was created in, project name, etc.<br />
<br />
* For each project, cd into the new directory<br />
* The script clear.sh will remove any existing tables for the current project name, then recreate it. Do<br />
<pre>./clear.sh</pre><br />
* To use the jproj.pl script that was checked in, add the directory to your path with<br />
<pre>source ../../scripts/setup.csh</pre><br />
or always specify the full path<br />
<pre>../../scripts/jproj.pl</pre><br />
* Now update the table of runs with<br />
<pre>jproj.pl <project name> update</pre><br />
This will fill the table with all files within /mss that are of the same form as what is in <project name>.jproj . If you want to register only on a subset of all such files, you can edit this file directly.<br />
* Once you have registered all of the files you would like to run over, do<br />
<pre>jproj.pl <project name> submit [max # of jobs] [run number]</pre><br />
where the additional options specify how many jobs to submit and which run number to run on. Without these options all files that are registered and have not been submitted yet will be submitted.<br />
<br />
At this stage you are ready to submit all files. It is a good idea to submit a few test jobs<br />
at first to check that all scripts are working and that the plugins do not crash. Once you are<br />
sure that this does not happen, you can send all jobs in. The remaining jobs are then the monitoring<br />
which will (among other things) put the results on the online webpage for the collaboration to view, and<br />
the analysis of the launch.<br />
<br />
=== Project File Overview ===<br />
<br />
An overview of each project file:<br />
* '''clear.sh:''' For the current project, deletes the job status and management database tables (if any), and creates new, empty ones. <br />
* '''<project_name>.jproj:''' Contains the path and file name format for the input files for the jobs. <br />
* '''<project_name>.jsub:''' The xml job submission script. The run number and file number variables are set during job submission for each input file. <br />
* '''script.sh:''' The script that is executed during the job. If output job directories are not pre-created manually, they should be created in this script with the proper permissions:<br />
<pre><br />
mkdir -p -m 775 my_directory<br />
</pre><br />
* '''setup_jlab-[run period].csh:''' The environment that is sourced at the beginning of the job execution. <br />
* '''status.sh:''' Updates the job status database table, and prints some of its columns to screen.<br />
<br />
=== Project Management ===<br />
<br />
* Delete (if any) and create the database table(s) for the current set of job submissions:<br />
<pre><br />
./clear.sh<br />
</pre><br />
Also, if testing was done with jobs, it is best to delete the output directory and the configuration files:<br />
<pre><br />
rm -frv /volatile/halld/offline_monitoring/RunPeriod-20YY-MM/verVV /group/halld/data_monitoring/run_conditions/soft_comm_20YY_MM_verVV.xml /group/halld/data_monitoring/run_conditions/jana_rawdata_comm_20YY_MM_verVV.conf<br />
</pre><br />
<br />
* Search for input files matching the string in the .jproj file, and create a row for each in the job management database table (called <project_name>). You can test by adding an optional argument at the end, which only selects files with a specific file number:<br />
<pre><br />
jproj.pl <project_name> update <optional_file_number><br />
</pre><br />
<br />
* Confirm that the job management database is accurate by printing it's contents to screen:<br />
<pre><br />
mysql -hhallddb -ufarmer farming -e "select * from <project_name>"<br />
</pre><br />
<br />
* ONLY if a mistake was made, to delete the tables from the database and recreate new, empty ones, run: <br />
<pre><br />
./clear.sh<br />
</pre><br />
<br />
* To look at the status of the submitted jobs, first query auger and update the job status database:<br />
<pre><br />
fill_in_job_details.pl <project_name><br />
</pre><br />
<br />
* The job status can then be viewed by submitting a query to the job status database (called <project_name>Job (no space in between)):<br />
<pre><br />
mysql -hhallddb -ufarmer farming -e "select id,run,file,jobId,hostname,status,timeSubmitted,timeActive,walltime,cput,timeComplete,result,error from <project_name>Job"<br />
</pre><br />
<br />
* These last two commands can instead be executed simultaneously by running:<br />
<pre><br />
./status.sh<br />
</pre><br />
<br />
=== Handy mysql Instructions ===<br />
<br />
* Handy mysql instructions:<br />
<pre><br />
mysql -hhallddb -ufarmer farming # Enter the "farming" mysql database on "hallddb" as user "farmer"<br />
quit; # Exit mysql<br />
show tables; # Show a list of the tables in the current database<br />
show columns from <project_name>; # show all of the columns for the given table<br />
select * from <project_name>; # show the contents of all rows from the given table<br />
</pre><br />
<br />
=== Backing Up Offline Monitoring Tables ===<br />
Tables created for offline monitoring can be backed up using the script backup_tables.sh which<br />
can be checked out with the other files from https://halldsvn.jlab.org/repos/trunk/scripts/monitoring/jproj/projects<br />
<br />
The script uses the command mysqldump to print out a file that can be executed to recreate the tables.<br />
Since executing this output file will drop the table when it exists, caution is advised.<br />
Example usage to backup all three tables created for run period 2014_10 ver 17:<br />
<pre>backup_tables.sh 2014_10 17</pre><br />
<br />
== Running Over Data As It Comes In ==<br />
<br />
A special user gxproj1 will have a cron job set up to run the plugins as new data appears on /mss.<br />
During the week, gxproj1 will submit offline plugin jobs with the same setup as the weekly jobs<br />
run the previous Friday. The procedure for this is shown below.<br />
<br />
<!--<br />
=== Setting up the environment ===<br />
The file<br />
/home/gxproj1/setup_jlab.csh<br />
is sourced through .tcshrc.<br />
This file is the same as what is linked to by<br />
/home/gluex/setup_jlab_commissioning.csh,<br />
except HALLD_HOME, HDDS_HOME, and JANA_CALIB_URL are set separately so that this<br />
user can have a separate build.<br />
<br />
To obtain the builds from the previous Friday's runs,<br />
execute<br />
/home/gxproj1/halld/monitoring/newruns/setup_previous.sh [year] [month] [day]<br />
The build revisions from the previous Friday are archived in files<br />
/work/halld/data_monitoring/run_conditions/soft_comm_[year]_[month]_[day].xml<br />
and the script will build libraries based on those stored revision numbers.<br />
--><br />
<br />
=== Running the cron job ===<br />
<br />
'''IMPORTANT:''' The cron job should not be running while you are manually submitting jobs using the jproj.pl script for the same project, or else you will probably multiply-submit a job. <br />
<br />
* Go to the cron job directory: <br />
<pre><br />
cd /u/home/gxproj1/halld/monitoring/newruns<br />
</pre><br />
<br />
* The cron_plugins file is the cronjob that will be executed. During execution, it runs the exec.sh command in the same folder. This command takes two arguments: the project name, and the maximum file number for each run. These fields should be updated in the cron_plugins file before running. <br />
<br />
* The exec.sh command updates the job management database table with any data that has arrived on tape since it was last updated, ignoring file numbers greater than the maximum file number. It then submits jobs for these files. <br />
<br />
* To start the cron job, run:<br />
<pre><br />
crontab cron_plugins<br />
</pre><br />
<br />
* To check whether the cron job is running, do<br />
<pre><br />
crontab -l<br />
</pre><br />
<br />
* To remove the cron job do<br />
<pre><br />
crontab -r<br />
</pre><br />
<br />
<!--<br />
The cron job will run the script scan_for_jobs.sh,<br />
which runs generatejobs_plugins_rawdata.sh for any<br />
new runs that it had not seen before. All previous<br />
runs are recorded in the file filelists/files_current.txt<br />
so clear this to run over runs, or set the parameters<br />
MINRUN and MAXRUN which will set the range of runs submitted.<br />
--><br />
<br />
==Post-Processing Procedures==<br />
<br />
To visualize the monitoring data, we save images of selected histograms and store time series of selected quantities in a database, which are then displayed on a web page. This section describes how to generate the monitoring images and database information.<br />
<br />
The scripts used to generate this summary data are primarily run from /home/gxprojN/halld/monitoring/process . If you want a new copy of the scripts, e.g., for a new monitoring run, you should check the scripts out from SVN:<br />
<syntaxhighlight><br />
svn co https://halldsvn.jlab.org/repos/trunk/scripts/monitoring/process<br />
</syntaxhighlight><br />
Note that these scripts currently have some parameters which must be periodically set by hand.<br />
<br />
The default python version on most JLab machine does not have the modules to allow these scripts to connect to the MySQL database. To run these scripts, load the environment with the following command<br />
<syntaxhighlight><br />
source /home/gxproj1/halld/monitoring/process/monitoring_env.csh<br />
</syntaxhighlight><br />
<br />
===Online Monitoring===<br />
<br />
There are two primary scripts for running over the monitoring data generated by the online system and offline reconstruction. The online script can be run with either of the following commands:<br />
<syntaxhighlight><br />
/home/gluex/halld/monitoring/process/check_new_runs.py<br />
<br />
OR <br />
<br />
/home/gluex/halld/monitoring/process/check_new_runs.csh<br />
</syntaxhighlight><br />
The shell script sets up the environment properly to run the python script. To connect to the monitoring database on the JLab CUE, modules included in the local installation of python >= 2.7 are needed. The shell script is appropriate to use in a cron job. The cronjob is currently run under the "gluex" account.<br />
<br />
The online monitoring system copies a ROOT file containing the results of the online monitoring, and other configuration files into a directory accessible outside the counting house. This python script automatically checks for new ROOT files, which it will then automatically process. It contains several configuration variables that must be correctly set, which contains the location of input/output directories, etc...<br />
<br />
===Offline Monitoring===<br />
<br />
After the data is run over, the results should be processed, so that summary data is entered into the monitoring database and plots are made for the monitoring webpages. Currently, this processing is controlled by a cronjob that runs the following script:<br />
<syntaxhighlight><br />
/home/gxproj1/halld/monitoring/process/check_monitoring_data.csh <br />
</syntaxhighlight><br />
This script checks for new ROOT files, and only runs over those it hasn't processed yet. Since one monitoring ROOT file is produced for each EVIO file, whenever a new file is produced, the plots for the corresponding run are recreated and all the ROOT files for a run are combined into one file. Information is stored in the database on a per-file basis. <br />
<br />
Plots for the monitoring web page can be made from single histograms or multiple histograms using RootSpy macros. If you want to change the list of plots made, you must modify one of the following files:<br />
* histograms_to_monitor - specify either the name of the histogram or its the full ROOT path <br />
* macros_to_monitor - specify the full path to the RootSpy macro .C file<br />
<br />
When a new monitoring run is started, or the conditions are changed, the following steps should be taken to process the new files:<br />
# Add a new data version, as described below:<br />
# Change the following parameters in check_monitoring_data.csh:<br />
## JOBDATE should correspond to the ouptut date used by the job submission script<br />
## OUTPUTDIR should correspond to the directory corresponding to the run period and revision corresponding to the new version you just submitted. Presumably, this directory will be empty at the beginning.<br />
## Once you create a new data version as defined below, you should pass the needed information as a command line option. Currently this is done by the ARGS variable. For example, the argument "-v RunPeriod-2014-10,8" tells the monitoring scripts to look up the version corresponding to revision 8 of RunPeriod-2014-10 in the monitoring DB and to use to store the results.<br />
<br />
<syntaxhighlight><br />
Example configuration parameters:<br />
set JOBDATE=2015-01-09<br />
set INPUTDIR=/volatile/halld/RunPeriod-2014-10/offline_monitoring<br />
set OUTPUTDIR=/w/halld-scifs1a/data_monitoring/RunPeriod-2014-10/ver08<br />
set ARGS=" -v RunPeriod-2014-10,8 "<br />
</syntaxhighlight><br />
If you want to process the results manually, the data is processed using the following script:<br />
<syntaxhighlight><br />
./process_new_offline_data.py <input directory> <output directory><br />
<br />
EXAMPLE:<br />
<br />
./process_new_offline_data.py 2014-11-14 /volatile/halld/RunPeriod-2014-10/offline_monitoring/ /w/halld-scifs1a/data_monitoring/RunPeriod-2014-10/ver02<br />
</syntaxhighlight><br />
The python script takes several options to enable/disable various steps in the processing. Of interest is the "--force" option, which will run over all monitoring ROOT files, whether or not they've been previously identified.<br />
<br />
Every time a new reconstruction pass is performed, a new version number must be generated. To do this, prepare a version file as described below. Then run the register_new_version.py script to store the information in the database. The script will return a version number, which then should be set by hand in process_new_offline_data.py - future versions of the script will streamline this part of the procedure. An example of how to generate a new version is:<br />
<syntaxhighlight><br />
./register_new_version.py add /home/gxproj1/halld/monitoring/process/versions/vers_RunPeriod-2014-10_pass1.txt<br />
</syntaxhighlight><br />
<br />
<b>If you are running the offline monitoring by checking out the files in trunk/scripts/monitoring/jproj/projects/<br />
of the svn repository</b>, and created a project with<br />
create_project.sh [project name] hd_rawdata<br />
Then go to the directory [project name]/processing/<br />
and execute<br />
./run_processing.sh<br />
which will run register_new_version.py as well as check_monitoring_data.csh for that project.<br />
<br />
===Step-by-Step Instructions For Processing a New Monitoring Run===<br />
<br />
The monitoring runs are current run out of the gxproj1 and gxproj5 accounts. After an offline monitoring run has been successfully started on the batch farm, the following steps should be followed to setup the post-processing for these runs.<br />
<br />
# The post-processing scripts are stored in $HOME/halld/monitoring/process and are automatically run by cron.<br />
# Run "svn update" to bring any changes in. Be sure that the list of histograms and macros to plot are current.<br />
# Edit check_monitoring_data.csh to point to the current revisions/directories<br />
#* VERSION<br />
#* ARGS<br />
#* Note that the environment depends on a standard script - $HOME/setup_jlab.csh<br />
# Update files in the web directory, so that the results are displayed on the web pages: /group/halld/www/halldweb/html/data_monitoring/textdata<br />
# Copy the REST files to more permanent locations:<br />
#* cp -a /volatile/halld/offline_monitoring/RunPeriod-YYYY-MM/verVV/REST /work/halld/data_monitoring/RunPeriod-YYYY-MM/REST/verVV<br />
#* cp -a /volatile/halld/offline_monitoring/RunPeriod-YYYY-MM/verVV/REST /cache/halld/RunPeriod-YYYY-MM/REST/verVV [under testing]<br />
<br />
Check log files in $HOME/halld/monitoring/process/log for more information on how each run went. If there are problems, check log files, and modify check_monitoring_data.csh to vary the verbosity of the output.<br />
<br />
==Data Versions==<br />
<br />
To document the conditions of the monitoring data that is created, for the sake of reproducability and further analysis we save several pieces of information. The format is intended to be comprehensive enough to document not just monitoring data, but versions of raw and reconstructed data, so that this database table can be used for the event database as well.<br />
<br />
We store one record per pass through one run period, with the following structure:<br />
<br />
{| class="wikitable"<br />
! Field !! Description<br />
|-<br />
| data_type || The level of data we are processing. For the purposes of monitoring, "rawdata" is the online monitoring, "recon" is the offline monitoring<br />
|- <br />
| run_period || The run period of the data<br />
|- <br />
| revision || An integer specifying which pass through the run period this data corresponds to<br />
|- <br />
| software_version || The name of the XML file that specifies the different software versions used<br />
|-<br />
| jana_config || The name of the text file that specifies which JANA options were passed to the reconstruction program<br />
|-<br />
| ccdb_context || The value of JANA_CALIB_CONTEXT, which specifies the version of calibration constants that were used<br />
|-<br />
| production_time || The data at which monitoring/reconstruction began<br />
|-<br />
| dataVersionString || A convenient string for identifying this version of the data<br />
|}<br />
<br />
<br />
An example file used as as input to ./register_new_version.py is:<br />
<syntaxhighlight><br />
data_type = recon<br />
run_period = RunPeriod-2014-10<br />
revision = 1<br />
software_version = soft_comm_2014_11_06.xml<br />
jana_config = jana_rawdata_comm_2014_11_06.conf<br />
ccdb_context = calibtime=2014-11-10<br />
production_time = 2014-11-10<br />
dataVersionString = recon_RunPeriod-2014-10_20141110_ver01<br />
</syntaxhighlight></div>Kmoriyahttps://halldweb.jlab.org/wiki/index.php?title=Data_Monitoring_Procedures&diff=70733Data Monitoring Procedures2015-10-14T20:22:06Z<p>Kmoriya: /* Starting the Launch and Submitting Jobs */</p>
<hr />
<div>__TOC__<br />
<br />
== Master List of File / Database / Webpage Locations ==<br />
=== Run Conditions ===<br />
* Online Run-by-run condition files (B-field, current, etc.): /work/halld/online_monitoring/conditions/<br />
* Offline monitoring run conditions (software versions, jana config): /group/halld/data_monitoring/run_conditions/<br />
*[http://www.jlab.org/Hall-D/test/RunInfo/ Run Info vers. 1]<br />
*[https://halldweb.jlab.org/cgi-bin/data_monitoring/run_conditions.pl Run Info vers. 2]<br />
<br />
=== Monitoring Output Files ===<br />
* Run Periods 201Y-MM is for example 2015-03, launch ver verVV is for example ver15<br />
* Online monitoring histograms: /work/halld/online_monitoring/root/<br />
* Offline monitoring histogram ROOT files: /work/halld/data_monitoring/RunPeriod-201Y-MM/verVV/rootfiles<br />
* REST files (most recent launch only): /work/halld/data_monitoring/RunPeriod-201Y-MM/REST/verVV<br />
* individual files for each job (ROOT, log, etc.): /volatile/halld/offline_monitoring/RunPeriod-201Y-MM/verVV/<br />
<br />
=== Monitoring Database ===<br />
* Accessing monitoring database (on ifarm): mysql -u datmon -h hallddb.jlab.org data_monitoring<br />
<br />
=== Monitoring Webpages ===<br />
*[https://halldweb.jlab.org/cgi-bin/data_monitoring/monitoring/plotBrowser.py Plot Browser]<br />
*[https://halldweb.jlab.org/cgi-bin/data_monitoring/monitoring/runBrowser.py Run Browser]<br />
*[https://halldweb.jlab.org/cgi-bin/data_monitoring/monitoring/timeSeries.py Time Series]<br />
*[https://halldweb.jlab.org/data_monitoring/launch_analysis/ Launch Analysis]<br />
<br />
== Job Monitoring Links ==<br />
* [http://scicomp.jlab.org/farm2/job.html Completed Job History]<br />
* [http://scicomp.jlab.org/farm2/project.html Job Stats By Project]<br />
* [http://scicomp.jlab.org/farm2/trackOrg.html Job Stats By Track]<br />
* [http://scicomp.jlab.org/farm2/report.html Cluster Report]<br />
* [http://scicomp.jlab.org/farm2/walltime.html Walltime Distribution]<br />
<br />
== Saving Online Monitoring Data ==<br />
<br />
The procedure for writing the data out is given in, e.g.,<br />
[https://halldweb.jlab.org/wiki/index.php/Raid-to-Silo_Transfer_Strategy Raid-to-Silo Transfer Strategy].<br />
<br />
Once the DAQ writes out the data to the raid disk, cron jobs will copy the file to tape,<br />
and within ~20 min., we will have access to the file on tape at<br />
/mss/halld/$RUN_PERIOD/rawdata/RunXXXXXX.<br />
<br />
All online monitoring plugins will be run as data is taken.<br />
They will be accessible within the counting house via RootSpy, and<br />
for each run and file, a ROOT file containing the histograms will be saved<br />
within a subdirectory for each run.<br />
<br />
For immediate access to these files, the raid disk files may be accessed directly<br />
from the counting house, or the tape files will be available within ~20 min. of the<br />
file being written out.<br />
<br />
== Offline Monitoring: Running Over Archived Data ==<br />
<br />
Once files are written to tape we run the online plugins on these files to confirm what we were seeing in the online monitoring, and also to update the results from the latest calibration and software. <br />
Manual scripts and cron jobs are set up to look for new data and run the plugins over a sample of files.<br />
<br />
Every other Friday (usually the Friday before the offline meetings) jobs will be started to run the newest software on all previous runs,and allows everybody to see improvements in each detector.<br />
For each launch, independent builds of hdds, sim-recon, the monitoring plugins, and an sqlite file will be generated.<br />
<br />
Below the procedures are described for<br />
# Preparing the software for the launch<br />
# Starting the launch (using hdswif)<br />
# Post-analysis of statistics of the launch<br />
<br />
Processing the results and making them available to the collaboration<br />
is handled in the section [[#Post-Processing_Procedures | Post-Processing_Procedures]] below.<br />
<br />
<!--<br />
==== Generating an offline plugin job ====<br />
<br />
The user gxproj1 should be used for official offline monitoring jobs.<br />
Within /home/gxproj1/halld/monitoring/batch/ there will be scripts to run the online monitoring plugins over tape files.<br />
The main script is generatejobs_plugins_rawdata.sh, which can be used as<br />
generatejobs_plugins_rawdata.sh [minrun] [maxrun] (minfile) (maxfile)<br />
where minrun, maxrun specify the range of the run #, and minfile, maxfile (optional) specify<br />
the file range for that run.<br />
<br />
This will generate a script run_rawdata_XXXXXX.sh, where the run # has now been formatted to be 6 digits.<br />
Executing this script will send the monitoring plugins job to the Auger batch system.<br />
<br />
There is also a script clean.sh which can be used as<br />
clean.sh XXX<br />
This will clean up all associated files created in association with run XXX.<br />
<br />
Internally, the xml file used to submit the job will be created, and the job to run<br />
will be given within script.sh. All run parameters should be specified in at the beginning<br />
of generatejobs_plugins_rawdata.sh<br />
Since we are running on tape, the tape file will first be copied over to the cache disk, and the job will run<br />
over this cached file.<br />
<br />
==== Using cron to run automatically ====<br />
Within /home/gluex/halld/monitoring/cron/ there is a file cron_plugins<br />
that can be executed via<br />
crontab cron_plugins<br />
This will set up a cron job to call the script scan_for_jobs.sh, which will<br />
check in the rawdata directory and call generatejobs_plugins_rawdata.sh for<br />
any run that is more than 5 min old. The cron job is set up to run every 10 min.<br />
<br />
==== Magnetic Field Settings ====<br />
For tracking, it is necessary to set the correct magnetic field settings.<br />
The below table is now obsolete. Please refer to the [https://halldweb.jlab.org/cgi-bin/data_monitoring/run_conditions.pl GlueX Run Conditions] webpage.<br />
<br />
The actual field settings for each run have not been documented well.<br />
Below are values based on going through entries in the [https://logbooks.jlab.org/book/halld halld log].<br />
<br />
{| border="1" cellpadding="1" style="text-align: center;"<br />
!width="100"| Run #s<br />
!width="150"| Solenoid Current (A)<br />
!width="300"| JANA option<br />
!width="300"| notes<br />
|-<br />
| 940 - 996 || 1000 || -PBFIELD_MAP=Magnets/Solenoid/solenoid_1000A_poisson_20141104 || At run 997, the solenoid started ramping down<br />
|-<br />
| 998 - 1448 || 0 || -PBFIELD_TYPE=NoField -PDEFTAG:DTrackCandidate=StraightLine || Solenoid started ramping up around run 1431. Runs 1432 - 1448 should not be used for tracking. See [https://logbooks.jlab.org/entry/3309039 this] and [https://logbooks.jlab.org/entry/3309092 this] entry.<br />
|-<br />
| 1449 - 1620 || 1200 || -PBFIELD_MAP=Magnets/Solenoid/solenoid_1200A_poisson_20140520 || <br />
|-<br />
|}<br />
<br />
'''Note:''' The following run ranges were taken with a 300A field, and could be analyzed with the magnetic field map Magnets/Solenoid/solenoid_300A_poisson_20140819 , more study is needed to see if the straight line track fitter is needed or not: 1036 - 1053, 1065 - 1121, 1212? - 1254, 1309 - 1318 [1307,1308,1319-1329 were taken while the magnet was ramping].<br />
--><br />
<br />
=== General Information on Procedures ===<br />
This section explains how the offline monitoring should be run.<br />
Since we may want to simultaneously run offline monitoring for different run periods that require<br />
different environment variables, the scripts are set up so that a generic user can download the<br />
scripts and run them from anywhere. Most output directories for offline monitoring are created<br />
with group read/write permissions so that any Hall D group user has access to the contents,<br />
but there are some cases where use of the account that created the launch is necessary.<br />
<br />
The accounts used for offline monitoring are the gxprojN accounts created and maintained by<br />
Mark Ito (see [https://halldweb.jlab.org/wiki/index.php/GlueX-related_shared_accounts_on_the_JLab_CUE here]).<br />
As of October 2015, we have been using<br />
* gxproj1 for running over Fall 2014 data (deprecated since June 2015)<br />
* gxproj5 for running over Spring 2015 data<br />
<br />
Since the summer of 2015 we have transitioned from a system using Mark Ito's jproj scripts<br />
to integrating the swif system that Chris Larrieu (SciComp) has been developing. For offline monitoring,<br />
the hdswif system that Kei developed is used for launching the jobs, and the jproj system is used<br />
for meta-analysis of launch statistics.<br />
<br />
Both hdswif and jproj are maintained in svn:<br />
* hdswif: https://halldsvn.jlab.org/repos/trunk/scripts/monitoring/hdswif<br />
* jproj : https://halldsvn.jlab.org/repos/trunk/scripts/monitoring/jproj<br />
<br />
To run each package the directories will need to be checked out and the necessary<br />
scripts will need to be run.<br />
<br />
=== Preparing the software for the launch ===<br />
<br />
To begin a new launch the software must be built to the latest versions.<br />
For the gxprojN user accounts used, all software builds are contained in the directory ~/builds<br />
(which are soft links to /work/halld/home/gxproj5/builds). When logging into these accounts<br />
the setup files ~/setup_jlab-2015-03.csh or similar files should be sourced.<br />
<br />
<b>Note that Mark Ito does <u>not</u> want you to change the contents of each .cshrc file.</b><br />
You should consult him if you feel the need.<br />
<br />
<span style="color:red">NOTE: IT IS A GOOD IDEA TO DO A COMPLETE WIPEOUT/CHECKOUT OF A NEW VERSION OF EACH SOFTWARE TO AVOID STALE HEADER FILES.</span><br />
<br />
# Building hdds: Go to ~/builds/hdds. The directory hdds is the one from git. Delete the contents, then download the newest version from git and build: <pre>cd ~/builds/hdds/</pre> <pre>rm -frv hdds</pre> <pre>git clone https://github.com/JeffersonLab/hdds</pre> <pre>scons install</pre><br />
# Building sim-recon: Go to ~/builds/sim-recon. The directory sim-recon is the one from git. Delete the contents, then download the newest version from git and build: <pre>cd ~/builds/sim-recon</pre> <pre>rm -frv sim-recon</pre> <pre>git clone https://github.com/JeffersonLab/sim-recon</pre> <pre>scons install -j8</pre><br />
# Prepare the latest sqlite file: The sqlite is set in the ~/setup_jlab-2015-03.csh script as sqlite:////home/gxproj5/builds/ccdb.sqlite through the environment variables <b>JANA_CALIB_URL</b> and <b>CCDB_CONNECTION</b>. Therefore, go to ~/builds,and create a new sqlite file. We create the sqlite file in a temporary directory since creating the sqlite file in a directory where the output file exists causes errors. Original documentation on creating sqlite files are [https://halldweb.jlab.org/wiki/index.php/SQLite-form_of_the_CCDB_database here]. <pre>cd ~/builds/tmp</pre> <pre>$CCDB_HOME/scripts/mysql2sqlite/mysql2sqlite.sh -hhallddb.jlab.org -uccdb_user ccdb | sqlite3 ccdb.sqlite</pre><br />
# Note that the above steps must be done <b>BEFORE</b> launch project creation. This is because we will track the revisions of the libraries used, and this is done by extracting the svn information in each directory. Also, note that the system assumes that we have the topmost build directory (usually called GLUEX_TOP) to be $HOME/builds . Such an assumption is necessary to be able to extract information about the library locations automatically.<br />
<br />
Create the appropriate project(s) and submit the jobs using hdswif, as detailed in the section below.<br />
<br />
=== Starting the Launch and Submitting Jobs ===<br />
<br />
Until the summer of 2015 we relied solely on Mark Ito's jproj system for submitting and keeping track of jobs. We have since moved to the swif system and use the hdswif wrapper for this. Below are instructions for how to use these.<br />
<br />
# Downloading hdswif: Download the hdswif directory from svn. For the gxprojN accounts, use the directory ~/halld/hdswif. <pre>cd ~/halld </pre> <pre> svn co https://halldsvn.jlab.org/repos/trunk/scripts/monitoring/hdswif</pre> <pre>cd hdswif</pre><br />
# Creating the workflow: Within SWIF jobs are registered into workflows. First create the workflow. For offline monitoring, the workflow names are of the form <b>offline_monitoring_RunPeriod201Y_MM_verVV_hd_rawdata</b> with suitable replacements for the run period and version number. The command "swif list" will list all existing workflows. Also, or most simple SWIF commands hdswif also provides a wrapper. <pre>swif list</pre> <pre>swif create offline_monitoring_RunPeriod2015_03_ver99_hd_rawdata</pre> Or equivalently, <pre>hdswif.py create offline_monitoring_RunPeriod2015_03_ver99_hd_rawdata </pre><br />
# Registering jobs in the workflow: To register jobs within the workflow, hdswif provides the use of config files. Jobs can be registered by specifying the workflow, config file (-c), run (-r) and file (-f) numbers if necessary. A typical config file will look this:<br />
<pre><br />
PROJECT gluex<br />
TRACK reconstruction<br />
OS centos65<br />
NCORES 4<br />
DISK 40<br />
RAM 8<br />
TIMELIMIT 16<br />
JOBNAMEBASE offmon_<br />
RUNPERIOD 2015-03<br />
VERSION 92<br />
OUTPUT_TOPDIR /volatile/halld/offline_monitoring/RunPeriod-[RUNPERIOD]/ver[VERSION] # Example of other variables included in variable<br />
SCRIPTFILE /home/gxproj5/halld/hdswif/script.sh # Must specify full path<br />
ENVFILE /home/gxproj5/halld/hdswif/setup_jlab-2015-03.csh # Must specify full path<br />
</pre><br />
The config file contains configuration parameters for each of the jobs.<br />
<span style="color:red">Note: Job configuration parameters can be set differently for jobs within the same workflow if necessary.</span><br />
Edit the config file and save as a new file if necessary. Once the configuration is set, jobs can be added via <pre>hdswif.py add offline_monitoring_RunPeriod2015_03_ver99_hd_rawdata -c input.config</pre><br />
By default, hdswif will add all files found within the directory /mss/halld/RunPeriod-201Y-MM/rawdata/ where 201Y-MM is specified by the RUNPERIOD parameter in the config file. If only some of the runs or files are needed, these can be specified for example with<br />
<pre>hdswif.py add offline_monitoring_RunPeriod2015_03_ver99_hd_rawdata -c input.config -r 3180 -f '00[0-4]'</pre><br />
to specify to register running only over run 3180 files 000 - 004 (Unix-style brackets and wildcards can be used).<br />
# Running the workflow: To run the workflow, simply use swif run: <pre>swif run offline_monitoring_RunPeriod2015_03_ver99_hd_rawdata</pre> or equivalently, using the hdswif wrapper, <pre>hdsswif.py run offline_monitoring_RunPeriod2015_03_ver99_hd_rawdata</pre><br />
<b>It is recommended that some jobs be tested over to make sure that everything is working rather than fail thousands of jobs.</b> For this purpose, hdswif will take an additional parameter to run which limits the number of jobs to submit: <pre>hdswif.py run offline_monitoring_RunPeriod2015_03_ver99_hd_rawdata 10</pre><br />
in which case only 10 jobs will be submitted. To submit all jobs after checking the results, do <pre>hdswif.py run offline_monitoring_RunPeriod2015_03_ver99_hd_rawdata</pre><br />
<br />
== Hall D Job Management System ==<br />
<br />
This section details instructions on how to create and launch a set of jobs using the Hall-D Job Management System developed by Mark Ito. These instructions are generic: this system can be used for the weekly monitoring jobs, but can also be used for other sets of job launches as well. <br />
<br />
=== Database Table Overview ===<br />
<br />
* Job management database table (<project_name>): For each input file, keeps track of whether or not a job for it has been submitted, along with other optional fields. <br />
<br />
* Job status database table (<project_name>Job (no space)): For each job, keeps track of the job-id, the job status, memory used, cpu & wall time, time taken to complete various stages (e.g. pending, dependency, active), and others. <br />
<br />
* Job metrics database table (<project_name>_aux (no space)): For each job, keeps track of the job-id, how many events were processed, the time it took to copy the cache file, and the time it took to run the plugin. This information is culled from the log files of each job, and is done within the analysis directory of each launch.<br />
<br />
=== Initialize Project Management ===<br />
<br />
* Log into the ifarm machine with one of the gxproj accounts. For this example we will use gxproj1.<br />
<pre><br />
ssh gxproj1@ifarm -Y<br />
</pre><br />
<br />
* Go to a directory to do the launch. In principle, any directory will work, but for gxproj1 this is usually done in /home/gxproj1/halld/<br />
<br />
* Check out the necessary scripts <pre>svn co https://halldsvn.jlab.org/repos/trunk/scripts/monitoring/jproj</pre> This will get all necessary scripts for launching. Once checked out, <pre>cd projects</pre><br />
<br />
* The script create_project.sh can be used to create a new project. It will take a single argument, the project name. It is assumed that the project name is of the form offline_monitoring_RunPeriod20YY_MM_verVV_hd_rawdata where this string will be parsed to give the run period 20YY_MM and version number VV. One thing to do BEFORE creation of a new project is to editthe conditions of the launch (plugins to run over, memory requested, disk space requested) within templates/template.jsub . This<br />
information is saved automatically at project creation time into files in /group/halld/data_monitoring/run_conditions .<br />
To create a project do <pre>./create_project.sh offline_monitoring_RunPeriod20YY_MM_verVV_hd_rawdata</pre><br />
The name has been chosen to be as consistent as possible with other directory structures.<br />
However, mysql requires that "-" be escaped in table names, so unfortunately run periods will be<br />
given as 2YYY_MM instead of 2YYY-MM.<br />
<br />
* A new directory with the project name (offline_monitoring_RunPeriod20YY_MM_verVV_hd_rawdata) and files will be copied and modified from the template directory to reflect the run period, the user, the directory that it was created in, project name, etc.<br />
<br />
* For each project, cd into the new directory<br />
* The script clear.sh will remove any existing tables for the current project name, then recreate it. Do<br />
<pre>./clear.sh</pre><br />
* To use the jproj.pl script that was checked in, add the directory to your path with<br />
<pre>source ../../scripts/setup.csh</pre><br />
or always specify the full path<br />
<pre>../../scripts/jproj.pl</pre><br />
* Now update the table of runs with<br />
<pre>jproj.pl <project name> update</pre><br />
This will fill the table with all files within /mss that are of the same form as what is in <project name>.jproj . If you want to register only on a subset of all such files, you can edit this file directly.<br />
* Once you have registered all of the files you would like to run over, do<br />
<pre>jproj.pl <project name> submit [max # of jobs] [run number]</pre><br />
where the additional options specify how many jobs to submit and which run number to run on. Without these options all files that are registered and have not been submitted yet will be submitted.<br />
<br />
At this stage you are ready to submit all files. It is a good idea to submit a few test jobs<br />
at first to check that all scripts are working and that the plugins do not crash. Once you are<br />
sure that this does not happen, you can send all jobs in. The remaining jobs are then the monitoring<br />
which will (among other things) put the results on the online webpage for the collaboration to view, and<br />
the analysis of the launch.<br />
<br />
=== Project File Overview ===<br />
<br />
An overview of each project file:<br />
* '''clear.sh:''' For the current project, deletes the job status and management database tables (if any), and creates new, empty ones. <br />
* '''<project_name>.jproj:''' Contains the path and file name format for the input files for the jobs. <br />
* '''<project_name>.jsub:''' The xml job submission script. The run number and file number variables are set during job submission for each input file. <br />
* '''script.sh:''' The script that is executed during the job. If output job directories are not pre-created manually, they should be created in this script with the proper permissions:<br />
<pre><br />
mkdir -p -m 775 my_directory<br />
</pre><br />
* '''setup_jlab-[run period].csh:''' The environment that is sourced at the beginning of the job execution. <br />
* '''status.sh:''' Updates the job status database table, and prints some of its columns to screen.<br />
<br />
=== Project Management ===<br />
<br />
* Delete (if any) and create the database table(s) for the current set of job submissions:<br />
<pre><br />
./clear.sh<br />
</pre><br />
Also, if testing was done with jobs, it is best to delete the output directory and the configuration files:<br />
<pre><br />
rm -frv /volatile/halld/offline_monitoring/RunPeriod-20YY-MM/verVV /group/halld/data_monitoring/run_conditions/soft_comm_20YY_MM_verVV.xml /group/halld/data_monitoring/run_conditions/jana_rawdata_comm_20YY_MM_verVV.conf<br />
</pre><br />
<br />
* Search for input files matching the string in the .jproj file, and create a row for each in the job management database table (called <project_name>). You can test by adding an optional argument at the end, which only selects files with a specific file number:<br />
<pre><br />
jproj.pl <project_name> update <optional_file_number><br />
</pre><br />
<br />
* Confirm that the job management database is accurate by printing it's contents to screen:<br />
<pre><br />
mysql -hhallddb -ufarmer farming -e "select * from <project_name>"<br />
</pre><br />
<br />
* ONLY if a mistake was made, to delete the tables from the database and recreate new, empty ones, run: <br />
<pre><br />
./clear.sh<br />
</pre><br />
<br />
* To look at the status of the submitted jobs, first query auger and update the job status database:<br />
<pre><br />
fill_in_job_details.pl <project_name><br />
</pre><br />
<br />
* The job status can then be viewed by submitting a query to the job status database (called <project_name>Job (no space in between)):<br />
<pre><br />
mysql -hhallddb -ufarmer farming -e "select id,run,file,jobId,hostname,status,timeSubmitted,timeActive,walltime,cput,timeComplete,result,error from <project_name>Job"<br />
</pre><br />
<br />
* These last two commands can instead be executed simultaneously by running:<br />
<pre><br />
./status.sh<br />
</pre><br />
<br />
=== Handy mysql Instructions ===<br />
<br />
* Handy mysql instructions:<br />
<pre><br />
mysql -hhallddb -ufarmer farming # Enter the "farming" mysql database on "hallddb" as user "farmer"<br />
quit; # Exit mysql<br />
show tables; # Show a list of the tables in the current database<br />
show columns from <project_name>; # show all of the columns for the given table<br />
select * from <project_name>; # show the contents of all rows from the given table<br />
</pre><br />
<br />
=== Backing Up Offline Monitoring Tables ===<br />
Tables created for offline monitoring can be backed up using the script backup_tables.sh which<br />
can be checked out with the other files from https://halldsvn.jlab.org/repos/trunk/scripts/monitoring/jproj/projects<br />
<br />
The script uses the command mysqldump to print out a file that can be executed to recreate the tables.<br />
Since executing this output file will drop the table when it exists, caution is advised.<br />
Example usage to backup all three tables created for run period 2014_10 ver 17:<br />
<pre>backup_tables.sh 2014_10 17</pre><br />
<br />
== Running Over Data As It Comes In ==<br />
<br />
A special user gxproj1 will have a cron job set up to run the plugins as new data appears on /mss.<br />
During the week, gxproj1 will submit offline plugin jobs with the same setup as the weekly jobs<br />
run the previous Friday. The procedure for this is shown below.<br />
<br />
<!--<br />
=== Setting up the environment ===<br />
The file<br />
/home/gxproj1/setup_jlab.csh<br />
is sourced through .tcshrc.<br />
This file is the same as what is linked to by<br />
/home/gluex/setup_jlab_commissioning.csh,<br />
except HALLD_HOME, HDDS_HOME, and JANA_CALIB_URL are set separately so that this<br />
user can have a separate build.<br />
<br />
To obtain the builds from the previous Friday's runs,<br />
execute<br />
/home/gxproj1/halld/monitoring/newruns/setup_previous.sh [year] [month] [day]<br />
The build revisions from the previous Friday are archived in files<br />
/work/halld/data_monitoring/run_conditions/soft_comm_[year]_[month]_[day].xml<br />
and the script will build libraries based on those stored revision numbers.<br />
--><br />
<br />
=== Running the cron job ===<br />
<br />
'''IMPORTANT:''' The cron job should not be running while you are manually submitting jobs using the jproj.pl script for the same project, or else you will probably multiply-submit a job. <br />
<br />
* Go to the cron job directory: <br />
<pre><br />
cd /u/home/gxproj1/halld/monitoring/newruns<br />
</pre><br />
<br />
* The cron_plugins file is the cronjob that will be executed. During execution, it runs the exec.sh command in the same folder. This command takes two arguments: the project name, and the maximum file number for each run. These fields should be updated in the cron_plugins file before running. <br />
<br />
* The exec.sh command updates the job management database table with any data that has arrived on tape since it was last updated, ignoring file numbers greater than the maximum file number. It then submits jobs for these files. <br />
<br />
* To start the cron job, run:<br />
<pre><br />
crontab cron_plugins<br />
</pre><br />
<br />
* To check whether the cron job is running, do<br />
<pre><br />
crontab -l<br />
</pre><br />
<br />
* To remove the cron job do<br />
<pre><br />
crontab -r<br />
</pre><br />
<br />
<!--<br />
The cron job will run the script scan_for_jobs.sh,<br />
which runs generatejobs_plugins_rawdata.sh for any<br />
new runs that it had not seen before. All previous<br />
runs are recorded in the file filelists/files_current.txt<br />
so clear this to run over runs, or set the parameters<br />
MINRUN and MAXRUN which will set the range of runs submitted.<br />
--><br />
<br />
==Post-Processing Procedures==<br />
<br />
To visualize the monitoring data, we save images of selected histograms and store time series of selected quantities in a database, which are then displayed on a web page. This section describes how to generate the monitoring images and database information.<br />
<br />
The scripts used to generate this summary data are primarily run from /home/gxprojN/halld/monitoring/process . If you want a new copy of the scripts, e.g., for a new monitoring run, you should check the scripts out from SVN:<br />
<syntaxhighlight><br />
svn co https://halldsvn.jlab.org/repos/trunk/scripts/monitoring/process<br />
</syntaxhighlight><br />
Note that these scripts currently have some parameters which must be periodically set by hand.<br />
<br />
The default python version on most JLab machine does not have the modules to allow these scripts to connect to the MySQL database. To run these scripts, load the environment with the following command<br />
<syntaxhighlight><br />
source /home/gxproj1/halld/monitoring/process/monitoring_env.csh<br />
</syntaxhighlight><br />
<br />
===Online Monitoring===<br />
<br />
There are two primary scripts for running over the monitoring data generated by the online system and offline reconstruction. The online script can be run with either of the following commands:<br />
<syntaxhighlight><br />
/home/gluex/halld/monitoring/process/check_new_runs.py<br />
<br />
OR <br />
<br />
/home/gluex/halld/monitoring/process/check_new_runs.csh<br />
</syntaxhighlight><br />
The shell script sets up the environment properly to run the python script. To connect to the monitoring database on the JLab CUE, modules included in the local installation of python >= 2.7 are needed. The shell script is appropriate to use in a cron job. The cronjob is currently run under the "gluex" account.<br />
<br />
The online monitoring system copies a ROOT file containing the results of the online monitoring, and other configuration files into a directory accessible outside the counting house. This python script automatically checks for new ROOT files, which it will then automatically process. It contains several configuration variables that must be correctly set, which contains the location of input/output directories, etc...<br />
<br />
===Offline Monitoring===<br />
<br />
After the data is run over, the results should be processed, so that summary data is entered into the monitoring database and plots are made for the monitoring webpages. Currently, this processing is controlled by a cronjob that runs the following script:<br />
<syntaxhighlight><br />
/home/gxproj1/halld/monitoring/process/check_monitoring_data.csh <br />
</syntaxhighlight><br />
This script checks for new ROOT files, and only runs over those it hasn't processed yet. Since one monitoring ROOT file is produced for each EVIO file, whenever a new file is produced, the plots for the corresponding run are recreated and all the ROOT files for a run are combined into one file. Information is stored in the database on a per-file basis. <br />
<br />
Plots for the monitoring web page can be made from single histograms or multiple histograms using RootSpy macros. If you want to change the list of plots made, you must modify one of the following files:<br />
* histograms_to_monitor - specify either the name of the histogram or its the full ROOT path <br />
* macros_to_monitor - specify the full path to the RootSpy macro .C file<br />
<br />
When a new monitoring run is started, or the conditions are changed, the following steps should be taken to process the new files:<br />
# Add a new data version, as described below:<br />
# Change the following parameters in check_monitoring_data.csh:<br />
## JOBDATE should correspond to the ouptut date used by the job submission script<br />
## OUTPUTDIR should correspond to the directory corresponding to the run period and revision corresponding to the new version you just submitted. Presumably, this directory will be empty at the beginning.<br />
## Once you create a new data version as defined below, you should pass the needed information as a command line option. Currently this is done by the ARGS variable. For example, the argument "-v RunPeriod-2014-10,8" tells the monitoring scripts to look up the version corresponding to revision 8 of RunPeriod-2014-10 in the monitoring DB and to use to store the results.<br />
<br />
<syntaxhighlight><br />
Example configuration parameters:<br />
set JOBDATE=2015-01-09<br />
set INPUTDIR=/volatile/halld/RunPeriod-2014-10/offline_monitoring<br />
set OUTPUTDIR=/w/halld-scifs1a/data_monitoring/RunPeriod-2014-10/ver08<br />
set ARGS=" -v RunPeriod-2014-10,8 "<br />
</syntaxhighlight><br />
If you want to process the results manually, the data is processed using the following script:<br />
<syntaxhighlight><br />
./process_new_offline_data.py <input directory> <output directory><br />
<br />
EXAMPLE:<br />
<br />
./process_new_offline_data.py 2014-11-14 /volatile/halld/RunPeriod-2014-10/offline_monitoring/ /w/halld-scifs1a/data_monitoring/RunPeriod-2014-10/ver02<br />
</syntaxhighlight><br />
The python script takes several options to enable/disable various steps in the processing. Of interest is the "--force" option, which will run over all monitoring ROOT files, whether or not they've been previously identified.<br />
<br />
Every time a new reconstruction pass is performed, a new version number must be generated. To do this, prepare a version file as described below. Then run the register_new_version.py script to store the information in the database. The script will return a version number, which then should be set by hand in process_new_offline_data.py - future versions of the script will streamline this part of the procedure. An example of how to generate a new version is:<br />
<syntaxhighlight><br />
./register_new_version.py add /home/gxproj1/halld/monitoring/process/versions/vers_RunPeriod-2014-10_pass1.txt<br />
</syntaxhighlight><br />
<br />
<b>If you are running the offline monitoring by checking out the files in trunk/scripts/monitoring/jproj/projects/<br />
of the svn repository</b>, and created a project with<br />
create_project.sh [project name] hd_rawdata<br />
Then go to the directory [project name]/processing/<br />
and execute<br />
./run_processing.sh<br />
which will run register_new_version.py as well as check_monitoring_data.csh for that project.<br />
<br />
===Step-by-Step Instructions For Processing a New Monitoring Run===<br />
<br />
The monitoring runs are current run out of the gxproj1 and gxproj5 accounts. After an offline monitoring run has been successfully started on the batch farm, the following steps should be followed to setup the post-processing for these runs.<br />
<br />
# The post-processing scripts are stored in $HOME/halld/monitoring/process and are automatically run by cron.<br />
# Run "svn update" to bring any changes in. Be sure that the list of histograms and macros to plot are current.<br />
# Edit check_monitoring_data.csh to point to the current revisions/directories<br />
#* VERSION<br />
#* ARGS<br />
#* Note that the environment depends on a standard script - $HOME/setup_jlab.csh<br />
# Update files in the web directory, so that the results are displayed on the web pages: /group/halld/www/halldweb/html/data_monitoring/textdata<br />
# Copy the REST files to more permanent locations:<br />
#* cp -a /volatile/halld/offline_monitoring/RunPeriod-YYYY-MM/verVV/REST /work/halld/data_monitoring/RunPeriod-YYYY-MM/REST/verVV<br />
#* cp -a /volatile/halld/offline_monitoring/RunPeriod-YYYY-MM/verVV/REST /cache/halld/RunPeriod-YYYY-MM/REST/verVV [under testing]<br />
<br />
Check log files in $HOME/halld/monitoring/process/log for more information on how each run went. If there are problems, check log files, and modify check_monitoring_data.csh to vary the verbosity of the output.<br />
<br />
==Data Versions==<br />
<br />
To document the conditions of the monitoring data that is created, for the sake of reproducability and further analysis we save several pieces of information. The format is intended to be comprehensive enough to document not just monitoring data, but versions of raw and reconstructed data, so that this database table can be used for the event database as well.<br />
<br />
We store one record per pass through one run period, with the following structure:<br />
<br />
{| class="wikitable"<br />
! Field !! Description<br />
|-<br />
| data_type || The level of data we are processing. For the purposes of monitoring, "rawdata" is the online monitoring, "recon" is the offline monitoring<br />
|- <br />
| run_period || The run period of the data<br />
|- <br />
| revision || An integer specifying which pass through the run period this data corresponds to<br />
|- <br />
| software_version || The name of the XML file that specifies the different software versions used<br />
|-<br />
| jana_config || The name of the text file that specifies which JANA options were passed to the reconstruction program<br />
|-<br />
| ccdb_context || The value of JANA_CALIB_CONTEXT, which specifies the version of calibration constants that were used<br />
|-<br />
| production_time || The data at which monitoring/reconstruction began<br />
|-<br />
| dataVersionString || A convenient string for identifying this version of the data<br />
|}<br />
<br />
<br />
An example file used as as input to ./register_new_version.py is:<br />
<syntaxhighlight><br />
data_type = recon<br />
run_period = RunPeriod-2014-10<br />
revision = 1<br />
software_version = soft_comm_2014_11_06.xml<br />
jana_config = jana_rawdata_comm_2014_11_06.conf<br />
ccdb_context = calibtime=2014-11-10<br />
production_time = 2014-11-10<br />
dataVersionString = recon_RunPeriod-2014-10_20141110_ver01<br />
</syntaxhighlight></div>Kmoriyahttps://halldweb.jlab.org/wiki/index.php?title=Data_Monitoring_Procedures&diff=70728Data Monitoring Procedures2015-10-14T18:45:27Z<p>Kmoriya: /* Offline Monitoring: Running Over Archived Data */</p>
<hr />
<div>__TOC__<br />
<br />
== Master List of File / Database / Webpage Locations ==<br />
=== Run Conditions ===<br />
* Online Run-by-run condition files (B-field, current, etc.): /work/halld/online_monitoring/conditions/<br />
* Offline monitoring run conditions (software versions, jana config): /group/halld/data_monitoring/run_conditions/<br />
*[http://www.jlab.org/Hall-D/test/RunInfo/ Run Info vers. 1]<br />
*[https://halldweb.jlab.org/cgi-bin/data_monitoring/run_conditions.pl Run Info vers. 2]<br />
<br />
=== Monitoring Output Files ===<br />
* Run Periods 201Y-MM is for example 2015-03, launch ver verVV is for example ver15<br />
* Online monitoring histograms: /work/halld/online_monitoring/root/<br />
* Offline monitoring histogram ROOT files: /work/halld/data_monitoring/RunPeriod-201Y-MM/verVV/rootfiles<br />
* REST files (most recent launch only): /work/halld/data_monitoring/RunPeriod-201Y-MM/REST/verVV<br />
* individual files for each job (ROOT, log, etc.): /volatile/halld/offline_monitoring/RunPeriod-201Y-MM/verVV/<br />
<br />
=== Monitoring Database ===<br />
* Accessing monitoring database (on ifarm): mysql -u datmon -h hallddb.jlab.org data_monitoring<br />
<br />
=== Monitoring Webpages ===<br />
*[https://halldweb.jlab.org/cgi-bin/data_monitoring/monitoring/plotBrowser.py Plot Browser]<br />
*[https://halldweb.jlab.org/cgi-bin/data_monitoring/monitoring/runBrowser.py Run Browser]<br />
*[https://halldweb.jlab.org/cgi-bin/data_monitoring/monitoring/timeSeries.py Time Series]<br />
*[https://halldweb.jlab.org/data_monitoring/launch_analysis/ Launch Analysis]<br />
<br />
== Job Monitoring Links ==<br />
* [http://scicomp.jlab.org/farm2/job.html Completed Job History]<br />
* [http://scicomp.jlab.org/farm2/project.html Job Stats By Project]<br />
* [http://scicomp.jlab.org/farm2/trackOrg.html Job Stats By Track]<br />
* [http://scicomp.jlab.org/farm2/report.html Cluster Report]<br />
* [http://scicomp.jlab.org/farm2/walltime.html Walltime Distribution]<br />
<br />
== Saving Online Monitoring Data ==<br />
<br />
The procedure for writing the data out is given in, e.g.,<br />
[https://halldweb.jlab.org/wiki/index.php/Raid-to-Silo_Transfer_Strategy Raid-to-Silo Transfer Strategy].<br />
<br />
Once the DAQ writes out the data to the raid disk, cron jobs will copy the file to tape,<br />
and within ~20 min., we will have access to the file on tape at<br />
/mss/halld/$RUN_PERIOD/rawdata/RunXXXXXX.<br />
<br />
All online monitoring plugins will be run as data is taken.<br />
They will be accessible within the counting house via RootSpy, and<br />
for each run and file, a ROOT file containing the histograms will be saved<br />
within a subdirectory for each run.<br />
<br />
For immediate access to these files, the raid disk files may be accessed directly<br />
from the counting house, or the tape files will be available within ~20 min. of the<br />
file being written out.<br />
<br />
== Offline Monitoring: Running Over Archived Data ==<br />
<br />
Once files are written to tape we run the online plugins on these files to confirm what we were seeing in the online monitoring, and also to update the results from the latest calibration and software. <br />
Manual scripts and cron jobs are set up to look for new data and run the plugins over a sample of files.<br />
<br />
Every other Friday (usually the Friday before the offline meetings) jobs will be started to run the newest software on all previous runs,and allows everybody to see improvements in each detector.<br />
For each launch, independent builds of hdds, sim-recon, the monitoring plugins, and an sqlite file will be generated.<br />
<br />
Below the procedures are described for<br />
# Preparing the software for the launch<br />
# Starting the launch (using hdswif)<br />
# Post-analysis of statistics of the launch<br />
<br />
Processing the results and making them available to the collaboration<br />
is handled in the section [[#Post-Processing_Procedures | Post-Processing_Procedures]] below.<br />
<br />
<!--<br />
==== Generating an offline plugin job ====<br />
<br />
The user gxproj1 should be used for official offline monitoring jobs.<br />
Within /home/gxproj1/halld/monitoring/batch/ there will be scripts to run the online monitoring plugins over tape files.<br />
The main script is generatejobs_plugins_rawdata.sh, which can be used as<br />
generatejobs_plugins_rawdata.sh [minrun] [maxrun] (minfile) (maxfile)<br />
where minrun, maxrun specify the range of the run #, and minfile, maxfile (optional) specify<br />
the file range for that run.<br />
<br />
This will generate a script run_rawdata_XXXXXX.sh, where the run # has now been formatted to be 6 digits.<br />
Executing this script will send the monitoring plugins job to the Auger batch system.<br />
<br />
There is also a script clean.sh which can be used as<br />
clean.sh XXX<br />
This will clean up all associated files created in association with run XXX.<br />
<br />
Internally, the xml file used to submit the job will be created, and the job to run<br />
will be given within script.sh. All run parameters should be specified in at the beginning<br />
of generatejobs_plugins_rawdata.sh<br />
Since we are running on tape, the tape file will first be copied over to the cache disk, and the job will run<br />
over this cached file.<br />
<br />
==== Using cron to run automatically ====<br />
Within /home/gluex/halld/monitoring/cron/ there is a file cron_plugins<br />
that can be executed via<br />
crontab cron_plugins<br />
This will set up a cron job to call the script scan_for_jobs.sh, which will<br />
check in the rawdata directory and call generatejobs_plugins_rawdata.sh for<br />
any run that is more than 5 min old. The cron job is set up to run every 10 min.<br />
<br />
==== Magnetic Field Settings ====<br />
For tracking, it is necessary to set the correct magnetic field settings.<br />
The below table is now obsolete. Please refer to the [https://halldweb.jlab.org/cgi-bin/data_monitoring/run_conditions.pl GlueX Run Conditions] webpage.<br />
<br />
The actual field settings for each run have not been documented well.<br />
Below are values based on going through entries in the [https://logbooks.jlab.org/book/halld halld log].<br />
<br />
{| border="1" cellpadding="1" style="text-align: center;"<br />
!width="100"| Run #s<br />
!width="150"| Solenoid Current (A)<br />
!width="300"| JANA option<br />
!width="300"| notes<br />
|-<br />
| 940 - 996 || 1000 || -PBFIELD_MAP=Magnets/Solenoid/solenoid_1000A_poisson_20141104 || At run 997, the solenoid started ramping down<br />
|-<br />
| 998 - 1448 || 0 || -PBFIELD_TYPE=NoField -PDEFTAG:DTrackCandidate=StraightLine || Solenoid started ramping up around run 1431. Runs 1432 - 1448 should not be used for tracking. See [https://logbooks.jlab.org/entry/3309039 this] and [https://logbooks.jlab.org/entry/3309092 this] entry.<br />
|-<br />
| 1449 - 1620 || 1200 || -PBFIELD_MAP=Magnets/Solenoid/solenoid_1200A_poisson_20140520 || <br />
|-<br />
|}<br />
<br />
'''Note:''' The following run ranges were taken with a 300A field, and could be analyzed with the magnetic field map Magnets/Solenoid/solenoid_300A_poisson_20140819 , more study is needed to see if the straight line track fitter is needed or not: 1036 - 1053, 1065 - 1121, 1212? - 1254, 1309 - 1318 [1307,1308,1319-1329 were taken while the magnet was ramping].<br />
--><br />
<br />
=== General Information on Procedures ===<br />
This section explains how the offline monitoring should be run.<br />
Since we may want to simultaneously run offline monitoring for different run periods that require<br />
different environment variables, the scripts are set up so that a generic user can download the<br />
scripts and run them from anywhere. Most output directories for offline monitoring are created<br />
with group read/write permissions so that any Hall D group user has access to the contents,<br />
but there are some cases where use of the account that created the launch is necessary.<br />
<br />
The accounts used for offline monitoring are the gxprojN accounts created and maintained by<br />
Mark Ito (see [https://halldweb.jlab.org/wiki/index.php/GlueX-related_shared_accounts_on_the_JLab_CUE here]).<br />
As of October 2015, we have been using<br />
* gxproj1 for running over Fall 2014 data (deprecated since June 2015)<br />
* gxproj5 for running over Spring 2015 data<br />
<br />
Since the summer of 2015 we have transitioned from a system using Mark Ito's jproj scripts<br />
to integrating the swif system that Chris Larrieu (SciComp) has been developing. For offline monitoring,<br />
the hdswif system that Kei developed is used for launching the jobs, and the jproj system is used<br />
for meta-analysis of launch statistics.<br />
<br />
Both hdswif and jproj are maintained in svn:<br />
* hdswif: https://halldsvn.jlab.org/repos/trunk/scripts/monitoring/hdswif<br />
* jproj : https://halldsvn.jlab.org/repos/trunk/scripts/monitoring/jproj<br />
<br />
To run each package the directories will need to be checked out and the necessary<br />
scripts will need to be run.<br />
<br />
=== Preparing the software for the launch ===<br />
<br />
To begin a new launch the software must be built to the latest versions.<br />
For the gxprojN user accounts used, all software builds are contained in the directory ~/builds<br />
(which are soft links to /work/halld/home/gxproj5/builds). When logging into these accounts<br />
the setup files ~/setup_jlab-2015-03.csh or similar files should be sourced.<br />
<br />
<b>Note that Mark Ito does <u>not</u> want you to change the contents of each .cshrc file.</b><br />
You should consult him if you feel the need.<br />
<br />
<span style="color:red">NOTE: IT IS A GOOD IDEA TO DO A COMPLETE WIPEOUT/CHECKOUT OF A NEW VERSION OF EACH SOFTWARE TO AVOID STALE HEADER FILES.</span><br />
<br />
# Building hdds: Go to ~/builds/hdds. The directory hdds is the one from git. Delete the contents, then download the newest version from git and build: <pre>cd ~/builds/hdds/</pre> <pre>rm -frv hdds</pre> <pre>git clone https://github.com/JeffersonLab/hdds</pre> <pre>scons install</pre><br />
# Building sim-recon: Go to ~/builds/sim-recon. The directory sim-recon is the one from git. Delete the contents, then download the newest version from git and build: <pre>cd ~/builds/sim-recon</pre> <pre>rm -frv sim-recon</pre> <pre>git clone https://github.com/JeffersonLab/sim-recon</pre> <pre>scons install -j8</pre><br />
# Prepare the latest sqlite file: The sqlite is set in the ~/setup_jlab-2015-03.csh script as sqlite:////home/gxproj5/builds/ccdb.sqlite through the environment variables <b>JANA_CALIB_URL</b> and <b>CCDB_CONNECTION</b>. Therefore, go to ~/builds,and create a new sqlite file. We create the sqlite file in a temporary directory since creating the sqlite file in a directory where the output file exists causes errors. Original documentation on creating sqlite files are [https://halldweb.jlab.org/wiki/index.php/SQLite-form_of_the_CCDB_database here]. <pre>cd ~/builds/tmp</pre> <pre>$CCDB_HOME/scripts/mysql2sqlite/mysql2sqlite.sh -hhallddb.jlab.org -uccdb_user ccdb | sqlite3 ccdb.sqlite</pre><br />
# Note that the above steps must be done <b>BEFORE</b> launch project creation. This is because we will track the revisions of the libraries used, and this is done by extracting the svn information in each directory. Also, note that the system assumes that we have the topmost build directory (usually called GLUEX_TOP) to be $HOME/builds . Such an assumption is necessary to be able to extract information about the library locations automatically.<br />
<br />
Create the appropriate project(s) and submit the jobs using hdswif, as detailed in the section below.<br />
<br />
=== Starting the Launch and Submitting Jobs ===<br />
<br />
Until the summer of 2015 we relied solely on Mark Ito's jproj system for submitting and keeping track of jobs. We have since moved to the swif system and use the hdswif wrapper for this. Below are instructions for how to use these.<br />
<br />
# Downloading hdswif: Download the hdswif directory from svn. For the gxprojN accounts, use the directory ~/halld/hdswif. <pre>cd ~/halld </pre> <pre> svn co https://halldsvn.jlab.org/repos/trunk/scripts/monitoring/hdswif</pre> <pre>cd hdswif</pre><br />
# Creating the workflow: Within SWIF jobs are registered into workflows. First create the workflow. For offline monitoring, the workflow names are of the form <b>offline_monitoring_RunPeriod201Y_MM_verVV_hd_rawdata</b> with suitable replacements for the run period and version number. The command "swif list" will list all existing workflows. Also, or most simple SWIF commands hdswif also provides a wrapper. <pre>swif list</pre> <pre>swif create offline_monitoring_RunPeriod2015_03_ver99_hd_rawdata</pre> Or equivalently, <pre>hdswif.py create offline_monitoring_RunPeriod2015_03_ver99_hd_rawdata </pre><br />
# Registering jobs in the workflow: To register jobs within the workflow, hdswif provides the use of config files. Jobs can be registered by specifying the workflow, config file (-c), run (-r) and file (-f) numbers if necessary. A typical config file will look this:<br />
<pre><br />
PROJECT gluex<br />
TRACK reconstruction<br />
OS centos65<br />
NCORES 4<br />
DISK 40<br />
RAM 8<br />
TIMELIMIT 16<br />
JOBNAMEBASE offmon_<br />
RUNPERIOD 2015-03<br />
VERSION 92<br />
OUTPUT_TOPDIR /volatile/halld/offline_monitoring/RunPeriod-[RUNPERIOD]/ver[VERSION] # Example of other variables included in variable<br />
SCRIPTFILE /home/gxproj5/halld/hdswif/script.sh # Must specify full path<br />
ENVFILE /home/gxproj5/halld/hdswif/setup_jlab-2015-03.csh # Must specify full path<br />
</pre><br />
<br />
== Hall D Job Management System ==<br />
<br />
This section details instructions on how to create and launch a set of jobs using the Hall-D Job Management System developed by Mark Ito. These instructions are generic: this system can be used for the weekly monitoring jobs, but can also be used for other sets of job launches as well. <br />
<br />
=== Database Table Overview ===<br />
<br />
* Job management database table (<project_name>): For each input file, keeps track of whether or not a job for it has been submitted, along with other optional fields. <br />
<br />
* Job status database table (<project_name>Job (no space)): For each job, keeps track of the job-id, the job status, memory used, cpu & wall time, time taken to complete various stages (e.g. pending, dependency, active), and others. <br />
<br />
* Job metrics database table (<project_name>_aux (no space)): For each job, keeps track of the job-id, how many events were processed, the time it took to copy the cache file, and the time it took to run the plugin. This information is culled from the log files of each job, and is done within the analysis directory of each launch.<br />
<br />
=== Initialize Project Management ===<br />
<br />
* Log into the ifarm machine with one of the gxproj accounts. For this example we will use gxproj1.<br />
<pre><br />
ssh gxproj1@ifarm -Y<br />
</pre><br />
<br />
* Go to a directory to do the launch. In principle, any directory will work, but for gxproj1 this is usually done in /home/gxproj1/halld/<br />
<br />
* Check out the necessary scripts <pre>svn co https://halldsvn.jlab.org/repos/trunk/scripts/monitoring/jproj</pre> This will get all necessary scripts for launching. Once checked out, <pre>cd projects</pre><br />
<br />
* The script create_project.sh can be used to create a new project. It will take a single argument, the project name. It is assumed that the project name is of the form offline_monitoring_RunPeriod20YY_MM_verVV_hd_rawdata where this string will be parsed to give the run period 20YY_MM and version number VV. One thing to do BEFORE creation of a new project is to editthe conditions of the launch (plugins to run over, memory requested, disk space requested) within templates/template.jsub . This<br />
information is saved automatically at project creation time into files in /group/halld/data_monitoring/run_conditions .<br />
To create a project do <pre>./create_project.sh offline_monitoring_RunPeriod20YY_MM_verVV_hd_rawdata</pre><br />
The name has been chosen to be as consistent as possible with other directory structures.<br />
However, mysql requires that "-" be escaped in table names, so unfortunately run periods will be<br />
given as 2YYY_MM instead of 2YYY-MM.<br />
<br />
* A new directory with the project name (offline_monitoring_RunPeriod20YY_MM_verVV_hd_rawdata) and files will be copied and modified from the template directory to reflect the run period, the user, the directory that it was created in, project name, etc.<br />
<br />
* For each project, cd into the new directory<br />
* The script clear.sh will remove any existing tables for the current project name, then recreate it. Do<br />
<pre>./clear.sh</pre><br />
* To use the jproj.pl script that was checked in, add the directory to your path with<br />
<pre>source ../../scripts/setup.csh</pre><br />
or always specify the full path<br />
<pre>../../scripts/jproj.pl</pre><br />
* Now update the table of runs with<br />
<pre>jproj.pl <project name> update</pre><br />
This will fill the table with all files within /mss that are of the same form as what is in <project name>.jproj . If you want to register only on a subset of all such files, you can edit this file directly.<br />
* Once you have registered all of the files you would like to run over, do<br />
<pre>jproj.pl <project name> submit [max # of jobs] [run number]</pre><br />
where the additional options specify how many jobs to submit and which run number to run on. Without these options all files that are registered and have not been submitted yet will be submitted.<br />
<br />
At this stage you are ready to submit all files. It is a good idea to submit a few test jobs<br />
at first to check that all scripts are working and that the plugins do not crash. Once you are<br />
sure that this does not happen, you can send all jobs in. The remaining jobs are then the monitoring<br />
which will (among other things) put the results on the online webpage for the collaboration to view, and<br />
the analysis of the launch.<br />
<br />
=== Project File Overview ===<br />
<br />
An overview of each project file:<br />
* '''clear.sh:''' For the current project, deletes the job status and management database tables (if any), and creates new, empty ones. <br />
* '''<project_name>.jproj:''' Contains the path and file name format for the input files for the jobs. <br />
* '''<project_name>.jsub:''' The xml job submission script. The run number and file number variables are set during job submission for each input file. <br />
* '''script.sh:''' The script that is executed during the job. If output job directories are not pre-created manually, they should be created in this script with the proper permissions:<br />
<pre><br />
mkdir -p -m 775 my_directory<br />
</pre><br />
* '''setup_jlab-[run period].csh:''' The environment that is sourced at the beginning of the job execution. <br />
* '''status.sh:''' Updates the job status database table, and prints some of its columns to screen.<br />
<br />
=== Project Management ===<br />
<br />
* Delete (if any) and create the database table(s) for the current set of job submissions:<br />
<pre><br />
./clear.sh<br />
</pre><br />
Also, if testing was done with jobs, it is best to delete the output directory and the configuration files:<br />
<pre><br />
rm -frv /volatile/halld/offline_monitoring/RunPeriod-20YY-MM/verVV /group/halld/data_monitoring/run_conditions/soft_comm_20YY_MM_verVV.xml /group/halld/data_monitoring/run_conditions/jana_rawdata_comm_20YY_MM_verVV.conf<br />
</pre><br />
<br />
* Search for input files matching the string in the .jproj file, and create a row for each in the job management database table (called <project_name>). You can test by adding an optional argument at the end, which only selects files with a specific file number:<br />
<pre><br />
jproj.pl <project_name> update <optional_file_number><br />
</pre><br />
<br />
* Confirm that the job management database is accurate by printing it's contents to screen:<br />
<pre><br />
mysql -hhallddb -ufarmer farming -e "select * from <project_name>"<br />
</pre><br />
<br />
* ONLY if a mistake was made, to delete the tables from the database and recreate new, empty ones, run: <br />
<pre><br />
./clear.sh<br />
</pre><br />
<br />
* To look at the status of the submitted jobs, first query auger and update the job status database:<br />
<pre><br />
fill_in_job_details.pl <project_name><br />
</pre><br />
<br />
* The job status can then be viewed by submitting a query to the job status database (called <project_name>Job (no space in between)):<br />
<pre><br />
mysql -hhallddb -ufarmer farming -e "select id,run,file,jobId,hostname,status,timeSubmitted,timeActive,walltime,cput,timeComplete,result,error from <project_name>Job"<br />
</pre><br />
<br />
* These last two commands can instead be executed simultaneously by running:<br />
<pre><br />
./status.sh<br />
</pre><br />
<br />
=== Handy mysql Instructions ===<br />
<br />
* Handy mysql instructions:<br />
<pre><br />
mysql -hhallddb -ufarmer farming # Enter the "farming" mysql database on "hallddb" as user "farmer"<br />
quit; # Exit mysql<br />
show tables; # Show a list of the tables in the current database<br />
show columns from <project_name>; # show all of the columns for the given table<br />
select * from <project_name>; # show the contents of all rows from the given table<br />
</pre><br />
<br />
=== Backing Up Offline Monitoring Tables ===<br />
Tables created for offline monitoring can be backed up using the script backup_tables.sh which<br />
can be checked out with the other files from https://halldsvn.jlab.org/repos/trunk/scripts/monitoring/jproj/projects<br />
<br />
The script uses the command mysqldump to print out a file that can be executed to recreate the tables.<br />
Since executing this output file will drop the table when it exists, caution is advised.<br />
Example usage to backup all three tables created for run period 2014_10 ver 17:<br />
<pre>backup_tables.sh 2014_10 17</pre><br />
<br />
== Running Over Data As It Comes In ==<br />
<br />
A special user gxproj1 will have a cron job set up to run the plugins as new data appears on /mss.<br />
During the week, gxproj1 will submit offline plugin jobs with the same setup as the weekly jobs<br />
run the previous Friday. The procedure for this is shown below.<br />
<br />
<!--<br />
=== Setting up the environment ===<br />
The file<br />
/home/gxproj1/setup_jlab.csh<br />
is sourced through .tcshrc.<br />
This file is the same as what is linked to by<br />
/home/gluex/setup_jlab_commissioning.csh,<br />
except HALLD_HOME, HDDS_HOME, and JANA_CALIB_URL are set separately so that this<br />
user can have a separate build.<br />
<br />
To obtain the builds from the previous Friday's runs,<br />
execute<br />
/home/gxproj1/halld/monitoring/newruns/setup_previous.sh [year] [month] [day]<br />
The build revisions from the previous Friday are archived in files<br />
/work/halld/data_monitoring/run_conditions/soft_comm_[year]_[month]_[day].xml<br />
and the script will build libraries based on those stored revision numbers.<br />
--><br />
<br />
=== Running the cron job ===<br />
<br />
'''IMPORTANT:''' The cron job should not be running while you are manually submitting jobs using the jproj.pl script for the same project, or else you will probably multiply-submit a job. <br />
<br />
* Go to the cron job directory: <br />
<pre><br />
cd /u/home/gxproj1/halld/monitoring/newruns<br />
</pre><br />
<br />
* The cron_plugins file is the cronjob that will be executed. During execution, it runs the exec.sh command in the same folder. This command takes two arguments: the project name, and the maximum file number for each run. These fields should be updated in the cron_plugins file before running. <br />
<br />
* The exec.sh command updates the job management database table with any data that has arrived on tape since it was last updated, ignoring file numbers greater than the maximum file number. It then submits jobs for these files. <br />
<br />
* To start the cron job, run:<br />
<pre><br />
crontab cron_plugins<br />
</pre><br />
<br />
* To check whether the cron job is running, do<br />
<pre><br />
crontab -l<br />
</pre><br />
<br />
* To remove the cron job do<br />
<pre><br />
crontab -r<br />
</pre><br />
<br />
<!--<br />
The cron job will run the script scan_for_jobs.sh,<br />
which runs generatejobs_plugins_rawdata.sh for any<br />
new runs that it had not seen before. All previous<br />
runs are recorded in the file filelists/files_current.txt<br />
so clear this to run over runs, or set the parameters<br />
MINRUN and MAXRUN which will set the range of runs submitted.<br />
--><br />
<br />
==Post-Processing Procedures==<br />
<br />
To visualize the monitoring data, we save images of selected histograms and store time series of selected quantities in a database, which are then displayed on a web page. This section describes how to generate the monitoring images and database information.<br />
<br />
The scripts used to generate this summary data are primarily run from /home/gxprojN/halld/monitoring/process . If you want a new copy of the scripts, e.g., for a new monitoring run, you should check the scripts out from SVN:<br />
<syntaxhighlight><br />
svn co https://halldsvn.jlab.org/repos/trunk/scripts/monitoring/process<br />
</syntaxhighlight><br />
Note that these scripts currently have some parameters which must be periodically set by hand.<br />
<br />
The default python version on most JLab machine does not have the modules to allow these scripts to connect to the MySQL database. To run these scripts, load the environment with the following command<br />
<syntaxhighlight><br />
source /home/gxproj1/halld/monitoring/process/monitoring_env.csh<br />
</syntaxhighlight><br />
<br />
===Online Monitoring===<br />
<br />
There are two primary scripts for running over the monitoring data generated by the online system and offline reconstruction. The online script can be run with either of the following commands:<br />
<syntaxhighlight><br />
/home/gluex/halld/monitoring/process/check_new_runs.py<br />
<br />
OR <br />
<br />
/home/gluex/halld/monitoring/process/check_new_runs.csh<br />
</syntaxhighlight><br />
The shell script sets up the environment properly to run the python script. To connect to the monitoring database on the JLab CUE, modules included in the local installation of python >= 2.7 are needed. The shell script is appropriate to use in a cron job. The cronjob is currently run under the "gluex" account.<br />
<br />
The online monitoring system copies a ROOT file containing the results of the online monitoring, and other configuration files into a directory accessible outside the counting house. This python script automatically checks for new ROOT files, which it will then automatically process. It contains several configuration variables that must be correctly set, which contains the location of input/output directories, etc...<br />
<br />
===Offline Monitoring===<br />
<br />
After the data is run over, the results should be processed, so that summary data is entered into the monitoring database and plots are made for the monitoring webpages. Currently, this processing is controlled by a cronjob that runs the following script:<br />
<syntaxhighlight><br />
/home/gxproj1/halld/monitoring/process/check_monitoring_data.csh <br />
</syntaxhighlight><br />
This script checks for new ROOT files, and only runs over those it hasn't processed yet. Since one monitoring ROOT file is produced for each EVIO file, whenever a new file is produced, the plots for the corresponding run are recreated and all the ROOT files for a run are combined into one file. Information is stored in the database on a per-file basis. <br />
<br />
Plots for the monitoring web page can be made from single histograms or multiple histograms using RootSpy macros. If you want to change the list of plots made, you must modify one of the following files:<br />
* histograms_to_monitor - specify either the name of the histogram or its the full ROOT path <br />
* macros_to_monitor - specify the full path to the RootSpy macro .C file<br />
<br />
When a new monitoring run is started, or the conditions are changed, the following steps should be taken to process the new files:<br />
# Add a new data version, as described below:<br />
# Change the following parameters in check_monitoring_data.csh:<br />
## JOBDATE should correspond to the ouptut date used by the job submission script<br />
## OUTPUTDIR should correspond to the directory corresponding to the run period and revision corresponding to the new version you just submitted. Presumably, this directory will be empty at the beginning.<br />
## Once you create a new data version as defined below, you should pass the needed information as a command line option. Currently this is done by the ARGS variable. For example, the argument "-v RunPeriod-2014-10,8" tells the monitoring scripts to look up the version corresponding to revision 8 of RunPeriod-2014-10 in the monitoring DB and to use to store the results.<br />
<br />
<syntaxhighlight><br />
Example configuration parameters:<br />
set JOBDATE=2015-01-09<br />
set INPUTDIR=/volatile/halld/RunPeriod-2014-10/offline_monitoring<br />
set OUTPUTDIR=/w/halld-scifs1a/data_monitoring/RunPeriod-2014-10/ver08<br />
set ARGS=" -v RunPeriod-2014-10,8 "<br />
</syntaxhighlight><br />
If you want to process the results manually, the data is processed using the following script:<br />
<syntaxhighlight><br />
./process_new_offline_data.py <input directory> <output directory><br />
<br />
EXAMPLE:<br />
<br />
./process_new_offline_data.py 2014-11-14 /volatile/halld/RunPeriod-2014-10/offline_monitoring/ /w/halld-scifs1a/data_monitoring/RunPeriod-2014-10/ver02<br />
</syntaxhighlight><br />
The python script takes several options to enable/disable various steps in the processing. Of interest is the "--force" option, which will run over all monitoring ROOT files, whether or not they've been previously identified.<br />
<br />
Every time a new reconstruction pass is performed, a new version number must be generated. To do this, prepare a version file as described below. Then run the register_new_version.py script to store the information in the database. The script will return a version number, which then should be set by hand in process_new_offline_data.py - future versions of the script will streamline this part of the procedure. An example of how to generate a new version is:<br />
<syntaxhighlight><br />
./register_new_version.py add /home/gxproj1/halld/monitoring/process/versions/vers_RunPeriod-2014-10_pass1.txt<br />
</syntaxhighlight><br />
<br />
<b>If you are running the offline monitoring by checking out the files in trunk/scripts/monitoring/jproj/projects/<br />
of the svn repository</b>, and created a project with<br />
create_project.sh [project name] hd_rawdata<br />
Then go to the directory [project name]/processing/<br />
and execute<br />
./run_processing.sh<br />
which will run register_new_version.py as well as check_monitoring_data.csh for that project.<br />
<br />
===Step-by-Step Instructions For Processing a New Monitoring Run===<br />
<br />
The monitoring runs are current run out of the gxproj1 and gxproj5 accounts. After an offline monitoring run has been successfully started on the batch farm, the following steps should be followed to setup the post-processing for these runs.<br />
<br />
# The post-processing scripts are stored in $HOME/halld/monitoring/process and are automatically run by cron.<br />
# Run "svn update" to bring any changes in. Be sure that the list of histograms and macros to plot are current.<br />
# Edit check_monitoring_data.csh to point to the current revisions/directories<br />
#* VERSION<br />
#* ARGS<br />
#* Note that the environment depends on a standard script - $HOME/setup_jlab.csh<br />
# Update files in the web directory, so that the results are displayed on the web pages: /group/halld/www/halldweb/html/data_monitoring/textdata<br />
<br />
Check log files in $HOME/halld/monitoring/process/log for more information on how each run went. If there are problems, check log files, and modify check_monitoring_data.csh to vary the verbosity of the output.<br />
<br />
==Data Versions==<br />
<br />
To document the conditions of the monitoring data that is created, for the sake of reproducability and further analysis we save several pieces of information. The format is intended to be comprehensive enough to document not just monitoring data, but versions of raw and reconstructed data, so that this database table can be used for the event database as well.<br />
<br />
We store one record per pass through one run period, with the following structure:<br />
<br />
{| class="wikitable"<br />
! Field !! Description<br />
|-<br />
| data_type || The level of data we are processing. For the purposes of monitoring, "rawdata" is the online monitoring, "recon" is the offline monitoring<br />
|- <br />
| run_period || The run period of the data<br />
|- <br />
| revision || An integer specifying which pass through the run period this data corresponds to<br />
|- <br />
| software_version || The name of the XML file that specifies the different software versions used<br />
|-<br />
| jana_config || The name of the text file that specifies which JANA options were passed to the reconstruction program<br />
|-<br />
| ccdb_context || The value of JANA_CALIB_CONTEXT, which specifies the version of calibration constants that were used<br />
|-<br />
| production_time || The data at which monitoring/reconstruction began<br />
|-<br />
| dataVersionString || A convenient string for identifying this version of the data<br />
|}<br />
<br />
<br />
An example file used as as input to ./register_new_version.py is:<br />
<syntaxhighlight><br />
data_type = recon<br />
run_period = RunPeriod-2014-10<br />
revision = 1<br />
software_version = soft_comm_2014_11_06.xml<br />
jana_config = jana_rawdata_comm_2014_11_06.conf<br />
ccdb_context = calibtime=2014-11-10<br />
production_time = 2014-11-10<br />
dataVersionString = recon_RunPeriod-2014-10_20141110_ver01<br />
</syntaxhighlight></div>Kmoriyahttps://halldweb.jlab.org/wiki/index.php?title=Data_Monitoring_Procedures&diff=70727Data Monitoring Procedures2015-10-14T18:14:35Z<p>Kmoriya: /* Offline Monitoring: Running Over Archived Data */</p>
<hr />
<div>__TOC__<br />
<br />
== Master List of File / Database / Webpage Locations ==<br />
=== Run Conditions ===<br />
* Online Run-by-run condition files (B-field, current, etc.): /work/halld/online_monitoring/conditions/<br />
* Offline monitoring run conditions (software versions, jana config): /group/halld/data_monitoring/run_conditions/<br />
*[http://www.jlab.org/Hall-D/test/RunInfo/ Run Info vers. 1]<br />
*[https://halldweb.jlab.org/cgi-bin/data_monitoring/run_conditions.pl Run Info vers. 2]<br />
<br />
=== Monitoring Output Files ===<br />
* Run Periods 201Y-MM is for example 2015-03, launch ver verVV is for example ver15<br />
* Online monitoring histograms: /work/halld/online_monitoring/root/<br />
* Offline monitoring histogram ROOT files: /work/halld/data_monitoring/RunPeriod-201Y-MM/verVV/rootfiles<br />
* REST files (most recent launch only): /work/halld/data_monitoring/RunPeriod-201Y-MM/REST/verVV<br />
* individual files for each job (ROOT, log, etc.): /volatile/halld/offline_monitoring/RunPeriod-201Y-MM/verVV/<br />
<br />
=== Monitoring Database ===<br />
* Accessing monitoring database (on ifarm): mysql -u datmon -h hallddb.jlab.org data_monitoring<br />
<br />
=== Monitoring Webpages ===<br />
*[https://halldweb.jlab.org/cgi-bin/data_monitoring/monitoring/plotBrowser.py Plot Browser]<br />
*[https://halldweb.jlab.org/cgi-bin/data_monitoring/monitoring/runBrowser.py Run Browser]<br />
*[https://halldweb.jlab.org/cgi-bin/data_monitoring/monitoring/timeSeries.py Time Series]<br />
*[https://halldweb.jlab.org/data_monitoring/launch_analysis/ Launch Analysis]<br />
<br />
== Job Monitoring Links ==<br />
* [http://scicomp.jlab.org/farm2/job.html Completed Job History]<br />
* [http://scicomp.jlab.org/farm2/project.html Job Stats By Project]<br />
* [http://scicomp.jlab.org/farm2/trackOrg.html Job Stats By Track]<br />
* [http://scicomp.jlab.org/farm2/report.html Cluster Report]<br />
* [http://scicomp.jlab.org/farm2/walltime.html Walltime Distribution]<br />
<br />
== Saving Online Monitoring Data ==<br />
<br />
The procedure for writing the data out is given in, e.g.,<br />
[https://halldweb.jlab.org/wiki/index.php/Raid-to-Silo_Transfer_Strategy Raid-to-Silo Transfer Strategy].<br />
<br />
Once the DAQ writes out the data to the raid disk, cron jobs will copy the file to tape,<br />
and within ~20 min., we will have access to the file on tape at<br />
/mss/halld/$RUN_PERIOD/rawdata/RunXXXXXX.<br />
<br />
All online monitoring plugins will be run as data is taken.<br />
They will be accessible within the counting house via RootSpy, and<br />
for each run and file, a ROOT file containing the histograms will be saved<br />
within a subdirectory for each run.<br />
<br />
For immediate access to these files, the raid disk files may be accessed directly<br />
from the counting house, or the tape files will be available within ~20 min. of the<br />
file being written out.<br />
<br />
== Offline Monitoring: Running Over Archived Data ==<br />
<br />
Once files are written to tape we run the online plugins on these files to confirm what we were seeing in the online monitoring, and also to update the results from the latest calibration and software. <br />
Manual scripts and cron jobs are set up to look for new data and run the plugins over a sample of files.<br />
<br />
Every other Friday (usually the Friday before the offline meetings) jobs will be started to run the newest software on all previous runs,and allows everybody to see improvements in each detector.<br />
For each launch, independent builds of hdds, sim-recon, the monitoring plugins, and an sqlite file will be generated.<br />
<br />
Below the procedures are described for<br />
# Preparing the software for the launch<br />
# Starting the launch (using hdswif)<br />
# Post-analysis of statistics of the launch<br />
<br />
Processing the results and making them available to the collaboration<br />
is handled in the section [[#Post-Processing_Procedures | Post-Processing_Procedures]] below.<br />
<br />
<!--<br />
==== Generating an offline plugin job ====<br />
<br />
The user gxproj1 should be used for official offline monitoring jobs.<br />
Within /home/gxproj1/halld/monitoring/batch/ there will be scripts to run the online monitoring plugins over tape files.<br />
The main script is generatejobs_plugins_rawdata.sh, which can be used as<br />
generatejobs_plugins_rawdata.sh [minrun] [maxrun] (minfile) (maxfile)<br />
where minrun, maxrun specify the range of the run #, and minfile, maxfile (optional) specify<br />
the file range for that run.<br />
<br />
This will generate a script run_rawdata_XXXXXX.sh, where the run # has now been formatted to be 6 digits.<br />
Executing this script will send the monitoring plugins job to the Auger batch system.<br />
<br />
There is also a script clean.sh which can be used as<br />
clean.sh XXX<br />
This will clean up all associated files created in association with run XXX.<br />
<br />
Internally, the xml file used to submit the job will be created, and the job to run<br />
will be given within script.sh. All run parameters should be specified in at the beginning<br />
of generatejobs_plugins_rawdata.sh<br />
Since we are running on tape, the tape file will first be copied over to the cache disk, and the job will run<br />
over this cached file.<br />
<br />
==== Using cron to run automatically ====<br />
Within /home/gluex/halld/monitoring/cron/ there is a file cron_plugins<br />
that can be executed via<br />
crontab cron_plugins<br />
This will set up a cron job to call the script scan_for_jobs.sh, which will<br />
check in the rawdata directory and call generatejobs_plugins_rawdata.sh for<br />
any run that is more than 5 min old. The cron job is set up to run every 10 min.<br />
<br />
==== Magnetic Field Settings ====<br />
For tracking, it is necessary to set the correct magnetic field settings.<br />
The below table is now obsolete. Please refer to the [https://halldweb.jlab.org/cgi-bin/data_monitoring/run_conditions.pl GlueX Run Conditions] webpage.<br />
<br />
The actual field settings for each run have not been documented well.<br />
Below are values based on going through entries in the [https://logbooks.jlab.org/book/halld halld log].<br />
<br />
{| border="1" cellpadding="1" style="text-align: center;"<br />
!width="100"| Run #s<br />
!width="150"| Solenoid Current (A)<br />
!width="300"| JANA option<br />
!width="300"| notes<br />
|-<br />
| 940 - 996 || 1000 || -PBFIELD_MAP=Magnets/Solenoid/solenoid_1000A_poisson_20141104 || At run 997, the solenoid started ramping down<br />
|-<br />
| 998 - 1448 || 0 || -PBFIELD_TYPE=NoField -PDEFTAG:DTrackCandidate=StraightLine || Solenoid started ramping up around run 1431. Runs 1432 - 1448 should not be used for tracking. See [https://logbooks.jlab.org/entry/3309039 this] and [https://logbooks.jlab.org/entry/3309092 this] entry.<br />
|-<br />
| 1449 - 1620 || 1200 || -PBFIELD_MAP=Magnets/Solenoid/solenoid_1200A_poisson_20140520 || <br />
|-<br />
|}<br />
<br />
'''Note:''' The following run ranges were taken with a 300A field, and could be analyzed with the magnetic field map Magnets/Solenoid/solenoid_300A_poisson_20140819 , more study is needed to see if the straight line track fitter is needed or not: 1036 - 1053, 1065 - 1121, 1212? - 1254, 1309 - 1318 [1307,1308,1319-1329 were taken while the magnet was ramping].<br />
--><br />
<br />
=== General Information on Procedures ===<br />
This section explains how the offline monitoring should be run.<br />
Since we may want to simultaneously run offline monitoring for different run periods that require<br />
different environment variables, the scripts are set up so that a generic user can download the<br />
scripts and run them from anywhere. Most output directories for offline monitoring are created<br />
with group read/write permissions so that any Hall D group user has access to the contents,<br />
but there are some cases where use of the account that created the launch is necessary.<br />
<br />
The accounts used for offline monitoring are the gxprojN accounts created and maintained by<br />
Mark Ito (see [https://halldweb.jlab.org/wiki/index.php/GlueX-related_shared_accounts_on_the_JLab_CUE here]).<br />
As of October 2015, we have been using<br />
* gxproj1 for running over Fall 2014 data (deprecated since June 2015)<br />
* gxproj5 for running over Spring 2015 data<br />
<br />
Since the summer of 2015 we have transitioned from a system using Mark Ito's jproj scripts<br />
to integrating the swif system that Chris Larrieu (SciComp) has been developing. For offline monitoring,<br />
the hdswif system that Kei developed is used for launching the jobs, and the jproj system is used<br />
for meta-analysis of launch statistics.<br />
<br />
Both hdswif and jproj are maintained in svn:<br />
* hdswif: https://halldsvn.jlab.org/repos/trunk/scripts/monitoring/hdswif<br />
* jproj : https://halldsvn.jlab.org/repos/trunk/scripts/monitoring/jproj<br />
<br />
To run each package the directories will need to be checked out and the necessary<br />
scripts will need to be run.<br />
<br />
=== Preparing the software for the launch ===<br />
<br />
To begin a new launch the software must be built to the latest versions.<br />
For the gxprojN user accounts used, all software builds are contained in the directory ~/builds<br />
(which are soft links to /work/halld/home/gxproj5/builds). When logging into these accounts<br />
the setup files ~/setup_jlab-2015-03.csh or similar files should be sourced.<br />
<br />
<b>Note that Mark Ito does <u>not</u> want you to change the contents of each .cshrc file.</b><br />
You should consult him if you feel the need.<br />
<br />
<span style="color:red">NOTE: IT IS A GOOD IDEA TO DO A COMPLETE WIPEOUT/CHECKOUT OF A NEW VERSION OF EACH SOFTWARE TO AVOID STALE HEADER FILES.</span><br />
<br />
# Building hdds: Go to ~/builds/hdds. The directory hdds is the one from git. Delete the contents, then download the newest version from git and build: <pre>cd ~/builds/hdds/</pre> <pre>rm -frv hdds</pre> <pre>git clone https://github.com/JeffersonLab/hdds</pre> <pre>scons install</pre><br />
# Building sim-recon: Go to ~/builds/sim-recon. The directory sim-recon is the one from git. Delete the contents, then download the newest version from git and build: <pre>cd ~/builds/sim-recon</pre> <pre>rm -frv sim-recon</pre> <pre>git clone https://github.com/JeffersonLab/sim-recon</pre> <pre>scons install -j8</pre><br />
# Prepare the latest sqlite file: The sqlite is set in the ~/setup_jlab-2015-03.csh script as sqlite:////home/gxproj5/builds/ccdb.sqlite through the environment variables <b>JANA_CALIB_URL</b> and <b>CCDB_CONNECTION</b>. Therefore, go to ~/builds,and create a new sqlite file. We create the sqlite file in a temporary directory since creating the sqlite file in a directory where the output file exists causes errors. Original documentation on creating sqlite files are [https://halldweb.jlab.org/wiki/index.php/SQLite-form_of_the_CCDB_database here]. <pre>cd ~/builds/tmp</pre> <pre>$CCDB_HOME/scripts/mysql2sqlite/mysql2sqlite.sh -hhallddb.jlab.org -uccdb_user ccdb | sqlite3 ccdb.sqlite</pre><br />
# Note that the above steps must be done <b>BEFORE</b> launch project creation. This is because we will track the revisions of the libraries used, and this is done by extracting the svn information in each directory. Also, note that the system assumes that we have the topmost build directory (usually called GLUEX_TOP) to be $HOME/builds . Such an assumption is necessary to be able to extract information about the library locations automatically.<br />
<br />
Create the appropriate project(s) and submit the jobs using hdswif, as detailed in the section below.<br />
<br />
== Hall D Job Management System ==<br />
<br />
This section details instructions on how to create and launch a set of jobs using the Hall-D Job Management System developed by Mark Ito. These instructions are generic: this system can be used for the weekly monitoring jobs, but can also be used for other sets of job launches as well. <br />
<br />
=== Database Table Overview ===<br />
<br />
* Job management database table (<project_name>): For each input file, keeps track of whether or not a job for it has been submitted, along with other optional fields. <br />
<br />
* Job status database table (<project_name>Job (no space)): For each job, keeps track of the job-id, the job status, memory used, cpu & wall time, time taken to complete various stages (e.g. pending, dependency, active), and others. <br />
<br />
* Job metrics database table (<project_name>_aux (no space)): For each job, keeps track of the job-id, how many events were processed, the time it took to copy the cache file, and the time it took to run the plugin. This information is culled from the log files of each job, and is done within the analysis directory of each launch.<br />
<br />
=== Initialize Project Management ===<br />
<br />
* Log into the ifarm machine with one of the gxproj accounts. For this example we will use gxproj1.<br />
<pre><br />
ssh gxproj1@ifarm -Y<br />
</pre><br />
<br />
* Go to a directory to do the launch. In principle, any directory will work, but for gxproj1 this is usually done in /home/gxproj1/halld/<br />
<br />
* Check out the necessary scripts <pre>svn co https://halldsvn.jlab.org/repos/trunk/scripts/monitoring/jproj</pre> This will get all necessary scripts for launching. Once checked out, <pre>cd projects</pre><br />
<br />
* The script create_project.sh can be used to create a new project. It will take a single argument, the project name. It is assumed that the project name is of the form offline_monitoring_RunPeriod20YY_MM_verVV_hd_rawdata where this string will be parsed to give the run period 20YY_MM and version number VV. One thing to do BEFORE creation of a new project is to editthe conditions of the launch (plugins to run over, memory requested, disk space requested) within templates/template.jsub . This<br />
information is saved automatically at project creation time into files in /group/halld/data_monitoring/run_conditions .<br />
To create a project do <pre>./create_project.sh offline_monitoring_RunPeriod20YY_MM_verVV_hd_rawdata</pre><br />
The name has been chosen to be as consistent as possible with other directory structures.<br />
However, mysql requires that "-" be escaped in table names, so unfortunately run periods will be<br />
given as 2YYY_MM instead of 2YYY-MM.<br />
<br />
* A new directory with the project name (offline_monitoring_RunPeriod20YY_MM_verVV_hd_rawdata) and files will be copied and modified from the template directory to reflect the run period, the user, the directory that it was created in, project name, etc.<br />
<br />
* For each project, cd into the new directory<br />
* The script clear.sh will remove any existing tables for the current project name, then recreate it. Do<br />
<pre>./clear.sh</pre><br />
* To use the jproj.pl script that was checked in, add the directory to your path with<br />
<pre>source ../../scripts/setup.csh</pre><br />
or always specify the full path<br />
<pre>../../scripts/jproj.pl</pre><br />
* Now update the table of runs with<br />
<pre>jproj.pl <project name> update</pre><br />
This will fill the table with all files within /mss that are of the same form as what is in <project name>.jproj . If you want to register only on a subset of all such files, you can edit this file directly.<br />
* Once you have registered all of the files you would like to run over, do<br />
<pre>jproj.pl <project name> submit [max # of jobs] [run number]</pre><br />
where the additional options specify how many jobs to submit and which run number to run on. Without these options all files that are registered and have not been submitted yet will be submitted.<br />
<br />
At this stage you are ready to submit all files. It is a good idea to submit a few test jobs<br />
at first to check that all scripts are working and that the plugins do not crash. Once you are<br />
sure that this does not happen, you can send all jobs in. The remaining jobs are then the monitoring<br />
which will (among other things) put the results on the online webpage for the collaboration to view, and<br />
the analysis of the launch.<br />
<br />
=== Project File Overview ===<br />
<br />
An overview of each project file:<br />
* '''clear.sh:''' For the current project, deletes the job status and management database tables (if any), and creates new, empty ones. <br />
* '''<project_name>.jproj:''' Contains the path and file name format for the input files for the jobs. <br />
* '''<project_name>.jsub:''' The xml job submission script. The run number and file number variables are set during job submission for each input file. <br />
* '''script.sh:''' The script that is executed during the job. If output job directories are not pre-created manually, they should be created in this script with the proper permissions:<br />
<pre><br />
mkdir -p -m 775 my_directory<br />
</pre><br />
* '''setup_jlab-[run period].csh:''' The environment that is sourced at the beginning of the job execution. <br />
* '''status.sh:''' Updates the job status database table, and prints some of its columns to screen.<br />
<br />
=== Project Management ===<br />
<br />
* Delete (if any) and create the database table(s) for the current set of job submissions:<br />
<pre><br />
./clear.sh<br />
</pre><br />
Also, if testing was done with jobs, it is best to delete the output directory and the configuration files:<br />
<pre><br />
rm -frv /volatile/halld/offline_monitoring/RunPeriod-20YY-MM/verVV /group/halld/data_monitoring/run_conditions/soft_comm_20YY_MM_verVV.xml /group/halld/data_monitoring/run_conditions/jana_rawdata_comm_20YY_MM_verVV.conf<br />
</pre><br />
<br />
* Search for input files matching the string in the .jproj file, and create a row for each in the job management database table (called <project_name>). You can test by adding an optional argument at the end, which only selects files with a specific file number:<br />
<pre><br />
jproj.pl <project_name> update <optional_file_number><br />
</pre><br />
<br />
* Confirm that the job management database is accurate by printing it's contents to screen:<br />
<pre><br />
mysql -hhallddb -ufarmer farming -e "select * from <project_name>"<br />
</pre><br />
<br />
* ONLY if a mistake was made, to delete the tables from the database and recreate new, empty ones, run: <br />
<pre><br />
./clear.sh<br />
</pre><br />
<br />
* To look at the status of the submitted jobs, first query auger and update the job status database:<br />
<pre><br />
fill_in_job_details.pl <project_name><br />
</pre><br />
<br />
* The job status can then be viewed by submitting a query to the job status database (called <project_name>Job (no space in between)):<br />
<pre><br />
mysql -hhallddb -ufarmer farming -e "select id,run,file,jobId,hostname,status,timeSubmitted,timeActive,walltime,cput,timeComplete,result,error from <project_name>Job"<br />
</pre><br />
<br />
* These last two commands can instead be executed simultaneously by running:<br />
<pre><br />
./status.sh<br />
</pre><br />
<br />
=== Handy mysql Instructions ===<br />
<br />
* Handy mysql instructions:<br />
<pre><br />
mysql -hhallddb -ufarmer farming # Enter the "farming" mysql database on "hallddb" as user "farmer"<br />
quit; # Exit mysql<br />
show tables; # Show a list of the tables in the current database<br />
show columns from <project_name>; # show all of the columns for the given table<br />
select * from <project_name>; # show the contents of all rows from the given table<br />
</pre><br />
<br />
=== Backing Up Offline Monitoring Tables ===<br />
Tables created for offline monitoring can be backed up using the script backup_tables.sh which<br />
can be checked out with the other files from https://halldsvn.jlab.org/repos/trunk/scripts/monitoring/jproj/projects<br />
<br />
The script uses the command mysqldump to print out a file that can be executed to recreate the tables.<br />
Since executing this output file will drop the table when it exists, caution is advised.<br />
Example usage to backup all three tables created for run period 2014_10 ver 17:<br />
<pre>backup_tables.sh 2014_10 17</pre><br />
<br />
== Running Over Data As It Comes In ==<br />
<br />
A special user gxproj1 will have a cron job set up to run the plugins as new data appears on /mss.<br />
During the week, gxproj1 will submit offline plugin jobs with the same setup as the weekly jobs<br />
run the previous Friday. The procedure for this is shown below.<br />
<br />
<!--<br />
=== Setting up the environment ===<br />
The file<br />
/home/gxproj1/setup_jlab.csh<br />
is sourced through .tcshrc.<br />
This file is the same as what is linked to by<br />
/home/gluex/setup_jlab_commissioning.csh,<br />
except HALLD_HOME, HDDS_HOME, and JANA_CALIB_URL are set separately so that this<br />
user can have a separate build.<br />
<br />
To obtain the builds from the previous Friday's runs,<br />
execute<br />
/home/gxproj1/halld/monitoring/newruns/setup_previous.sh [year] [month] [day]<br />
The build revisions from the previous Friday are archived in files<br />
/work/halld/data_monitoring/run_conditions/soft_comm_[year]_[month]_[day].xml<br />
and the script will build libraries based on those stored revision numbers.<br />
--><br />
<br />
=== Running the cron job ===<br />
<br />
'''IMPORTANT:''' The cron job should not be running while you are manually submitting jobs using the jproj.pl script for the same project, or else you will probably multiply-submit a job. <br />
<br />
* Go to the cron job directory: <br />
<pre><br />
cd /u/home/gxproj1/halld/monitoring/newruns<br />
</pre><br />
<br />
* The cron_plugins file is the cronjob that will be executed. During execution, it runs the exec.sh command in the same folder. This command takes two arguments: the project name, and the maximum file number for each run. These fields should be updated in the cron_plugins file before running. <br />
<br />
* The exec.sh command updates the job management database table with any data that has arrived on tape since it was last updated, ignoring file numbers greater than the maximum file number. It then submits jobs for these files. <br />
<br />
* To start the cron job, run:<br />
<pre><br />
crontab cron_plugins<br />
</pre><br />
<br />
* To check whether the cron job is running, do<br />
<pre><br />
crontab -l<br />
</pre><br />
<br />
* To remove the cron job do<br />
<pre><br />
crontab -r<br />
</pre><br />
<br />
<!--<br />
The cron job will run the script scan_for_jobs.sh,<br />
which runs generatejobs_plugins_rawdata.sh for any<br />
new runs that it had not seen before. All previous<br />
runs are recorded in the file filelists/files_current.txt<br />
so clear this to run over runs, or set the parameters<br />
MINRUN and MAXRUN which will set the range of runs submitted.<br />
--><br />
<br />
==Post-Processing Procedures==<br />
<br />
To visualize the monitoring data, we save images of selected histograms and store time series of selected quantities in a database, which are then displayed on a web page. This section describes how to generate the monitoring images and database information.<br />
<br />
The scripts used to generate this summary data are primarily run from /home/gxprojN/halld/monitoring/process . If you want a new copy of the scripts, e.g., for a new monitoring run, you should check the scripts out from SVN:<br />
<syntaxhighlight><br />
svn co https://halldsvn.jlab.org/repos/trunk/scripts/monitoring/process<br />
</syntaxhighlight><br />
Note that these scripts currently have some parameters which must be periodically set by hand.<br />
<br />
The default python version on most JLab machine does not have the modules to allow these scripts to connect to the MySQL database. To run these scripts, load the environment with the following command<br />
<syntaxhighlight><br />
source /home/gxproj1/halld/monitoring/process/monitoring_env.csh<br />
</syntaxhighlight><br />
<br />
===Online Monitoring===<br />
<br />
There are two primary scripts for running over the monitoring data generated by the online system and offline reconstruction. The online script can be run with either of the following commands:<br />
<syntaxhighlight><br />
/home/gluex/halld/monitoring/process/check_new_runs.py<br />
<br />
OR <br />
<br />
/home/gluex/halld/monitoring/process/check_new_runs.csh<br />
</syntaxhighlight><br />
The shell script sets up the environment properly to run the python script. To connect to the monitoring database on the JLab CUE, modules included in the local installation of python >= 2.7 are needed. The shell script is appropriate to use in a cron job. The cronjob is currently run under the "gluex" account.<br />
<br />
The online monitoring system copies a ROOT file containing the results of the online monitoring, and other configuration files into a directory accessible outside the counting house. This python script automatically checks for new ROOT files, which it will then automatically process. It contains several configuration variables that must be correctly set, which contains the location of input/output directories, etc...<br />
<br />
===Offline Monitoring===<br />
<br />
After the data is run over, the results should be processed, so that summary data is entered into the monitoring database and plots are made for the monitoring webpages. Currently, this processing is controlled by a cronjob that runs the following script:<br />
<syntaxhighlight><br />
/home/gxproj1/halld/monitoring/process/check_monitoring_data.csh <br />
</syntaxhighlight><br />
This script checks for new ROOT files, and only runs over those it hasn't processed yet. Since one monitoring ROOT file is produced for each EVIO file, whenever a new file is produced, the plots for the corresponding run are recreated and all the ROOT files for a run are combined into one file. Information is stored in the database on a per-file basis. <br />
<br />
Plots for the monitoring web page can be made from single histograms or multiple histograms using RootSpy macros. If you want to change the list of plots made, you must modify one of the following files:<br />
* histograms_to_monitor - specify either the name of the histogram or its the full ROOT path <br />
* macros_to_monitor - specify the full path to the RootSpy macro .C file<br />
<br />
When a new monitoring run is started, or the conditions are changed, the following steps should be taken to process the new files:<br />
# Add a new data version, as described below:<br />
# Change the following parameters in check_monitoring_data.csh:<br />
## JOBDATE should correspond to the ouptut date used by the job submission script<br />
## OUTPUTDIR should correspond to the directory corresponding to the run period and revision corresponding to the new version you just submitted. Presumably, this directory will be empty at the beginning.<br />
## Once you create a new data version as defined below, you should pass the needed information as a command line option. Currently this is done by the ARGS variable. For example, the argument "-v RunPeriod-2014-10,8" tells the monitoring scripts to look up the version corresponding to revision 8 of RunPeriod-2014-10 in the monitoring DB and to use to store the results.<br />
<br />
<syntaxhighlight><br />
Example configuration parameters:<br />
set JOBDATE=2015-01-09<br />
set INPUTDIR=/volatile/halld/RunPeriod-2014-10/offline_monitoring<br />
set OUTPUTDIR=/w/halld-scifs1a/data_monitoring/RunPeriod-2014-10/ver08<br />
set ARGS=" -v RunPeriod-2014-10,8 "<br />
</syntaxhighlight><br />
If you want to process the results manually, the data is processed using the following script:<br />
<syntaxhighlight><br />
./process_new_offline_data.py <input directory> <output directory><br />
<br />
EXAMPLE:<br />
<br />
./process_new_offline_data.py 2014-11-14 /volatile/halld/RunPeriod-2014-10/offline_monitoring/ /w/halld-scifs1a/data_monitoring/RunPeriod-2014-10/ver02<br />
</syntaxhighlight><br />
The python script takes several options to enable/disable various steps in the processing. Of interest is the "--force" option, which will run over all monitoring ROOT files, whether or not they've been previously identified.<br />
<br />
Every time a new reconstruction pass is performed, a new version number must be generated. To do this, prepare a version file as described below. Then run the register_new_version.py script to store the information in the database. The script will return a version number, which then should be set by hand in process_new_offline_data.py - future versions of the script will streamline this part of the procedure. An example of how to generate a new version is:<br />
<syntaxhighlight><br />
./register_new_version.py add /home/gxproj1/halld/monitoring/process/versions/vers_RunPeriod-2014-10_pass1.txt<br />
</syntaxhighlight><br />
<br />
<b>If you are running the offline monitoring by checking out the files in trunk/scripts/monitoring/jproj/projects/<br />
of the svn repository</b>, and created a project with<br />
create_project.sh [project name] hd_rawdata<br />
Then go to the directory [project name]/processing/<br />
and execute<br />
./run_processing.sh<br />
which will run register_new_version.py as well as check_monitoring_data.csh for that project.<br />
<br />
===Step-by-Step Instructions For Processing a New Monitoring Run===<br />
<br />
The monitoring runs are current run out of the gxproj1 and gxproj5 accounts. After an offline monitoring run has been successfully started on the batch farm, the following steps should be followed to setup the post-processing for these runs.<br />
<br />
# The post-processing scripts are stored in $HOME/halld/monitoring/process and are automatically run by cron.<br />
# Run "svn update" to bring any changes in. Be sure that the list of histograms and macros to plot are current.<br />
# Edit check_monitoring_data.csh to point to the current revisions/directories<br />
#* VERSION<br />
#* ARGS<br />
#* Note that the environment depends on a standard script - $HOME/setup_jlab.csh<br />
# Update files in the web directory, so that the results are displayed on the web pages: /group/halld/www/halldweb/html/data_monitoring/textdata<br />
<br />
Check log files in $HOME/halld/monitoring/process/log for more information on how each run went. If there are problems, check log files, and modify check_monitoring_data.csh to vary the verbosity of the output.<br />
<br />
==Data Versions==<br />
<br />
To document the conditions of the monitoring data that is created, for the sake of reproducability and further analysis we save several pieces of information. The format is intended to be comprehensive enough to document not just monitoring data, but versions of raw and reconstructed data, so that this database table can be used for the event database as well.<br />
<br />
We store one record per pass through one run period, with the following structure:<br />
<br />
{| class="wikitable"<br />
! Field !! Description<br />
|-<br />
| data_type || The level of data we are processing. For the purposes of monitoring, "rawdata" is the online monitoring, "recon" is the offline monitoring<br />
|- <br />
| run_period || The run period of the data<br />
|- <br />
| revision || An integer specifying which pass through the run period this data corresponds to<br />
|- <br />
| software_version || The name of the XML file that specifies the different software versions used<br />
|-<br />
| jana_config || The name of the text file that specifies which JANA options were passed to the reconstruction program<br />
|-<br />
| ccdb_context || The value of JANA_CALIB_CONTEXT, which specifies the version of calibration constants that were used<br />
|-<br />
| production_time || The data at which monitoring/reconstruction began<br />
|-<br />
| dataVersionString || A convenient string for identifying this version of the data<br />
|}<br />
<br />
<br />
An example file used as as input to ./register_new_version.py is:<br />
<syntaxhighlight><br />
data_type = recon<br />
run_period = RunPeriod-2014-10<br />
revision = 1<br />
software_version = soft_comm_2014_11_06.xml<br />
jana_config = jana_rawdata_comm_2014_11_06.conf<br />
ccdb_context = calibtime=2014-11-10<br />
production_time = 2014-11-10<br />
dataVersionString = recon_RunPeriod-2014-10_20141110_ver01<br />
</syntaxhighlight></div>Kmoriyahttps://halldweb.jlab.org/wiki/index.php?title=Data_Monitoring_Procedures&diff=70724Data Monitoring Procedures2015-10-14T17:55:01Z<p>Kmoriya: /* Procedures */</p>
<hr />
<div>__TOC__<br />
<br />
== Master List of File / Database / Webpage Locations ==<br />
=== Run Conditions ===<br />
* Online Run-by-run condition files (B-field, current, etc.): /work/halld/online_monitoring/conditions/<br />
* Offline monitoring run conditions (software versions, jana config): /group/halld/data_monitoring/run_conditions/<br />
*[http://www.jlab.org/Hall-D/test/RunInfo/ Run Info vers. 1]<br />
*[https://halldweb.jlab.org/cgi-bin/data_monitoring/run_conditions.pl Run Info vers. 2]<br />
<br />
=== Monitoring Output Files ===<br />
* Run Periods 201Y-MM is for example 2015-03, launch ver verVV is for example ver15<br />
* Online monitoring histograms: /work/halld/online_monitoring/root/<br />
* Offline monitoring histogram ROOT files: /work/halld/data_monitoring/RunPeriod-201Y-MM/verVV/rootfiles<br />
* REST files (most recent launch only): /work/halld/data_monitoring/RunPeriod-201Y-MM/REST/verVV<br />
* individual files for each job (ROOT, log, etc.): /volatile/halld/offline_monitoring/RunPeriod-201Y-MM/verVV/<br />
<br />
=== Monitoring Database ===<br />
* Accessing monitoring database (on ifarm): mysql -u datmon -h hallddb.jlab.org data_monitoring<br />
<br />
=== Monitoring Webpages ===<br />
*[https://halldweb.jlab.org/cgi-bin/data_monitoring/monitoring/plotBrowser.py Plot Browser]<br />
*[https://halldweb.jlab.org/cgi-bin/data_monitoring/monitoring/runBrowser.py Run Browser]<br />
*[https://halldweb.jlab.org/cgi-bin/data_monitoring/monitoring/timeSeries.py Time Series]<br />
*[https://halldweb.jlab.org/data_monitoring/launch_analysis/ Launch Analysis]<br />
<br />
== Job Monitoring Links ==<br />
* [http://scicomp.jlab.org/farm2/job.html Completed Job History]<br />
* [http://scicomp.jlab.org/farm2/project.html Job Stats By Project]<br />
* [http://scicomp.jlab.org/farm2/trackOrg.html Job Stats By Track]<br />
* [http://scicomp.jlab.org/farm2/report.html Cluster Report]<br />
* [http://scicomp.jlab.org/farm2/walltime.html Walltime Distribution]<br />
<br />
== Saving Online Monitoring Data ==<br />
<br />
The procedure for writing the data out is given in, e.g.,<br />
[https://halldweb.jlab.org/wiki/index.php/Raid-to-Silo_Transfer_Strategy Raid-to-Silo Transfer Strategy].<br />
<br />
Once the DAQ writes out the data to the raid disk, cron jobs will copy the file to tape,<br />
and within ~20 min., we will have access to the file on tape at<br />
/mss/halld/$RUN_PERIOD/rawdata/RunXXXXXX.<br />
<br />
All online monitoring plugins will be run as data is taken.<br />
They will be accessible within the counting house via RootSpy, and<br />
for each run and file, a ROOT file containing the histograms will be saved<br />
within a subdirectory for each run.<br />
<br />
For immediate access to these files, the raid disk files may be accessed directly<br />
from the counting house, or the tape files will be available within ~20 min. of the<br />
file being written out.<br />
<br />
== Offline Monitoring: Running Over Archived Data ==<br />
<br />
Once files are written to tape we run the online plugins on these files to confirm what we were seeing in the online monitoring, and also to update the results from the latest calibration and software. <br />
Manual scripts and cron jobs are set up to look for new data and run the plugins over a sample of files.<br />
<br />
Every other Friday (usually the Friday before the offline meetings) jobs will be started to run the newest software on all previous runs,and allows everybody to see improvements in each detector.<br />
For each launch, independent builds of hdds, sim-recon, the monitoring plugins, and an sqlite file will be generated.<br />
<br />
Below the procedures are described for<br />
# Preparing the software for the launch<br />
# Starting the launch (using hdswif)<br />
# Post-analysis of statistics of the launch<br />
<br />
Processing the results and making them available to the collaboration<br />
is handled in the section [[#Post-Processing_Procedures | Post-Processing_Procedures]] below.<br />
<br />
<!--<br />
==== Generating an offline plugin job ====<br />
<br />
The user gxproj1 should be used for official offline monitoring jobs.<br />
Within /home/gxproj1/halld/monitoring/batch/ there will be scripts to run the online monitoring plugins over tape files.<br />
The main script is generatejobs_plugins_rawdata.sh, which can be used as<br />
generatejobs_plugins_rawdata.sh [minrun] [maxrun] (minfile) (maxfile)<br />
where minrun, maxrun specify the range of the run #, and minfile, maxfile (optional) specify<br />
the file range for that run.<br />
<br />
This will generate a script run_rawdata_XXXXXX.sh, where the run # has now been formatted to be 6 digits.<br />
Executing this script will send the monitoring plugins job to the Auger batch system.<br />
<br />
There is also a script clean.sh which can be used as<br />
clean.sh XXX<br />
This will clean up all associated files created in association with run XXX.<br />
<br />
Internally, the xml file used to submit the job will be created, and the job to run<br />
will be given within script.sh. All run parameters should be specified in at the beginning<br />
of generatejobs_plugins_rawdata.sh<br />
Since we are running on tape, the tape file will first be copied over to the cache disk, and the job will run<br />
over this cached file.<br />
<br />
==== Using cron to run automatically ====<br />
Within /home/gluex/halld/monitoring/cron/ there is a file cron_plugins<br />
that can be executed via<br />
crontab cron_plugins<br />
This will set up a cron job to call the script scan_for_jobs.sh, which will<br />
check in the rawdata directory and call generatejobs_plugins_rawdata.sh for<br />
any run that is more than 5 min old. The cron job is set up to run every 10 min.<br />
<br />
==== Magnetic Field Settings ====<br />
For tracking, it is necessary to set the correct magnetic field settings.<br />
The below table is now obsolete. Please refer to the [https://halldweb.jlab.org/cgi-bin/data_monitoring/run_conditions.pl GlueX Run Conditions] webpage.<br />
<br />
The actual field settings for each run have not been documented well.<br />
Below are values based on going through entries in the [https://logbooks.jlab.org/book/halld halld log].<br />
<br />
{| border="1" cellpadding="1" style="text-align: center;"<br />
!width="100"| Run #s<br />
!width="150"| Solenoid Current (A)<br />
!width="300"| JANA option<br />
!width="300"| notes<br />
|-<br />
| 940 - 996 || 1000 || -PBFIELD_MAP=Magnets/Solenoid/solenoid_1000A_poisson_20141104 || At run 997, the solenoid started ramping down<br />
|-<br />
| 998 - 1448 || 0 || -PBFIELD_TYPE=NoField -PDEFTAG:DTrackCandidate=StraightLine || Solenoid started ramping up around run 1431. Runs 1432 - 1448 should not be used for tracking. See [https://logbooks.jlab.org/entry/3309039 this] and [https://logbooks.jlab.org/entry/3309092 this] entry.<br />
|-<br />
| 1449 - 1620 || 1200 || -PBFIELD_MAP=Magnets/Solenoid/solenoid_1200A_poisson_20140520 || <br />
|-<br />
|}<br />
<br />
'''Note:''' The following run ranges were taken with a 300A field, and could be analyzed with the magnetic field map Magnets/Solenoid/solenoid_300A_poisson_20140819 , more study is needed to see if the straight line track fitter is needed or not: 1036 - 1053, 1065 - 1121, 1212? - 1254, 1309 - 1318 [1307,1308,1319-1329 were taken while the magnet was ramping].<br />
--><br />
<br />
=== Procedures ===<br />
This section explains how the offline monitoring should be run.<br />
Since we may want to simultaneously run offline monitoring for different run periods that require<br />
different environment variables, the scripts are set up so that a generic user can download the<br />
scripts and run them from anywhere. Most output directories for offline monitoring are created<br />
with group read/write permissions so that any Hall D group user has access to the contents,<br />
but there are some cases where use of the account that created the launch is necessary.<br />
<br />
The accounts used for offline monitoring are the gxprojN accounts created and maintained by<br />
Mark Ito (see [https://halldweb.jlab.org/wiki/index.php/GlueX-related_shared_accounts_on_the_JLab_CUE here]).<br />
As of October 2015, we have been using<br />
* gxproj1 for running over Fall 2014 data (deprecated since June 2015)<br />
* gxproj5 for running over Spring 2015 data<br />
<br />
Since the summer of 2015 we have transitioned from a system using Mark Ito's jproj scripts<br />
to integrating the swif system that Chris Larrieu (SciComp) has been developing. For offline monitoring,<br />
the hdswif system that Kei developed is used for launching the jobs, and the jproj system is used<br />
for meta-analysis of launch statistics.<br />
<br />
Both hdswif and jproj are maintained in svn:<br />
* hdswif: https://halldsvn.jlab.org/repos/trunk/scripts/monitoring/hdswif<br />
* jproj : https://halldsvn.jlab.org/repos/trunk/scripts/monitoring/jproj<br />
<br />
To run each package the directories will need to be checked out and the necessary<br />
scripts will need to be run.<br />
<br />
Below the procedures for offline monitoring are explained.<br />
<br />
# To stop the incoming-data cron job, first kill the cron job with cron -r. Also, delete all jobs that have not started yet. If these jobs are still alive, they will cause confusion over which software version they ran. Do <pre>crontab -l</pre> to list the current cron jobs, and run <pre>crontab -r</pre> to kill them.<br />
# svn update & rebuild HDDS with <pre>cd $HDDS_HOME</pre><pre> svn up</pre><pre> scons install</pre><br />
# svn update & rebuild sim-recon with <pre>cd $HALLD_HOME</pre> <pre> svn up</pre> <pre>cd src</pre> <pre>scons -c install</pre> <pre>scons install</pre> The "scons -c install" is necessary to clean out any old header files. NOTE THAT IT IS A GOOD IDEA TO DO A COMPLETE WIPEOUT AND CHECK OUT A NEW VERSION SO THAT STALE HEADER FILES ARE NOT PRESENT.<br />
# svn update & rebuild monitoring plugins with <pre>cd /home/gxproj1/builds/online/packages/SBMS</pre> <pre> svn up</pre> <pre>cd /home/gxproj1/builds/online/packages/monitoring/src/plugins</pre> <pre> svn up</pre> <pre>scons -u install</pre><br />
# Prepare the latest sqlite file with: <pre>cd $HOME/builds/</pre> <pre>$CCDB_HOME/scripts/mysql2sqlite/mysql2sqlite.sh -hhallddb.jlab.org -uccdb_user ccdb | sqlite3 ccdb.sqlite</pre><br />
# Note that the above steps must be done BEFORE launch project creation. This is because we will track the svn revisions of the libraries used, and this is done by extracting the svn information in each directory. Also, note that the system assumes that we have the topmost build directory (usually called GLUEX_TOP) to be $HOME/builds . Such an assumption is necessary to be able to extract information about the library locations automatically.<br />
# Create the appropriate project(s) and submit the jobs using the Hall D Job Management System, as detailed in the section below.<br />
# Restart cron jobs for immediate processing of runs coming in.<br />
<br />
== Hall D Job Management System ==<br />
<br />
This section details instructions on how to create and launch a set of jobs using the Hall-D Job Management System developed by Mark Ito. These instructions are generic: this system can be used for the weekly monitoring jobs, but can also be used for other sets of job launches as well. <br />
<br />
=== Database Table Overview ===<br />
<br />
* Job management database table (<project_name>): For each input file, keeps track of whether or not a job for it has been submitted, along with other optional fields. <br />
<br />
* Job status database table (<project_name>Job (no space)): For each job, keeps track of the job-id, the job status, memory used, cpu & wall time, time taken to complete various stages (e.g. pending, dependency, active), and others. <br />
<br />
* Job metrics database table (<project_name>_aux (no space)): For each job, keeps track of the job-id, how many events were processed, the time it took to copy the cache file, and the time it took to run the plugin. This information is culled from the log files of each job, and is done within the analysis directory of each launch.<br />
<br />
=== Initialize Project Management ===<br />
<br />
* Log into the ifarm machine with one of the gxproj accounts. For this example we will use gxproj1.<br />
<pre><br />
ssh gxproj1@ifarm -Y<br />
</pre><br />
<br />
* Go to a directory to do the launch. In principle, any directory will work, but for gxproj1 this is usually done in /home/gxproj1/halld/<br />
<br />
* Check out the necessary scripts <pre>svn co https://halldsvn.jlab.org/repos/trunk/scripts/monitoring/jproj</pre> This will get all necessary scripts for launching. Once checked out, <pre>cd projects</pre><br />
<br />
* The script create_project.sh can be used to create a new project. It will take a single argument, the project name. It is assumed that the project name is of the form offline_monitoring_RunPeriod20YY_MM_verVV_hd_rawdata where this string will be parsed to give the run period 20YY_MM and version number VV. One thing to do BEFORE creation of a new project is to editthe conditions of the launch (plugins to run over, memory requested, disk space requested) within templates/template.jsub . This<br />
information is saved automatically at project creation time into files in /group/halld/data_monitoring/run_conditions .<br />
To create a project do <pre>./create_project.sh offline_monitoring_RunPeriod20YY_MM_verVV_hd_rawdata</pre><br />
The name has been chosen to be as consistent as possible with other directory structures.<br />
However, mysql requires that "-" be escaped in table names, so unfortunately run periods will be<br />
given as 2YYY_MM instead of 2YYY-MM.<br />
<br />
* A new directory with the project name (offline_monitoring_RunPeriod20YY_MM_verVV_hd_rawdata) and files will be copied and modified from the template directory to reflect the run period, the user, the directory that it was created in, project name, etc.<br />
<br />
* For each project, cd into the new directory<br />
* The script clear.sh will remove any existing tables for the current project name, then recreate it. Do<br />
<pre>./clear.sh</pre><br />
* To use the jproj.pl script that was checked in, add the directory to your path with<br />
<pre>source ../../scripts/setup.csh</pre><br />
or always specify the full path<br />
<pre>../../scripts/jproj.pl</pre><br />
* Now update the table of runs with<br />
<pre>jproj.pl <project name> update</pre><br />
This will fill the table with all files within /mss that are of the same form as what is in <project name>.jproj . If you want to register only on a subset of all such files, you can edit this file directly.<br />
* Once you have registered all of the files you would like to run over, do<br />
<pre>jproj.pl <project name> submit [max # of jobs] [run number]</pre><br />
where the additional options specify how many jobs to submit and which run number to run on. Without these options all files that are registered and have not been submitted yet will be submitted.<br />
<br />
At this stage you are ready to submit all files. It is a good idea to submit a few test jobs<br />
at first to check that all scripts are working and that the plugins do not crash. Once you are<br />
sure that this does not happen, you can send all jobs in. The remaining jobs are then the monitoring<br />
which will (among other things) put the results on the online webpage for the collaboration to view, and<br />
the analysis of the launch.<br />
<br />
=== Project File Overview ===<br />
<br />
An overview of each project file:<br />
* '''clear.sh:''' For the current project, deletes the job status and management database tables (if any), and creates new, empty ones. <br />
* '''<project_name>.jproj:''' Contains the path and file name format for the input files for the jobs. <br />
* '''<project_name>.jsub:''' The xml job submission script. The run number and file number variables are set during job submission for each input file. <br />
* '''script.sh:''' The script that is executed during the job. If output job directories are not pre-created manually, they should be created in this script with the proper permissions:<br />
<pre><br />
mkdir -p -m 775 my_directory<br />
</pre><br />
* '''setup_jlab-[run period].csh:''' The environment that is sourced at the beginning of the job execution. <br />
* '''status.sh:''' Updates the job status database table, and prints some of its columns to screen.<br />
<br />
=== Project Management ===<br />
<br />
* Delete (if any) and create the database table(s) for the current set of job submissions:<br />
<pre><br />
./clear.sh<br />
</pre><br />
Also, if testing was done with jobs, it is best to delete the output directory and the configuration files:<br />
<pre><br />
rm -frv /volatile/halld/offline_monitoring/RunPeriod-20YY-MM/verVV /group/halld/data_monitoring/run_conditions/soft_comm_20YY_MM_verVV.xml /group/halld/data_monitoring/run_conditions/jana_rawdata_comm_20YY_MM_verVV.conf<br />
</pre><br />
<br />
* Search for input files matching the string in the .jproj file, and create a row for each in the job management database table (called <project_name>). You can test by adding an optional argument at the end, which only selects files with a specific file number:<br />
<pre><br />
jproj.pl <project_name> update <optional_file_number><br />
</pre><br />
<br />
* Confirm that the job management database is accurate by printing it's contents to screen:<br />
<pre><br />
mysql -hhallddb -ufarmer farming -e "select * from <project_name>"<br />
</pre><br />
<br />
* ONLY if a mistake was made, to delete the tables from the database and recreate new, empty ones, run: <br />
<pre><br />
./clear.sh<br />
</pre><br />
<br />
* To look at the status of the submitted jobs, first query auger and update the job status database:<br />
<pre><br />
fill_in_job_details.pl <project_name><br />
</pre><br />
<br />
* The job status can then be viewed by submitting a query to the job status database (called <project_name>Job (no space in between)):<br />
<pre><br />
mysql -hhallddb -ufarmer farming -e "select id,run,file,jobId,hostname,status,timeSubmitted,timeActive,walltime,cput,timeComplete,result,error from <project_name>Job"<br />
</pre><br />
<br />
* These last two commands can instead be executed simultaneously by running:<br />
<pre><br />
./status.sh<br />
</pre><br />
<br />
=== Handy mysql Instructions ===<br />
<br />
* Handy mysql instructions:<br />
<pre><br />
mysql -hhallddb -ufarmer farming # Enter the "farming" mysql database on "hallddb" as user "farmer"<br />
quit; # Exit mysql<br />
show tables; # Show a list of the tables in the current database<br />
show columns from <project_name>; # show all of the columns for the given table<br />
select * from <project_name>; # show the contents of all rows from the given table<br />
</pre><br />
<br />
=== Backing Up Offline Monitoring Tables ===<br />
Tables created for offline monitoring can be backed up using the script backup_tables.sh which<br />
can be checked out with the other files from https://halldsvn.jlab.org/repos/trunk/scripts/monitoring/jproj/projects<br />
<br />
The script uses the command mysqldump to print out a file that can be executed to recreate the tables.<br />
Since executing this output file will drop the table when it exists, caution is advised.<br />
Example usage to backup all three tables created for run period 2014_10 ver 17:<br />
<pre>backup_tables.sh 2014_10 17</pre><br />
<br />
== Running Over Data As It Comes In ==<br />
<br />
A special user gxproj1 will have a cron job set up to run the plugins as new data appears on /mss.<br />
During the week, gxproj1 will submit offline plugin jobs with the same setup as the weekly jobs<br />
run the previous Friday. The procedure for this is shown below.<br />
<br />
<!--<br />
=== Setting up the environment ===<br />
The file<br />
/home/gxproj1/setup_jlab.csh<br />
is sourced through .tcshrc.<br />
This file is the same as what is linked to by<br />
/home/gluex/setup_jlab_commissioning.csh,<br />
except HALLD_HOME, HDDS_HOME, and JANA_CALIB_URL are set separately so that this<br />
user can have a separate build.<br />
<br />
To obtain the builds from the previous Friday's runs,<br />
execute<br />
/home/gxproj1/halld/monitoring/newruns/setup_previous.sh [year] [month] [day]<br />
The build revisions from the previous Friday are archived in files<br />
/work/halld/data_monitoring/run_conditions/soft_comm_[year]_[month]_[day].xml<br />
and the script will build libraries based on those stored revision numbers.<br />
--><br />
<br />
=== Running the cron job ===<br />
<br />
'''IMPORTANT:''' The cron job should not be running while you are manually submitting jobs using the jproj.pl script for the same project, or else you will probably multiply-submit a job. <br />
<br />
* Go to the cron job directory: <br />
<pre><br />
cd /u/home/gxproj1/halld/monitoring/newruns<br />
</pre><br />
<br />
* The cron_plugins file is the cronjob that will be executed. During execution, it runs the exec.sh command in the same folder. This command takes two arguments: the project name, and the maximum file number for each run. These fields should be updated in the cron_plugins file before running. <br />
<br />
* The exec.sh command updates the job management database table with any data that has arrived on tape since it was last updated, ignoring file numbers greater than the maximum file number. It then submits jobs for these files. <br />
<br />
* To start the cron job, run:<br />
<pre><br />
crontab cron_plugins<br />
</pre><br />
<br />
* To check whether the cron job is running, do<br />
<pre><br />
crontab -l<br />
</pre><br />
<br />
* To remove the cron job do<br />
<pre><br />
crontab -r<br />
</pre><br />
<br />
<!--<br />
The cron job will run the script scan_for_jobs.sh,<br />
which runs generatejobs_plugins_rawdata.sh for any<br />
new runs that it had not seen before. All previous<br />
runs are recorded in the file filelists/files_current.txt<br />
so clear this to run over runs, or set the parameters<br />
MINRUN and MAXRUN which will set the range of runs submitted.<br />
--><br />
<br />
==Post-Processing Procedures==<br />
<br />
To visualize the monitoring data, we save images of selected histograms and store time series of selected quantities in a database, which are then displayed on a web page. This section describes how to generate the monitoring images and database information.<br />
<br />
The scripts used to generate this summary data are primarily run from /home/gxprojN/halld/monitoring/process . If you want a new copy of the scripts, e.g., for a new monitoring run, you should check the scripts out from SVN:<br />
<syntaxhighlight><br />
svn co https://halldsvn.jlab.org/repos/trunk/scripts/monitoring/process<br />
</syntaxhighlight><br />
Note that these scripts currently have some parameters which must be periodically set by hand.<br />
<br />
The default python version on most JLab machine does not have the modules to allow these scripts to connect to the MySQL database. To run these scripts, load the environment with the following command<br />
<syntaxhighlight><br />
source /home/gxproj1/halld/monitoring/process/monitoring_env.csh<br />
</syntaxhighlight><br />
<br />
===Online Monitoring===<br />
<br />
There are two primary scripts for running over the monitoring data generated by the online system and offline reconstruction. The online script can be run with either of the following commands:<br />
<syntaxhighlight><br />
/home/gluex/halld/monitoring/process/check_new_runs.py<br />
<br />
OR <br />
<br />
/home/gluex/halld/monitoring/process/check_new_runs.csh<br />
</syntaxhighlight><br />
The shell script sets up the environment properly to run the python script. To connect to the monitoring database on the JLab CUE, modules included in the local installation of python >= 2.7 are needed. The shell script is appropriate to use in a cron job. The cronjob is currently run under the "gluex" account.<br />
<br />
The online monitoring system copies a ROOT file containing the results of the online monitoring, and other configuration files into a directory accessible outside the counting house. This python script automatically checks for new ROOT files, which it will then automatically process. It contains several configuration variables that must be correctly set, which contains the location of input/output directories, etc...<br />
<br />
===Offline Monitoring===<br />
<br />
After the data is run over, the results should be processed, so that summary data is entered into the monitoring database and plots are made for the monitoring webpages. Currently, this processing is controlled by a cronjob that runs the following script:<br />
<syntaxhighlight><br />
/home/gxproj1/halld/monitoring/process/check_monitoring_data.csh <br />
</syntaxhighlight><br />
This script checks for new ROOT files, and only runs over those it hasn't processed yet. Since one monitoring ROOT file is produced for each EVIO file, whenever a new file is produced, the plots for the corresponding run are recreated and all the ROOT files for a run are combined into one file. Information is stored in the database on a per-file basis. <br />
<br />
Plots for the monitoring web page can be made from single histograms or multiple histograms using RootSpy macros. If you want to change the list of plots made, you must modify one of the following files:<br />
* histograms_to_monitor - specify either the name of the histogram or its the full ROOT path <br />
* macros_to_monitor - specify the full path to the RootSpy macro .C file<br />
<br />
When a new monitoring run is started, or the conditions are changed, the following steps should be taken to process the new files:<br />
# Add a new data version, as described below:<br />
# Change the following parameters in check_monitoring_data.csh:<br />
## JOBDATE should correspond to the ouptut date used by the job submission script<br />
## OUTPUTDIR should correspond to the directory corresponding to the run period and revision corresponding to the new version you just submitted. Presumably, this directory will be empty at the beginning.<br />
## Once you create a new data version as defined below, you should pass the needed information as a command line option. Currently this is done by the ARGS variable. For example, the argument "-v RunPeriod-2014-10,8" tells the monitoring scripts to look up the version corresponding to revision 8 of RunPeriod-2014-10 in the monitoring DB and to use to store the results.<br />
<br />
<syntaxhighlight><br />
Example configuration parameters:<br />
set JOBDATE=2015-01-09<br />
set INPUTDIR=/volatile/halld/RunPeriod-2014-10/offline_monitoring<br />
set OUTPUTDIR=/w/halld-scifs1a/data_monitoring/RunPeriod-2014-10/ver08<br />
set ARGS=" -v RunPeriod-2014-10,8 "<br />
</syntaxhighlight><br />
If you want to process the results manually, the data is processed using the following script:<br />
<syntaxhighlight><br />
./process_new_offline_data.py <input directory> <output directory><br />
<br />
EXAMPLE:<br />
<br />
./process_new_offline_data.py 2014-11-14 /volatile/halld/RunPeriod-2014-10/offline_monitoring/ /w/halld-scifs1a/data_monitoring/RunPeriod-2014-10/ver02<br />
</syntaxhighlight><br />
The python script takes several options to enable/disable various steps in the processing. Of interest is the "--force" option, which will run over all monitoring ROOT files, whether or not they've been previously identified.<br />
<br />
Every time a new reconstruction pass is performed, a new version number must be generated. To do this, prepare a version file as described below. Then run the register_new_version.py script to store the information in the database. The script will return a version number, which then should be set by hand in process_new_offline_data.py - future versions of the script will streamline this part of the procedure. An example of how to generate a new version is:<br />
<syntaxhighlight><br />
./register_new_version.py add /home/gxproj1/halld/monitoring/process/versions/vers_RunPeriod-2014-10_pass1.txt<br />
</syntaxhighlight><br />
<br />
<b>If you are running the offline monitoring by checking out the files in trunk/scripts/monitoring/jproj/projects/<br />
of the svn repository</b>, and created a project with<br />
create_project.sh [project name] hd_rawdata<br />
Then go to the directory [project name]/processing/<br />
and execute<br />
./run_processing.sh<br />
which will run register_new_version.py as well as check_monitoring_data.csh for that project.<br />
<br />
===Step-by-Step Instructions For Processing a New Monitoring Run===<br />
<br />
The monitoring runs are current run out of the gxproj1 and gxproj5 accounts. After an offline monitoring run has been successfully started on the batch farm, the following steps should be followed to setup the post-processing for these runs.<br />
<br />
# The post-processing scripts are stored in $HOME/halld/monitoring/process and are automatically run by cron.<br />
# Run "svn update" to bring any changes in. Be sure that the list of histograms and macros to plot are current.<br />
# Edit check_monitoring_data.csh to point to the current revisions/directories<br />
#* VERSION<br />
#* ARGS<br />
#* Note that the environment depends on a standard script - $HOME/setup_jlab.csh<br />
# Update files in the web directory, so that the results are displayed on the web pages: /group/halld/www/halldweb/html/data_monitoring/textdata<br />
<br />
Check log files in $HOME/halld/monitoring/process/log for more information on how each run went. If there are problems, check log files, and modify check_monitoring_data.csh to vary the verbosity of the output.<br />
<br />
==Data Versions==<br />
<br />
To document the conditions of the monitoring data that is created, for the sake of reproducability and further analysis we save several pieces of information. The format is intended to be comprehensive enough to document not just monitoring data, but versions of raw and reconstructed data, so that this database table can be used for the event database as well.<br />
<br />
We store one record per pass through one run period, with the following structure:<br />
<br />
{| class="wikitable"<br />
! Field !! Description<br />
|-<br />
| data_type || The level of data we are processing. For the purposes of monitoring, "rawdata" is the online monitoring, "recon" is the offline monitoring<br />
|- <br />
| run_period || The run period of the data<br />
|- <br />
| revision || An integer specifying which pass through the run period this data corresponds to<br />
|- <br />
| software_version || The name of the XML file that specifies the different software versions used<br />
|-<br />
| jana_config || The name of the text file that specifies which JANA options were passed to the reconstruction program<br />
|-<br />
| ccdb_context || The value of JANA_CALIB_CONTEXT, which specifies the version of calibration constants that were used<br />
|-<br />
| production_time || The data at which monitoring/reconstruction began<br />
|-<br />
| dataVersionString || A convenient string for identifying this version of the data<br />
|}<br />
<br />
<br />
An example file used as as input to ./register_new_version.py is:<br />
<syntaxhighlight><br />
data_type = recon<br />
run_period = RunPeriod-2014-10<br />
revision = 1<br />
software_version = soft_comm_2014_11_06.xml<br />
jana_config = jana_rawdata_comm_2014_11_06.conf<br />
ccdb_context = calibtime=2014-11-10<br />
production_time = 2014-11-10<br />
dataVersionString = recon_RunPeriod-2014-10_20141110_ver01<br />
</syntaxhighlight></div>Kmoriyahttps://halldweb.jlab.org/wiki/index.php?title=Data_Monitoring_Procedures&diff=70723Data Monitoring Procedures2015-10-14T17:47:45Z<p>Kmoriya: /* Offline Monitoring: Running Over Archived Data */</p>
<hr />
<div>__TOC__<br />
<br />
== Master List of File / Database / Webpage Locations ==<br />
=== Run Conditions ===<br />
* Online Run-by-run condition files (B-field, current, etc.): /work/halld/online_monitoring/conditions/<br />
* Offline monitoring run conditions (software versions, jana config): /group/halld/data_monitoring/run_conditions/<br />
*[http://www.jlab.org/Hall-D/test/RunInfo/ Run Info vers. 1]<br />
*[https://halldweb.jlab.org/cgi-bin/data_monitoring/run_conditions.pl Run Info vers. 2]<br />
<br />
=== Monitoring Output Files ===<br />
* Run Periods 201Y-MM is for example 2015-03, launch ver verVV is for example ver15<br />
* Online monitoring histograms: /work/halld/online_monitoring/root/<br />
* Offline monitoring histogram ROOT files: /work/halld/data_monitoring/RunPeriod-201Y-MM/verVV/rootfiles<br />
* REST files (most recent launch only): /work/halld/data_monitoring/RunPeriod-201Y-MM/REST/verVV<br />
* individual files for each job (ROOT, log, etc.): /volatile/halld/offline_monitoring/RunPeriod-201Y-MM/verVV/<br />
<br />
=== Monitoring Database ===<br />
* Accessing monitoring database (on ifarm): mysql -u datmon -h hallddb.jlab.org data_monitoring<br />
<br />
=== Monitoring Webpages ===<br />
*[https://halldweb.jlab.org/cgi-bin/data_monitoring/monitoring/plotBrowser.py Plot Browser]<br />
*[https://halldweb.jlab.org/cgi-bin/data_monitoring/monitoring/runBrowser.py Run Browser]<br />
*[https://halldweb.jlab.org/cgi-bin/data_monitoring/monitoring/timeSeries.py Time Series]<br />
*[https://halldweb.jlab.org/data_monitoring/launch_analysis/ Launch Analysis]<br />
<br />
== Job Monitoring Links ==<br />
* [http://scicomp.jlab.org/farm2/job.html Completed Job History]<br />
* [http://scicomp.jlab.org/farm2/project.html Job Stats By Project]<br />
* [http://scicomp.jlab.org/farm2/trackOrg.html Job Stats By Track]<br />
* [http://scicomp.jlab.org/farm2/report.html Cluster Report]<br />
* [http://scicomp.jlab.org/farm2/walltime.html Walltime Distribution]<br />
<br />
== Saving Online Monitoring Data ==<br />
<br />
The procedure for writing the data out is given in, e.g.,<br />
[https://halldweb.jlab.org/wiki/index.php/Raid-to-Silo_Transfer_Strategy Raid-to-Silo Transfer Strategy].<br />
<br />
Once the DAQ writes out the data to the raid disk, cron jobs will copy the file to tape,<br />
and within ~20 min., we will have access to the file on tape at<br />
/mss/halld/$RUN_PERIOD/rawdata/RunXXXXXX.<br />
<br />
All online monitoring plugins will be run as data is taken.<br />
They will be accessible within the counting house via RootSpy, and<br />
for each run and file, a ROOT file containing the histograms will be saved<br />
within a subdirectory for each run.<br />
<br />
For immediate access to these files, the raid disk files may be accessed directly<br />
from the counting house, or the tape files will be available within ~20 min. of the<br />
file being written out.<br />
<br />
== Offline Monitoring: Running Over Archived Data ==<br />
<br />
Once files are written to tape we run the online plugins on these files to confirm what we were seeing in the online monitoring, and also to update the results from the latest calibration and software. <br />
Manual scripts and cron jobs are set up to look for new data and run the plugins over a sample of files.<br />
<br />
Every other Friday (usually the Friday before the offline meetings) jobs will be started to run the newest software on all previous runs,and allows everybody to see improvements in each detector.<br />
For each launch, independent builds of hdds, sim-recon, the monitoring plugins, and an sqlite file will be generated.<br />
<br />
Below the procedures are described for<br />
# Preparing the software for the launch<br />
# Starting the launch (using hdswif)<br />
# Post-analysis of statistics of the launch<br />
<br />
Processing the results and making them available to the collaboration<br />
is handled in the section [[#Post-Processing_Procedures | Post-Processing_Procedures]] below.<br />
<br />
<!--<br />
==== Generating an offline plugin job ====<br />
<br />
The user gxproj1 should be used for official offline monitoring jobs.<br />
Within /home/gxproj1/halld/monitoring/batch/ there will be scripts to run the online monitoring plugins over tape files.<br />
The main script is generatejobs_plugins_rawdata.sh, which can be used as<br />
generatejobs_plugins_rawdata.sh [minrun] [maxrun] (minfile) (maxfile)<br />
where minrun, maxrun specify the range of the run #, and minfile, maxfile (optional) specify<br />
the file range for that run.<br />
<br />
This will generate a script run_rawdata_XXXXXX.sh, where the run # has now been formatted to be 6 digits.<br />
Executing this script will send the monitoring plugins job to the Auger batch system.<br />
<br />
There is also a script clean.sh which can be used as<br />
clean.sh XXX<br />
This will clean up all associated files created in association with run XXX.<br />
<br />
Internally, the xml file used to submit the job will be created, and the job to run<br />
will be given within script.sh. All run parameters should be specified in at the beginning<br />
of generatejobs_plugins_rawdata.sh<br />
Since we are running on tape, the tape file will first be copied over to the cache disk, and the job will run<br />
over this cached file.<br />
<br />
==== Using cron to run automatically ====<br />
Within /home/gluex/halld/monitoring/cron/ there is a file cron_plugins<br />
that can be executed via<br />
crontab cron_plugins<br />
This will set up a cron job to call the script scan_for_jobs.sh, which will<br />
check in the rawdata directory and call generatejobs_plugins_rawdata.sh for<br />
any run that is more than 5 min old. The cron job is set up to run every 10 min.<br />
<br />
==== Magnetic Field Settings ====<br />
For tracking, it is necessary to set the correct magnetic field settings.<br />
The below table is now obsolete. Please refer to the [https://halldweb.jlab.org/cgi-bin/data_monitoring/run_conditions.pl GlueX Run Conditions] webpage.<br />
<br />
The actual field settings for each run have not been documented well.<br />
Below are values based on going through entries in the [https://logbooks.jlab.org/book/halld halld log].<br />
<br />
{| border="1" cellpadding="1" style="text-align: center;"<br />
!width="100"| Run #s<br />
!width="150"| Solenoid Current (A)<br />
!width="300"| JANA option<br />
!width="300"| notes<br />
|-<br />
| 940 - 996 || 1000 || -PBFIELD_MAP=Magnets/Solenoid/solenoid_1000A_poisson_20141104 || At run 997, the solenoid started ramping down<br />
|-<br />
| 998 - 1448 || 0 || -PBFIELD_TYPE=NoField -PDEFTAG:DTrackCandidate=StraightLine || Solenoid started ramping up around run 1431. Runs 1432 - 1448 should not be used for tracking. See [https://logbooks.jlab.org/entry/3309039 this] and [https://logbooks.jlab.org/entry/3309092 this] entry.<br />
|-<br />
| 1449 - 1620 || 1200 || -PBFIELD_MAP=Magnets/Solenoid/solenoid_1200A_poisson_20140520 || <br />
|-<br />
|}<br />
<br />
'''Note:''' The following run ranges were taken with a 300A field, and could be analyzed with the magnetic field map Magnets/Solenoid/solenoid_300A_poisson_20140819 , more study is needed to see if the straight line track fitter is needed or not: 1036 - 1053, 1065 - 1121, 1212? - 1254, 1309 - 1318 [1307,1308,1319-1329 were taken while the magnet was ramping].<br />
--><br />
<br />
=== Procedures ===<br />
This section explains how the offline monitoring should be run.<br />
Since we may want to simultaneously run offline monitoring for different run periods that require<br />
different environment variables, the scripts are set up so that a generic user can download the<br />
scripts and run them from anywhere.<br />
<br />
To do this, you will check out a directory from svn that contains all the necessary scripts,<br />
and running a generation script will generate all the necessary files for the present launch.<br />
The main engine behind keeping track of all the files submitted and their status is the jproj<br />
system created by Mark Ito.<br />
<br />
Below is the process of creating and running a launch.<br />
<br />
<br />
# To stop the incoming-data cron job, first kill the cron job with cron -r. Also, delete all jobs that have not started yet. If these jobs are still alive, they will cause confusion over which software version they ran. Do <pre>crontab -l</pre> to list the current cron jobs, and run <pre>crontab -r</pre> to kill them.<br />
# svn update & rebuild HDDS with <pre>cd $HDDS_HOME</pre><pre> svn up</pre><pre> scons install</pre><br />
# svn update & rebuild sim-recon with <pre>cd $HALLD_HOME</pre> <pre> svn up</pre> <pre>cd src</pre> <pre>scons -c install</pre> <pre>scons install</pre> The "scons -c install" is necessary to clean out any old header files. NOTE THAT IT IS A GOOD IDEA TO DO A COMPLETE WIPEOUT AND CHECK OUT A NEW VERSION SO THAT STALE HEADER FILES ARE NOT PRESENT.<br />
# svn update & rebuild monitoring plugins with <pre>cd /home/gxproj1/builds/online/packages/SBMS</pre> <pre> svn up</pre> <pre>cd /home/gxproj1/builds/online/packages/monitoring/src/plugins</pre> <pre> svn up</pre> <pre>scons -u install</pre><br />
# Prepare the latest sqlite file with: <pre>cd $HOME/builds/</pre> <pre>$CCDB_HOME/scripts/mysql2sqlite/mysql2sqlite.sh -hhallddb.jlab.org -uccdb_user ccdb | sqlite3 ccdb.sqlite</pre><br />
# Note that the above steps must be done BEFORE launch project creation. This is because we will track the svn revisions of the libraries used, and this is done by extracting the svn information in each directory. Also, note that the system assumes that we have the topmost build directory (usually called GLUEX_TOP) to be $HOME/builds . Such an assumption is necessary to be able to extract information about the library locations automatically.<br />
# Create the appropriate project(s) and submit the jobs using the Hall D Job Management System, as detailed in the section below.<br />
# Restart cron jobs for immediate processing of runs coming in.<br />
<br />
== Hall D Job Management System ==<br />
<br />
This section details instructions on how to create and launch a set of jobs using the Hall-D Job Management System developed by Mark Ito. These instructions are generic: this system can be used for the weekly monitoring jobs, but can also be used for other sets of job launches as well. <br />
<br />
=== Database Table Overview ===<br />
<br />
* Job management database table (<project_name>): For each input file, keeps track of whether or not a job for it has been submitted, along with other optional fields. <br />
<br />
* Job status database table (<project_name>Job (no space)): For each job, keeps track of the job-id, the job status, memory used, cpu & wall time, time taken to complete various stages (e.g. pending, dependency, active), and others. <br />
<br />
* Job metrics database table (<project_name>_aux (no space)): For each job, keeps track of the job-id, how many events were processed, the time it took to copy the cache file, and the time it took to run the plugin. This information is culled from the log files of each job, and is done within the analysis directory of each launch.<br />
<br />
=== Initialize Project Management ===<br />
<br />
* Log into the ifarm machine with one of the gxproj accounts. For this example we will use gxproj1.<br />
<pre><br />
ssh gxproj1@ifarm -Y<br />
</pre><br />
<br />
* Go to a directory to do the launch. In principle, any directory will work, but for gxproj1 this is usually done in /home/gxproj1/halld/<br />
<br />
* Check out the necessary scripts <pre>svn co https://halldsvn.jlab.org/repos/trunk/scripts/monitoring/jproj</pre> This will get all necessary scripts for launching. Once checked out, <pre>cd projects</pre><br />
<br />
* The script create_project.sh can be used to create a new project. It will take a single argument, the project name. It is assumed that the project name is of the form offline_monitoring_RunPeriod20YY_MM_verVV_hd_rawdata where this string will be parsed to give the run period 20YY_MM and version number VV. One thing to do BEFORE creation of a new project is to editthe conditions of the launch (plugins to run over, memory requested, disk space requested) within templates/template.jsub . This<br />
information is saved automatically at project creation time into files in /group/halld/data_monitoring/run_conditions .<br />
To create a project do <pre>./create_project.sh offline_monitoring_RunPeriod20YY_MM_verVV_hd_rawdata</pre><br />
The name has been chosen to be as consistent as possible with other directory structures.<br />
However, mysql requires that "-" be escaped in table names, so unfortunately run periods will be<br />
given as 2YYY_MM instead of 2YYY-MM.<br />
<br />
* A new directory with the project name (offline_monitoring_RunPeriod20YY_MM_verVV_hd_rawdata) and files will be copied and modified from the template directory to reflect the run period, the user, the directory that it was created in, project name, etc.<br />
<br />
* For each project, cd into the new directory<br />
* The script clear.sh will remove any existing tables for the current project name, then recreate it. Do<br />
<pre>./clear.sh</pre><br />
* To use the jproj.pl script that was checked in, add the directory to your path with<br />
<pre>source ../../scripts/setup.csh</pre><br />
or always specify the full path<br />
<pre>../../scripts/jproj.pl</pre><br />
* Now update the table of runs with<br />
<pre>jproj.pl <project name> update</pre><br />
This will fill the table with all files within /mss that are of the same form as what is in <project name>.jproj . If you want to register only on a subset of all such files, you can edit this file directly.<br />
* Once you have registered all of the files you would like to run over, do<br />
<pre>jproj.pl <project name> submit [max # of jobs] [run number]</pre><br />
where the additional options specify how many jobs to submit and which run number to run on. Without these options all files that are registered and have not been submitted yet will be submitted.<br />
<br />
At this stage you are ready to submit all files. It is a good idea to submit a few test jobs<br />
at first to check that all scripts are working and that the plugins do not crash. Once you are<br />
sure that this does not happen, you can send all jobs in. The remaining jobs are then the monitoring<br />
which will (among other things) put the results on the online webpage for the collaboration to view, and<br />
the analysis of the launch.<br />
<br />
=== Project File Overview ===<br />
<br />
An overview of each project file:<br />
* '''clear.sh:''' For the current project, deletes the job status and management database tables (if any), and creates new, empty ones. <br />
* '''<project_name>.jproj:''' Contains the path and file name format for the input files for the jobs. <br />
* '''<project_name>.jsub:''' The xml job submission script. The run number and file number variables are set during job submission for each input file. <br />
* '''script.sh:''' The script that is executed during the job. If output job directories are not pre-created manually, they should be created in this script with the proper permissions:<br />
<pre><br />
mkdir -p -m 775 my_directory<br />
</pre><br />
* '''setup_jlab-[run period].csh:''' The environment that is sourced at the beginning of the job execution. <br />
* '''status.sh:''' Updates the job status database table, and prints some of its columns to screen.<br />
<br />
=== Project Management ===<br />
<br />
* Delete (if any) and create the database table(s) for the current set of job submissions:<br />
<pre><br />
./clear.sh<br />
</pre><br />
Also, if testing was done with jobs, it is best to delete the output directory and the configuration files:<br />
<pre><br />
rm -frv /volatile/halld/offline_monitoring/RunPeriod-20YY-MM/verVV /group/halld/data_monitoring/run_conditions/soft_comm_20YY_MM_verVV.xml /group/halld/data_monitoring/run_conditions/jana_rawdata_comm_20YY_MM_verVV.conf<br />
</pre><br />
<br />
* Search for input files matching the string in the .jproj file, and create a row for each in the job management database table (called <project_name>). You can test by adding an optional argument at the end, which only selects files with a specific file number:<br />
<pre><br />
jproj.pl <project_name> update <optional_file_number><br />
</pre><br />
<br />
* Confirm that the job management database is accurate by printing it's contents to screen:<br />
<pre><br />
mysql -hhallddb -ufarmer farming -e "select * from <project_name>"<br />
</pre><br />
<br />
* ONLY if a mistake was made, to delete the tables from the database and recreate new, empty ones, run: <br />
<pre><br />
./clear.sh<br />
</pre><br />
<br />
* To look at the status of the submitted jobs, first query auger and update the job status database:<br />
<pre><br />
fill_in_job_details.pl <project_name><br />
</pre><br />
<br />
* The job status can then be viewed by submitting a query to the job status database (called <project_name>Job (no space in between)):<br />
<pre><br />
mysql -hhallddb -ufarmer farming -e "select id,run,file,jobId,hostname,status,timeSubmitted,timeActive,walltime,cput,timeComplete,result,error from <project_name>Job"<br />
</pre><br />
<br />
* These last two commands can instead be executed simultaneously by running:<br />
<pre><br />
./status.sh<br />
</pre><br />
<br />
=== Handy mysql Instructions ===<br />
<br />
* Handy mysql instructions:<br />
<pre><br />
mysql -hhallddb -ufarmer farming # Enter the "farming" mysql database on "hallddb" as user "farmer"<br />
quit; # Exit mysql<br />
show tables; # Show a list of the tables in the current database<br />
show columns from <project_name>; # show all of the columns for the given table<br />
select * from <project_name>; # show the contents of all rows from the given table<br />
</pre><br />
<br />
=== Backing Up Offline Monitoring Tables ===<br />
Tables created for offline monitoring can be backed up using the script backup_tables.sh which<br />
can be checked out with the other files from https://halldsvn.jlab.org/repos/trunk/scripts/monitoring/jproj/projects<br />
<br />
The script uses the command mysqldump to print out a file that can be executed to recreate the tables.<br />
Since executing this output file will drop the table when it exists, caution is advised.<br />
Example usage to backup all three tables created for run period 2014_10 ver 17:<br />
<pre>backup_tables.sh 2014_10 17</pre><br />
<br />
== Running Over Data As It Comes In ==<br />
<br />
A special user gxproj1 will have a cron job set up to run the plugins as new data appears on /mss.<br />
During the week, gxproj1 will submit offline plugin jobs with the same setup as the weekly jobs<br />
run the previous Friday. The procedure for this is shown below.<br />
<br />
<!--<br />
=== Setting up the environment ===<br />
The file<br />
/home/gxproj1/setup_jlab.csh<br />
is sourced through .tcshrc.<br />
This file is the same as what is linked to by<br />
/home/gluex/setup_jlab_commissioning.csh,<br />
except HALLD_HOME, HDDS_HOME, and JANA_CALIB_URL are set separately so that this<br />
user can have a separate build.<br />
<br />
To obtain the builds from the previous Friday's runs,<br />
execute<br />
/home/gxproj1/halld/monitoring/newruns/setup_previous.sh [year] [month] [day]<br />
The build revisions from the previous Friday are archived in files<br />
/work/halld/data_monitoring/run_conditions/soft_comm_[year]_[month]_[day].xml<br />
and the script will build libraries based on those stored revision numbers.<br />
--><br />
<br />
=== Running the cron job ===<br />
<br />
'''IMPORTANT:''' The cron job should not be running while you are manually submitting jobs using the jproj.pl script for the same project, or else you will probably multiply-submit a job. <br />
<br />
* Go to the cron job directory: <br />
<pre><br />
cd /u/home/gxproj1/halld/monitoring/newruns<br />
</pre><br />
<br />
* The cron_plugins file is the cronjob that will be executed. During execution, it runs the exec.sh command in the same folder. This command takes two arguments: the project name, and the maximum file number for each run. These fields should be updated in the cron_plugins file before running. <br />
<br />
* The exec.sh command updates the job management database table with any data that has arrived on tape since it was last updated, ignoring file numbers greater than the maximum file number. It then submits jobs for these files. <br />
<br />
* To start the cron job, run:<br />
<pre><br />
crontab cron_plugins<br />
</pre><br />
<br />
* To check whether the cron job is running, do<br />
<pre><br />
crontab -l<br />
</pre><br />
<br />
* To remove the cron job do<br />
<pre><br />
crontab -r<br />
</pre><br />
<br />
<!--<br />
The cron job will run the script scan_for_jobs.sh,<br />
which runs generatejobs_plugins_rawdata.sh for any<br />
new runs that it had not seen before. All previous<br />
runs are recorded in the file filelists/files_current.txt<br />
so clear this to run over runs, or set the parameters<br />
MINRUN and MAXRUN which will set the range of runs submitted.<br />
--><br />
<br />
==Post-Processing Procedures==<br />
<br />
To visualize the monitoring data, we save images of selected histograms and store time series of selected quantities in a database, which are then displayed on a web page. This section describes how to generate the monitoring images and database information.<br />
<br />
The scripts used to generate this summary data are primarily run from /home/gxprojN/halld/monitoring/process . If you want a new copy of the scripts, e.g., for a new monitoring run, you should check the scripts out from SVN:<br />
<syntaxhighlight><br />
svn co https://halldsvn.jlab.org/repos/trunk/scripts/monitoring/process<br />
</syntaxhighlight><br />
Note that these scripts currently have some parameters which must be periodically set by hand.<br />
<br />
The default python version on most JLab machine does not have the modules to allow these scripts to connect to the MySQL database. To run these scripts, load the environment with the following command<br />
<syntaxhighlight><br />
source /home/gxproj1/halld/monitoring/process/monitoring_env.csh<br />
</syntaxhighlight><br />
<br />
===Online Monitoring===<br />
<br />
There are two primary scripts for running over the monitoring data generated by the online system and offline reconstruction. The online script can be run with either of the following commands:<br />
<syntaxhighlight><br />
/home/gluex/halld/monitoring/process/check_new_runs.py<br />
<br />
OR <br />
<br />
/home/gluex/halld/monitoring/process/check_new_runs.csh<br />
</syntaxhighlight><br />
The shell script sets up the environment properly to run the python script. To connect to the monitoring database on the JLab CUE, modules included in the local installation of python >= 2.7 are needed. The shell script is appropriate to use in a cron job. The cronjob is currently run under the "gluex" account.<br />
<br />
The online monitoring system copies a ROOT file containing the results of the online monitoring, and other configuration files into a directory accessible outside the counting house. This python script automatically checks for new ROOT files, which it will then automatically process. It contains several configuration variables that must be correctly set, which contains the location of input/output directories, etc...<br />
<br />
===Offline Monitoring===<br />
<br />
After the data is run over, the results should be processed, so that summary data is entered into the monitoring database and plots are made for the monitoring webpages. Currently, this processing is controlled by a cronjob that runs the following script:<br />
<syntaxhighlight><br />
/home/gxproj1/halld/monitoring/process/check_monitoring_data.csh <br />
</syntaxhighlight><br />
This script checks for new ROOT files, and only runs over those it hasn't processed yet. Since one monitoring ROOT file is produced for each EVIO file, whenever a new file is produced, the plots for the corresponding run are recreated and all the ROOT files for a run are combined into one file. Information is stored in the database on a per-file basis. <br />
<br />
Plots for the monitoring web page can be made from single histograms or multiple histograms using RootSpy macros. If you want to change the list of plots made, you must modify one of the following files:<br />
* histograms_to_monitor - specify either the name of the histogram or its the full ROOT path <br />
* macros_to_monitor - specify the full path to the RootSpy macro .C file<br />
<br />
When a new monitoring run is started, or the conditions are changed, the following steps should be taken to process the new files:<br />
# Add a new data version, as described below:<br />
# Change the following parameters in check_monitoring_data.csh:<br />
## JOBDATE should correspond to the ouptut date used by the job submission script<br />
## OUTPUTDIR should correspond to the directory corresponding to the run period and revision corresponding to the new version you just submitted. Presumably, this directory will be empty at the beginning.<br />
## Once you create a new data version as defined below, you should pass the needed information as a command line option. Currently this is done by the ARGS variable. For example, the argument "-v RunPeriod-2014-10,8" tells the monitoring scripts to look up the version corresponding to revision 8 of RunPeriod-2014-10 in the monitoring DB and to use to store the results.<br />
<br />
<syntaxhighlight><br />
Example configuration parameters:<br />
set JOBDATE=2015-01-09<br />
set INPUTDIR=/volatile/halld/RunPeriod-2014-10/offline_monitoring<br />
set OUTPUTDIR=/w/halld-scifs1a/data_monitoring/RunPeriod-2014-10/ver08<br />
set ARGS=" -v RunPeriod-2014-10,8 "<br />
</syntaxhighlight><br />
If you want to process the results manually, the data is processed using the following script:<br />
<syntaxhighlight><br />
./process_new_offline_data.py <input directory> <output directory><br />
<br />
EXAMPLE:<br />
<br />
./process_new_offline_data.py 2014-11-14 /volatile/halld/RunPeriod-2014-10/offline_monitoring/ /w/halld-scifs1a/data_monitoring/RunPeriod-2014-10/ver02<br />
</syntaxhighlight><br />
The python script takes several options to enable/disable various steps in the processing. Of interest is the "--force" option, which will run over all monitoring ROOT files, whether or not they've been previously identified.<br />
<br />
Every time a new reconstruction pass is performed, a new version number must be generated. To do this, prepare a version file as described below. Then run the register_new_version.py script to store the information in the database. The script will return a version number, which then should be set by hand in process_new_offline_data.py - future versions of the script will streamline this part of the procedure. An example of how to generate a new version is:<br />
<syntaxhighlight><br />
./register_new_version.py add /home/gxproj1/halld/monitoring/process/versions/vers_RunPeriod-2014-10_pass1.txt<br />
</syntaxhighlight><br />
<br />
<b>If you are running the offline monitoring by checking out the files in trunk/scripts/monitoring/jproj/projects/<br />
of the svn repository</b>, and created a project with<br />
create_project.sh [project name] hd_rawdata<br />
Then go to the directory [project name]/processing/<br />
and execute<br />
./run_processing.sh<br />
which will run register_new_version.py as well as check_monitoring_data.csh for that project.<br />
<br />
===Step-by-Step Instructions For Processing a New Monitoring Run===<br />
<br />
The monitoring runs are current run out of the gxproj1 and gxproj5 accounts. After an offline monitoring run has been successfully started on the batch farm, the following steps should be followed to setup the post-processing for these runs.<br />
<br />
# The post-processing scripts are stored in $HOME/halld/monitoring/process and are automatically run by cron.<br />
# Run "svn update" to bring any changes in. Be sure that the list of histograms and macros to plot are current.<br />
# Edit check_monitoring_data.csh to point to the current revisions/directories<br />
#* VERSION<br />
#* ARGS<br />
#* Note that the environment depends on a standard script - $HOME/setup_jlab.csh<br />
# Update files in the web directory, so that the results are displayed on the web pages: /group/halld/www/halldweb/html/data_monitoring/textdata<br />
<br />
Check log files in $HOME/halld/monitoring/process/log for more information on how each run went. If there are problems, check log files, and modify check_monitoring_data.csh to vary the verbosity of the output.<br />
<br />
==Data Versions==<br />
<br />
To document the conditions of the monitoring data that is created, for the sake of reproducability and further analysis we save several pieces of information. The format is intended to be comprehensive enough to document not just monitoring data, but versions of raw and reconstructed data, so that this database table can be used for the event database as well.<br />
<br />
We store one record per pass through one run period, with the following structure:<br />
<br />
{| class="wikitable"<br />
! Field !! Description<br />
|-<br />
| data_type || The level of data we are processing. For the purposes of monitoring, "rawdata" is the online monitoring, "recon" is the offline monitoring<br />
|- <br />
| run_period || The run period of the data<br />
|- <br />
| revision || An integer specifying which pass through the run period this data corresponds to<br />
|- <br />
| software_version || The name of the XML file that specifies the different software versions used<br />
|-<br />
| jana_config || The name of the text file that specifies which JANA options were passed to the reconstruction program<br />
|-<br />
| ccdb_context || The value of JANA_CALIB_CONTEXT, which specifies the version of calibration constants that were used<br />
|-<br />
| production_time || The data at which monitoring/reconstruction began<br />
|-<br />
| dataVersionString || A convenient string for identifying this version of the data<br />
|}<br />
<br />
<br />
An example file used as as input to ./register_new_version.py is:<br />
<syntaxhighlight><br />
data_type = recon<br />
run_period = RunPeriod-2014-10<br />
revision = 1<br />
software_version = soft_comm_2014_11_06.xml<br />
jana_config = jana_rawdata_comm_2014_11_06.conf<br />
ccdb_context = calibtime=2014-11-10<br />
production_time = 2014-11-10<br />
dataVersionString = recon_RunPeriod-2014-10_20141110_ver01<br />
</syntaxhighlight></div>Kmoriyahttps://halldweb.jlab.org/wiki/index.php?title=Data_Monitoring_Procedures&diff=70722Data Monitoring Procedures2015-10-14T17:44:23Z<p>Kmoriya: /* Offline Monitoring: Running Over Archived Data */</p>
<hr />
<div>__TOC__<br />
<br />
== Master List of File / Database / Webpage Locations ==<br />
=== Run Conditions ===<br />
* Online Run-by-run condition files (B-field, current, etc.): /work/halld/online_monitoring/conditions/<br />
* Offline monitoring run conditions (software versions, jana config): /group/halld/data_monitoring/run_conditions/<br />
*[http://www.jlab.org/Hall-D/test/RunInfo/ Run Info vers. 1]<br />
*[https://halldweb.jlab.org/cgi-bin/data_monitoring/run_conditions.pl Run Info vers. 2]<br />
<br />
=== Monitoring Output Files ===<br />
* Run Periods 201Y-MM is for example 2015-03, launch ver verVV is for example ver15<br />
* Online monitoring histograms: /work/halld/online_monitoring/root/<br />
* Offline monitoring histogram ROOT files: /work/halld/data_monitoring/RunPeriod-201Y-MM/verVV/rootfiles<br />
* REST files (most recent launch only): /work/halld/data_monitoring/RunPeriod-201Y-MM/REST/verVV<br />
* individual files for each job (ROOT, log, etc.): /volatile/halld/offline_monitoring/RunPeriod-201Y-MM/verVV/<br />
<br />
=== Monitoring Database ===<br />
* Accessing monitoring database (on ifarm): mysql -u datmon -h hallddb.jlab.org data_monitoring<br />
<br />
=== Monitoring Webpages ===<br />
*[https://halldweb.jlab.org/cgi-bin/data_monitoring/monitoring/plotBrowser.py Plot Browser]<br />
*[https://halldweb.jlab.org/cgi-bin/data_monitoring/monitoring/runBrowser.py Run Browser]<br />
*[https://halldweb.jlab.org/cgi-bin/data_monitoring/monitoring/timeSeries.py Time Series]<br />
*[https://halldweb.jlab.org/data_monitoring/launch_analysis/ Launch Analysis]<br />
<br />
== Job Monitoring Links ==<br />
* [http://scicomp.jlab.org/farm2/job.html Completed Job History]<br />
* [http://scicomp.jlab.org/farm2/project.html Job Stats By Project]<br />
* [http://scicomp.jlab.org/farm2/trackOrg.html Job Stats By Track]<br />
* [http://scicomp.jlab.org/farm2/report.html Cluster Report]<br />
* [http://scicomp.jlab.org/farm2/walltime.html Walltime Distribution]<br />
<br />
== Saving Online Monitoring Data ==<br />
<br />
The procedure for writing the data out is given in, e.g.,<br />
[https://halldweb.jlab.org/wiki/index.php/Raid-to-Silo_Transfer_Strategy Raid-to-Silo Transfer Strategy].<br />
<br />
Once the DAQ writes out the data to the raid disk, cron jobs will copy the file to tape,<br />
and within ~20 min., we will have access to the file on tape at<br />
/mss/halld/$RUN_PERIOD/rawdata/RunXXXXXX.<br />
<br />
All online monitoring plugins will be run as data is taken.<br />
They will be accessible within the counting house via RootSpy, and<br />
for each run and file, a ROOT file containing the histograms will be saved<br />
within a subdirectory for each run.<br />
<br />
For immediate access to these files, the raid disk files may be accessed directly<br />
from the counting house, or the tape files will be available within ~20 min. of the<br />
file being written out.<br />
<br />
== Offline Monitoring: Running Over Archived Data ==<br />
<br />
Once files are written to tape we run the online plugins on these files to confirm what we were seeing in the online monitoring, and also to update the results from the latest calibration and software. <br />
Manual scripts and cron jobs are set up to look for new data and run the plugins over a sample of files.<br />
<br />
Every other Friday (usually the Friday before the offline meetings) jobs will be started to run the newest software on all previous runs,and allows everybody to see improvements in each detector.<br />
For each launch, independent builds of hdds, sim-recon, the monitoring plugins, and an sqlite file will be generated.<br />
<br />
Below the procedures are described for<br />
# Preparing the software for the launch<br />
# Starting the launch (using hdswif)<br />
# Post-analysis of statistics of the launch<br />
<br />
Processing the results and making them available to the collaboration<br />
is handled in the section below<br />
<br />
<!--<br />
==== Generating an offline plugin job ====<br />
<br />
The user gxproj1 should be used for official offline monitoring jobs.<br />
Within /home/gxproj1/halld/monitoring/batch/ there will be scripts to run the online monitoring plugins over tape files.<br />
The main script is generatejobs_plugins_rawdata.sh, which can be used as<br />
generatejobs_plugins_rawdata.sh [minrun] [maxrun] (minfile) (maxfile)<br />
where minrun, maxrun specify the range of the run #, and minfile, maxfile (optional) specify<br />
the file range for that run.<br />
<br />
This will generate a script run_rawdata_XXXXXX.sh, where the run # has now been formatted to be 6 digits.<br />
Executing this script will send the monitoring plugins job to the Auger batch system.<br />
<br />
There is also a script clean.sh which can be used as<br />
clean.sh XXX<br />
This will clean up all associated files created in association with run XXX.<br />
<br />
Internally, the xml file used to submit the job will be created, and the job to run<br />
will be given within script.sh. All run parameters should be specified in at the beginning<br />
of generatejobs_plugins_rawdata.sh<br />
Since we are running on tape, the tape file will first be copied over to the cache disk, and the job will run<br />
over this cached file.<br />
<br />
==== Using cron to run automatically ====<br />
Within /home/gluex/halld/monitoring/cron/ there is a file cron_plugins<br />
that can be executed via<br />
crontab cron_plugins<br />
This will set up a cron job to call the script scan_for_jobs.sh, which will<br />
check in the rawdata directory and call generatejobs_plugins_rawdata.sh for<br />
any run that is more than 5 min old. The cron job is set up to run every 10 min.<br />
<br />
==== Magnetic Field Settings ====<br />
For tracking, it is necessary to set the correct magnetic field settings.<br />
The below table is now obsolete. Please refer to the [https://halldweb.jlab.org/cgi-bin/data_monitoring/run_conditions.pl GlueX Run Conditions] webpage.<br />
<br />
The actual field settings for each run have not been documented well.<br />
Below are values based on going through entries in the [https://logbooks.jlab.org/book/halld halld log].<br />
<br />
{| border="1" cellpadding="1" style="text-align: center;"<br />
!width="100"| Run #s<br />
!width="150"| Solenoid Current (A)<br />
!width="300"| JANA option<br />
!width="300"| notes<br />
|-<br />
| 940 - 996 || 1000 || -PBFIELD_MAP=Magnets/Solenoid/solenoid_1000A_poisson_20141104 || At run 997, the solenoid started ramping down<br />
|-<br />
| 998 - 1448 || 0 || -PBFIELD_TYPE=NoField -PDEFTAG:DTrackCandidate=StraightLine || Solenoid started ramping up around run 1431. Runs 1432 - 1448 should not be used for tracking. See [https://logbooks.jlab.org/entry/3309039 this] and [https://logbooks.jlab.org/entry/3309092 this] entry.<br />
|-<br />
| 1449 - 1620 || 1200 || -PBFIELD_MAP=Magnets/Solenoid/solenoid_1200A_poisson_20140520 || <br />
|-<br />
|}<br />
<br />
'''Note:''' The following run ranges were taken with a 300A field, and could be analyzed with the magnetic field map Magnets/Solenoid/solenoid_300A_poisson_20140819 , more study is needed to see if the straight line track fitter is needed or not: 1036 - 1053, 1065 - 1121, 1212? - 1254, 1309 - 1318 [1307,1308,1319-1329 were taken while the magnet was ramping].<br />
--><br />
<br />
=== Procedures ===<br />
This section explains how the offline monitoring should be run.<br />
Since we may want to simultaneously run offline monitoring for different run periods that require<br />
different environment variables, the scripts are set up so that a generic user can download the<br />
scripts and run them from anywhere.<br />
<br />
To do this, you will check out a directory from svn that contains all the necessary scripts,<br />
and running a generation script will generate all the necessary files for the present launch.<br />
The main engine behind keeping track of all the files submitted and their status is the jproj<br />
system created by Mark Ito.<br />
<br />
Below is the process of creating and running a launch.<br />
<br />
<br />
# To stop the incoming-data cron job, first kill the cron job with cron -r. Also, delete all jobs that have not started yet. If these jobs are still alive, they will cause confusion over which software version they ran. Do <pre>crontab -l</pre> to list the current cron jobs, and run <pre>crontab -r</pre> to kill them.<br />
# svn update & rebuild HDDS with <pre>cd $HDDS_HOME</pre><pre> svn up</pre><pre> scons install</pre><br />
# svn update & rebuild sim-recon with <pre>cd $HALLD_HOME</pre> <pre> svn up</pre> <pre>cd src</pre> <pre>scons -c install</pre> <pre>scons install</pre> The "scons -c install" is necessary to clean out any old header files. NOTE THAT IT IS A GOOD IDEA TO DO A COMPLETE WIPEOUT AND CHECK OUT A NEW VERSION SO THAT STALE HEADER FILES ARE NOT PRESENT.<br />
# svn update & rebuild monitoring plugins with <pre>cd /home/gxproj1/builds/online/packages/SBMS</pre> <pre> svn up</pre> <pre>cd /home/gxproj1/builds/online/packages/monitoring/src/plugins</pre> <pre> svn up</pre> <pre>scons -u install</pre><br />
# Prepare the latest sqlite file with: <pre>cd $HOME/builds/</pre> <pre>$CCDB_HOME/scripts/mysql2sqlite/mysql2sqlite.sh -hhallddb.jlab.org -uccdb_user ccdb | sqlite3 ccdb.sqlite</pre><br />
# Note that the above steps must be done BEFORE launch project creation. This is because we will track the svn revisions of the libraries used, and this is done by extracting the svn information in each directory. Also, note that the system assumes that we have the topmost build directory (usually called GLUEX_TOP) to be $HOME/builds . Such an assumption is necessary to be able to extract information about the library locations automatically.<br />
# Create the appropriate project(s) and submit the jobs using the Hall D Job Management System, as detailed in the section below.<br />
# Restart cron jobs for immediate processing of runs coming in.<br />
<br />
== Hall D Job Management System ==<br />
<br />
This section details instructions on how to create and launch a set of jobs using the Hall-D Job Management System developed by Mark Ito. These instructions are generic: this system can be used for the weekly monitoring jobs, but can also be used for other sets of job launches as well. <br />
<br />
=== Database Table Overview ===<br />
<br />
* Job management database table (<project_name>): For each input file, keeps track of whether or not a job for it has been submitted, along with other optional fields. <br />
<br />
* Job status database table (<project_name>Job (no space)): For each job, keeps track of the job-id, the job status, memory used, cpu & wall time, time taken to complete various stages (e.g. pending, dependency, active), and others. <br />
<br />
* Job metrics database table (<project_name>_aux (no space)): For each job, keeps track of the job-id, how many events were processed, the time it took to copy the cache file, and the time it took to run the plugin. This information is culled from the log files of each job, and is done within the analysis directory of each launch.<br />
<br />
=== Initialize Project Management ===<br />
<br />
* Log into the ifarm machine with one of the gxproj accounts. For this example we will use gxproj1.<br />
<pre><br />
ssh gxproj1@ifarm -Y<br />
</pre><br />
<br />
* Go to a directory to do the launch. In principle, any directory will work, but for gxproj1 this is usually done in /home/gxproj1/halld/<br />
<br />
* Check out the necessary scripts <pre>svn co https://halldsvn.jlab.org/repos/trunk/scripts/monitoring/jproj</pre> This will get all necessary scripts for launching. Once checked out, <pre>cd projects</pre><br />
<br />
* The script create_project.sh can be used to create a new project. It will take a single argument, the project name. It is assumed that the project name is of the form offline_monitoring_RunPeriod20YY_MM_verVV_hd_rawdata where this string will be parsed to give the run period 20YY_MM and version number VV. One thing to do BEFORE creation of a new project is to editthe conditions of the launch (plugins to run over, memory requested, disk space requested) within templates/template.jsub . This<br />
information is saved automatically at project creation time into files in /group/halld/data_monitoring/run_conditions .<br />
To create a project do <pre>./create_project.sh offline_monitoring_RunPeriod20YY_MM_verVV_hd_rawdata</pre><br />
The name has been chosen to be as consistent as possible with other directory structures.<br />
However, mysql requires that "-" be escaped in table names, so unfortunately run periods will be<br />
given as 2YYY_MM instead of 2YYY-MM.<br />
<br />
* A new directory with the project name (offline_monitoring_RunPeriod20YY_MM_verVV_hd_rawdata) and files will be copied and modified from the template directory to reflect the run period, the user, the directory that it was created in, project name, etc.<br />
<br />
* For each project, cd into the new directory<br />
* The script clear.sh will remove any existing tables for the current project name, then recreate it. Do<br />
<pre>./clear.sh</pre><br />
* To use the jproj.pl script that was checked in, add the directory to your path with<br />
<pre>source ../../scripts/setup.csh</pre><br />
or always specify the full path<br />
<pre>../../scripts/jproj.pl</pre><br />
* Now update the table of runs with<br />
<pre>jproj.pl <project name> update</pre><br />
This will fill the table with all files within /mss that are of the same form as what is in <project name>.jproj . If you want to register only on a subset of all such files, you can edit this file directly.<br />
* Once you have registered all of the files you would like to run over, do<br />
<pre>jproj.pl <project name> submit [max # of jobs] [run number]</pre><br />
where the additional options specify how many jobs to submit and which run number to run on. Without these options all files that are registered and have not been submitted yet will be submitted.<br />
<br />
At this stage you are ready to submit all files. It is a good idea to submit a few test jobs<br />
at first to check that all scripts are working and that the plugins do not crash. Once you are<br />
sure that this does not happen, you can send all jobs in. The remaining jobs are then the monitoring<br />
which will (among other things) put the results on the online webpage for the collaboration to view, and<br />
the analysis of the launch.<br />
<br />
=== Project File Overview ===<br />
<br />
An overview of each project file:<br />
* '''clear.sh:''' For the current project, deletes the job status and management database tables (if any), and creates new, empty ones. <br />
* '''<project_name>.jproj:''' Contains the path and file name format for the input files for the jobs. <br />
* '''<project_name>.jsub:''' The xml job submission script. The run number and file number variables are set during job submission for each input file. <br />
* '''script.sh:''' The script that is executed during the job. If output job directories are not pre-created manually, they should be created in this script with the proper permissions:<br />
<pre><br />
mkdir -p -m 775 my_directory<br />
</pre><br />
* '''setup_jlab-[run period].csh:''' The environment that is sourced at the beginning of the job execution. <br />
* '''status.sh:''' Updates the job status database table, and prints some of its columns to screen.<br />
<br />
=== Project Management ===<br />
<br />
* Delete (if any) and create the database table(s) for the current set of job submissions:<br />
<pre><br />
./clear.sh<br />
</pre><br />
Also, if testing was done with jobs, it is best to delete the output directory and the configuration files:<br />
<pre><br />
rm -frv /volatile/halld/offline_monitoring/RunPeriod-20YY-MM/verVV /group/halld/data_monitoring/run_conditions/soft_comm_20YY_MM_verVV.xml /group/halld/data_monitoring/run_conditions/jana_rawdata_comm_20YY_MM_verVV.conf<br />
</pre><br />
<br />
* Search for input files matching the string in the .jproj file, and create a row for each in the job management database table (called <project_name>). You can test by adding an optional argument at the end, which only selects files with a specific file number:<br />
<pre><br />
jproj.pl <project_name> update <optional_file_number><br />
</pre><br />
<br />
* Confirm that the job management database is accurate by printing it's contents to screen:<br />
<pre><br />
mysql -hhallddb -ufarmer farming -e "select * from <project_name>"<br />
</pre><br />
<br />
* ONLY if a mistake was made, to delete the tables from the database and recreate new, empty ones, run: <br />
<pre><br />
./clear.sh<br />
</pre><br />
<br />
* To look at the status of the submitted jobs, first query auger and update the job status database:<br />
<pre><br />
fill_in_job_details.pl <project_name><br />
</pre><br />
<br />
* The job status can then be viewed by submitting a query to the job status database (called <project_name>Job (no space in between)):<br />
<pre><br />
mysql -hhallddb -ufarmer farming -e "select id,run,file,jobId,hostname,status,timeSubmitted,timeActive,walltime,cput,timeComplete,result,error from <project_name>Job"<br />
</pre><br />
<br />
* These last two commands can instead be executed simultaneously by running:<br />
<pre><br />
./status.sh<br />
</pre><br />
<br />
=== Handy mysql Instructions ===<br />
<br />
* Handy mysql instructions:<br />
<pre><br />
mysql -hhallddb -ufarmer farming # Enter the "farming" mysql database on "hallddb" as user "farmer"<br />
quit; # Exit mysql<br />
show tables; # Show a list of the tables in the current database<br />
show columns from <project_name>; # show all of the columns for the given table<br />
select * from <project_name>; # show the contents of all rows from the given table<br />
</pre><br />
<br />
=== Backing Up Offline Monitoring Tables ===<br />
Tables created for offline monitoring can be backed up using the script backup_tables.sh which<br />
can be checked out with the other files from https://halldsvn.jlab.org/repos/trunk/scripts/monitoring/jproj/projects<br />
<br />
The script uses the command mysqldump to print out a file that can be executed to recreate the tables.<br />
Since executing this output file will drop the table when it exists, caution is advised.<br />
Example usage to backup all three tables created for run period 2014_10 ver 17:<br />
<pre>backup_tables.sh 2014_10 17</pre><br />
<br />
== Running Over Data As It Comes In ==<br />
<br />
A special user gxproj1 will have a cron job set up to run the plugins as new data appears on /mss.<br />
During the week, gxproj1 will submit offline plugin jobs with the same setup as the weekly jobs<br />
run the previous Friday. The procedure for this is shown below.<br />
<br />
<!--<br />
=== Setting up the environment ===<br />
The file<br />
/home/gxproj1/setup_jlab.csh<br />
is sourced through .tcshrc.<br />
This file is the same as what is linked to by<br />
/home/gluex/setup_jlab_commissioning.csh,<br />
except HALLD_HOME, HDDS_HOME, and JANA_CALIB_URL are set separately so that this<br />
user can have a separate build.<br />
<br />
To obtain the builds from the previous Friday's runs,<br />
execute<br />
/home/gxproj1/halld/monitoring/newruns/setup_previous.sh [year] [month] [day]<br />
The build revisions from the previous Friday are archived in files<br />
/work/halld/data_monitoring/run_conditions/soft_comm_[year]_[month]_[day].xml<br />
and the script will build libraries based on those stored revision numbers.<br />
--><br />
<br />
=== Running the cron job ===<br />
<br />
'''IMPORTANT:''' The cron job should not be running while you are manually submitting jobs using the jproj.pl script for the same project, or else you will probably multiply-submit a job. <br />
<br />
* Go to the cron job directory: <br />
<pre><br />
cd /u/home/gxproj1/halld/monitoring/newruns<br />
</pre><br />
<br />
* The cron_plugins file is the cronjob that will be executed. During execution, it runs the exec.sh command in the same folder. This command takes two arguments: the project name, and the maximum file number for each run. These fields should be updated in the cron_plugins file before running. <br />
<br />
* The exec.sh command updates the job management database table with any data that has arrived on tape since it was last updated, ignoring file numbers greater than the maximum file number. It then submits jobs for these files. <br />
<br />
* To start the cron job, run:<br />
<pre><br />
crontab cron_plugins<br />
</pre><br />
<br />
* To check whether the cron job is running, do<br />
<pre><br />
crontab -l<br />
</pre><br />
<br />
* To remove the cron job do<br />
<pre><br />
crontab -r<br />
</pre><br />
<br />
<!--<br />
The cron job will run the script scan_for_jobs.sh,<br />
which runs generatejobs_plugins_rawdata.sh for any<br />
new runs that it had not seen before. All previous<br />
runs are recorded in the file filelists/files_current.txt<br />
so clear this to run over runs, or set the parameters<br />
MINRUN and MAXRUN which will set the range of runs submitted.<br />
--><br />
<br />
==Post-Processing Procedures==<br />
<br />
To visualize the monitoring data, we save images of selected histograms and store time series of selected quantities in a database, which are then displayed on a web page. This section describes how to generate the monitoring images and database information.<br />
<br />
The scripts used to generate this summary data are primarily run from /home/gxprojN/halld/monitoring/process . If you want a new copy of the scripts, e.g., for a new monitoring run, you should check the scripts out from SVN:<br />
<syntaxhighlight><br />
svn co https://halldsvn.jlab.org/repos/trunk/scripts/monitoring/process<br />
</syntaxhighlight><br />
Note that these scripts currently have some parameters which must be periodically set by hand.<br />
<br />
The default python version on most JLab machine does not have the modules to allow these scripts to connect to the MySQL database. To run these scripts, load the environment with the following command<br />
<syntaxhighlight><br />
source /home/gxproj1/halld/monitoring/process/monitoring_env.csh<br />
</syntaxhighlight><br />
<br />
===Online Monitoring===<br />
<br />
There are two primary scripts for running over the monitoring data generated by the online system and offline reconstruction. The online script can be run with either of the following commands:<br />
<syntaxhighlight><br />
/home/gluex/halld/monitoring/process/check_new_runs.py<br />
<br />
OR <br />
<br />
/home/gluex/halld/monitoring/process/check_new_runs.csh<br />
</syntaxhighlight><br />
The shell script sets up the environment properly to run the python script. To connect to the monitoring database on the JLab CUE, modules included in the local installation of python >= 2.7 are needed. The shell script is appropriate to use in a cron job. The cronjob is currently run under the "gluex" account.<br />
<br />
The online monitoring system copies a ROOT file containing the results of the online monitoring, and other configuration files into a directory accessible outside the counting house. This python script automatically checks for new ROOT files, which it will then automatically process. It contains several configuration variables that must be correctly set, which contains the location of input/output directories, etc...<br />
<br />
===Offline Monitoring===<br />
<br />
After the data is run over, the results should be processed, so that summary data is entered into the monitoring database and plots are made for the monitoring webpages. Currently, this processing is controlled by a cronjob that runs the following script:<br />
<syntaxhighlight><br />
/home/gxproj1/halld/monitoring/process/check_monitoring_data.csh <br />
</syntaxhighlight><br />
This script checks for new ROOT files, and only runs over those it hasn't processed yet. Since one monitoring ROOT file is produced for each EVIO file, whenever a new file is produced, the plots for the corresponding run are recreated and all the ROOT files for a run are combined into one file. Information is stored in the database on a per-file basis. <br />
<br />
Plots for the monitoring web page can be made from single histograms or multiple histograms using RootSpy macros. If you want to change the list of plots made, you must modify one of the following files:<br />
* histograms_to_monitor - specify either the name of the histogram or its the full ROOT path <br />
* macros_to_monitor - specify the full path to the RootSpy macro .C file<br />
<br />
When a new monitoring run is started, or the conditions are changed, the following steps should be taken to process the new files:<br />
# Add a new data version, as described below:<br />
# Change the following parameters in check_monitoring_data.csh:<br />
## JOBDATE should correspond to the ouptut date used by the job submission script<br />
## OUTPUTDIR should correspond to the directory corresponding to the run period and revision corresponding to the new version you just submitted. Presumably, this directory will be empty at the beginning.<br />
## Once you create a new data version as defined below, you should pass the needed information as a command line option. Currently this is done by the ARGS variable. For example, the argument "-v RunPeriod-2014-10,8" tells the monitoring scripts to look up the version corresponding to revision 8 of RunPeriod-2014-10 in the monitoring DB and to use to store the results.<br />
<br />
<syntaxhighlight><br />
Example configuration parameters:<br />
set JOBDATE=2015-01-09<br />
set INPUTDIR=/volatile/halld/RunPeriod-2014-10/offline_monitoring<br />
set OUTPUTDIR=/w/halld-scifs1a/data_monitoring/RunPeriod-2014-10/ver08<br />
set ARGS=" -v RunPeriod-2014-10,8 "<br />
</syntaxhighlight><br />
If you want to process the results manually, the data is processed using the following script:<br />
<syntaxhighlight><br />
./process_new_offline_data.py <input directory> <output directory><br />
<br />
EXAMPLE:<br />
<br />
./process_new_offline_data.py 2014-11-14 /volatile/halld/RunPeriod-2014-10/offline_monitoring/ /w/halld-scifs1a/data_monitoring/RunPeriod-2014-10/ver02<br />
</syntaxhighlight><br />
The python script takes several options to enable/disable various steps in the processing. Of interest is the "--force" option, which will run over all monitoring ROOT files, whether or not they've been previously identified.<br />
<br />
Every time a new reconstruction pass is performed, a new version number must be generated. To do this, prepare a version file as described below. Then run the register_new_version.py script to store the information in the database. The script will return a version number, which then should be set by hand in process_new_offline_data.py - future versions of the script will streamline this part of the procedure. An example of how to generate a new version is:<br />
<syntaxhighlight><br />
./register_new_version.py add /home/gxproj1/halld/monitoring/process/versions/vers_RunPeriod-2014-10_pass1.txt<br />
</syntaxhighlight><br />
<br />
<b>If you are running the offline monitoring by checking out the files in trunk/scripts/monitoring/jproj/projects/<br />
of the svn repository</b>, and created a project with<br />
create_project.sh [project name] hd_rawdata<br />
Then go to the directory [project name]/processing/<br />
and execute<br />
./run_processing.sh<br />
which will run register_new_version.py as well as check_monitoring_data.csh for that project.<br />
<br />
===Step-by-Step Instructions For Processing a New Monitoring Run===<br />
<br />
The monitoring runs are current run out of the gxproj1 and gxproj5 accounts. After an offline monitoring run has been successfully started on the batch farm, the following steps should be followed to setup the post-processing for these runs.<br />
<br />
# The post-processing scripts are stored in $HOME/halld/monitoring/process and are automatically run by cron.<br />
# Run "svn update" to bring any changes in. Be sure that the list of histograms and macros to plot are current.<br />
# Edit check_monitoring_data.csh to point to the current revisions/directories<br />
#* VERSION<br />
#* ARGS<br />
#* Note that the environment depends on a standard script - $HOME/setup_jlab.csh<br />
# Update files in the web directory, so that the results are displayed on the web pages: /group/halld/www/halldweb/html/data_monitoring/textdata<br />
<br />
Check log files in $HOME/halld/monitoring/process/log for more information on how each run went. If there are problems, check log files, and modify check_monitoring_data.csh to vary the verbosity of the output.<br />
<br />
==Data Versions==<br />
<br />
To document the conditions of the monitoring data that is created, for the sake of reproducability and further analysis we save several pieces of information. The format is intended to be comprehensive enough to document not just monitoring data, but versions of raw and reconstructed data, so that this database table can be used for the event database as well.<br />
<br />
We store one record per pass through one run period, with the following structure:<br />
<br />
{| class="wikitable"<br />
! Field !! Description<br />
|-<br />
| data_type || The level of data we are processing. For the purposes of monitoring, "rawdata" is the online monitoring, "recon" is the offline monitoring<br />
|- <br />
| run_period || The run period of the data<br />
|- <br />
| revision || An integer specifying which pass through the run period this data corresponds to<br />
|- <br />
| software_version || The name of the XML file that specifies the different software versions used<br />
|-<br />
| jana_config || The name of the text file that specifies which JANA options were passed to the reconstruction program<br />
|-<br />
| ccdb_context || The value of JANA_CALIB_CONTEXT, which specifies the version of calibration constants that were used<br />
|-<br />
| production_time || The data at which monitoring/reconstruction began<br />
|-<br />
| dataVersionString || A convenient string for identifying this version of the data<br />
|}<br />
<br />
<br />
An example file used as as input to ./register_new_version.py is:<br />
<syntaxhighlight><br />
data_type = recon<br />
run_period = RunPeriod-2014-10<br />
revision = 1<br />
software_version = soft_comm_2014_11_06.xml<br />
jana_config = jana_rawdata_comm_2014_11_06.conf<br />
ccdb_context = calibtime=2014-11-10<br />
production_time = 2014-11-10<br />
dataVersionString = recon_RunPeriod-2014-10_20141110_ver01<br />
</syntaxhighlight></div>Kmoriyahttps://halldweb.jlab.org/wiki/index.php?title=Data_Monitoring_Procedures&diff=70721Data Monitoring Procedures2015-10-14T17:37:08Z<p>Kmoriya: /* Master List of File / Database / Webpage Locations */</p>
<hr />
<div>__TOC__<br />
<br />
== Master List of File / Database / Webpage Locations ==<br />
=== Run Conditions ===<br />
* Online Run-by-run condition files (B-field, current, etc.): /work/halld/online_monitoring/conditions/<br />
* Offline monitoring run conditions (software versions, jana config): /group/halld/data_monitoring/run_conditions/<br />
*[http://www.jlab.org/Hall-D/test/RunInfo/ Run Info vers. 1]<br />
*[https://halldweb.jlab.org/cgi-bin/data_monitoring/run_conditions.pl Run Info vers. 2]<br />
<br />
=== Monitoring Output Files ===<br />
* Run Periods 201Y-MM is for example 2015-03, launch ver verVV is for example ver15<br />
* Online monitoring histograms: /work/halld/online_monitoring/root/<br />
* Offline monitoring histogram ROOT files: /work/halld/data_monitoring/RunPeriod-201Y-MM/verVV/rootfiles<br />
* REST files (most recent launch only): /work/halld/data_monitoring/RunPeriod-201Y-MM/REST/verVV<br />
* individual files for each job (ROOT, log, etc.): /volatile/halld/offline_monitoring/RunPeriod-201Y-MM/verVV/<br />
<br />
=== Monitoring Database ===<br />
* Accessing monitoring database (on ifarm): mysql -u datmon -h hallddb.jlab.org data_monitoring<br />
<br />
=== Monitoring Webpages ===<br />
*[https://halldweb.jlab.org/cgi-bin/data_monitoring/monitoring/plotBrowser.py Plot Browser]<br />
*[https://halldweb.jlab.org/cgi-bin/data_monitoring/monitoring/runBrowser.py Run Browser]<br />
*[https://halldweb.jlab.org/cgi-bin/data_monitoring/monitoring/timeSeries.py Time Series]<br />
*[https://halldweb.jlab.org/data_monitoring/launch_analysis/ Launch Analysis]<br />
<br />
== Job Monitoring Links ==<br />
* [http://scicomp.jlab.org/farm2/job.html Completed Job History]<br />
* [http://scicomp.jlab.org/farm2/project.html Job Stats By Project]<br />
* [http://scicomp.jlab.org/farm2/trackOrg.html Job Stats By Track]<br />
* [http://scicomp.jlab.org/farm2/report.html Cluster Report]<br />
* [http://scicomp.jlab.org/farm2/walltime.html Walltime Distribution]<br />
<br />
== Saving Online Monitoring Data ==<br />
<br />
The procedure for writing the data out is given in, e.g.,<br />
[https://halldweb.jlab.org/wiki/index.php/Raid-to-Silo_Transfer_Strategy Raid-to-Silo Transfer Strategy].<br />
<br />
Once the DAQ writes out the data to the raid disk, cron jobs will copy the file to tape,<br />
and within ~20 min., we will have access to the file on tape at<br />
/mss/halld/$RUN_PERIOD/rawdata/RunXXXXXX.<br />
<br />
All online monitoring plugins will be run as data is taken.<br />
They will be accessible within the counting house via RootSpy, and<br />
for each run and file, a ROOT file containing the histograms will be saved<br />
within a subdirectory for each run.<br />
<br />
For immediate access to these files, the raid disk files may be accessed directly<br />
from the counting house, or the tape files will be available within ~20 min. of the<br />
file being written out.<br />
<br />
== Offline Monitoring: Running Over Archived Data ==<br />
<br />
Once the files are written to take we can run the online plugins on these files to confirm what we were seeing in the online monitoring.<br />
Manual scripts and cron jobs are set up to look for new data and run the plugins over a sample of files.<br />
<br />
Every Friday jobs will be started to run the newest software on all previous runs,<br />
and allows everybody to see improvements in each detector over the week.<br />
When launching, independent builds of hdds, sim-recon, the monitoring plugins, and an sqlite file will<br />
be generated.<br />
<br />
<!--<br />
==== Generating an offline plugin job ====<br />
<br />
The user gxproj1 should be used for official offline monitoring jobs.<br />
Within /home/gxproj1/halld/monitoring/batch/ there will be scripts to run the online monitoring plugins over tape files.<br />
The main script is generatejobs_plugins_rawdata.sh, which can be used as<br />
generatejobs_plugins_rawdata.sh [minrun] [maxrun] (minfile) (maxfile)<br />
where minrun, maxrun specify the range of the run #, and minfile, maxfile (optional) specify<br />
the file range for that run.<br />
<br />
This will generate a script run_rawdata_XXXXXX.sh, where the run # has now been formatted to be 6 digits.<br />
Executing this script will send the monitoring plugins job to the Auger batch system.<br />
<br />
There is also a script clean.sh which can be used as<br />
clean.sh XXX<br />
This will clean up all associated files created in association with run XXX.<br />
<br />
Internally, the xml file used to submit the job will be created, and the job to run<br />
will be given within script.sh. All run parameters should be specified in at the beginning<br />
of generatejobs_plugins_rawdata.sh<br />
Since we are running on tape, the tape file will first be copied over to the cache disk, and the job will run<br />
over this cached file.<br />
<br />
==== Using cron to run automatically ====<br />
Within /home/gluex/halld/monitoring/cron/ there is a file cron_plugins<br />
that can be executed via<br />
crontab cron_plugins<br />
This will set up a cron job to call the script scan_for_jobs.sh, which will<br />
check in the rawdata directory and call generatejobs_plugins_rawdata.sh for<br />
any run that is more than 5 min old. The cron job is set up to run every 10 min.<br />
<br />
==== Magnetic Field Settings ====<br />
For tracking, it is necessary to set the correct magnetic field settings.<br />
The below table is now obsolete. Please refer to the [https://halldweb.jlab.org/cgi-bin/data_monitoring/run_conditions.pl GlueX Run Conditions] webpage.<br />
<br />
The actual field settings for each run have not been documented well.<br />
Below are values based on going through entries in the [https://logbooks.jlab.org/book/halld halld log].<br />
<br />
{| border="1" cellpadding="1" style="text-align: center;"<br />
!width="100"| Run #s<br />
!width="150"| Solenoid Current (A)<br />
!width="300"| JANA option<br />
!width="300"| notes<br />
|-<br />
| 940 - 996 || 1000 || -PBFIELD_MAP=Magnets/Solenoid/solenoid_1000A_poisson_20141104 || At run 997, the solenoid started ramping down<br />
|-<br />
| 998 - 1448 || 0 || -PBFIELD_TYPE=NoField -PDEFTAG:DTrackCandidate=StraightLine || Solenoid started ramping up around run 1431. Runs 1432 - 1448 should not be used for tracking. See [https://logbooks.jlab.org/entry/3309039 this] and [https://logbooks.jlab.org/entry/3309092 this] entry.<br />
|-<br />
| 1449 - 1620 || 1200 || -PBFIELD_MAP=Magnets/Solenoid/solenoid_1200A_poisson_20140520 || <br />
|-<br />
|}<br />
<br />
'''Note:''' The following run ranges were taken with a 300A field, and could be analyzed with the magnetic field map Magnets/Solenoid/solenoid_300A_poisson_20140819 , more study is needed to see if the straight line track fitter is needed or not: 1036 - 1053, 1065 - 1121, 1212? - 1254, 1309 - 1318 [1307,1308,1319-1329 were taken while the magnet was ramping].<br />
--><br />
<br />
=== Procedures ===<br />
This section explains how the offline monitoring should be run.<br />
Since we may want to simultaneously run offline monitoring for different run periods that require<br />
different environment variables, the scripts are set up so that a generic user can download the<br />
scripts and run them from anywhere.<br />
<br />
To do this, you will check out a directory from svn that contains all the necessary scripts,<br />
and running a generation script will generate all the necessary files for the present launch.<br />
The main engine behind keeping track of all the files submitted and their status is the jproj<br />
system created by Mark Ito.<br />
<br />
Below is the process of creating and running a launch.<br />
<br />
<br />
# To stop the incoming-data cron job, first kill the cron job with cron -r. Also, delete all jobs that have not started yet. If these jobs are still alive, they will cause confusion over which software version they ran. Do <pre>crontab -l</pre> to list the current cron jobs, and run <pre>crontab -r</pre> to kill them.<br />
# svn update & rebuild HDDS with <pre>cd $HDDS_HOME</pre><pre> svn up</pre><pre> scons install</pre><br />
# svn update & rebuild sim-recon with <pre>cd $HALLD_HOME</pre> <pre> svn up</pre> <pre>cd src</pre> <pre>scons -c install</pre> <pre>scons install</pre> The "scons -c install" is necessary to clean out any old header files. NOTE THAT IT IS A GOOD IDEA TO DO A COMPLETE WIPEOUT AND CHECK OUT A NEW VERSION SO THAT STALE HEADER FILES ARE NOT PRESENT.<br />
# svn update & rebuild monitoring plugins with <pre>cd /home/gxproj1/builds/online/packages/SBMS</pre> <pre> svn up</pre> <pre>cd /home/gxproj1/builds/online/packages/monitoring/src/plugins</pre> <pre> svn up</pre> <pre>scons -u install</pre><br />
# Prepare the latest sqlite file with: <pre>cd $HOME/builds/</pre> <pre>$CCDB_HOME/scripts/mysql2sqlite/mysql2sqlite.sh -hhallddb.jlab.org -uccdb_user ccdb | sqlite3 ccdb.sqlite</pre><br />
# Note that the above steps must be done BEFORE launch project creation. This is because we will track the svn revisions of the libraries used, and this is done by extracting the svn information in each directory. Also, note that the system assumes that we have the topmost build directory (usually called GLUEX_TOP) to be $HOME/builds . Such an assumption is necessary to be able to extract information about the library locations automatically.<br />
# Create the appropriate project(s) and submit the jobs using the Hall D Job Management System, as detailed in the section below.<br />
# Restart cron jobs for immediate processing of runs coming in.<br />
<br />
== Hall D Job Management System ==<br />
<br />
This section details instructions on how to create and launch a set of jobs using the Hall-D Job Management System developed by Mark Ito. These instructions are generic: this system can be used for the weekly monitoring jobs, but can also be used for other sets of job launches as well. <br />
<br />
=== Database Table Overview ===<br />
<br />
* Job management database table (<project_name>): For each input file, keeps track of whether or not a job for it has been submitted, along with other optional fields. <br />
<br />
* Job status database table (<project_name>Job (no space)): For each job, keeps track of the job-id, the job status, memory used, cpu & wall time, time taken to complete various stages (e.g. pending, dependency, active), and others. <br />
<br />
* Job metrics database table (<project_name>_aux (no space)): For each job, keeps track of the job-id, how many events were processed, the time it took to copy the cache file, and the time it took to run the plugin. This information is culled from the log files of each job, and is done within the analysis directory of each launch.<br />
<br />
=== Initialize Project Management ===<br />
<br />
* Log into the ifarm machine with one of the gxproj accounts. For this example we will use gxproj1.<br />
<pre><br />
ssh gxproj1@ifarm -Y<br />
</pre><br />
<br />
* Go to a directory to do the launch. In principle, any directory will work, but for gxproj1 this is usually done in /home/gxproj1/halld/<br />
<br />
* Check out the necessary scripts <pre>svn co https://halldsvn.jlab.org/repos/trunk/scripts/monitoring/jproj</pre> This will get all necessary scripts for launching. Once checked out, <pre>cd projects</pre><br />
<br />
* The script create_project.sh can be used to create a new project. It will take a single argument, the project name. It is assumed that the project name is of the form offline_monitoring_RunPeriod20YY_MM_verVV_hd_rawdata where this string will be parsed to give the run period 20YY_MM and version number VV. One thing to do BEFORE creation of a new project is to editthe conditions of the launch (plugins to run over, memory requested, disk space requested) within templates/template.jsub . This<br />
information is saved automatically at project creation time into files in /group/halld/data_monitoring/run_conditions .<br />
To create a project do <pre>./create_project.sh offline_monitoring_RunPeriod20YY_MM_verVV_hd_rawdata</pre><br />
The name has been chosen to be as consistent as possible with other directory structures.<br />
However, mysql requires that "-" be escaped in table names, so unfortunately run periods will be<br />
given as 2YYY_MM instead of 2YYY-MM.<br />
<br />
* A new directory with the project name (offline_monitoring_RunPeriod20YY_MM_verVV_hd_rawdata) and files will be copied and modified from the template directory to reflect the run period, the user, the directory that it was created in, project name, etc.<br />
<br />
* For each project, cd into the new directory<br />
* The script clear.sh will remove any existing tables for the current project name, then recreate it. Do<br />
<pre>./clear.sh</pre><br />
* To use the jproj.pl script that was checked in, add the directory to your path with<br />
<pre>source ../../scripts/setup.csh</pre><br />
or always specify the full path<br />
<pre>../../scripts/jproj.pl</pre><br />
* Now update the table of runs with<br />
<pre>jproj.pl <project name> update</pre><br />
This will fill the table with all files within /mss that are of the same form as what is in <project name>.jproj . If you want to register only on a subset of all such files, you can edit this file directly.<br />
* Once you have registered all of the files you would like to run over, do<br />
<pre>jproj.pl <project name> submit [max # of jobs] [run number]</pre><br />
where the additional options specify how many jobs to submit and which run number to run on. Without these options all files that are registered and have not been submitted yet will be submitted.<br />
<br />
At this stage you are ready to submit all files. It is a good idea to submit a few test jobs<br />
at first to check that all scripts are working and that the plugins do not crash. Once you are<br />
sure that this does not happen, you can send all jobs in. The remaining jobs are then the monitoring<br />
which will (among other things) put the results on the online webpage for the collaboration to view, and<br />
the analysis of the launch.<br />
<br />
=== Project File Overview ===<br />
<br />
An overview of each project file:<br />
* '''clear.sh:''' For the current project, deletes the job status and management database tables (if any), and creates new, empty ones. <br />
* '''<project_name>.jproj:''' Contains the path and file name format for the input files for the jobs. <br />
* '''<project_name>.jsub:''' The xml job submission script. The run number and file number variables are set during job submission for each input file. <br />
* '''script.sh:''' The script that is executed during the job. If output job directories are not pre-created manually, they should be created in this script with the proper permissions:<br />
<pre><br />
mkdir -p -m 775 my_directory<br />
</pre><br />
* '''setup_jlab-[run period].csh:''' The environment that is sourced at the beginning of the job execution. <br />
* '''status.sh:''' Updates the job status database table, and prints some of its columns to screen.<br />
<br />
=== Project Management ===<br />
<br />
* Delete (if any) and create the database table(s) for the current set of job submissions:<br />
<pre><br />
./clear.sh<br />
</pre><br />
Also, if testing was done with jobs, it is best to delete the output directory and the configuration files:<br />
<pre><br />
rm -frv /volatile/halld/offline_monitoring/RunPeriod-20YY-MM/verVV /group/halld/data_monitoring/run_conditions/soft_comm_20YY_MM_verVV.xml /group/halld/data_monitoring/run_conditions/jana_rawdata_comm_20YY_MM_verVV.conf<br />
</pre><br />
<br />
* Search for input files matching the string in the .jproj file, and create a row for each in the job management database table (called <project_name>). You can test by adding an optional argument at the end, which only selects files with a specific file number:<br />
<pre><br />
jproj.pl <project_name> update <optional_file_number><br />
</pre><br />
<br />
* Confirm that the job management database is accurate by printing it's contents to screen:<br />
<pre><br />
mysql -hhallddb -ufarmer farming -e "select * from <project_name>"<br />
</pre><br />
<br />
* ONLY if a mistake was made, to delete the tables from the database and recreate new, empty ones, run: <br />
<pre><br />
./clear.sh<br />
</pre><br />
<br />
* To look at the status of the submitted jobs, first query auger and update the job status database:<br />
<pre><br />
fill_in_job_details.pl <project_name><br />
</pre><br />
<br />
* The job status can then be viewed by submitting a query to the job status database (called <project_name>Job (no space in between)):<br />
<pre><br />
mysql -hhallddb -ufarmer farming -e "select id,run,file,jobId,hostname,status,timeSubmitted,timeActive,walltime,cput,timeComplete,result,error from <project_name>Job"<br />
</pre><br />
<br />
* These last two commands can instead be executed simultaneously by running:<br />
<pre><br />
./status.sh<br />
</pre><br />
<br />
=== Handy mysql Instructions ===<br />
<br />
* Handy mysql instructions:<br />
<pre><br />
mysql -hhallddb -ufarmer farming # Enter the "farming" mysql database on "hallddb" as user "farmer"<br />
quit; # Exit mysql<br />
show tables; # Show a list of the tables in the current database<br />
show columns from <project_name>; # show all of the columns for the given table<br />
select * from <project_name>; # show the contents of all rows from the given table<br />
</pre><br />
<br />
=== Backing Up Offline Monitoring Tables ===<br />
Tables created for offline monitoring can be backed up using the script backup_tables.sh which<br />
can be checked out with the other files from https://halldsvn.jlab.org/repos/trunk/scripts/monitoring/jproj/projects<br />
<br />
The script uses the command mysqldump to print out a file that can be executed to recreate the tables.<br />
Since executing this output file will drop the table when it exists, caution is advised.<br />
Example usage to backup all three tables created for run period 2014_10 ver 17:<br />
<pre>backup_tables.sh 2014_10 17</pre><br />
<br />
== Running Over Data As It Comes In ==<br />
<br />
A special user gxproj1 will have a cron job set up to run the plugins as new data appears on /mss.<br />
During the week, gxproj1 will submit offline plugin jobs with the same setup as the weekly jobs<br />
run the previous Friday. The procedure for this is shown below.<br />
<br />
<!--<br />
=== Setting up the environment ===<br />
The file<br />
/home/gxproj1/setup_jlab.csh<br />
is sourced through .tcshrc.<br />
This file is the same as what is linked to by<br />
/home/gluex/setup_jlab_commissioning.csh,<br />
except HALLD_HOME, HDDS_HOME, and JANA_CALIB_URL are set separately so that this<br />
user can have a separate build.<br />
<br />
To obtain the builds from the previous Friday's runs,<br />
execute<br />
/home/gxproj1/halld/monitoring/newruns/setup_previous.sh [year] [month] [day]<br />
The build revisions from the previous Friday are archived in files<br />
/work/halld/data_monitoring/run_conditions/soft_comm_[year]_[month]_[day].xml<br />
and the script will build libraries based on those stored revision numbers.<br />
--><br />
<br />
=== Running the cron job ===<br />
<br />
'''IMPORTANT:''' The cron job should not be running while you are manually submitting jobs using the jproj.pl script for the same project, or else you will probably multiply-submit a job. <br />
<br />
* Go to the cron job directory: <br />
<pre><br />
cd /u/home/gxproj1/halld/monitoring/newruns<br />
</pre><br />
<br />
* The cron_plugins file is the cronjob that will be executed. During execution, it runs the exec.sh command in the same folder. This command takes two arguments: the project name, and the maximum file number for each run. These fields should be updated in the cron_plugins file before running. <br />
<br />
* The exec.sh command updates the job management database table with any data that has arrived on tape since it was last updated, ignoring file numbers greater than the maximum file number. It then submits jobs for these files. <br />
<br />
* To start the cron job, run:<br />
<pre><br />
crontab cron_plugins<br />
</pre><br />
<br />
* To check whether the cron job is running, do<br />
<pre><br />
crontab -l<br />
</pre><br />
<br />
* To remove the cron job do<br />
<pre><br />
crontab -r<br />
</pre><br />
<br />
<!--<br />
The cron job will run the script scan_for_jobs.sh,<br />
which runs generatejobs_plugins_rawdata.sh for any<br />
new runs that it had not seen before. All previous<br />
runs are recorded in the file filelists/files_current.txt<br />
so clear this to run over runs, or set the parameters<br />
MINRUN and MAXRUN which will set the range of runs submitted.<br />
--><br />
<br />
==Post-Processing Procedures==<br />
<br />
To visualize the monitoring data, we save images of selected histograms and store time series of selected quantities in a database, which are then displayed on a web page. This section describes how to generate the monitoring images and database information.<br />
<br />
The scripts used to generate this summary data are primarily run from /home/gxprojN/halld/monitoring/process . If you want a new copy of the scripts, e.g., for a new monitoring run, you should check the scripts out from SVN:<br />
<syntaxhighlight><br />
svn co https://halldsvn.jlab.org/repos/trunk/scripts/monitoring/process<br />
</syntaxhighlight><br />
Note that these scripts currently have some parameters which must be periodically set by hand.<br />
<br />
The default python version on most JLab machine does not have the modules to allow these scripts to connect to the MySQL database. To run these scripts, load the environment with the following command<br />
<syntaxhighlight><br />
source /home/gxproj1/halld/monitoring/process/monitoring_env.csh<br />
</syntaxhighlight><br />
<br />
===Online Monitoring===<br />
<br />
There are two primary scripts for running over the monitoring data generated by the online system and offline reconstruction. The online script can be run with either of the following commands:<br />
<syntaxhighlight><br />
/home/gluex/halld/monitoring/process/check_new_runs.py<br />
<br />
OR <br />
<br />
/home/gluex/halld/monitoring/process/check_new_runs.csh<br />
</syntaxhighlight><br />
The shell script sets up the environment properly to run the python script. To connect to the monitoring database on the JLab CUE, modules included in the local installation of python >= 2.7 are needed. The shell script is appropriate to use in a cron job. The cronjob is currently run under the "gluex" account.<br />
<br />
The online monitoring system copies a ROOT file containing the results of the online monitoring, and other configuration files into a directory accessible outside the counting house. This python script automatically checks for new ROOT files, which it will then automatically process. It contains several configuration variables that must be correctly set, which contains the location of input/output directories, etc...<br />
<br />
===Offline Monitoring===<br />
<br />
After the data is run over, the results should be processed, so that summary data is entered into the monitoring database and plots are made for the monitoring webpages. Currently, this processing is controlled by a cronjob that runs the following script:<br />
<syntaxhighlight><br />
/home/gxproj1/halld/monitoring/process/check_monitoring_data.csh <br />
</syntaxhighlight><br />
This script checks for new ROOT files, and only runs over those it hasn't processed yet. Since one monitoring ROOT file is produced for each EVIO file, whenever a new file is produced, the plots for the corresponding run are recreated and all the ROOT files for a run are combined into one file. Information is stored in the database on a per-file basis. <br />
<br />
Plots for the monitoring web page can be made from single histograms or multiple histograms using RootSpy macros. If you want to change the list of plots made, you must modify one of the following files:<br />
* histograms_to_monitor - specify either the name of the histogram or its the full ROOT path <br />
* macros_to_monitor - specify the full path to the RootSpy macro .C file<br />
<br />
When a new monitoring run is started, or the conditions are changed, the following steps should be taken to process the new files:<br />
# Add a new data version, as described below:<br />
# Change the following parameters in check_monitoring_data.csh:<br />
## JOBDATE should correspond to the ouptut date used by the job submission script<br />
## OUTPUTDIR should correspond to the directory corresponding to the run period and revision corresponding to the new version you just submitted. Presumably, this directory will be empty at the beginning.<br />
## Once you create a new data version as defined below, you should pass the needed information as a command line option. Currently this is done by the ARGS variable. For example, the argument "-v RunPeriod-2014-10,8" tells the monitoring scripts to look up the version corresponding to revision 8 of RunPeriod-2014-10 in the monitoring DB and to use to store the results.<br />
<br />
<syntaxhighlight><br />
Example configuration parameters:<br />
set JOBDATE=2015-01-09<br />
set INPUTDIR=/volatile/halld/RunPeriod-2014-10/offline_monitoring<br />
set OUTPUTDIR=/w/halld-scifs1a/data_monitoring/RunPeriod-2014-10/ver08<br />
set ARGS=" -v RunPeriod-2014-10,8 "<br />
</syntaxhighlight><br />
If you want to process the results manually, the data is processed using the following script:<br />
<syntaxhighlight><br />
./process_new_offline_data.py <input directory> <output directory><br />
<br />
EXAMPLE:<br />
<br />
./process_new_offline_data.py 2014-11-14 /volatile/halld/RunPeriod-2014-10/offline_monitoring/ /w/halld-scifs1a/data_monitoring/RunPeriod-2014-10/ver02<br />
</syntaxhighlight><br />
The python script takes several options to enable/disable various steps in the processing. Of interest is the "--force" option, which will run over all monitoring ROOT files, whether or not they've been previously identified.<br />
<br />
Every time a new reconstruction pass is performed, a new version number must be generated. To do this, prepare a version file as described below. Then run the register_new_version.py script to store the information in the database. The script will return a version number, which then should be set by hand in process_new_offline_data.py - future versions of the script will streamline this part of the procedure. An example of how to generate a new version is:<br />
<syntaxhighlight><br />
./register_new_version.py add /home/gxproj1/halld/monitoring/process/versions/vers_RunPeriod-2014-10_pass1.txt<br />
</syntaxhighlight><br />
<br />
<b>If you are running the offline monitoring by checking out the files in trunk/scripts/monitoring/jproj/projects/<br />
of the svn repository</b>, and created a project with<br />
create_project.sh [project name] hd_rawdata<br />
Then go to the directory [project name]/processing/<br />
and execute<br />
./run_processing.sh<br />
which will run register_new_version.py as well as check_monitoring_data.csh for that project.<br />
<br />
===Step-by-Step Instructions For Processing a New Monitoring Run===<br />
<br />
The monitoring runs are current run out of the gxproj1 and gxproj5 accounts. After an offline monitoring run has been successfully started on the batch farm, the following steps should be followed to setup the post-processing for these runs.<br />
<br />
# The post-processing scripts are stored in $HOME/halld/monitoring/process and are automatically run by cron.<br />
# Run "svn update" to bring any changes in. Be sure that the list of histograms and macros to plot are current.<br />
# Edit check_monitoring_data.csh to point to the current revisions/directories<br />
#* VERSION<br />
#* ARGS<br />
#* Note that the environment depends on a standard script - $HOME/setup_jlab.csh<br />
# Update files in the web directory, so that the results are displayed on the web pages: /group/halld/www/halldweb/html/data_monitoring/textdata<br />
<br />
Check log files in $HOME/halld/monitoring/process/log for more information on how each run went. If there are problems, check log files, and modify check_monitoring_data.csh to vary the verbosity of the output.<br />
<br />
==Data Versions==<br />
<br />
To document the conditions of the monitoring data that is created, for the sake of reproducability and further analysis we save several pieces of information. The format is intended to be comprehensive enough to document not just monitoring data, but versions of raw and reconstructed data, so that this database table can be used for the event database as well.<br />
<br />
We store one record per pass through one run period, with the following structure:<br />
<br />
{| class="wikitable"<br />
! Field !! Description<br />
|-<br />
| data_type || The level of data we are processing. For the purposes of monitoring, "rawdata" is the online monitoring, "recon" is the offline monitoring<br />
|- <br />
| run_period || The run period of the data<br />
|- <br />
| revision || An integer specifying which pass through the run period this data corresponds to<br />
|- <br />
| software_version || The name of the XML file that specifies the different software versions used<br />
|-<br />
| jana_config || The name of the text file that specifies which JANA options were passed to the reconstruction program<br />
|-<br />
| ccdb_context || The value of JANA_CALIB_CONTEXT, which specifies the version of calibration constants that were used<br />
|-<br />
| production_time || The data at which monitoring/reconstruction began<br />
|-<br />
| dataVersionString || A convenient string for identifying this version of the data<br />
|}<br />
<br />
<br />
An example file used as as input to ./register_new_version.py is:<br />
<syntaxhighlight><br />
data_type = recon<br />
run_period = RunPeriod-2014-10<br />
revision = 1<br />
software_version = soft_comm_2014_11_06.xml<br />
jana_config = jana_rawdata_comm_2014_11_06.conf<br />
ccdb_context = calibtime=2014-11-10<br />
production_time = 2014-11-10<br />
dataVersionString = recon_RunPeriod-2014-10_20141110_ver01<br />
</syntaxhighlight></div>Kmoriyahttps://halldweb.jlab.org/wiki/index.php?title=GlueX-Collaboration-Oct-2015&diff=70675GlueX-Collaboration-Oct-20152015-10-09T13:22:23Z<p>Kmoriya: /* Friday October 9, 2015 */</p>
<hr />
<div>== GlueX Collaboration Meeting ==<br />
<br />
<font size="+1">October 8 to 10, 2015 at Jefferson Lab</font><br />
<br />
The following template has been set up to allow people to identify what needs to be presented at the meeting. It has been roughly broken out by working group with suggestions for topics within each group. The working group chairs should work on adding the relevant talks with an estimated time (including questions) and the speaker's name. If there is a talk that does not fit into this template, please add it at the bottom.<br />
<br />
== Registration ==<br />
Everyone participating in the collaboration meeting whether in person at JLab or remotely via electronic media are encouraged to register. Please visit the<br />
[https://misportal.jlab.org/Ul/conferences/generic_conference/registration.cfm?conference_id=COLLAB-GLUEX-OCT2015 Registration Page].<br />
<br />
To see who else will be attending either in person or via electronic media, please see the<br />
[https://misportal.jlab.org/Ul/conferences/generic_conference/participants.cfm?conference_id=COLLAB-GLUEX-OCT2015 Current List of Participants Page]<br />
<br />
== Location ==<br />
CEBAF Center L102<br />
<br />
== Remote Access ==<br />
# To join via a Web Browser, go to the page [https://bluejeans.com/660743227] https://bluejeans.com/660743227.<br />
# To join via Polycom room system go to the IP Address: 199.48.152.152 ([http://bjn.vc bjn.vc]) and enter the meeting ID: 660743227.<br />
# To join via phone, use one of the following numbers and the Conference ID: 660743227.<br />
#* US or Canada: +1 408 740 7256 or <br />
#* US or Canada: +1 888 240 2560<br />
# More information on connecting to [[Connect to the Data Challenge Meetings|bluejeans]] is available.<br />
<br />
== Talks on the DocDB ==<br />
<br />
The DocDB talk-upload template for the October 2015 Collaboration Meeting can be found here:<br />
<br />
http://argus.phys.uregina.ca/cgi-bin/private/DocDB/DisplayMeeting?conferenceid=46<br />
<br />
'''Upload your talks at the link above and then edit the agenda below to enter the appropriate link to your talk based on its DocDB document number'''. <br />
<br />
Example (from the May 2008 meeting):<br />
* 15:15 Session I (90) --- Offline Working Group Meeting --- Chair Curtis Meyer<br />
** (20) [http://argus.phys.uregina.ca/cgi-bin/private/DocDB/ShowDocument?docid=1042 Offline Software Status] -- D. Lawrence<br />
<br />
-----<br />
<br />
== AGENDA == <br />
<br />
== Thursday October 8, 2015 ==<br />
* 8:30 Session Ia (110) --- [http://argus.phys.uregina.ca/cgi-bin/private/DocDB/DisplayMeeting?sessionid=569 Opening Session] - Chair: Matt Shepherd<br />
** 8:30 (10) --- Welcome --- Curtis Meyer<br />
** 8:40 (30) --- [http://argus.phys.uregina.ca/cgi-bin/private/DocDB/ShowDocument?docid=2830 Hall-D Update] --- Eugene Chudakov<br />
** 9:10 (10) --- [http://argus.phys.uregina.ca/cgi-bin/private/DocDB/ShowDocument?docid=2831 JLab IT] --- Amber Boehnlein<br />
** 9:20 (25) --- [http://argus.phys.uregina.ca/cgi-bin/private/DocDB/ShowDocument?docid=2829 Accelerator Update] --- Todd Satogata<br />
** 9:45 (25) --- [http://argus.phys.uregina.ca/cgi-bin/private/DocDB/ShowDocument?docid=2835 Engineering Update] --- Tim Whitlatch<br />
** 10:00 (30) --- [http://argus.phys.uregina.ca/cgi-bin/private/DocDB/ShowDocument?docid=2826 Electronics Update] --- Fernando Barbosa<br />
* 10:30 (30) Coffee<br />
* 11:00 Session Ib (100) --- [http://argus.phys.uregina.ca/cgi-bin/private/DocDB/DisplayMeeting?sessionid=571 DAQ/Trigger/Electronics] - (Organizer:David L. ) Chair: - Elton Smith <br />
** 11:00 (20) --- [http://argus.phys.uregina.ca/cgi-bin/private/DocDB/ShowDocument?docid=2828 FA125 Firmware] --- Naomi Jarvis<br />
** 11:20 (20) --- [http://argus.phys.uregina.ca/cgi-bin/private/DocDB/ShowDocument?docid=2836 DAQ Status] --- Sergey Furletov<br />
** 11:40 (20) --- [http://argus.phys.uregina.ca/cgi-bin/private/DocDB/ShowDocument?docid=2837 L1 Trigger Status] --- Alex Somov<br />
** 12:00 (20) --- [http://argus.phys.uregina.ca/cgi-bin/private/DocDB/ShowDocument?docid=2834 Control Status] --- Hovanes Egiyan<br />
** 12:20 (20) --- [http://argus.phys.uregina.ca/cgi-bin/private/DocDB/ShowDocument?docid=2827 Online/L3 Status] --- David Lawrence<br />
* 12:40 (80) Lunch <br />
* 12:40 (80) Collaboration Board Meeting<br />
* 14:00 Session IIa (125) --- [http://argus.phys.uregina.ca/cgi-bin/private/DocDB/DisplayMeeting?sessionid=576 Beamline and Tagger I] - (Organizer: R. Jones ) - Chair: - Zisis Papandreou<br />
** 14:00 (25) --- [http://argus.phys.uregina.ca/cgi-bin/private/DocDB/ShowDocument?docid=2839 Polarimeter Update] --- Kei Moriya<br />
** 14:25 (25) --- [http://argus.phys.uregina.ca/cgi-bin/private/DocDB/ShowDocument?docid=2840 Diamond radiator fabrication] --- Brendan Pratt<br />
** 14:50 (25) --- [http://argus.phys.uregina.ca/cgi-bin/private/DocDB/ShowDocument?docid=2841 Tagger microscope calibration] --- Alex Barnes<br />
** 15:15 (25) --- [http://argus.phys.uregina.ca/cgi-bin/private/DocDB/ShowDocument?docid=2842 Tagger hodoscope calibration] --- Nathan Sparks<br />
** 15:40 (25) --- [http://argus.phys.uregina.ca/cgi-bin/private/DocDB/ShowDocument?docid=2843 Pair Spectrometer calibration] --- Alex Somov<br />
* 16:05 (20) Coffee<br />
* 16:25 Session IIb (50) --- [http://argus.phys.uregina.ca/cgi-bin/private/DocDB/DisplayMeeting?sessionid=577 Beamline and Tagger II] - (Organizer: R. Jones ) - Chair: - Zisis Papandreou<br />
** 16:25 (25) --- [http://argus.phys.uregina.ca/gluex/DocDB/0028/002833/001/beamline_radmon_spring15.pdf Photon beam monitoring survey] --- Alexandre Deur<br />
** 16:50 (25) --- [http://argus.phys.uregina.ca/cgi-bin/private/DocDB/ShowDocument?docid=2848 Fast feedback control system] --- Trent Allison<br />
* 17:15 Session Iic (50) --- Run Planning for Fall 2015 - (Organizer: ) Chair: Kei Moriya<br />
** 17:15 (50) --- [http://argus.phys.uregina.ca/cgi-bin/private/DocDB/ShowDocument?docid=2832 Discussion/All]<br />
** [[Run Coordination Meetings: Fall 2015 Run]]<br />
* 18:15 --- Reception in the Atrium<br />
<br />
== Friday October 9, 2015 ==<br />
* 9:00 Session IIIa (75) --- [http://argus.phys.uregina.ca/cgi-bin/private/DocDB/DisplayMeeting?sessionid=576 Calorimeters] - (Organizer: Zisis Papandreou) - Chair: Beni Zihlmann<br />
** 9:00 (25) --- [http://argus.phys.uregina.ca/cgi-bin/private/DocDB/ShowDocument?docid=2849 BCAL Status, Simulations and Analysis] -- Zisis Papandreou<br />
** 9:25 (25) --- [http://argus.phys.uregina.ca/cgi-bin/private/DocDB/ShowDocument?docid=2850 BCAL Calibration/Reconstruction] -- Mark Dalton<br />
** 9:50 (25) --- [http://argus.phys.uregina.ca/cgi-bin/private/DocDB/ShowDocument?docid=2851 FCAL Update] -- Adesh Subedi<br />
* 10:15 (30) Coffee<br />
* 10:45 Session IIIb (100) --- [http://argus.phys.uregina.ca/cgi-bin/private/DocDB/DisplayMeeting?sessionid=581 Tracking] - (Organizer: Naomi Jarvis) - Chair: Justin Stevens<br />
** 10:45 (25) --- [http://argus.phys.uregina.ca/cgi-bin/private/DocDB/ShowDocument?docid=2852 CDC Update] -- Mike Staib<br />
** 11:10 (15) --- [http://argus.phys.uregina.ca/cgi-bin/private/DocDB/ShowDocument?docid=2825 Tracking Update] -- Simon Taylor<br />
** 11:25 (10) --- [http://argus.phys.uregina.ca/cgi-bin/private/DocDB/ShowDocument?docid=2838 Tracking Efficiency and Resolution] -- Paul Mattione<br />
** 11:35 (25) --- [http://argus.phys.uregina.ca/cgi-bin/private/DocDB/ShowDocument?docid=2853 FDC Update] -- Lubomir Pentchev<br />
** 12:00 (25) --- Transition Radiation Detector -- Sergey Furletov<br />
* 12:25 Lunch (95)<br />
* 14:00 Session IVa (150) --- [http://argus.phys.uregina.ca/cgi-bin/private/DocDB/DisplayMeeting?sessionid=584 Offline/Analysis] (Organizer: Mark Ito ) Chair: Dave Mack<br />
** 14:00 (25) --- [http://argus.phys.uregina.ca/cgi-bin/public/DocDB/ShowDocument?docid=2845 Overview] -- Mark Ito<br />
** 14:25 (25) --- [http://argus.phys.uregina.ca/cgi-bin/public/DocDB/ShowDocument?docid=2844 Software Version Management System] -- Mark Ito<br />
** 14:50 (25) --- [http://argus.phys.uregina.ca/cgi-bin/private/DocDB/ShowDocument?docid=2847 Software/Analysis Tutorial: &gamma;p &rarr; p&omega;] -- Paul Mattione<br />
** 15:15 (25) --- [http://argus.phys.uregina.ca/cgi-bin/private/DocDB/ShowDocument?docid=2854 Offline Monitoring and SWIF] -- Kei Moriya<br />
** 15:40 (25) --- Geant4 Development Update -- Richard Jones<br />
** 16:05 (25) --- Calibration Status -- Sean Dobbs<br />
* 16:30 Coffee (30)<br />
* 17:00 Session IVb (90) --- [http://argus.phys.uregina.ca/cgi-bin/private/DocDB/DisplayMeeting?sessionid=586 Particle ID] - (Organizer: Justin Stevens ) - Chair: Sean Dobbs<br />
** 17:00 (15) --- Start Counter Efficiencies -- Mahmoud Kamel <br />
** 17:15 (15) --- Start Counter Calibration & Performance -- Eric Pooser <br />
** 17:30 (30) --- [http://argus.phys.uregina.ca/cgi-bin/public/DocDB/ShowDocument?docid=2846 TOF Performance] -- Brad Cannon<br />
** 18:00 (30) --- DIRC Update -- John Hardin<br />
<br />
== Saturday October 10, 2015 ==<br />
* 9:00 Session Va (120) --- [http://argus.phys.uregina.ca/cgi-bin/private/DocDB/DisplayMeeting?sessionid=588 Physics I] - (Organizer: Volker Crede) - Chair: David Lawrence<br />
** 9:00 (30) --- Update on GlueX analysis efforts -- Justin Stevens <br />
** 9:30 (30) --- [http://argus.phys.uregina.ca/cgi-bin/private/DocDB/ShowDocument?docid=2820 Survey of multi-photon final states from the Spring data] -- Simon Taylor<br />
** 10:00 (30) --- &omega; Photoproduction off Nuclei: Updates and plans for a PAC proposal -- Alexander Somov<br />
** 10:30 (30) --- Recent results on meson spectroscopy from CLAS -- Paul Eugenio<br />
* 11:00 (20) Coffee<br />
* 11:20 Session Vb (30) --- [http://argus.phys.uregina.ca/cgi-bin/private/DocDB/DisplayMeeting?sessionid=588 Physics II] - (Organizer: Volker Crede) - Chair: David Lawrence<br />
** 11:20 (15) --- Omega Decays -- Michael Staib<br />
** 11:35 (15) --- Eta Decays -- Will McGinley <br />
* 11:50 (30) Session Vc (30) --- [http://argus.phys.uregina.ca/cgi-bin/private/DocDB/DisplayMeeting?sessionid=592 Business Meeting] - Chair: Matt Shepherd<br />
** 11:50 (25) --- Report from the collaboration board - David Lawrence<br />
** 12:15 (15) --- Moving forward and closeout - Curtis Meyer<br />
* 12:30 Adjourn</div>Kmoriyahttps://halldweb.jlab.org/wiki/index.php?title=GlueX-Collaboration-Oct-2015&diff=70641GlueX-Collaboration-Oct-20152015-10-08T17:38:11Z<p>Kmoriya: /* Thursday October 8, 2014 */</p>
<hr />
<div>== GlueX Collaboration Meeting ==<br />
<br />
<font size="+1">October 8 to 10, 2015 at Jefferson Lab</font><br />
<br />
The following template has been set up to allow people to identify what needs to be presented at the meeting. It has been roughly broken out by working group with suggestions for topics within each group. The working group chairs should work on adding the relevant talks with an estimated time (including questions) and the speaker's name. If there is a talk that does not fit into this template, please add it at the bottom.<br />
<br />
== Registration ==<br />
Everyone participating in the collaboration meeting whether in person at JLab or remotely via electronic media are encouraged to register. Please visit the<br />
[https://misportal.jlab.org/Ul/conferences/generic_conference/registration.cfm?conference_id=COLLAB-GLUEX-OCT2015 Registration Page].<br />
<br />
To see who else will be attending either in person or via electronic media, please see the<br />
[https://misportal.jlab.org/Ul/conferences/generic_conference/participants.cfm?conference_id=COLLAB-GLUEX-OCT2015 Current List of Participants Page]<br />
<br />
== Location ==<br />
CEBAF Center L102<br />
<br />
== Remote Access ==<br />
# To join via a Web Browser, go to the page [https://bluejeans.com/660743227] https://bluejeans.com/660743227.<br />
# To join via Polycom room system go to the IP Address: 199.48.152.152 ([http://bjn.vc bjn.vc]) and enter the meeting ID: 660743227.<br />
# To join via phone, use one of the following numbers and the Conference ID: 660743227.<br />
#* US or Canada: +1 408 740 7256 or <br />
#* US or Canada: +1 888 240 2560<br />
# More information on connecting to [[Connect to the Data Challenge Meetings|bluejeans]] is available.<br />
<br />
== Talks on the DocDB ==<br />
<br />
The DocDB talk-upload template for the October 2015 Collaboration Meeting can be found here:<br />
<br />
http://argus.phys.uregina.ca/cgi-bin/private/DocDB/DisplayMeeting?conferenceid=46<br />
<br />
'''Upload your talks at the link above and then edit the agenda below to enter the appropriate link to your talk based on its DocDB document number'''. <br />
<br />
Example (from the May 2008 meeting):<br />
* 15:15 Session I (90) --- Offline Working Group Meeting --- Chair Curtis Meyer<br />
** (20) [http://argus.phys.uregina.ca/cgi-bin/private/DocDB/ShowDocument?docid=1042 Offline Software Status] -- D. Lawrence<br />
<br />
-----<br />
<br />
== AGENDA == <br />
<br />
== Thursday October 8, 2014 ==<br />
* 8:30 Session Ia (110) --- [http://argus.phys.uregina.ca/cgi-bin/private/DocDB/DisplayMeeting?sessionid=499 Opening Session] - Chair: Matt Shepherd<br />
** 8:30 (10) --- Welcome --- Curtis Meyer<br />
** 8:40 (30) --- [http://argus.phys.uregina.ca/cgi-bin/private/DocDB/ShowDocument?docid=2830 Hall-D Update] --- Eugene Chudakov<br />
** 9:10 (10) --- [http://argus.phys.uregina.ca/cgi-bin/private/DocDB/ShowDocument?docid=2831 JLab IT] --- Amber Boehnlein<br />
** 9:20 (25) --- [http://argus.phys.uregina.ca/cgi-bin/private/DocDB/ShowDocument?docid=2829 Accelerator Update] --- Todd Satogata<br />
** 9:45 (25) --- [http://argus.phys.uregina.ca/cgi-bin/private/DocDB/ShowDocument?docid=2835 Engineering Update] --- Tim Whitlatch<br />
** 10:00 (30) --- [http://argus.phys.uregina.ca/cgi-bin/private/DocDB/ShowDocument?docid=2826 Electronics Update] --- Fernando Barbosa<br />
* 10:30 (30) Coffee<br />
* 11:00 Session Ib (100) --- DAQ/Trigger/Electronics - (Organizer:David L. ) Chair: - Elton Smith <br />
** 11:00 (20) --- [http://argus.phys.uregina.ca/cgi-bin/private/DocDB/ShowDocument?docid=2828 FA125 Firmware] --- Naomi Jarvis<br />
** 11:20 (20) --- [http://argus.phys.uregina.ca/cgi-bin/private/DocDB/ShowDocument?docid=2836 DAQ Status] --- Sergey Furletov<br />
** 11:40 (20) --- [http://argus.phys.uregina.ca/cgi-bin/private/DocDB/ShowDocument?docid=2837 L1 Trigger Status] --- Alex Somov<br />
** 12:00 (20) --- [http://argus.phys.uregina.ca/cgi-bin/private/DocDB/ShowDocument?docid=2834 Control Status] --- Hovanes Egiyan<br />
** 12:20 (20) --- [http://argus.phys.uregina.ca/cgi-bin/private/DocDB/ShowDocument?docid=2827 Online/L3 Status] --- David Lawrence<br />
* 12:40 (80) Lunch <br />
* 12:40 (80) Collaboration Board Meeting<br />
* 14:00 Session IIa (160) --- Beamline and Tagger - (Organizer: R. Jones ) - Chair: - Zisis Papandreou<br />
** 14:00 (25) --- Diamond radiator fabrication --- Brendan Pratt<br />
** 14:25 (25) --- [http://argus.phys.uregina.ca/cgi-bin/private/DocDB/ShowDocument?docid=2839 Polarimeter Update] --- Kei Moriya<br />
** 14:50 (25) --- Tagger hodoscope calibration --- Nathan Sparks<br />
** 15:15 (25) --- Tagger microscope calibration --- Alex Barnes<br />
** 15:40 (25) --- Pair Spectrometer calibration --- Alex Somov<br />
* 16:05 (20) Coffee<br />
** 16:25 (25) --- [http://argus.phys.uregina.ca/gluex/DocDB/0028/002833/001/beamline_radmon_spring15.pdf Photon beam monitoring survey] --- Alexandre Deur<br />
** 16:50 (25) --- Fast feedback control system --- Trent Allison<br />
* 17:15 Session IIb (50) --- Run Planning for Fall 2015 - (Organizer: ) Chair: Kei Moriya<br />
** 17:15 (50) --- [http://argus.phys.uregina.ca/cgi-bin/private/DocDB/ShowDocument?docid=2832 Discussion/All]<br />
** [[Run Coordination Meetings: Fall 2015 Run]]<br />
* 18:15 --- Reception in the Atrium<br />
<br />
== Friday October 9, 2015 ==<br />
* 9:00 Session IIIa (75) --- Calorimeters - (Organizer: Zisis Papandreou) - Chair: Beni Zihlmann<br />
** 9:00 (25) --- BCAL Status, Simulations and Analysis -- Zisis Papandreou<br />
** 9:25 (25) --- BCAL Calibration/Reconstruction -- Mark Dalton<br />
** 9:50 (25) --- FCAL Update -- Adesh Subedi<br />
* 10:15 (30) Coffee<br />
* 10:45 Session IIIb (100) --- Tracking - (Organizer: Naomi Jarvis) - Chair: Justin Stevens<br />
** 10:45 (25) --- CDC Update -- Mike Staib<br />
** 11:10 (15) --- Tracking Update -- Simon Taylor<br />
** 11:25 (10) --- [http://argus.phys.uregina.ca/cgi-bin/private/DocDB/ShowDocument?docid=2838 Tracking Efficiency and Resolution] -- Paul Mattione<br />
** 11:35 (25) --- FDC Update -- Lubomir Pentchev<br />
** 12:00 (25) --- Transition Radiation Detector -- Sergey Furletov<br />
* 12:25 Lunch (95)<br />
* 14:00 Session IVa (150) --- Offline/Analysis (Organizer: Mark Ito ) Chair: Dave Mack<br />
** 14:00 (25) --- Overview -- Mark Ito<br />
** 14:25 (25) --- Software Version Management System -- Mark Ito<br />
** 14:50 (25) --- Software/Analysis Tutorial: &gamma;p &rarr; p&omega; -- Paul Mattione<br />
** 15:15 (25) --- Offline Monitoring and SWIF -- Kei Moriya<br />
** 15:40 (25) --- Geant4 Development Update -- Richard Jones<br />
** 16:05 (25) --- Calibration Status -- Sean Dobbs<br />
* 16:30 Coffee (30)<br />
* 17:00 Session IVb (90) --- Particle ID - (Organizer: Justin Stevens ) - Chair: Sean Dobbs<br />
** 17:00 (15) --- Start Counter Efficiencies -- Mahmoud Kamel <br />
** 17:15 (15) --- Start Counter Calibration & Performance -- Eric Pooser <br />
** 17:30 (30) --- TOF Performance -- Brad Cannon<br />
** 18:00 (30) --- DIRC Update -- John Hardin<br />
<br />
== Saturday October 10, 2015 ==<br />
* 9:00 Session Va (120) --- Physics - (Organizer: Volker Crede) - Chair: David Lawrence<br />
** 9:00 (30) --- Update on GlueX analysis efforts -- Justin Stevens <br />
** 9:30 (30) --- Survey of multi-photon final states from the Spring data -- Simon Taylor<br />
** 10:00 (30) --- &omega; Photoproduction off Nuclei: Updates and plans for a PAC proposal -- Alexander Somov<br />
** 10:30 (30) --- Recent results on meson spectroscopy from CLAS -- Paul Eugenio<br />
* 11:00 (20) Coffee<br />
** 11:20 (15) --- Omega Decays -- Michael Staib<br />
** 11:35 (15) --- Eta Decays -- Will McGinley <br />
* 11:50 (30) Session Vb (90) --- Business Meeting - Chair: Matt Shepherd<br />
** 11:50 (25) --- Report from the collaboration board - David Lawrence<br />
** 12:15 (15) --- Moving forward and closeout - Curtis Meyer<br />
* 12:30 Adjourn</div>Kmoriyahttps://halldweb.jlab.org/wiki/index.php?title=GlueX_Offline_Meeting,_September_30,_2015&diff=70471GlueX Offline Meeting, September 30, 20152015-09-30T17:41:03Z<p>Kmoriya: /* Agenda */</p>
<hr />
<div>GlueX Offline Software Meeting<br><br />
Wednesday, September 30, 2015<br><br />
1:30 pm EDT<br><br />
JLab: CEBAF Center F326/327<br />
<br />
==Agenda==<br />
<br />
# Announcements<br />
## [https://github.com/orgs/JeffersonLab/teams?utf8=%E2%9C%93&query=%40markito3 Team Maintainers and Admins] (Mark)<br />
## [[GlueX-Collaboration-Oct-2015|Collaboration Meeting]] October 8-10, 2015 at Jefferson Lab<br />
# Review of [[GlueX Offline Meeting, September 16, 2015#Minutes|minutes from September 16]] (all)<br />
# Geant4 Update (Richard, David)<br />
# [https://halldweb1.jlab.org/wiki/images/e/e5/2015-09-30-offline_monitoring.pdf Offline Monitoring (Kei)] [https://halldweb.jlab.org/data_monitoring/launch_analysis/tmp/summary_swif_output_offline_monitoring_RunPeriod2015_03_ver15_hd_rawdata.html SWIF output]<br />
# [[Data Challenge 3]] update (Mark)<br />
# [[Spring 2015 Commissioning Simulations]]<br />
# Fall 2015 Commissioning Simulations (all)<br />
# Auto-Build on Pull Request (Sean)<br />
# [https://halldweb.jlab.org/wiki/images/b/b9/Sdobbs_OfflineMtg_20150930.pdf Noise Studies] (Sean)<br />
# [[Automatic Tests of GlueX Software|b1pi results review]]<br />
# Review of [https://github.com/JeffersonLab/sim-recon/pulls?q=is%3Aopen+is%3Apr recent pull requests]<br />
#* comments on merge<br />
#* alternate workflows for submitting pull requests<br />
#* rebasing?<br />
# Action Item Review<br />
<br />
==Communication Information==<br />
<br />
===Remote Connection===<br />
<br />
* The BlueJeans meeting number is 968 592 007 .<br />
* [http://bluejeans.com/968592007 Join the Meeting] via BlueJeans<br />
<br />
===Slides===<br />
<br />
Talks can be deposited in the directory <code>/group/halld/www/halldweb/html/talks/2015</code> on the JLab CUE. This directory is accessible from the web at https://halldweb.jlab.org/talks/2015/ .</div>Kmoriyahttps://halldweb.jlab.org/wiki/index.php?title=GlueX_Offline_Meeting,_September_30,_2015&diff=70470GlueX Offline Meeting, September 30, 20152015-09-30T17:32:52Z<p>Kmoriya: /* Agenda */</p>
<hr />
<div>GlueX Offline Software Meeting<br><br />
Wednesday, September 30, 2015<br><br />
1:30 pm EDT<br><br />
JLab: CEBAF Center F326/327<br />
<br />
==Agenda==<br />
<br />
# Announcements<br />
## [https://github.com/orgs/JeffersonLab/teams?utf8=%E2%9C%93&query=%40markito3 Team Maintainers and Admins] (Mark)<br />
## [[GlueX-Collaboration-Oct-2015|Collaboration Meeting]] October 8-10, 2015 at Jefferson Lab<br />
# Review of [[GlueX Offline Meeting, September 16, 2015#Minutes|minutes from September 16]] (all)<br />
# Geant4 Update (Richard, David)<br />
# [https://halldweb1.jlab.org/wiki/images/e/e5/2015-09-30-offline_monitoring.pdf Offline Monitoring (Kei)]<br />
# [[Data Challenge 3]] update (Mark)<br />
# [[Spring 2015 Commissioning Simulations]]<br />
# Fall 2015 Commissioning Simulations (all)<br />
# Auto-Build on Pull Request (Sean)<br />
# [https://halldweb.jlab.org/wiki/images/b/b9/Sdobbs_OfflineMtg_20150930.pdf Noise Studies] (Sean)<br />
# [[Automatic Tests of GlueX Software|b1pi results review]]<br />
# Review of [https://github.com/JeffersonLab/sim-recon/pulls?q=is%3Aopen+is%3Apr recent pull requests]<br />
#* comments on merge<br />
#* alternate workflows for submitting pull requests<br />
#* rebasing?<br />
# Action Item Review<br />
<br />
==Communication Information==<br />
<br />
===Remote Connection===<br />
<br />
* The BlueJeans meeting number is 968 592 007 .<br />
* [http://bluejeans.com/968592007 Join the Meeting] via BlueJeans<br />
<br />
===Slides===<br />
<br />
Talks can be deposited in the directory <code>/group/halld/www/halldweb/html/talks/2015</code> on the JLab CUE. This directory is accessible from the web at https://halldweb.jlab.org/talks/2015/ .</div>Kmoriyahttps://halldweb.jlab.org/wiki/index.php?title=File:2015-09-30-offline_monitoring.pdf&diff=70469File:2015-09-30-offline monitoring.pdf2015-09-30T17:32:23Z<p>Kmoriya: Offline monitoring report by Kei for offline meeting 2015/09/30</p>
<hr />
<div>Offline monitoring report by Kei for offline meeting 2015/09/30</div>Kmoriyahttps://halldweb.jlab.org/wiki/index.php?title=GlueX_Offline_Meeting,_September_16,_2015&diff=70103GlueX Offline Meeting, September 16, 20152015-09-16T17:26:36Z<p>Kmoriya: /* Agenda */</p>
<hr />
<div>GlueX Offline Software Meeting<br><br />
Wednesday, September 16, 2015<br><br />
1:30 pm EDT<br><br />
JLab: CEBAF Center F326/327<br />
<br />
==Agenda==<br />
<br />
# Announcements<br />
## [https://mailman.jlab.org/pipermail/halld-offline/2015-September/002144.html "git_update" simple email list] (Mark)<br />
## [[Scripts for Installing GlueX Software|gluex_install]] moved to GitHub (Mark)<br />
## [https://mailman.jlab.org/pipermail/halld-offline/2015-September/002143.html ROOT TTree Format Overhaul] (Paul)<br />
## [https://halldweb.jlab.org/wiki/images/1/15/2015-09-16-offline_monitoring.pdf Offline Monitoring] (Kei)<br />
# Review of [[GlueX Offline Meeting, September 2, 2015#Minutes|minutes from September 2]] (all)<br />
# [[GlueX-Collaboration-Oct-2015|Collaboration Meeting]] October 8-10, 2015 at Jefferson Lab<br />
# Geant4 Update (Richard, David)<br />
# [[Data Challenge 3]] update (Mark)<br />
# [[Spring 2015 Commissioning Simulations]]<br />
# Fall 2015 Commissioning Simulations (all)<br />
# Auto-Build on Pull Request (Sean)<br />
# Review of [https://github.com/JeffersonLab/sim-recon/pulls?q=is%3Aopen+is%3Apr recent pull requests]<br />
# Action Item Review<br />
<br />
==Communication Information==<br />
<br />
===Remote Connection===<br />
<br />
* The BlueJeans meeting number is 968 592 007 .<br />
* [http://bluejeans.com/968592007 Join the Meeting] via BlueJeans<br />
<br />
===Slides===<br />
<br />
Talks can be deposited in the directory <code>/group/halld/www/halldweb/html/talks/2015</code> on the JLab CUE. This directory is accessible from the web at https://halldweb.jlab.org/talks/2015/ .</div>Kmoriyahttps://halldweb.jlab.org/wiki/index.php?title=File:2015-09-16-offline_monitoring.pdf&diff=70102File:2015-09-16-offline monitoring.pdf2015-09-16T17:26:09Z<p>Kmoriya: Talk about offline monitoring 2015-03 ver14 given at offline meeting September 16 2015 by Kei</p>
<hr />
<div>Talk about offline monitoring 2015-03 ver14 given at offline meeting September 16 2015 by Kei</div>Kmoriya