Difference between revisions of "Offline Monitoring Archived Data"

From GlueXWiki
Jump to: navigation, search
(→‎Starting a new run period)
Line 1: Line 1:
 
= Setup the Software & Environment =
 
= Setup the Software & Environment =
  
−
'''1)''' Update the version xml file to indicate which versions of the software will be used. For hdds & sim-recon, set the names to those of the tags that you will be about to create (in a later step below):
+
'''1)''' Checkout (or svn update) the launch scripts if needed:
 
<pre>
 
<pre>
−
~/builds/version_monitoring_launch.xml
+
cd ~/
 +
svn co https://halldsvn.jlab.org/repos/trunk/scripts/monitoring/launch
 
</pre>
 
</pre>
  
−
'''2)''' Source the environment.  It may be tempting, but do NOT put this in the auto-exec-on-login scripts. It is dangerous, in case you need to switch between environments. Just call:  
+
'''2)''' You will need only the following 3 files:
 
<pre>
 
<pre>
−
bash
+
launch.py
−
source ~/env_monitoring_launch.sh
+
script.sh
 +
jobs_analysis_bggen.config
 
</pre>
 
</pre>
  
−
'''3)''' Updating & building hdds:
 
−
<pre>
 
−
cd $HDDS_HOME
 
−
git pull                # Get latest software
 
−
scons -c install        # Clean out the old install: EXTREMELY IMPORTANT for cleaning out stale headers
 
−
scons install -j4      # Rebuild and re-install with 4 threads
 
−
</pre>
 
  
−
'''4)''' Updating & building halld_recon:
+
= Prepare the Launch =
−
<pre>
+
−
cd $HALLD_RECON_HOME/src
+
−
git pull                # Get latest software
+
−
scons -c install        # Clean out the old install: EXTREMELY IMPORTANT for cleaning out stale headers
+
−
scons install -j4      # Rebuild and re-install with 4 threads
+
−
</pre>
+
  
−
'''5)''' Create a new sqlite file containing the very latest calibration constants. Original documentation on creating sqlite files are [https://halldweb.jlab.org/wiki/index.php/SQLite-form_of_the_CCDB_database here].  
+
'''1)''' Update the jobs_analysis_bggen.config file. Definitely be sure to update '''<span style="color:red">RUNPERIOD</span>''', '''<span style="color:red">VERSION</span>''', and '''<span style="color:red">BATCH</span>''' where appropriate (if jobs submitted in batches).
 
<pre>
 
<pre>
−
cd `dirname $SQLITE_PATH`
+
ENVFILE                analysis-2017_01-ver46.xml
−
$CCDB_HOME/scripts/mysql2sqlite/mysql2sqlite.sh -hhallddb.jlab.org -uccdb_user ccdb | sqlite3 ccdb.sqlite
+
−
mv ccdb.sqlite ccdb_monitoring_launch.sqlite
+
 
</pre>
 
</pre>
  
−
'''6)''' Tag the software, where '''<span style="color:red">"<type>"</span>''' below is either '''<span style="color:red">"offmon"</span>''' for offline monitoring launches, or '''<span style="color:red">"recon"</span>''' for full reconstruction launches:  
+
'''2)''' Make a new jana_analysis.config with your channel:
 
<pre>
 
<pre>
−
cd $HALLD_HOME/src/
+
PLUGINS monitoring_hists,ReactionFilter
−
git tag -a <type>-20YY_MM-verVV -m "Used for offline monitoring 20YY-MM verVV"
+
NTHREADS 16
−
git push origin <type>-20YY_MM-verVV
+
COMBO:MAX_NEUTRALS 15
−
 
+
Reaction1  1_14__8_9_14                # pi+ pi- p
−
cd $HDDS_HOME/
+
Reaction1:Flags B4
−
git tag -a <type>-20YY_MM-verVV -m "Used for offline monitoring 20YY-MM verVV"
+
−
git push origin <type>-20YY_MM-verVV
+
 
</pre>
 
</pre>
  
−
'''7)''' Checkout (or svn update) the launch scripts if needed:
 
−
<pre>
 
−
cd ~/
 
−
svn co https://halldsvn.jlab.org/repos/trunk/scripts/monitoring/
 
−
</pre>
 
−
 
−
= Starting a new run period =
 
−
When a new run period is started, it is best to make sure all top-level directories are created with the right permissions. This can save headaches later on when a different gxprojN account is used for offline monitoring.
 
−
 
−
'''1)''' Create top-level directories
 
−
<pre>
 
−
/cache/halld/RunPeriod-20YY-MM/recon/
 
−
/cache/halld/offline_monitoring/RunPeriod-20YY-MM/
 
−
/work/halld2/recon/RunPeriod-20YY-MM/
 
−
/work/halld/data_monitoring/RunPeriod-20YY-MM/
 
−
/work/halld2/data_monitoring/RunPeriod-20YY-MM/
 
−
/group/halld/data_monitoring/run_conditions/RunPeriod-20YY-MM/
 
−
</pre>
 
−
 
−
'''2)''' Make sure other gxprojN users can write in with '''<span style="color:red">chmod g+w</span>''' [dir name] (NB: use '''<span style="color:red">jcache chmod</span>''' for /cache).<br> Check that permissions match those from previous run periods.
 
−
 
−
'''3)''' ???Deprecated??? Since the publish-to-web scripts are webpage-update scripts, you need to have pre-existing, template versions of a few files for the new run period.  It also expects to find comment hooks that include the new run period name, so make sure those are edited in the new template files.
 
−
 
−
= Prepare and Do the Launch =
 
−
 
−
'''1)''' Update the appropriate job config file, depending on the type of launch (e.g. '''<span style="color:red">jobs_offmon.config</span>''' for monitoring launches, '''<span style="color:red">jobs_recon.config</span>''' for full reconstruction). Definitely be sure to update '''<span style="color:red">RUNPERIOD</span>''', '''<span style="color:red">VERSION</span>''', and '''<span style="color:red">BATCH</span>''' where appropriate (if jobs submitted in batches).
 
−
<pre>
 
−
~/monitoring/launch/jobs_offmon.config
 
−
</pre>
 
−
 
−
'''2)''' Update the appropriate jana config file, depending on the type of launch (e.g. '''<span style="color:red">jana_offmon.config</span>''' for monitoring launches, '''<span style="color:red">jana_recon.config</span>''' for full reconstruction). This contains the command line arguments given to JANA. Definitely be sure to update '''<span style="color:red">REST:DATAVERSIONSTRING</span>''' and '''<span style="color:red">JANA_CALIB_CONTEXT</span>'''.
 
−
<pre>
 
−
~/monitoring/launch/jana_offmon.config
 
−
</pre>
 
  
 
'''3)''' Create the SWIF workflow.  The workflow should have a name like '''<span style="color:red">"recon_2016-02_ver05"</span>''' for monitoring launches and '''<span style="color:red">"recon_2016-02_ver01_batch01"</span>''' for full reconstruction launches.  It should also match the workflow name in the job config file (e.g. jobs_offmon.config).  
 
'''3)''' Create the SWIF workflow.  The workflow should have a name like '''<span style="color:red">"recon_2016-02_ver05"</span>''' for monitoring launches and '''<span style="color:red">"recon_2016-02_ver01_batch01"</span>''' for full reconstruction launches.  It should also match the workflow name in the job config file (e.g. jobs_offmon.config).  
Line 86: Line 37:
 
</pre>
 
</pre>
  
−
'''4)''' Backup the software versions & appropriate jana config script, '''<span style="color:red">using appropriate names</span>''' for the launch, '''<span style="color:red">e.g.</span>''':
 
−
<pre>
 
−
cp ~/builds/version_monitoring_launch.xml /group/halld/data_monitoring/run_conditions/RunPeriod-2016-02/version_recon_2016_02_ver01.xml
 
−
cp ~/monitoring/launch/jana_recon.config /group/halld/data_monitoring/run_conditions/RunPeriod-2016-02/jana_recon_2016_02_ver01_batch01.config
 
−
cp ~/monitoring/launch/jobs_recon.config /group/halld/data_monitoring/run_conditions/RunPeriod-2016-02/jobs_recon_2016_02_ver01_batch01.config
 
−
</pre>
 
  
−
'''5)''' Register jobs for the workflow, where '''<span style="color:red"><job_config_file></span>''' is (e.g.) '''<span style="color:red">"~/monitoring/launch/jobs_offmon.config"</span>''':
+
'''4)''' Register jobs for the workflow, where '''<span style="color:red"><job_config_file></span>''' is (e.g.) '''<span style="color:red">"~/monitoring/launch/jobs_offmon.config"</span>''':
 
<pre>
 
<pre>
 
~/monitoring/launch/launch.py <job_config_file> <run_min> <run_max>
 
~/monitoring/launch/launch.py <job_config_file> <run_min> <run_max>
Line 102: Line 47:
 
</pre>
 
</pre>
  
−
'''6)''' Run the workflow:
+
'''5)''' Run the workflow:
 
<pre>
 
<pre>
 
swif run -workflow <my_workflow>
 
swif run -workflow <my_workflow>
 
</pre>
 
</pre>

Revision as of 18:04, 12 November 2021

Setup the Software & Environment

1) Checkout (or svn update) the launch scripts if needed: 

cd ~/
svn co https://halldsvn.jlab.org/repos/trunk/scripts/monitoring/launch

2) You will need only the following 3 files: 

launch.py
script.sh
jobs_analysis_bggen.config


Prepare the Launch

1) Update the jobs_analysis_bggen.config file. Definitely be sure to update RUNPERIOD, VERSION, and BATCH where appropriate (if jobs submitted in batches). 

ENVFILE                analysis-2017_01-ver46.xml

2) Make a new jana_analysis.config with your channel: 

PLUGINS monitoring_hists,ReactionFilter
NTHREADS 16
COMBO:MAX_NEUTRALS 15
Reaction1  1_14__8_9_14                 # pi+ pi- p
Reaction1:Flags B4


3) Create the SWIF workflow. The workflow should have a name like "recon_2016-02_ver05" for monitoring launches and "recon_2016-02_ver01_batch01" for full reconstruction launches. It should also match the workflow name in the job config file (e.g. jobs_offmon.config). 

swif create -workflow <my_workflow>


4) Register jobs for the workflow, where <job_config_file> is (e.g.) "~/monitoring/launch/jobs_offmon.config": 

~/monitoring/launch/launch.py <job_config_file> <run_min> <run_max>

You can optionally specify specific file numbers to use. For example, to submit jobs for the first 5 files of each run: 

~/monitoring/launch/launch.py <job_config_file> <run_min> <run_max> -f '00[0-4]'

5) Run the workflow: 

swif run -workflow <my_workflow>
Retrieved from "https://halldweb.jlab.org/wiki/index.php?title=Offline_Monitoring_Archived_Data&oldid=108850"