Locking in JANA

From GlueXWiki
Jump to: navigation, search

General Comments

  • Be careful when creating / filling histograms & trees.
    • Be VERY careful when doing so inside of a factory or action (follow the example code!).
  • Be careful when modifying / reading JEventProcessor class member data.
  • Be careful when returning early from functions: Make sure you release the lock first.
  • Avoid nested locks.

Basics: Factories & Plugin Processors

  • Plugin Processors: For each plugin, there is a total of one instance of the processor object, regardless of how many threads you run with.
    • Thus, since the same object is shared amongst all threads, you must be careful about modifying it.
  • Factories: Each thread has it's own instance of each factory.
    • Thus, you can easily modify them without worrying about interfering with other threads.
    • However, sharing/modifying objects for a factory between the threads is difficult, and you must be careful about doing so.

Plugin Processor Variables

Basics

  • Do not place use any locks in JEventProcessor::init() or ::fini(): they are unnecessary, and a waste of time.
    • JANA is guaranteed to be single-threaded in these functions.
  • Do not use or modify any global variables, unless you are in a lock.
    • Or even better, just avoid them at all possible. It's terrible programming practice.

Class members

  • Do NOT use or modify any class member-variables that you define for your JEventProcessor (i.e. in header file), unless you are in a lock (or in init(), fini()). These variables can be modified by other threads. This locking can be done via:
LockState(); //ACQUIRE PROCESSOR LOCK
UnlockState(); //RELEASE PROCESSOR LOCK
  • For data updated in brun() and read in evnt(), it is sometimes unfeasible to work around this, such as when working with calibration constants. As long as the run number doesn't change, this should still be OK (they should just be overwriting the same value).
    • However, running your program over data with two different runs may result in unpredictable behavior.
  • Just to be safe, have as few class member variables as possible, and reduce the amount of time spent in locks. In other words, use function-scope (local) variables instead.
    • Where possible of course. E.g. you can't avoid it with histograms, but you need to lock to fill those anyway.

DTreeInterface

  • It's designed to make saving data to a TTree (and managing the TFile) thread-safe and robust. See instructions at: Link

Creating Histograms, TTrees, TDirectories, or TFiles in a plugin processor

  • If you are creating them inside of JEventProcessor::init() or ::fini(), you do not need a lock. Just create them.
  • If you are creating them outside of these function (e.g. brun()): Use the global ROOT JANA lock functions below.
    • This is because they modify gDirectory, or depend on gDirectory being constant.
japp->RootWriteLock(); //ACQUIRE ROOT LOCK
japp->RootUnLock(); //RELEASE ROOT LOCK

Creating Histograms, TTrees, TDirectories, or TFiles somewhere else (e.g. a factory, or an analysis action)

  • Remember, each thread has it's own instance of each factory.
    • This is also true of analysis actions, unless they are created directly in the plugin processor.
  • Thus, you must be careful to only create these objects once, and not once for each thread.
    • Init() and fini() are not inherently safe: They are called once on each object, so once by each thread.

Example: Creating a histogram

  • Directories and files can be created similarly.
//Factory header file:
TH1I* dMyHist;
 
//Creation code:
japp->RootWriteLock(); //ACQUIRE ROOT LOCK
{
   dMyHist = (TH1I*)gDirectory->Get("my_hist");
   if(dMyHist == NULL) //else already created by another thread, and already retrieved
      dMyHist = new TH1I("my_hist", "title", 100, 0.0, 1.0);
}
japp->RootUnLock(); //RELEASE ROOT LOCK

Example: Creating a TTree

  • You must be extra careful here. Not only do you need to do something similar to the histogram-creation code, but you must be careful about the branch memory.
  • Each factory/action must have access to the same branch memory objects for the TTree, so they can set them prior to filling. Either that, or you have to change all of the branch addresses prior to filling every time.
    • If you are simply filling branches with fundamental data types (floats, ints, etc.) or arrays, then you can just let the TTree keep track of the memory for you.
    • For other objects, you must keep track of the memory objects yourself. To safely share them amongst threads, you must use static variables that are defined locally within functions, and return references to them. These functions should only be called within a lock. See src/libraries/ANALYSIS/DEventWriterROOT.* for an example.
  • Example code:
//Factory header file:
TTree* dTree;
 
//Creation code:
japp->RootWriteLock(); //ACQUIRE ROOT LOCK
{
   dTree = (TH1I*)gDirectory->Get("my_tree");
   if(dTree == NULL) //else already created by another thread, and already retrieved
   {
      dTree = new TTree("my_tree", "title");
      dTree->Branch("my_branch", new Float_t[my_size], "my_branch[my_size_name]/F");
   }
}
japp->RootUnLock(); //RELEASE ROOT LOCK
 
//Partial Filling Code (assume already in a lock)
Float_t* locBranchAddress = (Float_t*)locTree->GetBranch("my_branch")->GetAddress();
locBranchAddress[loc_i] = value;

Filling Histograms

  • If you are filling histograms, you do NOT need to grab the global ROOT JANA lock functions. Try to instead grab a lock with a smaller scope, so that other threads can write to other histograms that don't interfere with yours.

Within a plugin processor

  • In you are directly in the plugin processor, there's a special function to do this, that locks for the current plugin only. It is:
japp->RootFillLock(this); //ACQUIRE ROOT FILL LOCK //this: JEventProcessor pointer
{
   my_hist->Fill(my_value);
}
japp->RootFillUnLock(this); //RELEASE ROOT FILL LOCK //this: JEventProcessor pointer

Within an analysis action

  • Each action has it's own unique lock. Simply use it when filling with:
//Since we are filling histograms local to this action, it will not interfere with other ROOT operations: can use action-wide ROOT lock
//Note, the mutex is unique to this DReaction + action_string combo: actions of same class with different hists will have a different mutex
Lock_Action(); //ACQUIRE ACTION LOCK!!
{
   my_hist->Fill(my_value);
}
Unlock_Action(); //RELEASE ACTION LOCK!!

Within a factory

  • You can acquire a lock unique to this factory, by calling something similar to the below:
//If the lock below hasn't been created yet, it is created prior to being grabbed. 
japp->WriteLock("my_factory"); //ACQUIRE FACTORY LOCK!! //technically, just the lock corresponding to this name
{
   my_hist->Fill(my_value);
}
japp->Unlock("my_factory"); //RELEASE FACTORY LOCK!!

Filling TTrees

  • You should really use the DTreeInterface instead. See instructions in that section.

hd_root.root

  • If you are filling TTrees to the global ROOT file (i.e. hd_root.root), write to the TTrees with the global ROOT JANA lock functions below:
    • This is because TTree::Fill() periodically writes to the file, modifying it.
japp->RootWriteLock(); //ACQUIRE ROOT LOCK
{
   //Set branch data
   dTree->Fill();
}
japp->RootUnLock(); //RELEASE ROOT LOCK

Other Output Files

  • If you are filling TTrees to a local ROOT file (i.e. NOT hd_root.root), you do NOT need to grab the global ROOT JANA lock functions. Try to instead grab a lock with a smaller scope, so that other threads can write to other trees that don't interfere with yours.
    • Grab the same for every tree that is written to the same output file. This is because TTree::Fill() periodically writes to the file, modifying it.
  • This can be done with:
japp->WriteLock("MyFileName"); //ACQUIRE FILE LOCK
{
   //Set branch data
   dTree->Fill();
}
japp->Unlock("MyFileName"); //RELEASE FILE LOCK

Writing-to and Closing TFiles in a plugin processor

  • Simply write-to and close the file in fini(), without a lock.

Writing-to and Closing TFiles in a factory or action

  • You must be VERY careful here. You don't want to close the file until all threads have finished writing.
  • To do this, you must keep track of the number of threads:
    • Increment a static counter in the factory/action constructor.
      • For safety, define the counter as a local static variable within a function, return it by reference, and only read/modify it in a lock.
    • Decrement it in the destructor. When the counter drops to zero, write-to and close the file. All within a lock, of course.
  • For an example, see sim-recon/src/libraries/ANALYSIS/DEventWriterROOT.*

Nested Locks

Overview

  • Don't EVER acquire a lock from within another lock. This is extremely dangerous, and can easily result in deadlock: program hangs, because two threads are waiting for each other to unlock.
    • If you think you need to do a nested lock, you are probably doing it wrong.
    • If you absolutely must have a nested lock, be sure to acquire & release the locks in the exact same order everywhere. This is of course, very difficult to guarantee, because once someone starts doing it one plugin, someone else may have a different convention in another plugin ...

Avoiding Nested Locks

  • To avoid nested locking, when you're in a lock:
    • Never query JANA for calibration constants
    • Never query JANA for global parameters
    • Never query JANA for objects (JEventLoop::Get()) //This is also potentially incredibly slow
    • Never call the event writers to save your EVIO skims, REST files, ROOT trees, etc.
    • Never call any function that you don't understand exactly what it's doing. It might be acquiring a lock underneath.