Difference between revisions of "HOWTO install and run HDGeant4"

Latest revision as of 08:47, 22 February 2017

Download the package sources

The HDGeant4 simulation is still in alpha release. During the present active development phase, it is currently being maintained in a read-only public git repository under the ownership of Richard Jones. The repository is being updated frequently as new features are being added and bugs corrected. Every effort is being made to ensure that the package can be successfully checked out and built after every commit that gets pushed to the master branch on github.

To check out the HDGeant source with the latest updates, go to a work directory where you would like to build the package and type the following command.

$ git clone git@github.com:rjones30/HDGeant4.git
$ cd HDGeant4 && ls
GNUmakefile  INSTALL  README.md  SConstruct  VERSIONS  g4py  src  test  vis

The README file gives an overview of the purpose and function of the HDGeant4 physics simulation package, and gives a bit of history regarding its design and early development. The VERSIONS file contains that release notes and dates for each major release of the HDGeant4 package. The INSTALL file contains the same information as conveyed on this wiki page concerning how to download, build, and run hdgeant4 and associated python modules. The GNUmakefile and SConstruct files contain the instructions for building package executables from source files; see below for further details. The g4py directory contains the python sources that comprise the python modules that are a part of HDGeant4. The src directory contains the C++ sources. The test directory contains a sample control.in file that is useful for initial tests that the hdgeant4 executable functions correctly after it is built. It also contains a suite of eps image files showing various cutaway views of the GlueX detector geometry, which are useful for comparison with the same views that can be generated using hdgeant4 once it is built. The vis directory contains a suite of graphics macros that can be invoked from the interactive command line in hdgeant4 to produce these same cutaway views. These images take a few minutes each to create, but it is worthwhile to take the time to make them because they provide a sensitive test that the GlueX geometry is faithfully represented within the Geant4 release against which you are building hdgeant4.

Supported platforms

The build instructions in this document have been developed for the Linux platform, running a 2.6 kernel. They have only been extensively tested on machines running the Centos 6 distribution, but they should work pretty much without modification on a host running a flavor of Debian, Scientific Linux, Gentoo, Ubuntu, or even Raspian, should anyone want to try such a thing. The code has been compiled and tested with the gnu c++ 4.9.2 compiler/linker suite. It should also work with more recent releases of the gnu compiler, and may even work with version 4.8.2 that is the default release for Centos 7. The only requirement is that it support the commandline option -std=c++11 which enables most of the features included in the C++11 standard.

The default gnu compiler that comes with Centos 6 is gcc-4.4.7 which does not support -std=c++11. Before attempting the instructions below to install an updated compiler version, you can check the currently installed version on your system by typing "gcc -v". If the version reported by this command is older than 4.8 then you will need to upgrade. Version 4.9.2 of the gcc compiler suite can be installed on a Centos 6 machine as a part of the bundle package "devtoolset-3". This package is not available from the standard Centos rpm repositories. I can be found by direct search on the internet, but the simplest means to obtain it is to enable the SCL repositories in your supported yum repositories, and then install devtoolset-3 using yum.

$ yum install centos-release-scl
$ yum install centos-release-scl-rh
$ yum install devtoolset-3

After this completes successfully, all you need to do in order to switch from the base gnu compiler to version 4.9.2 is to type the following from within your working shell,

$ scl enable devtoolset-3 bash

where "bash" can be replaced with the name of your working shell, eg. tcsh or zsh. This command cannot easily be embedded in a script because it results in adding a shell layer to the working environment. To see this, type the command in an interactive shell and then type "ps" to see the additional shell has been introduced. After this, any invocations of the gcc/g++ compilers will bring up the version 4.9.2 compiler (eg. try "g++ -v") until the user types "exit" to return to the original shell; once the exit from the daughter shell completes, the compiler resets automatically back to the default system version.

External package dependencies

1. sim-recon - The core components of the GlueX software stack are distributed in a single unified bundle called "sim-recon". This package has a number of external packages upon which it depends. Users who want to build and use HDGeant4 should already be familiar with sim-recon. Installation instructions for sim-recon can be found on the page Getting started with GlueX Software.
2. python - The current releases of Geant4 supports python 2.6 or 2.7, and in the near future will be extended to support version 3 as well. Normally one of these versions of python 2 should be already installed on any up-to-date Linux system. If not, you will need to install packages python-devel and python-libs in addition to python-runtime.
3. geant4 - The Geant4 simulation toolkit is maintained separately from the HDGeant4 sources. It should be downloaded from the main geant4 download site, and installed using the instructions provided by the distributors. The minimum recommended version of Geant4 that is supported by HDGeant4 is 4.9.6, although only version numbers starting with 4.10 provide multi-threading capability. Whatever version you decide to use, it must be one of those listed in the HDGeant4/src directory, in the form of a folder named G4.VV.RR-pPPfixes corresponding to version Geant4.VV.RR-pPP. You may try to build and run hdgeant4 against any other patch release of the Geant4 libraries, but it is not guaranteed to build without errors nor to run without crashing. The software dependencies of the Geant4 libraries are described in detail in the Installation Guide, but the key dependencies are listed below for convenience.
   • cmake 3.3 - This is the recommended build system for compiling the geant4 libraries from sources. On RH/Centos systems, this utility is available from the standard repositories as package "cmake3".
   • CLHEP - Prior to version 4.9, one needed to download and install CLHEP on the build host before attempting to build Geant4, but lately they have begun distributing a compatible CLHEP release inside the Geant4 sources tree at source/externals/clhep. Unless you have a specific reason to do so, you should not define any of the CLHEP_*** environment variables when you build Geant4, and instead let the cmake system discover the built-in version that is distributed with Geant4. This tends to be a relatively recent release, and is guaranteed to be compatible with the Geant4 release you are installing.
   • xerces-c 3 - This is a general xml parser library distributed under the apache license. It is also a dependency of sim-recon, so if you have already installed sim-recon you should already have xerces-c installed. If your sim-recon is an old one, you may have the older version 2 of xerces-c installed, in which case you should upgrade to one of the version 3 releases before attempting to build Geant4.
   • boost libraries - The boost libraries are widely-used extensions to the c++ standard that provide classes and macros for a variety of purposes. Geant4 uses boost to provide a python interface to the Geant4 toolkit. Building the python interface to Geant4 requires a special build step after the cmake build of the G4 libraries has completed successfully. Instructions for building the python extensions to Geant4 are found in an appendix of the Geant4 User's Guide for Application Developers.
   • Motif and Qt libraries - These packages are not required for building a production-ready executable, but if you might ever be interested in using the geometry and event visualization facilities of Geant4 to run the simulation interactively, generate dynamic views of tracks as they traverse the detector, and make on-the-fly adjustments to the simulation parameters, these packages will be useful. Up to date releases of the Motif and Qt libraries are available for all of the common flavors of Linux, and can be installed using the package manager that goes with your system, eg. yum for RH/Centos, apt for Debian, etc.
   • X11/Qt with OpenGL extensions - These packages are basic to the visualization package in Geant4. Generally you will want to build Geant4 with at least one of these two visualization facilities turned on, and probably both. OpenGL is a high-level 3D graphics package that provides efficient means to generate 2D views of a 3D geometry, and zoom/rotate/pan within that geometry in response to mouse input. Both X11 and Qt graphics drivers support OpenGL operations whenever the output display has hardware graphics acceleration available. This is generally the case on modern Intel workstations, provided that you are working on the main console. If you want to work over a remote desktop graphics screen (eg. vnc) then you will need to find a way to associate your virtual display with the console graphics device so that its hardware accelerator can be used to implement the OpenGL calls. See below for a practical means to do this while working remotely using a vncserver on a Linux host.

There are many options not listed above that can be selected during the cmake step of building the Geant4 libraries, most of which are optional as far as HDGeant4 is concerned. The following example shows the cmake command that I used to build a 4.10.02-p02 release of Geant4 for use with HDGeant4.

cmake3 -DCMAKE_INSTALL_PREFIX=/nfs/cern.ch/asis/geant4.10.02.p02/x86_64-debug \
       -DCMAKE_INSTALL_DATADIR=/nfs/cern.ch/asis/geant4.10.02.p02/data \
       -DXERCESC_ROOT_DIR=/usr/local/xerces \
       -DGEANT4_USE_QT=ON \
       -DQT_QMAKE_EXECUTABLE=/usr/lib64/qt4/bin/qmake \
       -DGEANT4_USE_XM=ON \
       -DGEANT4_USE_OPENGL_X11=ON \
       -DGEANT4_USE_RAYTRACER_X11=ON \
       -DGEANT4_USE_NETWORKDAWN=ON \
       -DGEANT4_BUILD_MULTITHREADED=ON \
       -DCMAKE_BUILD_TYPE=DEBUG \
../src

The above command was executed from within a directory /nfs/cern.ch/asis/geant4.10.02.p02/x86_64-debug-build, which is located in a top-level installation directory together with src which contains the complete G4 source tree that is unzipped from the distribution tar.gz file.

Note added by the author: I found that I was unable to build the python extensions to Geant4 if I had GEANT4_USE_MULTITHREADED set to ON. Because of this, I made a separate build in a directory called geant4.10.02.p02/x86_64-noMT and then built the python extensions within that install. After this, whenever I need the python extensions I switch my environment to point to the noMT library, and whenever I want to run production I switch back to the default build that was done with GEANT4_USE_MULTITHREADED=ON.

Building binaries from sources

Once you have downloaded the HDGeant4 sources from git and verified that you have installed all of the necessary external packages, you are ready to build the executables. Building consists of 3 steps: configure, setup, and make. For the configure step, go into the HDGeant4 directory that resulted from the "git clone" command described above, and look for GNUmakefile. Edit this file with your favorite text editor, and look for a list of symbols defined near the top of the file. These are the options that define the user-selectable features in the build. The CPPFLAGS settings that can be customized are found in a sequence of lines beginning with CPPFLAGS += -D<symbol>. The only settings that you will normally need to customize are G4MULTITHREADED and G4DEBUG, whose meanings are obvious. Once you are satisfied with the settings in GNUmakefile, the following steps will complete the build of hdgeant4. Before you execute the following commands, do what is necessary (see above) to make sure that when you type "g++ -v" you see a compiler version string that is 4.8.2 or greater.

$ source <your_sim-recon_environment_setup_script>
$ ln -s G4.<VV>.<RR>.p<PP>fixes src/G4fixes
$ unset CLHEP_INCLUDE_DIR
$ cd HDGeant4
$ make
$ ls -R
bin:
Linux-g++

bin/Linux-g++:
hdgeant4

The symbol <VV>.<RR>.p<PP> is the version.release.patch sufffix for your build of the Geant4 toolkit, which must match one of the G4*fixes in the HDGeant4 src directory. The unset step (unsetenv if you use csh) makes sure that you do not have a standalone version of CLHEP in your setup script that might override the version that is distributed with the G4 libraries. Only the one that comes with the G4 libraries is needed for building HDGeant4.

Testing the build

If the build procedure described in the previous section completed successfully, the new hdgeant4 executable should be ready to run. To try it out, descend from HDGeant4 into the directory "test" and simply type the command "hdgeant4."

$ cd test
$ hdgeant4 -h
hdgeant4: invalid option -- 'h'

Usage: hdgeant4 [options] [<batch.mac>]
 where options include:
    -v : open a graphics window for visualization
    -tN : start N worker threads, default 1
    -rN : set run to N, default taken from control.in

Simply typing "hdgeant4" in this same test directory without any commandline arguments initializes the G4 toolkit and brings you to an interactive prompt "Idle>" as illustrated below. Typing the command "/run/beamOn 100" generates 100 minimum-bias events using the built-in coherent bremsstrahlung generator as a source of the simulated particles.

$ hdgeant4

        ############################################
        !!! WARNING - FPE detection is activated !!!
        ############################################

*************************************************************
 Geant4 version Name: geant4-10-02-patch-02 [MT]   (17-June-2016)
  << in Multi-threaded mode >> 
                      Copyright : Geant4 Collaboration
                      Reference : NIM A 506 (2003), 250-303
                            WWW : http://cern.ch/geant4
*************************************************************

JANA >>Created JCalibration object of type: JCalibrationCCDB

>>> multiple lines of initialization output suppressed here <<<

Idle>

For batch mode processing, it is convenient to create a two-line file called run.mac as illustrated below, and then specify run.mac on the commandline when you start hdgeant4.

$ cat <<EOI >run.mac
/run/beamOn 10000
exit
EOI
$ hdgeant4 run.mac

If you built hdgeant4 with multithreading enabled, you should be able to use the -tN commandline option to request N parallel execution threads. The simulation runs single-threaded for the first couple of minutes, as it initializes the geometry and physics lookup tables, and after that completes, multi-threaded event simulation begins. Use the linux "time" command to measure the amount of cpu and wall clock time that is needed for initialization, and for the simulation of each minimum-bias event. On a 3.5GHz core2 i7 machines, I find that 170s are used to initialize the simulation, and then on the order of 200 ms of cpu time are used for each single beam particle event that is simulated. To test that hdgeant4 is behaving properly, run the Geant3 simulation by typing "hdgeant" in the same directory. Both binaries should exist in your path, and the two should produce output that is the same, within normal Monte Carlo statistical fluctuations. Use the hddm-xml tool to generate plain-text listings of both of these small output files called bgtest.hddm, and do a visual comparison to check that the two simulations are behaving the same way.

Visualizing the geometry and fields

Normally when the hdgeant4 application starts up, it proceeds directly to initialize itself and then to simulate events in batch mode, governed by the contents of the control.in input file. If an optional Geant4 command file (denoted "batch.mac" in the usage summary above) is given on the commandline then the commands in that file are used to direct the simulation processing, otherwise the command If instead you want to start the visualization system and generate graphics output then two additional things are needed,

1. the -v option must be given on the commandline;
2. a special file called vis.mac must exist in your working directory, that contains commands needed to initialize the graphics system.

If these two conditions are met, the simulation starts the visualization engine and opens an interactive prompt requesting commands from the terminal. Geant4 offers a large variety of different graphics engines with different capabilities and performance characteristics. It is beyond the scope of this document to review all of these, but a good choice that is supported by a broad variety of setups and gives reasonable performance is the one listed in the default vis.mac file found in the test directory of the HDGeant4 source distribution. To try other choices, modify your local copy of vis.mac and restart hdgeant4 with the -v option. If all goes as expected, as soon as the simulation geometry and fields are initialized, a graphics window should open automatically on your desktop and you should be given the interactive command prompt "Idle>". If you built hdgeant4 with the G4UI_USE_EXECUTIVE option enabled then instead of the command prompt appearing on your shell terminal, a control GUI will open on your desktop which integrates the graphics screen together with other panes containing the session log, command input, a file browser, etc. Personally, I do not find this command GUI very helpful, and prefer to have standard text terminal input when working in interactive mode, so this GUI option is disabled by default in the build system for HDGeant4.

Special notice for vnc users: If you are working directly from the primary display of your Linux workstation then this paragraph does not apply to you. But if you find yourself needing to work remotely even part of the time, chances are you are using vnc to provide a persistent session with remote connectivity. In case this is your situation, you should be aware that vnc replaces the graphics hardware device of your workstation with a virtual device that supports only a subset of the full capabilities of a modern graphics display. In particular, most vnc implementations do not support OpenGL. As a result, you might get an error like the following when you try to start hdgeant4 with the -v option.

$ hdgeant4 -v

>>> initialization printout suppressed here <<<

X Error of failed request:  BadValue (integer parameter out of range for operation)
  Major opcode of failed request:  148 (GLX)
  Minor opcode of failed request:  3 (X_GLXCreateContext)
  Value in failed request:  0x0
  Serial number of failed request:  20
  Current serial number in output stream:  21

There is more than one way to get around this limitation of vnc, but the most convenient and powerful one that I have found is to install and use the VirtualGL package. This allows any virtual screen to "borrow" the graphics hardware accelerator that is associated with the primary display for this host and use it to support OpenGL graphics on the virtual desktop. If more than one person attempts to do this on the same workstation at the same time there may be a conflict, but this problem is relatively rare in practice. VirtualGL is available from most standard Linux repositories. Once it is installed, you can prefix the hdgeant4 command with the "vglrun" command, as illustrated below, which opens a new graphics window on the vnc terminal and waits for interactive commands.

$ vglrun -d :0 hdgeant4 -v

>>> initialization printout suppressed here <<<

/vis/open OGLSX 600x600+10+50
/vis/sceneHandler/create OGLSX
/vis/viewer/create ! ! 600x600+10+50
/vis/viewer/refresh
/vis/ogl/set/displayListLimit 1500000
/vis/viewer/set/autoRefresh false
/vis/viewer/set/background white
/vis/viewer/set/lineSegmentsPerCircle 60
Idle> 

A complete description of all of the visualization commands that are supported by Geant4 is beyond the scope of this manual. G4 visualization is well known for producing very nice 3D views of experimental setups. However, I find 2D sections to be more powerful than 3D views for performing a detailed examination of a setup. For this purpose, the command /vis/viewer/set/sectionPlane command is extremely powerful. I suspect that this command is not used very heavily by the broader community because, when I first began to work with validating the GlueX geometry in Geant4, I found that the views created with sectionPlane defined were very often incorrect and inconsistent with what the G4 tracking was actually seeing. Fixes for these bugs in the G4 visualization system are a significant part of what is in the G4*fixes directories. The directory vis/stdviews contains a number of G4 macro files that can serve as examples for how to make sectionPlane views of the GlueX detector and beamline.

A number of standard planar sections of the GlueX spectrometer and beamline have been created to serve as references against which you can compare the output obtained when you visualize the GlueX geometry using your HDGeant4 build. The images are found in the HDGeant4/test directory, with names like zNNN_0.eps, where NNN is the location in simulation coordinates along the GlueX beamline where the view is centered. Some of these sections are taken in a plane perpendicular to the beam axis, while others are taken in a plane perpendicular to x. The commands that were used to generate these images are found in the directory HDGeant4/vis/stdviews, with names zNNN.mac that make obvious connections with the names of the image files. You can execute these commands directly from the interactive command prompt in a hdgeant4 visualization session, as in "/control/execute ../vis/stdviews/z65.mac". Comparing the image generated by this command with the one found in z65_0.eps allows you to make sure that your simulation geometry is what you expect it to be.

Using the python interface

The python interface to the Geant4 toolkit is available for anyone who would like to create custom simulations using a rapid prototyping language. Most of the G4 objects that are accessible to user code in Geant4 applications are exposed as python objects through the python interface. This interface has been extended in the context of HDGeant4 to include GlueX-specific classes, many of which are subclasses of G4 classes. One example of where this might be useful would be to sample the magnetic field over a grid defined to cover in a certain region of the geometry, and output the results as a multi-column text file or a root tree. Another example would be to plot the interaction length or cross section for a given physics process as a function of particle type and energy. All of this can be done easily using the python interface, requiring only a few lines of user python code to accomplish. This functionality is achieved by adding the directory HDGeant4/g4py to the PYTHONPATH environment variable and then running python with "import hdgeant4".

Due to popular demand, the coherent bremsstrahlung generator class CobremsGenerator has been instrumented with a set of user-friendly python functions and exported as a stand-alone python module found in the same location HDGeant4/g4py together with the hdgeant4 module. To discover what methods are provided by the cobrems module, you can issue the usual "pydoc cobrems" command from the shell commandline and read the brief descriptions that accompany the listing of all of the class methods and their argument lists.

Performance and benchmarks

As a comparative performance benchmark between hdgeant and hdgeant4, I prepared a control.in file requesting 10000 single-track events consisting of a single pi+ track generated at a random position along the liquid hydrogen target at kinetic energy 0.5 GeV and distributed uniformly in azimuthal angle and polar angles between 30 and 120 degrees in the lab. The hdgeant simulation was single-threaded, while hdgeant4 was run with 4 threads (commandline option -t4). The timing measurements were carried out on a 3.4GHz Intel i7 with no significant load other than the hdgeant/hdgeant4 process.

• hdgeant: total processing time: 257s (real), 245s (user), 0.3s (sys)

  per-event processing time: 0.0240 s/event

• hdgeant4: total processing time 726s (real), 2300s (user), 0.6s (sys)

  per-event processing time: 0.213 s/event

From these statistics, one sees that total event processing time in HDGeant4 is a factor ~9 more expensive in terms of cpu time than it was in HDGeant. In the tests above, the hdgeant4 runs encountered an initialization overhead of 170s in single-threaded processing prior to the startup of the event simulation phase. After that, the speedup seen in multi-threaded mode was essentially 100% of the number of processing threads, indicating that the simulation scales extremely well with the number of cores. Even so, more study is needed to understand the reasons for the slowdown by almost a factor 10 for exactly the same geometry, fields, and event types being simulated.