Subsections

7 The Build System

Each of the packages have their own native build system and each build system has its own set of details that have to be understood. In addition, the technology used to do the build varies from system to system. It may be make, imake, cmake, SCons, or something else. The BUILD SCRIPTS system makes a choice of build options for each package so that the user need not master these details.

7.1 The Makefiles

The BUILD SCRIPTS build system is implemented in GNU Make. These makefiles invoke the native package-specific build system. The top-level makefile builds packages into the standard directory structure described in Section [*]. It in turn uses a “package makefile” for each package (e. g., Makefile_jana, Makefile_sim-recon). Invoking make with a package makefile will build that package with the home directory placed in the current working directory. In other words, the package makefiles have no knowledge of the directory structure within which they are used; they just build locally. Only the top-level makefile knows about the directory structure.


7.1.1 The Top-Level: Makefile_all

Complete builds are orchestrated by Makefile_all. The highest-level targets of Makefile_all, shown with their dependencies are:

all: gluex_pass2

gluex_pass1: xerces_build lapack_build cernlib_build root_build \
	     geant4_build evio_build sqlite_build sqlitecpp_build \
	     rcdb_build ccdb_build

gluex_pass2: gluex_pass1 jana_build hdds_build amptools_build \
	     hepmc_build photos_build evtgen_build halld_recon_build \
	     halld_sim_build hd_utilities_build hdgeant4_build \
	     gluex_root_analysis_build mcwrapper_build

The all target builds all officially supported packages. These are broken into two sets, gluex_pass1 and gluex_pass2 to solve the issue of packages that can only be set-up after they are built. This is a problem when starting from scratch. In that case the build must proceed in four steps.

  1. Initial environment set-up.
  2. Build the gluex_pass1 target.
      make -f $BUILD_SCRIPTS/Makefile_all gluex_pass1
    
  3. Final environment set-up using contents of packages built in pass 1.
  4. Build the gluex_pass2 target.
      make -f $BUILD_SCRIPTS/Makefile_all gluex_pass2
    

Makefile_all should always be invoked from the $GLUEX_TOP directory.

If you would like the build to proceed in parallel (for those packages that support parallel builds), set an environment variable (eight threads in this example):

  export NTHREADS=8
or set the variable on the make commmand line. For example:
  make -f $BUILD_SCRIPTS/Makefile_all gluex_pass2 NTHREADS=8

7.1.2 Individual Package Targets

Each of the individual package targets (e. g., evio_build and hdds_build) use the corresponding package makefile. Directories are created and the package makes are executed in way that gives the directory structure described in Section [*]. To do this, the package container directory is created if it does not exist and the requested package makefile is invoked from within the package container directory.

Of course, each individual package build target can be invoked directly. More on this in Section [*]

7.1.3 Special Package Target

The target cernlib_debug is non-standard. If this target is invoked, it well create a separate container directory cernlib_debug for the debug versions. Also it does not have its own package makefile, rather on 64-bit machines, it invokes Makefile_cernlib_Vogt with command line options that cause appropriate debug compiler flags to be used.

To use the resulting debug version of CERNLIB, the CERN variable must be set to point to the cernlib_debug directory, either explicitly as an environment variable (as described in Section [*] or by using the home attribute in the package element of a version set file (as described in Section [*].


7.2 The Package Makefiles

Each of the package makefiles is sensitive to environment variables that control which version of the package to build. The makefiles themselves take care of obtaining the source code. In general, there are two ways to get the code: downloading a tarball or checking the code out from a version control repository, although the later option is not available for all packages.

7.2.1 Specifying the Version of a Tarball

Each packages respects a version-specifying environment variable. Here is an example of how they might be set in the C-shell:

setenv JANA_VERSION 0.7.2
setenv SIM_RECON_VERSION 1.2.0
setenv HDDS_VERSION 3.2
setenv CERNLIB_VERSION 2005
setenv XERCES_C_VERSION 3.1.1
setenv CLHEP_VERSION 2.0.4.5
setenv ROOT_VERSION 5.34.26
setenv CCDB_VERSION 1.05
setenv RCDB_VERSION 0.00
setenv EVIO_VERSION 4.3.1

In each case there is standard system for distributing tarballs marked with the version name. Each package has different conventions, but the package makefiles have that knowledge of the appropriate convention coded in. Also, the name of home directory created depends on the name that appears in the tarball (with exceptions as mentioned in Section [*]. Note that the version variable can be set on the make command line as well.

7.2.2 Checking-Out from a Repository

Some packages can be checked out from a Subversion or Git repository. If that is the desired source of the code, the version variable should not be set. Instead a URL variable should be used to specify the location of the repository. For example:

  setenv HDDS_URL https://github.com/jeffersonlab/hdds

will cause the HDDS package makefile to check out the master branch of the HDDS Git repository at GitHub. The names of the variables for other packages are JANA_URL (Subversion), CCDB_URL (Git), RCDB_URL (Git), and SIM_RECON_URL (Git) respectively.

For packages that use a Git repository, there are two additional variables that can be used to control the checkout. SIM_RECON_BRANCH is used to check-out a specific branch and SIM_RECON_HASH is used to check-out a specific commit of sim-recon. If one is set, the other should not be. There are analogous variables for CCDB, RCDB, and HDDS.


7.2.3 Extra Features for Specific Packages

SCons options for halld_recon, halld_sim, and sim-recon

For the halld_recon, halld_sim, and sim-recon packages, there are variables, one per packages, that can be defined, either in the shell environment on on the make command line, that will supply optional arguments to the native scons command invoked by make. They are

    HALLD_RECON_SCONS_OPTIONS
    HALLD_SIM_SCONS_OPTIONS
    SIM_RECON_SCONS_OPTIONS
respectively. For example,
        make -f $BUILD_SCRIPTS/Makefile_halld_recon \
            HALLD_RECON_SCONS_OPTIONS='SHOWBUILD=1 -j4'
will cause SCons to show the compiler commands explicitly and build halld_recon with four threads.

Building LAPACK/BLAS

The LAPACK and BLAS libraries are needed by CERNLIB. They are downloaded and build automatically, but rather than being installed in their own home directory, The “install” target of Makefile_lapack adds them to the lib directory of your CERNLIB build.

Forcing use of Python 2 for HDGeant4

For systems that do not have Python 2 as the default, setting the make variable PYTHON2_EXPLICIT will force the use of Python-2-specific versions of the boost-python library and the python-config command in Makefile_hdgeant4.


7.3 Adding New Package Builds to an Existing Tree

There are two ways:

7.3.1 Use the Top-Level Makefile_all

After setting the desired values of the version environment variables and/or the URL environment variables (see Section [*] you can invoke Makefile_all with the target(s) needed or with a high-level target like gluex,

         cd $GLUEX_TOP
         make -f $BUILD_SCRIPTS/Makefile_all gluex
If some of the versions of individual packages requested already exist, then make will do the usual thing: try to remake them and find that there is nothing to do.

7.3.2 Use Individual Package Makefiles

Since the individual package makefiles build in the local directory, they can be used directly by going to the appropriate container directory. For example,

cd $GLUEX_TOP/sim-recon
make -f $BUILD_SCRIPTS/Makefile_sim-recon SIM_RECON_VERSION=1.4.0
Note that in this example the version is specified on the make command line rather than through an environment variable. That is not necessary; it is an option supported by make and defining SIM_RECON_VERSION in the environment would work as well. Also note that doing the build in the sim-recon container directory is not necessary for the build to succeed; any directory will work. In this example however we are adding to an existing standard directory structure so we cd to the standard directory.

7.4 Building Custom Versions of Packages

7.4.1 my_halld_build_jlab

Script location at JLab:

/group/halld/Software/build_scripts/my_halld_build_jlab

Usage message:

my_halld_build_jlab [-h] [-x filename] [-n nthreads] [package_1] [package_2] ...

where:
    -h        show this help text
    -x        set name of version set XML file
              (default: /group/halld/www/halldweb/html/dist/version_jlab.xml)
    -n        set number of threads to use in builds (default: 1)
    -k        keep potentially inconsistent packages in environment
              (default: remove them from the environment)
    package_n package name, choose from:
                  hdds
                  sim-recon
                  halld_recon
                  halld_sim
                  hdgeant4
                  gluex_root_analysis
              if no package name supplied all except sim-recon will be built

Examples:

  1. Build all GlueX packages (hdds, halld_recon, halld_sim, hdgeant4, and gluex_root_analysis):

    my_halld_build_jlab
    

  2. Build halld_recon only, using hdds from the default global build:

    my_halld_build_jlab halld_recon
    

  3. Build halld_recon, halld_sim, and hdgeant4, use hdds from the global build defined in version_3.1_jlab.xml:

    DIST=group/halld/www/halldweb/html/dist
    my_halld_build_jlab -x $DIST/version_3.1_jlab.xml \
      halld_recon halld_sim hdgeant4
    

  4. Build halld_sim, using halld_recon from the default global build, using 4 threads:

    my_halld_build_jlab -n 4 halld_sim
    

  5. Build halld_recon, using hdds from the default global build, and retaining environment set-up of halld_sim, hdgeant4, and gluex_root_analysis from the default global build even though the latter three were built against a different version of halld_recon

    my_halld_build_jlab -k halld_recon
    

7.4.2 my_halld_update.py

Usage message:

usage: my_halld_update.py [-h] [-x XML] [-n NTHREADS]
                          [inputPackages [inputPackages ...]]

positional arguments:
  inputPackages         list of packages to update

optional arguments:
  -h, --help            show this help message and exit
  -x XML, --xml XML     version set xml file name (default: version.xml)
  -n NTHREADS, --nthreads NTHREADS
                        number of threads to use in rebuild (default: 1)

Examples:

  1. Update all private packages as identified by version.xml in the current working directory. Private packages contain a "home" attribute:

    my_halld_update.py
    

  2. Update halld_sim only:

    my_halld_update.py halld_sim
    

  3. Update all private packages as identified by /home/user/project/version_special.xml:

    my_halld_update.py -x /home/user/project/version_special.xml