GlueX Offline FAQ
This is the GlueX Offline Frequently-Asked Questions list. If you cannot find an answer to your question, please send it to the Offline Software Coordinator (currently Mark Ito).
- 1 General
- 2 GlueX Administrative Tasks
- 3 Building GlueX Software
- 4 JLab
- 4.1 How do I get a computer account at JLab?
- 4.2 How do I install a certificate in my browser so that I can view secure web sites at JLab (without creating a security exception)?
- 4.3 I cannot log into the ifarm machines. How do I get access?
- 4.4 How do I become a member of the "halld" Unix group?
- 4.5 How do I subscribe to a GlueX email list?
- 4.6 How do I get personal web space at JLab?
- 4.7 What is the Volatile disk?
- 4.8 What is the difference between /work/halld and /work/halld2?
- 4.9 How do I check how much space Hall D is using on the Lustre disks?
- 4.10 Why does sudo not work for me?
- 4.11 How do I kill all of my farm jobs?
- 4.12 How to access RCDB with python (e.g. plot accumulated triggers)
- 4.13 How do I get a Scientific Computing Certificate?
- 4.14 How do I find out what share of the Farm is allocated to Hall D?
- 4.15 SWIF
- 5 sim-recon
- 5.1 I have a build of sim-recon that I want to use. How do I set up my environment?
- 5.2 How do I get debug symbols in my sim-recon binary?
- 5.3 How do I create EVIO data from an HDDM file produced by HDGeant?
- 5.4 When unstable particles appear in the simulation, which ones are decayed by the event generator versus hdgeant/hdgeant4?
- 6 CCDB
- 7 Git
- 7.1 Where can I get help with Git?
- 7.2 What are the basic git commands that I need to know?
- 7.3 What happens when I do a git clone?
- 7.4 What is a remote repository?
- 7.5 What is a tracking branch?
- 7.6 How do I create a tracking branch?
- 7.7 How do I make an existing branch track a remote branch?
- 7.8 What is a topic branch?
- 7.9 What is the difference between Git and GitHub?
- 7.10 How do I get privilege to create pull requests on GitHub?
- 7.11 Why is there no "git pull-request" command?
- 7.12 How are we notified of pull requests?
- 7.13 What is the difference between a clone and a fork?
- 7.14 What happens to forks when a repository is deleted or changes visibility?
- 7.15 What happens when I checkout another branch when there are uncommitted changes on the current branch?
- 7.16 I cannot clone a GitHub repository on the Gluon Cluster. What is the trick?
- 7.17 I am getting a "403 Forbidden" error when trying to push to a GitHub repository from JLab. What is wrong?
- 7.18 After a branch is deleted on GitHub, I still see it when I do a git branch -r on my local repository, even after a git fetch. How do I get rid of it?
- 7.19 How do I get to the compare view on GitHub?
- 7.20 Tags have changed on a remote repository. How do I update them on my local repository?
- 7.21 I am getting SSL errors when trying to access GitHub from JLab. What is wrong?
- 7.22 How do I use secure shell (ssh) to access GitHub?
- 7.23 How do I restart the automatic pull-request test?
Where do I find information about GlueX software?
GlueX Administrative Tasks
How do I get an account on the wiki?
All members of the
halld group at JLab can log into the wiki. Use your JLab CUE username and password.
How do I get privilege to check stuff into the Subversion repository?
Become a member of the halld unix group.
How do I get emails when files are checked into the Subversion repository? Stop the emails from coming?
If you look in the directory /group/halld/Repositories/svnroot/hooks there is a file called post-commit, with several backups. That is the file you need to change. If you look at it it is pretty clear what to do. People in the halld group should have privilege to modify it.
Note that you can "subscribe" to only a subset of check-ins as well, selection by directory.
Any GlueXer with the writer privilege can add a new author. Go to https://halldweb.jlab.org/doc-private/DocDB/AuthorAddForm . (Thanks Zisis!)
Building GlueX Software
How do I do a complete build of GlueX Software, starting from scratch?
See the instructions in "The gluex_install System" section of the Version Management System document.
How do I upgrade my $GLUEX_TOP?
New versions of the packages come out all of the time. Sometimes you have to upgrade.
1. Identify a version set to use for the upgrade.
Version set XML files can be found at
or at JLab in
and have names like version_2.22_jlab.xml, for example. If you select a "_jlab.xml" file then you will likely not have any inconsistencies in the build. Copy the file into your $GLUEX_TOP directory. For definiteness let's call the file
2. Create an environment set-up file for the upgraded system
For definiteness, let us assume you are using tcsh as your shell. The file $GLUEX_TOP/setup.csh should look like:
setenv GLUEX_TOP /your/location/for/gluex_top setenv BUILD_SCRIPTS $GLUEX_TOP/build_scripts source $BUILD_SCRIPTS/gluex_env_version.csh /your/location/for/gluex_top/version.xml
Make a copy in $GLUEX_TOP, for definiteness call it setup_upgrade.csh. Edit the last line to be
source $BUILD_SCRIPTS/gluex_env_version.csh /your/location/for/gluex_top/version_upgrade.xml
Now the command
will set-up the environment for the upgraded version set (not yet built).
3. Do the build
After setting up the environment as described in the previous step, build the updates
cd $GLUEX_TOP make -f $BUILD_SCRIPTS/Makefile_all gluex
How do I get a computer account at JLab?
You need to register as a JLab user. The registration form has a box for "I want an account" (or something like that). In any case registration must be completed before getting an account. Also you will need a sponsor. Your sponsor must be a member of JLab staff. The process starts with a link on this page
Click on "online" in the first sentence of the first paragraph.
How do I install a certificate in my browser so that I can view secure web sites at JLab (without creating a security exception)?
Go to https://pki.jlab.org/ , select the browser you are using, and follow the instructions.
I cannot log into the ifarm machines. How do I get access?
You have to be member of the "halld" Unix group (or an analogous Unix group associated with one of the other experimental halls).
How do I become a member of the "halld" Unix group?
Ask the Software Coordinator to add you.
How do I subscribe to a GlueX email list?
There is a wiki page listing email lists of interest to GlueX collaboarators. To subscribe to any of them, click on the appropriate "information page".
How do I get personal web space at JLab?
The standard /home/username/public_html has been deprecated at the Lab as of August 2011. Another system has been set up on the /userweb partition. There is a HOWTO that explains the procedure for getting space there.
What is the Volatile disk?
We now have access to up to 40 TB of disk space on the "volatile" file system.
- storage is temporary; there are scripts which run to delete old files
- a quota system is enforced by the operation of these cleaning scripts
- the system has two disk usage level triggers, called "quota" and "reservation", the former is a global maximum per group, the latter is used when the disk as a whole runs out of space (the sum of all quotas exceeds the available space, by design)
- the system is explained at the bottom of the page referenced in Sandy's message
The /volatile partition is only available from the farm nodes at the Lab. This space is independent of our "work" space (/work/halld). Work space is not subject to deletion by scripts.
Two possible use cases for this space are (1) for staging files into and out of farm nodes and (2) for storage of data that are permanently stored elsewhere (e. g., on tape) for repeated analysis at the Lab.
There is a directory, /volatile/halld/home, for users to put their personal directories.
See http://scicomp.jlab.org/disk/volatile.faces for details of who has what space and how the space is managed.
What is the difference between /work/halld and /work/halld2?
How do I check how much space Hall D is using on the Lustre disks?
From a recent CCPR:
You can see the current lustre quota, and usage, for Hall D with the following:
> lfs quota -hg halld /lustre Disk quotas for group halld (gid 267): Filesystem used quota limit grace files quota limit grace /lustre 103.6T 115T 120T - 7666476 0 0 -
Note that this shows the Hall D quota for all of the Lustre-based disks; the quota has no knowledge of cache/volatile/work.
Why does sudo not work for me?
At JLab, /apps/bin/sudo is configured for use by Computer Center personnel. It does not reference your local sudoers file. To use sudo locally, make sure you are using /usr/bin/sudo (i. e., check your PATH variable).
How do I kill all of my farm jobs?
jkill 0 will kill all farm jobs submitted by the logged in user, independent of the state of the jobs.
How to access RCDB with python (e.g. plot accumulated triggers)
- On most machines at JLab
- Source /group/halld/Software/build_scripts/gluex_env_jlab.csh
- Use /apps/anaconda/PRO/bin/python2
- Example: > python2 plot_rcdb3.py # run script that plots accumulated triggers over the run
How do I get a Scientific Computing Certificate?
For more details see SciComp's network certificate page.
The computer center maintains a page on that here. You need to scroll down to see the table.
How do I resolve problem jobs so I can start the workflow going again?
When you have hit the error limit, SWIF will stop submitting jobs from your workflow. By default, there is no limit set. If you have set a limit and have reached the limit, you can use 'modify-jobs' if you want to retry them with altered cores/time/ram/disk. If you simply want to try again, use 'retry-jobs'. To just fail them, use 'abandon-jobs'.
If you want to put off thinking about it, you can up the error threshold to, say, 1000 by using 'swif run -workflow <> -errorlimit 1000'.
Suppose a successful job, upon scrutiny later, looks suspicious and I want to re-run it. Is there a way to to this?
You can use the undocumented '-resurrect' option of 'swif retry-jobs' then specify which jobs to resurrect either by name or id. You can also specify job names and ids by pattern, but you need to make sure the pattern isn't too broad because this command will search all jobs, not just ones that were successful.
I have a build of sim-recon that I want to use. How do I set up my environment?
Source either the setenv.csh or the the setenv.sh that was created when sim-recon was built. For the details, see the setenv section of the wiki page Setting Up the GlueX Environment.
How do I get debug symbols in my sim-recon binary?
Debug symbols are put in by default by the SBMS system. You don't have to do anything.
How do I create EVIO data from an HDDM file produced by HDGeant?
hd_ana -PPLUGINS=rawevent file.hddm
When unstable particles appear in the simulation, which ones are decayed by the event generator versus hdgeant/hdgeant4?
In general, the choice is up to you. If you are writing an event generator and you want to produce an unstable particle like a τ- lepton or an η meson, you can either convert it immediately into the daughter particles and pass those to hdgeant/hdgeant4 for tracking, or you can pass the unstable particle directly to hdgeant/hdgeant4 and let it decay in the simulation. In order for hdgeant or hdgeant4 to handle the decay, the mother particle needs to be included in the list of known particles to the underlying Geant library. The known particle lists for the two Geant versions are similar, but not identical. See the following references for details.
- Geant3 Reference Manual, see Table CONS-301 on page 58-59.
- Geant4 User's Guide, separate tables for leptons, mesons, baryons, ions, and others.
Unstable particles that appear in these lists are those that the Geant/Geant4 library has some knowledge of, and can generate the major decay modes with the correct branching ratios. The fact that these libraries can simulate the in-flight decays of these particles does not mean that it should be done that way. There are limitations to the decay models employed in these simulation libraries, particularly the fact that the daughter particle momenta are distributed uniformly in phase space. For example, if you inject a bunch of ω(783) mesons into hdgeant or hdgeant4 and then generate a Dalitz plot of their 3-pion decays, you will see that the interior of the Dalitz ellipse is uniformly populated rather than weighted by the physical amplitude for isoscalar,vector -> 3π decays. One way to address this would be to generate the decays instead within the event generator, using the proper physics amplitude. Then hdgeant/hdgeant4 would only see the 3 pions from omega decay and would propagate/decay those according to their lifetimes and branching fractions. The more standard approach to this problem in GlueX analyses would be to generate all of the hadronic decays using a uniform phase-space distribution, and then apply the physical decay distribution later on in the analysis as a factor in the PWA amplitude. However, that is beyond the scope of this FAQ. For the present purposes, you can either embrace phase-space decay distributions and let hdgeant/hdgeant4 perform the decays, or else take control of the decays directly in the event generator and make the decay distributions as realistic as you wish. In general, I recommend the following rule of thumb as a best practice for GlueX simulations.
If the unstable particle has a sufficiently long lifetime that it can travel a measurable distance (>0.1 mm) in the detector before decaying, the decay should be handled by hdgeant/hdgeant4. If on the other hand, the unstable particle lifetime is too short to allow its decay vertex to be distinguished from the primary event vertex then the decays should be handled by the event generator. This second rule gives the experimenter more flexibility in terms of which decay branches are selected for simulation, and saves time that otherwise would be spent in the simulation following products of decay branches that are not the focus of the study.
"But," someone asks, "I don't want to write my own generator. I want to use bggen (pythia) or genr8, or some other channel-specific generator that I found in the sim-recon codebase. What happens then?"
The answer depends on the specific event generator tool you are using, but in general they tend to conform to the best-practice rule given above. In particular, bggen only generates the following unstable objects as final state particles.
You can verify that all of the unstable particles in this list are known particle types to both hdgeant and hdgeant4. Any unstable objects not in this list, that appear in bggen as products from the primary interaction vertex, are tagged in the generated output file as intermediate resonances, not final state particles. When reading from the bggen event source, the hdgeant/hdgeant4 simulation tracks only those that are tagged as final state particles, ignoring the intermediate nodes in the decay tree. This way there is no double-counting of parent and daughter particles in the simulation, yet the record of which final-state particles conceptually belong to which parent resonances is still available in the simulation output file in case the user wants to check it.
How do I use a CCDB SQLite file?
There are instructions on this wiki page.
I am getting a disk I/O error reading an SQLite version of the CCDB. What is wrong?
If your error looks like
> ccdb ls CCDB provider unable to connect to sqlite:////work/halld/user/ccdb.sqlite. Aborting command. Exception details: (sqlite3.OperationalError) disk I/O error [SQL: u'SELECT "schemaVersions"."schemaVersion" AS "schemaVersions_schemaVersion", "schemaVersions".id AS "schemaVersions_id" \nFROM "schemaVersions"\n LIMIT ? OFFSET ?'] [parameters: (1, 0)]
it is likely you are trying to use a SQLite file from a Lustre-based disk system. SQLite is incompatible with Lustre. You must move your file to a different, non-Lustre file system.
Where can I get help with Git?
See Git Help Resources on this wiki.
What are the basic git commands that I need to know?
- git help: get a help message (e. g., git help clone)
- git clone: get a complete copy of a repository
- git status: show the status of your files (changed, added to index, etc.)
- git checkout: change to another branch
- git add: add modified files to the index
- git commit: commit files in the index to your repository
- git branch: show, create, delete branches
- git tag: show, create, delete tags
- git fetch: get remote repository information, do not change files
- git pull: get changed files from remote repository
- git push: write changed files to remote repository
What happens when I do a git clone?
When you clone a repository to your local disk, you get not only get the main line of code (in SVN: the trunk, in Git: the master branch), but all branches and tags and the history of all of the above. The directory that is created contains the actual files being controlled consistent with the master branch. In the top level directory there is a hidden directory, .git, that contains all of the versioning information. There are no .git directories in the subdirectories. You are meant to work with the files you see. You do not checkout the code to a separate directory as you do with SVN or CVS. The repository and the working files co-exist in the same tree.
What is a remote repository?
A remote repository is any repository outside of your repository that your repository is aware of. A network of remote repositories will have a common ancestor. When you clone a repository, your new repository is automatically aware if one remote repository: the repository from which it was cloned. The remote repository is called "origin" on your local repository by default. Remote repositories can be added and removed. They can be pushed to or pulled from. To see your remote repositories do a "git remote".
What is a tracking branch?
If you're on a tracking branch and type git push, Git automatically knows which server and branch to push to. Also, running git pull while on one of these branches fetches all the remote references and then automatically merges in the corresponding remote branch.
How do I create a tracking branch?
When branches are created using the --track option, they will be set up linked to the remote branch. For example, if you wanted to create a new branch from the master branch from the origin remote, using this would set it up so it would pull from the remote and branch automatically:
$ git branch --track feature1 origin/master
Branch feature1 set up to track remote branch refs/remotes/origin/master.
How do I make an existing branch track a remote branch?
As of Git 1.8.0:
git branch -u upstream/foo
Or, if local branch foo is not the current branch:
git branch -u upstream/foo foo
Or, if you like to type longer commands, these are equivalent to the above two:
git branch --set-upstream-to=upstream/foo
git branch --set-upstream-to=upstream/foo foo
As of Git 1.7.0:
git branch --set-upstream foo upstream/foo
All of the above commands will cause local branch foo to track remote branch foo from remote upstream. The old (1.7.x) syntax is deprecated in favor of the new (1.8+) syntax. The new syntax is intended to be more intuitive and easier to remember.
What is a topic branch?
A topic branch is a branch that is created, and possibly shared, to work on some particular issue. The name of the branch should be descriptive of the issue, of course. The concept emphasizes separation of work on one issue from work on others. One may have several topic branches going at the same time. The word "topic" is descriptive and has no operational significance.
What is the difference between Git and GitHub?
Git is the underlying software package that performs the direct operations on Git repositories. It is open source. As a software project, it is completely independent of GitHub. One can profitably run a project under Git without ever using a web browser.
GitHub is a commercial site that is in the business of hosting Git repositories and providing web tools for working with them. Jefferson Lab has a contract with GitHub to host repositories for doing Lab-related work. Public repositories are free; private repositories can potentially incur charges.
How do I get privilege to create pull requests on GitHub?
Find the instructions here.
Why is there no "git pull-request" command?
Generically, a pull request is any communication from one developer to another asking that changes on the one's branch be incorporated on the others remote branch via a "git pull". As your question implies, there is no native Git command for doing this.
On GitHub, there is a site-specific set of tools (web pages and web forms) for issuing pull requests which includes a conversation between developers on the proposed change. These tools work only for the case when the source and the target branch are both hosted on GitHub. The repositories need not be owned by the same account.
How are we notified of pull requests?
Those interested in acting on pull requests should "watch" the appropriate repository on GitHub. Go to settings/notification to configure how you are notified (email, website, or both). Then go to the repository page to mark the repository as being watched/not-watched/ignored by you.
What is the difference between a clone and a fork?
A fork is a clone from a GitHub-hosted repository to another GitHub-hosted repository. It is a GitHub-specific term.
What happens to forks when a repository is deleted or changes visibility?
When you delete a public repository, one of the existing public forks is chosen to be the new parent repository. All other repositories are forked off of this new parent and subsequent pull requests go to this new parent.
What happens when I checkout another branch when there are uncommitted changes on the current branch?
If the new branch contains edits that are different from the current branch for that particular changed file, then it will not allow you to switch branches until the change is committed or stashed. If the changed file is the same on both branches (that is, the committed version of that file), then you can switch freely.
I cannot clone a GitHub repository on the Gluon Cluster. What is the trick?
setenv HTTPS_PROXY https://jprox:8082
I am getting a "403 Forbidden" error when trying to push to a GitHub repository from JLab. What is wrong?
You might be using an old version of git. On the JLab CUE or on the Gluon Cluster, use the version of git from /apps/bin and not the one in /usr/bin. (Version 1.7.1 has given problems; version 188.8.131.52 seems OK. See https://help.github.com/articles/https-cloning-errors/). In addition the version of /apps/bin/git on the CentOS 6.2 farm nodes (e. g., ifarm62) gives error messages. It is not known if commands are executing correctly or not.
After a branch is deleted on GitHub, I still see it when I do a
git branch -r on my local repository, even after a
git fetch. How do I get rid of it?
git remote prune origin will remove all such stale branches.
How do I get to the compare view on GitHub?
To get to the compare view, append /compare to your repository's path.
Tags have changed on a remote repository. How do I update them on my local repository?
On the local clone:
git fetch --prune --tags
I am getting SSL errors when trying to access GitHub from JLab. What is wrong?
For example, you might see the error:
Peer certificate cannot be authenticated with known CA certificates.
Peer's certificate issuer has been marked as not trusted by the user.
In late 2016 through early 2017, after JLab enacted stricter security measures for encrypted web traffic (e.g. HTTPS), users had to reference a JLab security certificate in their git configuration when authenticating to GitHub. After that period, the reference is not only unnecessary, but prevents git from working with GitHub repositories using https.
To get rid of the reference remove the "sslCAInfo" line from your ~/.gitconfig file.
How do I use secure shell (ssh) to access GitHub?
You can clone a repository, for example sim-recon, by typing
at the command line. Note that you have to specify user "git" in the URL. This will not work unless you have previously added your public ssh key to your GitHub account. To add your key:
- Log into GitHub
- Navigate to "settings" (click on far upper right icon, select "settings")
- Click on "SSH and GPG keys"
- Click on "New SSH key" (near the upper right)
- Enter a title (reminder about where the key is from) and your public key.
- Click on "Add SSH key"
If you run an ssh-agent that provides your passphrase, you can operate password free now.
What if after this I get the "X11 forwarding request failed on channel 0" message?
That does not affect the success of your command. To get rid of it create a file ~/.ssh/config that contains
Host github.com ForwardX11 no
with only user read/write permissions. After that you will not get the message.
How do I restart the automatic pull-request test?
From Sean Dobbs:
From the main GitHub page:
Settings -> Webhooks -> [select the script, it's the only one]
Scroll down to "Recent Deliveries", and if you click on the hashes, it gives you details about the messages that were recently sent to the script. If you look at those, and pick the one corresponding to the pull request you are interested in, with an "action" of "opened", you can click the "Redeliver" button to trigger the action again.