Difference between revisions of "GlueX Software Meeting, February 2, 2021"

From GlueXWiki
Jump to: navigation, search
(initial draft)
 
m (Standardized Tests)
 
(4 intermediate revisions by the same user not shown)
Line 8: Line 8:
 
# Announcements
 
# Announcements
 
## [https://mailman.jlab.org/pipermail/halld-offline/2021-January/008443.html New version set: 4.35.0] (Mark)
 
## [https://mailman.jlab.org/pipermail/halld-offline/2021-January/008443.html New version set: 4.35.0] (Mark)
## [https://mailman.jlab.org/pipermail/halld-offline/2021-January/008447.html Tape Recovery News]
+
## [https://mailman.jlab.org/pipermail/halld-offline/2021-January/008447.html Tape Recovery News] (Mark)
 
## [https://mailman.jlab.org/pipermail/halld-offline/2021-February/008449.html Old packages backed up and deleted from group disk] (Mark)
 
## [https://mailman.jlab.org/pipermail/halld-offline/2021-February/008449.html Old packages backed up and deleted from group disk] (Mark)
 
## [https://mailman.jlab.org/pipermail/jlab-scicomp-briefs/2021q1/000833.html HSF Training: Continuous Integration in GitHub, February 16-19] (Mark)
 
## [https://mailman.jlab.org/pipermail/jlab-scicomp-briefs/2021q1/000833.html HSF Training: Continuous Integration in GitHub, February 16-19] (Mark)
# Review of [[GlueX Software Meeting, January 5, 2021#Minutes|Minutes from the Last Software Meeting]] (all)
+
# Review of [[GlueX Software Meeting, January 19, 2021#Minutes|Minutes from the Last Software Meeting]] (all)
 
# [[HDGeant4_Meeting, January 26, 2021#Minutes|Minutes from the Last HDGeant4 Meeting]] (all)
 
# [[HDGeant4_Meeting, January 26, 2021#Minutes|Minutes from the Last HDGeant4 Meeting]] (all)
 +
# Collaboration Meeting Planning
 +
## [[GlueX-Collaboration-Feb-2021|Upcoming Meeting]]
 +
## [[GlueX-Collaboration-Oct-2020|Last Meeting]]
 +
# [https://docs.google.com/presentation/d/1GzmYiLmANsP-xxgHHGObSAam3pKSX6wFNNhEL25vcXg/edit#slide=id.p Reconstruction on the OSG] (Richard)
 
# Review of recent issues and pull requests:
 
# Review of recent issues and pull requests:
 
## halld_recon
 
## halld_recon
Line 31: Line 35:
 
# Review of [https://groups.google.com/forum/#!forum/gluex-software recent discussion on the GlueX Software Help List] (all)
 
# Review of [https://groups.google.com/forum/#!forum/gluex-software recent discussion on the GlueX Software Help List] (all)
 
# Action Item Review (all)
 
# Action Item Review (all)
 +
 +
== Minutes ==
 +
 +
Present: Alex Austregesilo, Thomas Britton, Sean Dobbs, Mark Ito (chair), Igal Jaegle, Richard Jones, David Lawrence, Simon Taylor, Nilanga Wickramaarachchi
 +
 +
There is a [https://bluejeans.com/s/8jPxgwuScQc/ recording of this meeting]. Log into the [https://jlab.bluejeans.com BlueJeans site] first to gain access (use your JLab credentials).
 +
 +
=== Announcements ===
 +
 +
# [https://mailman.jlab.org/pipermail/halld-offline/2021-January/008443.html New version set: 4.35.0]. This one has the latest release of AmpTools.
 +
# [https://mailman.jlab.org/pipermail/halld-offline/2021-January/008447.html Tape Recovery News]. The latest from Bryan Hess from last week.
 +
# [https://mailman.jlab.org/pipermail/halld-offline/2021-February/008449.html Old packages backed up and deleted from group disk]. Peter Pauli reported that halld_recon 4.5.0 needed to put back for simulations on the OSG. Mark did that this morning.
 +
# [https://mailman.jlab.org/pipermail/jlab-scicomp-briefs/2021q1/000833.html HSF Training: Continuous Integration in GitHub, February 16-19]. Please see the Indico page for details.
 +
 +
=== Review of Minutes from the Last Software Meeting ===
 +
 +
We went over the [[GlueX Software Meeting, January 19, 2021#Minutes|minutes from January 19]] without significant comment.
 +
 +
=== Minutes from the Last HDGeant4 Meeting ===
 +
 +
We went over the [[HDGeant4_Meeting, January 26, 2021#Minutes|minutes from January 26]]. Mark mentioned that he is close to incorporating [https://github.com/JeffersonLab/Diracxx Richard's Diracxx package] into our set of standard software.
 +
 +
=== Collaboration Meeting Planning ===
 +
 +
We decided that it was premature to make a definite plan for software-related talks at the upcoming [[GlueX-Collaboration-Feb-2021|collaboration meeting]]. We will revisit the topic once the agenda is announced.
 +
 +
=== Reconstruction on the OSG ===
 +
 +
Richard gave us an update on progress toward doing large-scale reconstruction on the Open Science Grid (OSG). He described issues with running the jobs, which have largely been solved. In the process of comparing the results on the OSG with those from the JLab farm, he did not find agreement and traced it to non-reproducibility of the resulting reconstructed events from repeated runs on identical data.
 +
 +
Further investigation reveals that most of the problems are in the Analysis Library. He continues to pursue the painstaking exploration of the problem.
 +
 +
Please see [https://docs.google.com/presentation/d/1GzmYiLmANsP-xxgHHGObSAam3pKSX6wFNNhEL25vcXg/edit#slide=id.p Richard's slides] for the details.
 +
 +
=== Standardized Tests ===
 +
 +
Sean brought up the need to step up our testing game. We discussed ideas to both augment our current procedures and develop new ones.
 +
 +
* We do a build test and simple execution tests on new pull requests. The output of the execution tests are not examined.
 +
* We do a nightly build test of all GlueX software. There is no nightly execution test done.
 +
* Every three days we do a reconstruction test with the latest nightly build. There is no automated verification of the results, just an email with a link to selected histograms.
 +
* Twice a week we do a simulation test, from event generation to reconstruction and histogram production. There is no automatic verification of results, although a small number of [https://halldweb.jlab.org/disk_management/b1pi_mean.png quantities are plotted as function of calendar time]. The test also suffers from low statistics.
 +
* There is no test of output reproducibility.
 +
 +
Sean proposed we add this to future software meeting agendas. Richard suggested starting a wiki page to collect ideas.
 +
 +
=== Action Item Review ===
 +
 +
# Ask the Computer Center about staging the start of jobs in time. (Mark I.)
 +
# Make sure that the automatic tests of HDGeant4 pull requests have been fully implemented. (Mark I., Sean)
 +
# Finish conversion of halld_recon to use JANA2. (Nathan)
 +
# Release CCDB 2.0 (Dmitry, Mark I.)
 +
# Start a wiki page to collect ideas on software testing. (Mark)

Latest revision as of 20:38, 30 March 2021

GlueX Software Meeting
Tuesday, February 2, 2021
3:00 pm EST
BlueJeans: 968 592 007

Agenda

  1. Announcements
    1. New version set: 4.35.0 (Mark)
    2. Tape Recovery News (Mark)
    3. Old packages backed up and deleted from group disk (Mark)
    4. HSF Training: Continuous Integration in GitHub, February 16-19 (Mark)
  2. Review of Minutes from the Last Software Meeting (all)
  3. Minutes from the Last HDGeant4 Meeting (all)
  4. Collaboration Meeting Planning
    1. Upcoming Meeting
    2. Last Meeting
  5. Reconstruction on the OSG (Richard)
  6. Review of recent issues and pull requests:
    1. halld_recon
    2. halld_sim
    3. CCDB
    4. RCDB
    5. MCwrapper
  7. Review of recent discussion on the GlueX Software Help List (all)
  8. Action Item Review (all)

Minutes

Present: Alex Austregesilo, Thomas Britton, Sean Dobbs, Mark Ito (chair), Igal Jaegle, Richard Jones, David Lawrence, Simon Taylor, Nilanga Wickramaarachchi

There is a recording of this meeting. Log into the BlueJeans site first to gain access (use your JLab credentials).

Announcements

  1. New version set: 4.35.0. This one has the latest release of AmpTools.
  2. Tape Recovery News. The latest from Bryan Hess from last week.
  3. Old packages backed up and deleted from group disk. Peter Pauli reported that halld_recon 4.5.0 needed to put back for simulations on the OSG. Mark did that this morning.
  4. HSF Training: Continuous Integration in GitHub, February 16-19. Please see the Indico page for details.

Review of Minutes from the Last Software Meeting

We went over the minutes from January 19 without significant comment.

Minutes from the Last HDGeant4 Meeting

We went over the minutes from January 26. Mark mentioned that he is close to incorporating Richard's Diracxx package into our set of standard software.

Collaboration Meeting Planning

We decided that it was premature to make a definite plan for software-related talks at the upcoming collaboration meeting. We will revisit the topic once the agenda is announced.

Reconstruction on the OSG

Richard gave us an update on progress toward doing large-scale reconstruction on the Open Science Grid (OSG). He described issues with running the jobs, which have largely been solved. In the process of comparing the results on the OSG with those from the JLab farm, he did not find agreement and traced it to non-reproducibility of the resulting reconstructed events from repeated runs on identical data.

Further investigation reveals that most of the problems are in the Analysis Library. He continues to pursue the painstaking exploration of the problem.

Please see Richard's slides for the details.

Standardized Tests

Sean brought up the need to step up our testing game. We discussed ideas to both augment our current procedures and develop new ones.

  • We do a build test and simple execution tests on new pull requests. The output of the execution tests are not examined.
  • We do a nightly build test of all GlueX software. There is no nightly execution test done.
  • Every three days we do a reconstruction test with the latest nightly build. There is no automated verification of the results, just an email with a link to selected histograms.
  • Twice a week we do a simulation test, from event generation to reconstruction and histogram production. There is no automatic verification of results, although a small number of quantities are plotted as function of calendar time. The test also suffers from low statistics.
  • There is no test of output reproducibility.

Sean proposed we add this to future software meeting agendas. Richard suggested starting a wiki page to collect ideas.

Action Item Review

  1. Ask the Computer Center about staging the start of jobs in time. (Mark I.)
  2. Make sure that the automatic tests of HDGeant4 pull requests have been fully implemented. (Mark I., Sean)
  3. Finish conversion of halld_recon to use JANA2. (Nathan)
  4. Release CCDB 2.0 (Dmitry, Mark I.)
  5. Start a wiki page to collect ideas on software testing. (Mark)