Difference between revisions of "OWG Meeting 10-Aug-2016"

From GlueXWiki
Jump to: navigation, search
(Agenda)
 
(5 intermediate revisions by 2 users not shown)
Line 30: Line 30:
 
#* Farm nodes delivered (installation pending)
 
#* Farm nodes delivered (installation pending)
 
#* gluon43 remains down
 
#* gluon43 remains down
 +
#* [[High Intensity Running|High Intensity Running]]
 
# '''Network Bandwidth to CC'''
 
# '''Network Bandwidth to CC'''
#* [File:GlueX DAQ Transfer.pdf | Note from Chip Watson]
+
#* [https://halldweb.jlab.org/wiki/images/c/c3/GlueX_DAQ_Transfer.pdf Note from Chip Watson]
 
# '''DAQ'''
 
# '''DAQ'''
 
#* Single crate data rates
 
#* Single crate data rates
#* fADC125 Testing
+
#* fADC125 Testing (Sergey)
 +
#* [[FA125 chip occupancy]] (Naomi)
 
#* Event size
 
#* Event size
 
# '''Front-end Firmware Status'''
 
# '''Front-end Firmware Status'''
Line 47: Line 49:
 
= Minutes =
 
= Minutes =
  
''TBD''
+
Attendees: David L. (chair), Simon T., Naomi, J., Eugene C., Sandy P., Chip W., Beni Z., Bryan M., Dave A., Vardan G., Mark I., Curtis M., Cody D.
 +
 
 +
== Announcements ==
 +
* New RAID Storage node is also delivered and is in Dat Center
 +
** David will be meeting with Paul L. next week to discuss where the nodes will be installed in rack room relative to switch and allowing for future farm expansion
 +
* High Intensity Running
 +
** Wiki page is up and gathering information
 +
** Need a dedicated meeting to help generate answers to questions (or form plan to generate answers)
 +
** Need new numbers by end of month to share with management and help with our own planning
 +
** Main issues:
 +
*** Reduce L1 trigger rate
 +
*** Bandwidth limitation in VME
 +
 
 +
== Network Bandwidth to CC ==
 +
* Chip sent out an e-mail with some details on how we could go about maximizing are bandwidth to permanent tape storage. He went over some highlights and gave a few other ideas.
 +
* We currently have two 10Gbit fiber links from Hall-D to CEBAF center that have been tested to give an aggregate of ~1.8GB/sec sustained rate
 +
** Other fibers were strung and are available, but would need to be outfitted on both ends
 +
** Hall-B has a 40GBit link. Our fibers could be used for this, but there is significant cost involved for the hardware needed on both ends.
 +
** May be cheaper to implement 4x10Gbit links with clever scripting to ensure all links are being utilized
 +
* Lustre file system is capable of consuming a couple of Gbyte/sec
 +
** To maximize data rate, need to setup 2 file transfers per link for a total of 4 simultaneous file transfers
 +
* New RAID disk should be capable of supplying 1.6Gbyte/sec. (Old ones probably not, though we may improve their capability if we reconfigured them with the right stripes)
 +
** May need to configure new RAID to have 4 stripes to make it easy for DAQ to efficiently write to disks at maximum rate. (Mount these as 4 separate file systems?)
 +
** Will need to limit how much is written to each disk so that only outer cylinders of disk are used which have higher bandwidth
 +
* Slightly less than 2GB/sec currently available into tape library
 +
** If we intend to write 1.6GByte/sec sustained, Chip would need 1 month lead time to procure additional tape drives
 +
* Chip suggested option: Record 1.6GByte/sec to Data Center of "pre-raw" data. After run, submit farm jobs to apply L3 filtering/reformatting and generate "raw" data which will then have 2 copies made for permanent storage
 +
** The "pre-raw" data will then be discarded and the tapes reused for the "raw" data.
 +
 
 +
== DAQ ==
 +
* Sergey has been testing single crate data rates
 +
** Modified trigger rate, holdoff time, and occupancy (via threshold and noise)
 +
* Used 3 CDC crates with fADC125 modules
 +
** 256kHz period trigger
 +
*** 200kHz, 80% livetime, 35MB/sec/crate (no hit occupancy, headers only)
 +
** 110kHz random trigger
 +
*** Required ~2.5μs holdoff time (suspect this is due to minimum holdoff needed for system)
 +
*** Lowered thresholds to implement non-zero hit occupancy
 +
* Realistic readout limit is 50-60MB/s per fADC125 crate (theoretical with current firmware is 66MB/s)
 +
* Realistic readout limit is ~120MB/s per fADC250 crate (theoretical with current firmware is200MB/s)
 +
** Cody thinks he knows why and can squeeze close to a factor of 2 out of it
 +
* fADC250 readout limited to 200MB/s
 +
** Also requires 2.5μs holdoff
 +
* Asked Cody if it would be possible to drop Trigger Time words from all but 1st and last events in block. He thought is was possible but will need to look at it.
 +
* Dave A. has been testing 10Gbit ethernet boards for ROCs.
 +
** Able to achieve ~630MB/s
 +
** Does not know current cost of cards, but likely around $500 each (we would probably need 14)
 +
** Switches in hall have 1Gbit links with a 10Gbit uplink.
 +
*** May have second 10Gbit uplink that is currently unused
 +
*** Options: Replace switches with one(s) containing multiple 10Gbit ports or run fibers directly from ROCs upstairs to switch in Counting House
 +
* Naomi showed some results of a brief study of the single chip occupancy in the fADC125
 +
** Chip occupancy seems to be around 5% for 80nA PARA run with 50μm diamond (210nA run needs to be looked at again)
 +
 
 +
== AOT ==
 +
* Beni noted that we need to ask all subsystem groups to minimize their readout windows so as to minimize data footprint and therefore, bandwidth requirement
 +
* CDC and FDC have been looked at and are already at minimum

Latest revision as of 16:07, 10 August 2016

Location and Time

Room: CC F326

Time: 2:00pm-3:00pm

Connection

You can connect using BlueJeans Video conferencing (ID: 120 390 084). (Click "Expand" to the right for details -->):

(if problems, call phone in conference room: 757-269-6460)

  1. To join via Polycom room system go to the IP Address: 199.48.152.152 (bjn.vc) and enter the meeting ID: 120390084.
  2. To join via a Web Browser, go to the page https://bluejeans.com/120390084.
  3. To join via phone, use one of the following numbers and the Conference ID: 120390084
    • US or Canada: +1 408 740 7256 or
    • US or Canada: +1 888 240 2560
  4. More information on connecting to bluejeans is available.


Previous Meeting

Agenda

  1. Announcements
  2. Network Bandwidth to CC
  3. DAQ
  4. Front-end Firmware Status
  5. AOT


Recharge Wednesday: ice cream novelties

Rwanda Pictures

Minutes

Attendees: David L. (chair), Simon T., Naomi, J., Eugene C., Sandy P., Chip W., Beni Z., Bryan M., Dave A., Vardan G., Mark I., Curtis M., Cody D.

Announcements

  • New RAID Storage node is also delivered and is in Dat Center
    • David will be meeting with Paul L. next week to discuss where the nodes will be installed in rack room relative to switch and allowing for future farm expansion
  • High Intensity Running
    • Wiki page is up and gathering information
    • Need a dedicated meeting to help generate answers to questions (or form plan to generate answers)
    • Need new numbers by end of month to share with management and help with our own planning
    • Main issues:
      • Reduce L1 trigger rate
      • Bandwidth limitation in VME

Network Bandwidth to CC

  • Chip sent out an e-mail with some details on how we could go about maximizing are bandwidth to permanent tape storage. He went over some highlights and gave a few other ideas.
  • We currently have two 10Gbit fiber links from Hall-D to CEBAF center that have been tested to give an aggregate of ~1.8GB/sec sustained rate
    • Other fibers were strung and are available, but would need to be outfitted on both ends
    • Hall-B has a 40GBit link. Our fibers could be used for this, but there is significant cost involved for the hardware needed on both ends.
    • May be cheaper to implement 4x10Gbit links with clever scripting to ensure all links are being utilized
  • Lustre file system is capable of consuming a couple of Gbyte/sec
    • To maximize data rate, need to setup 2 file transfers per link for a total of 4 simultaneous file transfers
  • New RAID disk should be capable of supplying 1.6Gbyte/sec. (Old ones probably not, though we may improve their capability if we reconfigured them with the right stripes)
    • May need to configure new RAID to have 4 stripes to make it easy for DAQ to efficiently write to disks at maximum rate. (Mount these as 4 separate file systems?)
    • Will need to limit how much is written to each disk so that only outer cylinders of disk are used which have higher bandwidth
  • Slightly less than 2GB/sec currently available into tape library
    • If we intend to write 1.6GByte/sec sustained, Chip would need 1 month lead time to procure additional tape drives
  • Chip suggested option: Record 1.6GByte/sec to Data Center of "pre-raw" data. After run, submit farm jobs to apply L3 filtering/reformatting and generate "raw" data which will then have 2 copies made for permanent storage
    • The "pre-raw" data will then be discarded and the tapes reused for the "raw" data.

DAQ

  • Sergey has been testing single crate data rates
    • Modified trigger rate, holdoff time, and occupancy (via threshold and noise)
  • Used 3 CDC crates with fADC125 modules
    • 256kHz period trigger
      • 200kHz, 80% livetime, 35MB/sec/crate (no hit occupancy, headers only)
    • 110kHz random trigger
      • Required ~2.5μs holdoff time (suspect this is due to minimum holdoff needed for system)
      • Lowered thresholds to implement non-zero hit occupancy
  • Realistic readout limit is 50-60MB/s per fADC125 crate (theoretical with current firmware is 66MB/s)
  • Realistic readout limit is ~120MB/s per fADC250 crate (theoretical with current firmware is200MB/s)
    • Cody thinks he knows why and can squeeze close to a factor of 2 out of it
  • fADC250 readout limited to 200MB/s
    • Also requires 2.5μs holdoff
  • Asked Cody if it would be possible to drop Trigger Time words from all but 1st and last events in block. He thought is was possible but will need to look at it.
  • Dave A. has been testing 10Gbit ethernet boards for ROCs.
    • Able to achieve ~630MB/s
    • Does not know current cost of cards, but likely around $500 each (we would probably need 14)
    • Switches in hall have 1Gbit links with a 10Gbit uplink.
      • May have second 10Gbit uplink that is currently unused
      • Options: Replace switches with one(s) containing multiple 10Gbit ports or run fibers directly from ROCs upstairs to switch in Counting House
  • Naomi showed some results of a brief study of the single chip occupancy in the fADC125
    • Chip occupancy seems to be around 5% for 80nA PARA run with 50μm diamond (210nA run needs to be looked at again)

AOT

  • Beni noted that we need to ask all subsystem groups to minimize their readout windows so as to minimize data footprint and therefore, bandwidth requirement
  • CDC and FDC have been looked at and are already at minimum