Difference between revisions of "Online farm testing 5-dec-2012"

From GlueXWiki
Jump to: navigation, search
m
m
 
(18 intermediate revisions by the same user not shown)
Line 1: Line 1:
 
'''Current status of online farm tests in the counting house'''
 
'''Current status of online farm tests in the counting house'''
  
−
* Main GBit Ethernet switch working (has 10 GBit capability).
+
* Infrastructure
−
* Big UPS working.
+
** Main GBit Ethernet switch working (has 10 GBit capability).
−
* Borrowed nodes gluon02-gluon05 working (16 cores each, lots of memory).
+
** Big UPS working.
−
* Dave A will install borrowed 40 GBit/sec InfiniBand switch (will eventually purchase 60 GBit).
+
** Borrowed nodes gluon02-gluon05 working (64-bit RHEL6, 16 cores each, lots of memory).
−
* gluon02-05 have both GBit and IB interfaces.
+
** Dave A will install borrowed 40 GBit/sec InfiniBand switch (will eventually purchase 60 GBit).
−
* DAQ software installed and working (EVIO, cMsg, ET).
+
** gluon02-05 have both GBit and IB interfaces.
−
* Used rawevent plugin to convert Dave L's hddm file to EVIO format using mc2coda package from Dave A.
+
 
−
* Generated compressed EVIO file using my mc2coda_compress program (to 8 GB/evt vs 24 GB/evt).
+
 
−
* Created ET system on gluon02.
+
* Software
−
* Fake event builder (ET producer) montest injects EVIO events from a file into ET system.
+
** DAQ software installed and working (EVIO, cMsg, ET).
−
* Local injection rate about 110 kHz per producer (5 producers gives 550 kHz), each producer takes 100% of one core.
+
** rawevent plugin converts hddm file to EVIO format using mc2coda package from Dave A.
−
* Minimal ET system overhead, maybe 5% of a core.
+
** Event size is 24 kB/event out of mc2coda.
−
* Local consumer can keep up with producer rate.
+
** mc2coda_compress compresses raw EVIO file, 8 GB/evt.
−
* Remote consumer saturates GBit Ethernet at 95% level!
+
** montest reads EVIO files and inserts events into ET system (fake EB).
−
* Further tests when InfiniBand working, expect factor 10 increase, not factor 40.
+
** Online monitoring plugins ready, make occupancy plots for all supported detectors.
 +
** Waiting for JANA improvements.
 +
 
 +
 
 +
* Results for local transfers
 +
** montest injection rate about 110 kHz per instance for uncompressed events, 350 kHz for compressed events.
 +
** Injection rate scales with the number of instances.
 +
** Each producer instance takes 100% of one core.
 +
** ET system overhead about 5% of a core.
 +
** Local consumers can keep up with producer rate.
 +
 
 +
 
 +
* Results for one-way network transfers
 +
** Remote consumer saturates GBit Ethernet at 95% level
 +
** 5 kHz for uncompressed events, 15 kHz for compressed events.
 +
** Consumers use about 25% of a core.
 +
** ET system about 20% of a core.
 +
** Results mostly independent of the number of consumer nodes.
 +
 
 +
 
 +
* InfiniBand testing
 +
** Expect factor 10 increase, not factor 40.
 
** DAQ tests show you get 25% of 40 GBit using TCP over IB.
 
** DAQ tests show you get 25% of 40 GBit using TCP over IB.
−
** we will purchase 60 GBit IB.
+
** Can trunk IB ports.
−
* Remote event rate 5 kHz for uncompressed 25 kB events, expect 20 kHz for 8 kB compressed events.
+
** We expect to purchase 60 GBit IB.
−
* Another step is to introduce JANA analysis process. Dave L:
+
 
−
** is waiting for development of evioETchannel class (me, soon).
+
 
−
** has updated JANA to read uncompressed EVIO files.
+
* JANA analysis processes
−
** is working on reading compressed EVIO data.
+
** Waiting for development of evioETchannel class (me, soon).
 +
** Dave L has updated JANA to read uncompressed EVIO files.
 +
** Dave L is working on reading compressed EVIO data.
 +
** Full test requires implementing translation table in JANA in a standard way.
 +
 
 +
 
 +
* Farm manager
 +
** Vardan and I will develop this.
 +
** CODA component manages farm processes, which are NOT CODA components.
 +
** CODA components must be running and partake in run control state machine.
 +
 
 +
 
 +
* Additional testing
 +
** RootSpy when JANA online processes working.
 +
** Test different ET architectures, more nodes, etc.
 +
** Two-way network transfers.
 +
** Simulated event recorder?

Latest revision as of 09:57, 5 December 2012

Current status of online farm tests in the counting house

  • Infrastructure
    • Main GBit Ethernet switch working (has 10 GBit capability).
    • Big UPS working.
    • Borrowed nodes gluon02-gluon05 working (64-bit RHEL6, 16 cores each, lots of memory).
    • Dave A will install borrowed 40 GBit/sec InfiniBand switch (will eventually purchase 60 GBit).
    • gluon02-05 have both GBit and IB interfaces.


  • Software
    • DAQ software installed and working (EVIO, cMsg, ET).
    • rawevent plugin converts hddm file to EVIO format using mc2coda package from Dave A.
    • Event size is 24 kB/event out of mc2coda.
    • mc2coda_compress compresses raw EVIO file, 8 GB/evt.
    • montest reads EVIO files and inserts events into ET system (fake EB).
    • Online monitoring plugins ready, make occupancy plots for all supported detectors.
    • Waiting for JANA improvements.


  • Results for local transfers
    • montest injection rate about 110 kHz per instance for uncompressed events, 350 kHz for compressed events.
    • Injection rate scales with the number of instances.
    • Each producer instance takes 100% of one core.
    • ET system overhead about 5% of a core.
    • Local consumers can keep up with producer rate.


  • Results for one-way network transfers
    • Remote consumer saturates GBit Ethernet at 95% level
    • 5 kHz for uncompressed events, 15 kHz for compressed events.
    • Consumers use about 25% of a core.
    • ET system about 20% of a core.
    • Results mostly independent of the number of consumer nodes.


  • InfiniBand testing
    • Expect factor 10 increase, not factor 40.
    • DAQ tests show you get 25% of 40 GBit using TCP over IB.
    • Can trunk IB ports.
    • We expect to purchase 60 GBit IB.


  • JANA analysis processes
    • Waiting for development of evioETchannel class (me, soon).
    • Dave L has updated JANA to read uncompressed EVIO files.
    • Dave L is working on reading compressed EVIO data.
    • Full test requires implementing translation table in JANA in a standard way.


  • Farm manager
    • Vardan and I will develop this.
    • CODA component manages farm processes, which are NOT CODA components.
    • CODA components must be running and partake in run control state machine.


  • Additional testing
    • RootSpy when JANA online processes working.
    • Test different ET architectures, more nodes, etc.
    • Two-way network transfers.
    • Simulated event recorder?
Retrieved from "https://halldweb.jlab.org/wiki/index.php?title=Online_farm_testing_5-dec-2012&oldid=42742"