Online farm testing 5-dec-2012

Current status of online farm tests in the counting house

• Main GBit Ethernet switch working (has 10 GBit capability).
• Big UPS working.
• Borrowed nodes gluon02-gluon05 working (64-bit RHEL6, 16 cores each, lots of memory).
• Dave A will install borrowed 40 GBit/sec InfiniBand switch (will eventually purchase 60 GBit).
• gluon02-05 have both GBit and IB interfaces.
• DAQ software installed and working (EVIO, cMsg, ET).
• Used rawevent plugin to convert Dave L's hddm file to EVIO format using mc2coda package from Dave A.
• Generated compressed EVIO file using my mc2coda_compress program (to 8 GB/evt vs 24 GB/evt).
• Created ET system on gluon02.
• Fake event builder (ET producer) montest injects EVIO events from a file into ET system.
• Local injection rate about 110 kHz per producer (5 producers gives 550 kHz), each producer takes 100% of one core.
• Minimal ET system overhead, maybe 5% of a core.
• Local consumer can keep up with producer rate.
• Remote consumer saturates GBit Ethernet at 95% level!
• Further tests when InfiniBand working, expect factor 10 increase, not factor 40.
  • DAQ tests show you get 25% of 40 GBit using TCP over IB.
  • we will purchase 60 GBit IB.
• Remote event rate 5 kHz for uncompressed 25 kB events, expect 20 kHz for 8 kB compressed events.
• Another step is to introduce JANA analysis process. Dave L:
  • is waiting for development of evioETchannel class (me, soon).
  • has updated JANA to read uncompressed EVIO files.
  • is working on reading compressed EVIO data.
• Vardan and I will develop farm manager CODA component.