OWG Meeting 13-Jul-2016

Location and Time

Room: CC F326

Time: 2:00pm-3:00pm

Connection

You can connect using BlueJeans Video conferencing (ID: 120 390 084). (Click "Expand" to the right for details -->):

(if problems, call phone in conference room: 757-269-6460)

1. To join via Polycom room system go to the IP Address: 199.48.152.152 (bjn.vc) and enter the meeting ID: 120390084.
2. To join via a Web Browser, go to the page https://bluejeans.com/120390084.
3. To join via phone, use one of the following numbers and the Conference ID: 120390084
   • US or Canada: +1 408 740 7256 or
   • US or Canada: +1 888 240 2560
4. More information on connecting to bluejeans is available.

Previous Meeting

• Previous Meeting (15-Jun-2016)

Agenda

1. Announcements
   • 8 farm nodes ordered
   • 36-port IB switch (56GB/s) w/ cables : "Buyer Assigned"
   • gluon43 sent for repair
2. RAID Capacity
   • 72 buffer (200TB at 800MB/s)
   • Possible downtime of tape library
3. DAQ
   • CODA 3.0.6
4. Front-end Firmware Status
5. L3 Status (meetings)
6. AOT

Background Info. for RAID Discussion

• Existing gluon RAID specs (Click "Expand" to the right for details -->):

Here is the description of our existing RAID servers from the PO attached to PR #334477:

ITEM 005B:
DATA STORAGE NODES NODES AS PER THE TABLE IN THE STATEMENT OF WORK ENTITLED "JEFFERSON LAB PHYSICS SERVERS 2013 DATED JULY 24, 2013”
 SANDY BRIDGE E5-2630 CPU
 30*3TB SATA ENTERPRISE
 4*500 GB SATA ENTERPRISE
 2*QDR CONNECT X3 (ONE MAY BE LANDED ON THE MOTHERBOARD, OR MAY USE DUAL PORT CARD)
 LSI RAID 9271-8I RAID CARD (OR BETTER) WITH BATTERY BACKUP.

• Chip e-mail describing options. (Click "Expand" to the right for details -->):

David et al,

One thing we are doing new with file servers is configuring them as fault tolerant pairs.
For the last year we've been buying two computers plus two 44 disk chassis (front and
back disks, so a total of 4 back planes), and putting 2 disk controllers in each file server
(2 cables per controller, so 4 cables per file server).  Each server controls connects to
each back plane, and if a server dies, the other can control all of each back plane.  The
pair easily can move 1 GB/s in and 1 GB/s out concurrently, and if a node dies, the sole
survivor can manage 800 MB/s in and out.  We have been using 8 TB disks for a year now;
smaller disks are cheaper, performance scales with number of spindles.  Everything is
12 Gbps SAS3.

If you want to scale down cost and capacity but keep fault tolerance and performance,
here is a small version of the above:

* two file servers, each with one controller

* two 44 disk chassis each with 30 4-TB disks

This yields high bandwidth, 240 TB raw, 192 TB before file system, ~170 with file system,
135 TB at about 80% full.  In a pinch you could run 90% full so this gives 150 TB on top of
your current 50.

When you retire your two old servers, you could buy another 20 4 TB drives and add them
to the two disk chassis.  If you want more future growth potential then use 6 TB drives
instead of 4 TB drives.  If you want more bandwidth, start with 40+40 disks.  Maybe plan
to buy a matched pair every 2-3 years for both capacity and performance growth.  Don't
run anything longer than 5 years.

The pair loaded with 8 TB drives will be around $50K (direct), so under populating the
chassis and using smaller disks will make it a good bit cheaper.  For our pair, we get
380 TB at 80% full (to prevent fragmentation), $130 per usable TB.

Shoestring version: buy half a pair with 40*6 TB drives => 135 TB at lower performance,
perhaps $20K.  Add the mate a year later. They do still make the 36 disk all-in-one (no
active-active pairing) for a little bit less per TB, but I think going for active-active
would be best.

Chip

Recharge Wednesday: hand crafted ice cream sandwiches

Minutes

Attendees: David L. (chair), Eugene C., Chip W., Sandy P., Graham H., Dave A., Bryan M., Vardan G., Carl T., Beni Z., Curtis M., Sergey F., Sean D., Mark I.

Announcements

• 8 new farm nodes might be here around mid-August
• IB switch has now been ordered
• * Lots of cables ordered with it. Need to check on cabinet space for when they are delivered (David L. volunteered)

RAID Capacity

• Chip suggested we purchase a system similar to what the CC has recently been purchasing.
  • This consists of large a large chassis that can hold 44 disks split on 2 backplanes.
  • Computer is in separate housing with RAID controller connected via cable to disk backplane
  • System can be run "active-active" where two servers can control both backplanes simultaneously
  • This provides automatic failover where full disk is still available(at reduced rate) if one server node dies
• Consensus that we should go for Chip's "shoestring" option where we buy single server and single chassis with 6TB disks to give us ~135TB additional, usable space
  • Chip will check with vendor who recently won contract and procurement to see if we can get this through them
  • Worst case scenario: system is spec'd and put out for bids with short time scale. In either case, we should expect to have the new disk sometime in Sept.
  • Cost expected to be in $20k range

Tape Library Reconfiguration

• Computer center floor plan will be reconfigured to run at 45 degree angle to current configuration.
• Contract constraints currently have this in the middle of the Spring run
• Initial schedule has this for one week in first half of March. Vendor though may want to shift it by several weeks. Clarity of schedule will improve as we get closer to activity
• Staging disk capacity is plenty for storing several days worth of data at high rate (800MB/s)
• Option available to make 2nd copy on another part of staging disk while tape library unavailable.
  • Possible to use our 72 hour contingency capacity to store 2nd copy for safety though we'd have to delete the 2nd copy if the contingency required activation.
• Eugene noted that accelerator cost ~$4-$5 million / week so we have strong interest in ensuring redundancy so that no beam data is lost.

DAQ

• CODA 3.0.6 has been tested and shown to work, though no stress/performance testing done in Hall-D yet
• Both ET and direct connection tested and working
• f125 Busy mechanism not performance tested in Hall-D
  • Sergey F. and Cody D. will need to schedule time and coordinate with Alex S. who is currently testing f250 firmware using TS
• Jinflux system is now integrated into CODA 3.0.6 and can be accessed via web
  • (http://claraweb.jlab.org:3000)
  • Full java API is available. Could be used to transfer EPICS values on current run conditions to display for easy offsite viewing of current run conditions.

L3 Status

• From Spring 2016 data, roughly 1/4 of L1 events are reconstructable
• Original design goal was for 200kHz event rate
  • Currently approved experiments have maximum of 100kHz
• Eugene predicts we will run at 150kHz and L3 will be able to knock that down to about 60kHz to disk
  • Dave A. noted that with current event size, that will mean 1.6GB/s which will require 2 event streams
• Beni suggested we could reduce event size by cutting out-of-time hits when reforming single events
• David L. noted that we may also be able to make some gains by dropping Trigger Time or other redundant data that appears in multiple places within the same event.

Quote of the day:
David L. : "That number was just pulled out of the air"
Dave A. : "Better there than from somewhere else."