Barrel Calorimeter Expert

From Hall D Ops Wiki
Jump to: navigation, search

The Barrel Calorimeter

The barrel calorimeter (BCAL) is a lead-scintillating fiber matrix readout with 3840 S12045 Hamamatsu multi-pixel photon counters (MPPCs). The MPPC light sensors operate a bias voltage less than 76 V. Liquid coolant is circulated through the readout assemblies to set and maintain the temperature of the sensors at their operating temperature between 5 and 25 degrees C.

Routine operation

Instructions for shift personnel are available in the Barrel Calorimeter instructions for shift personnel.

Voltage Setting for Specific Overvoltage and Temperature

Fig. 1. Graphical interface for setting the voltages on the BCAL. Indicated are the pulldown menus for retrieving saved voltages from a file and the menus for turning the voltages on and off.

Changes to the voltage settings are only allowed by an expert, or under his/her guidance. The bias voltages for the MPPC sensors can be set using the GUI show in Fig. 1. The bias settings must be set separately for upstream and downstream. For upstream, use the "ALL U Bias" button to select the menu and choose "BCAL U: Set Bias Parameters." Set the "ramp up/down"= 5A/s and "Set Voltage Offset" (overbias) to the desired value (nominal=1.4 V). For production running, make sure the chillers are cooling at the production temperature (nominal=5 degC), then use the "Set Voltages Using the Measured Temperature" button to set the voltages. [Alternatively, one may set the voltages for a fixed temperature by entering a temperature and using the "Temperature for Voltage Settings" option.] A <carriage return> must be entered for the value to register. Once complete, click on the "Set Voltage" button. Wait for the rainbow "disk" to finish turning (patience, it takes time to load all values). If the voltages are already on, you are done, otherwise you need to use the "BCAL U: Turn ON Bias Channels" menu item to turn them on. The voltage read back and the current draw of each channel are histogramed and plotted at the bottom of the GUI. When all voltages are on at their set values, the display turns green.

Saving and Comparing Save/Restore Files

Specific voltage settings can be saved (from electronics to file) or restored (from file to electronics). This is done via the "Save/Restore" button on the top right of the GUI. The files are automatically saved with the date/time to /gluex/data/burt/voltages/BCAL_BIAS/. We have renamed the files incorporating the date/time temperature and 10 times the overvoltage (to avoid an additional "." in the name). A soft link is created in that directory to point to the nominal file for the current run. The voltages from different files can be compared using the script compare_snap_voltages.C which is located in /gluex/Subsystems/BCAL/scripts.

Chiller Operation

The chiller temperature should be set to 2 degC less than the desired SiPM temperature. The chillers are normally in 'remote' mode. (You can still change the temperature manually). To change the 'set' temperature, use the up/down arrows on the front key pad to the desired temperature. The values are in deg F. Wait for the 'readback' temperature to settle to the 'set' value, as it may overshoot at first. At startup it may be necessary to pull the reset (green) switch forward that is accessed through the hole on the side of the chiller closest to the front.

LED Pulsing Sequence for Production

We take BCAL LED triggers during production, with the pulsing sequence is asynchronous to starting and stopping runs.

The LEDs are pulsed using a stand-alone script which runs in the background. This command will start an infinite loop cycling through preset configurations of the LED pulser, currently just switching between pulsing the upstream and downstream LEDs at 6.25 V.

The actual pulsing is performed by the 'BCAL_2side_pulsing.py' python script, which must be running in the background. Under normal circumstances, the script is always running and no action is needed from the user. The script checks the start/stop button on the GUI to determine whether it should be pulsing.

Background Script: Auto Start

A cron job is started up by hdops@gluon27, which checks that the background script is running and if it is not starts it.

  • The crontab command is 
 */5 * * * *  /home/hdops/BCAL/bcal_column_pulsing/cron_bcal_column_pulsing.csh > /dev/null
  • To list the command: > crontab -l
  • To edit the command and run: > crontab -e

Background Script: Manual Operation

ssh hdops@gluon27
cd /home/hdops/BCAL/bcal_column_pulsing
BCAL_2side_pulsing.py

Note:

  • this will run the script in the foreground and will die if the terminal is exited, but can be used for debugging.


If the BCAL LEDs on 1 side stop triggering

Rack U1-1 in Hall D. The crate U1-1-TOP contains the hardware for pulsing the BCAL LEDs.

It is possible for LED pulses from one side the the BCAL to not show up in the data stream. This would be seen in the monitoring plots called HistMacro_LEDTrigger. This might be corrected by rebooting the crate which houses the BCAL LED pulsing electronics.

The instruction are as follows:

  1. Make an access into Hall D.
  2. After entering the hall find the crate U1-1-TOP. Look at the stairs going up to the platform, it is the top crate in the rack right at the foot of the stairs (see figure.)
  3. Turn the crate power off using the switch.
  4. Once the switch is off, go behind the crate and unplug it for about 5 seconds.
  5. Power up the crate.
  6. Make a log entry and record the run number.

Restore after a long shutdown

Hot Checkout

BCAL Hot Checkout

Low Voltage

After a long shutdown (e.g.summer), we have found that some of the MPOD channels for the LV will trip upon restoration. The following sequence should be followed:

  • Power cycle the crates either locally or remotely
    • LOCAL:
      • Turn off the Bias/LV crate with the front button.
      • Unplug the power from the back of the crate. Wait about 10 s.
      • Plug power back and turn crate on with front button.
    • REMOTE: Use the PDU, see below #Remote Control PDU
  • Restore voltages from saved file
  • 'Reset and Turn OFF Bias Channels' and 'Reset and Turn OFF LV Channels'
  • Turn channels back ON. If everything comes up you are finished.

If some LV channels trip when coming back, it may be that some of their hardware parameters (not available to EPICS) have changed. (Don't know why). I this case contact Nick or Chris and let them trouble shoot the modules. See Log Entry 3486939.

LED Runs for BCAL Checklist

Power Up

There is a peculiarity of the V495 VME modules that we use to generate trigger pulses. Upon power up, the controller needs to be reset for the modules to operate.

If the LED bias or LV is not working and the VME crate was cycled properly, turn on the green switch on U1-1-MID MPOD crate.

History of Results

Up to now, the DAQ runs are taken independently using the local master trigger interface (TI) for north and south, but should be consolidated by feeding the LED triggers to the trigger supervisor (TS).

  • The checklist entry for the 2023 Sprimg production run is recorded in Log Entries 4107444 4106865
  • The checklist entry for the 2021 Summer production run is recorded in Log Entries 3893606 3893960 3894258
  • The checklist entry for the 2019 fall production run is recorded in Log Entries 3735113 3740112 3738357
  • The checklist entries for the 2019 spring production run are recorded in Log Entries 3646562, 3646447, 3646186
  • Updated checks following Hurricane Florence shutdown for fall 2018 Fall production are recorded in Log Entries 3595377,3595364
  • The checklist entries for the 2018 Fall production run are recorded in Log Entries 3584631, 3583676, 3584833, 3584865, 3584874
  • The checklist entries for the 2018 spring production run are recorded in Log Entry 3489168, 3497205 and 3497398
  • The checklist entries for the 2017 spring production run are recorded in Log Entry 3453056.
  • The checklist entries for the 2016 fall production run are recorded in Log Entry 3422892.
  • The checklist entries for the 2016 spring production run are recorded in Log Entry 3373213.
  • The checklist entries for the 2015 spring commissioning run are recorded in Log Entries 3322823 and 3322905.
  • The checklist entry for the fall 2014 commissioning run is recorded in Log Entry 3297440.


Status Occupancy Plots

Recall that raw data that is recently collected can be found with the symbolic linked directory: 

/gluex/data/rawdata/all/

After the raw data were taken, the following procedure was used to generate root files starting from the /gluex/Subsystems/BCAL/data/ directory:

  1. Log onto computer OTHER than gluon01-gluon05.
  2. generate root file to dump wave forms
    • hd_root -PPLUGINS=DAQTree /gluonraid4/data4/rawdata/active/RunPeriod-2019-11/rawdata/Run0*/hd_rawdata_070234_000.evio -o /gluex/Subsystems/BCAL/rootfiles/hd_rawdata_070234_000.root
    • display waveforms
    • root [0] .L /gluex/Subsystems/BCAL/scripts/dump_tree_waveform_dalton.C+
    • root [0] dump_tree_waveform_dalton("/gluex/Subsystems/BCAL/rootfiles/hd_rawdata_070234_000.root",20,1,0);
    • (first argument is filename, second argument is minimum threshold above pedestal required to plot, third argument is whether to pedestal subtract, fourth argument is first event to plot)
  3. generate root file to create occupancy plots
    • hd_root -PPLUGINS=DAQTreeBCAL /gluonraid4/data4/rawdata/active/RunPeriod-2019-11/rawdata/Run0*/hd_rawdata_070234_000.evio -o /gluex/Subsystems/BCAL/rootfiles/hd_rawdata_070234_000_status.root
    • create occupancy plot for FADCs
    • root [0] .L /gluex/Subsystems/BCAL/scripts/SiPM_Test_fADC.C+
    • root [0] SiPM_Test_fADC("/gluex/Subsystems/BCAL/rootfiles/hd_rawdata_070234_000_status.root");
    • root [0] c1->SaveAs("/gluex/Subsystems/BCAL/plots/LED_status/hd_bcal_n-led_up_mode8-ADC_R70234.pdf");
  4. create occupancy plot for TDCs
    • root [0] .L /gluex/Subsystems/BCAL/scripts/SiPM_Test_TDC.C+
    • root [0] SiPM_Test_TDC("/gluex/Subsystems/BCAL/rootfiles/hd_rawdata_070234_000_status.root");
    • root [0] c1->SaveAs("/gluex/Subsystems/BCAL/plots/LED_status/hd_bcal_n-led_up_mode8-TDC_R70234.pdf");

LED Occupancy Runs: Spring 2019 to PRESENT (using TS method)

Consolidated by feeding the LED triggers to the trigger supervisor (TS).

  • Make sure that SiPM and LED voltages are off
    • Turn off SiPM bias voltage
    • Turn off LED bias voltage
  • Start the DAQ system. 
ssh hdops@gluon02
rcm.sh
    • choose EXPERT and hd_bcal.tsg and bcal_led_m10.conf
    • Complete prestart/start sequence, leaving system ready and waiting for triggers
 ssh hdops@gluon47
 cd /gluondaqfs/hdops/CDAQ/daq_dev_v0.31/daq/tools/bcal_pulse_sequence
 BCAL_test.py -s U -n 10 -b 6.5 -t 5 -o 1.4
 BCAL_test.py -s D -n 10 -b 6.5 -t 5 -o 1.4
 BCAL_test.py -h

Note that because of the low event rate and low number of events, the buffers might not flush until the run is ended. i.e. There might be no events until the run is ended.

  • Troubleshooting
    • It's been observed that the python script sometimes loses necessary modules, giving errors such as "cannot find mysql/sqlalchemy". If this is encountered, try reinstalling them with commands of the form
python -m pip install --trusted-host files.pythonhosted.org mysql --user

OLD METHOD: Led Occupancy Runs (using TI)

The DAQ runs are taken independently using the local master trigger interface (TI) for north and south.

Data were first taken with all LEDs on so that the timing could be checked. Once offsets were adjusted, data were taken using Orlando's script (BCAL_test.py) to pulse the LED. (More information can be found at BCAL DAQ Instructions). The following procedure was followed.

  1. Make sure that SiPM and LED voltages are off
    • Turn off SiPM bias voltage
    • Turn off LED bias voltage
    • Make sure no triggers are enabled or running.
  2. Start the DAQ system. 
ssh hdbcalops@gluon02
cd DAQ/daq_run/work
source online_setup.cshrc
rcm.sh
    • With new pedestal checks at begining of run, one must make sure that the SiPM low voltage is ON during the go transition.
    • Use hd_bcal_n.ti or hd_bcal_s.ti setups, and the appropriate up/down mode9/mode10 configuration files such as led_downstream_mode10.cnf.
    • Complete prestart/start sequence, leaving system ready and waiting for triggers
  2. Open a terminal and login to hdops@gluon02-05
    • cd $DAQ_HOME/tools/bcal_pulse_sequence
    • BCAL_test.py -s U -q 1,4 -n 10 -b 6.5 -t 5 -o 1.4 // Upstream, North quadrants, 10 pulses, LED bias=7 V, T=5 degC, Vover=1.4 V]
    • BCAL_test.py -s D -q 2,3 -n 10 -b 6.5 -t 5 -o 1.4 // Downstream, South quadrants, 10 pulses, LED bias=7 V, T=5 degC, Vover=1.4 V]
    • BCAL_test.py –h // Help option
  3. The script with the above arguments should loop through a total of 320 triggers (10 pulses x 4 voltages x 4 columns x 2 quadrants).
    • End run


Status Occupancy Plots

 ssh hdops@gluon47
 cd /gluex/Subsystems/BCAL/data/
 hd_root -PPLUGINS=DAQTreeBCAL /gluex/data/rawdata/all/Run060398/hd_rawdata_060398_000.evio -o /gluex/Subsystems/BCAL/rootfiles/hd_rawdata_060398_000.root

To analyze upstream LED data: 

 root -q '/gluex/Subsystems/BCAL/scripts/SiPM_Test_fADC.C("/gluex/Subsystems/BCAL/rootfiles/hd_rawdata_060398_000.root",10,4,1)'

To analyze downstream LED data: 

 root -q '/gluex/Subsystems/BCAL/scripts/SiPM_Test_fADC.C("/gluex/Subsystems/BCAL/rootfiles/hd_rawdata_060398_000.root",10,4,0)'

Crate temperature feedback

Enable

  • Enable fan feedback in CSS
Main Action Bar -> DAQ -> VME/VXS Crates -> EPICS PID SETTINGS -> PID Parameters for BCAL Crates -> Crate DISABLE Mode -> Enable
  • Start temperature set-point correction
ssh  gluon47
cd /home/dalton/work/epics/cratetemp/
./crate_AdjustTempSetpoint_BCAL_FCAL.pl
  • Protect temperature set-point correction with cron
crontab -e
*/5 * * * *  $HOME/work/epics/cratetemp/cron_cratesettemps_BCALFCAL.csh > /dev/null

Disable

  • Disable fan feedback in CSS
 Main Action Bar -> DAQ -> VME/VXS Crates -> EPICS PID SETTINGS -> PID Parameters for BCAL Crates -> Crate DISABLE Mode -> DISABLE
  • comment out respawning from the crontab on gluon47
  • Kill temperature set-point correction (crate_AdjustTempSetpoint_BCAL_FCAL.pl on gluon47)

Remote Control PDU

Fig. 2. Screen shot of the PDU web interface.


The bias (ISEG) and low voltage (MPOD) supplies providing power to the SiPMs are plugged into remotely controllable power distribution units (PDUs). This allows us to recycle the power to the crates via a web interface. Each of the four ISEG/MPOD crates is plugged into its own PDU. Each crate provides the bias and LV for one quarter of the detector. At the present time, only the first output of the distribution strip is used. The following PDUs are being used, given by the name of the rack followed by -pdu. If there are problems loading the page, try opening a firefox tab logged in as a user other than hdops

n1-1-pdu n1-1-top voltage webpage upstream, left
n1-6-pdu n1-6-top voltage webpage downstream, left
s1-1-pdu s1-1-top voltage webpage upstream, right
s1-6-pdu s1-6-top voltage webpage downstream, right

The remote access to the crate is via a web browser in the Counting House pointing to the name of the PDU (e.g. type s1-1-pdu into browser and confirm with user/pass=admin/admin). See the screen shot of the web interface. To cycle the power

  • Click on the Actions, then Loads button.
  • Highlight the first line (Load 1). The On/Off/Cycle buttons should become enabled
  • Click on the Off and confirm preference. It will take about half a minute to complete the command (patience!). After "Output Current Changed" message appears under Alarm Status, the status should turn from 'On' to 'Off', then message disappears.
  • Repeat reverse action to turn PDU input back on.
  • In the CSS GUI load the bias voltages if the bias load was cycled.

SiPM Dark Rate Measurements

The dark rate of SiPMs can be measured by reading out no-pulse waveforms while SiPM bias voltages are set. The dark rate is then related to the RMS variation about the pedestal subtracted mean ADC value. Performing this study requires the DAQ to read out all channels, rather than only reading out channels above a given ADC threshold as normally is done. The DAQ system also requires a source of triggers. For the 2019 measurement (and presumably onwards) this is achieved by pulsing LEDs normally in quadrant 3, then excluding this quadrant from later analysis. Quadrants 1, 2, and 4 are also read out as a result. Nominally, quadrant 1 only is used later for later study. We take data for 12 voltage (bias = OFF, 0.0, 0.2, 0.4, ..., 2.0 V) settings and repeat for three different chiller temperatures (18 C, 10 C, 5 C); thus 36 runs are needed for this study.

Performing the pedestal study requires a few different windows up at a time:

  1. DAQ screen with special no threshold configuration.
  2. LED pulser screen from CSS GUI
  3. SiPM bias voltage set screen (two windows: one for upstream, one for downstream)
  4. MyaViewer with selected channels, to verify voltage settings make it to the detector properly

Afterwards, we run a plugin over the raw data, and a python script to analyze and compare runs.

DAQ Setup

The DAQ should be started in the following configuration 

RUN: EXPERT
SETUP: hd_bcal.tsg
CONFIG: bcal_nothresh_m10.conf

LED Pulser setup

Use the pulser with the following settings:

  • Turned ON for quadrant 3, OFF for quadrants 1, 2, and 4.
  • LED bias voltage: 6.25 (standard setting)
  • 50 Hz frequency
  • 1000 total pulses
  • It doesn't matter whether upstream pulsers or downstream pulsers are used, but it's recommended that one be consistent.

SiPM bias voltages

Use the CSS GUI to set bias voltages. From the Main Action Bar, go to BCAL -> BCAL Voltages. Keep this screen open. Use ALL U Bias -> BCAL U: Turn ON(OFF) Bias Channels to turn SiPMs on(off). Repeat for downstream. Verify that all LV channels are turned on. Take one data point with SiPM bias turned off. To set SiPM bias voltages to a given setpoint, starting from the BCAL voltages css screen, go to ALL U bias -> BCAL U: Set Bias Parameters and ALL U bias -> BCAL U: Set Bias Parameters. Keep these two windows up. Set "Ramp Up" and "Ramp Down" to 5 A/s, hitting enter to save information. Input the desired bias voltage, hit enter, and select "Set Voltage using Measured Temperature -> Set Voltage -> Yes". Repeat for upstream and downstream. A console will appear in this window printing current voltage settings, finishing with module 48 upstream/downstream. This can take 7-10 minutes.

Check Voltages on MyaViewer

The css interface to set SiPM voltages is somewhat buggy: you should double check that both upstream and downstream channels have been set properly according the the terminal window output. To verify bias voltages get properly set, it's recommended that one open a MyaViewer window and plot "BCAL:bias:U:48:4:v_term" and "BCAL:bias:D:48:4:v_term". Right click on one and select "Move graph" to overlay on the other. Start auto-stepping forward in time with the keyboard command ctrl+a and use ctrl+t to select an appropriate time window (a few minutes to few hours). Upstream and downstream roughly line up and have the expected number of steps up/down in voltage, one can check that voltage set commands were sent properly.

Collecting Data

Once ready, use the BCAL GUI to fire BCAL quadrant 3 LEDs for 1,000 events. Make a log entry with run numbers and corresponding bias voltage settings (example).

Analyzing Data

The raw data should be copied to a directory where one can run the F250_mode10_pedestal plugin over it. The plugin outputs should be analyzed (see analysis scripts in hd_utilities) and compared to previous years.

More reference material can be found at: gluex-doc-4285

Restarting memory for discriminator scalers

These are the instructions from Paul (2/8/2016): If the discriminator scalers are showing zero, reboot the procServ's on these crates. There's no way to do this through the GUIs. In the meantime, you can ssh to the ROC in question (these are rocbcal13, rocbcal14, rocbecal15 and rocbcal16), and do:

  • telnet localhost 25061 (connect to procServ)
  • ctrl + X (reboot procServ)
  • ctrl + ]
  • quit (exit the telnet connection)

Roc References:

  • rocbcal13: Modules North downstream
  • rocbcal14: Modules North upstream
  • rocbcal15: Modules South upstream
  • rocbcal16: Modules South downstream

Note: "Readout 2" in the BCAL-Scaler GUI is not implemented!

External Cosmic-Ray Trigger

The external cosmic-ray trigger was setup using some scintillation counters located above and below the magnet yoke to signal the passing of cosmic rays through the central detectors. Log Entry 3275972 and Log Entry 3285969 contain detailed information about the setup. To control the high voltage using the CAEN power supply use the following procedure starting from a terminal on the local network:

  • telnet u1-9-top 1527
  • login: user/user
  • Use 'tab' to move from one column menu heading to another.
  • Use keypad 'up' and 'down' arrows to select submenus under the header. Select Main/Channels to view voltages.
  • Use 'tab' to move back to Main menu, 'down' key to 'Log Out' and exit

Expert personnel

The individuals responsible for checking that the BCAL is ready to take data and setting its operating parameters are shown in following table. Problems with normal operation of the BCAL should be referred to those individuals and any changes to their settings must be approved by them. Additional experts may be trained by the system owner and their name and date added to this table.

Table: Expert personnel for the BCAL system
Name Extension Date of qualification
Elton Smith 269-7625 May 20, 2014
Mark Dalton 269-6931  ?
Retrieved from "https://halldweb.jlab.org/hdops/wiki/index.php?title=Barrel_Calorimeter_Expert&oldid=4538"