Difference between revisions of "Forward Calorimeter Shift"

From Hall D Ops Wiki
Jump to: navigation, search
(→‎FCAL Base lockup)
 
(81 intermediate revisions by 5 users not shown)
Line 1: Line 1:
−
== The Forward Calorimeter ==
+
= Forward Calorimeter Summary =
−
The Forward Calorimeter (FCAL) is a 2800 element lead glass detector consisting of 4cm x 4cm x 45cm bars.  The light produced in each element is measured using FEU 84-3 PMTs. The PMTs are readout using custom Cockcroft-Walton bases and operate up to 1800 V.  The entire detector exists inside an environmentally controlled and light-tight darkroom on the FCAL platform.
+
  
−
== Rates Monitoring==
+
[[Image:FCAL_construction.JPG | thumb |left| 300px | Fig. 1. FCAL during construction, view looking downstream]]
−
[[Image:FCAL_Crates_2D.png | thumb | 400px | Fig. 1. Upstream view of the FCAL with crates colored different colors]]
+
−
The rates will be monitored using the rcm.sh script. There is a tab labeled "FCAL g" which shows the rates for each crate. Figure 1 shows the correspondence with crate and FCAL location where the colors represent the rocfcal number. The bottom of "FCAL g" gives a more detailed view of 12 specific channels symmetric about the beam pipe.
+
  
−
The plot title is formatted like FCAL_X_Y where X and Y are in the coordinates of the hall and the origin is along the beam axis. The coordinates used are:
+
The hall D Forward Calorimeter (FCAL) is an electromagnetic calorimeter designed to detect neutrals in the forward direction from approximately 2-11 degrees in lab theta, measuring from the center of the target.
  
−
* Row 1:
+
The Forward Calorimeter is a 2800 element lead glass detector consisting of 4cm x 4cm x 45cm bars.  The light produced in each element is measured using FEU 84-3 PMTs. The PMTs are powered using custom Cockcroft-Walton bases and operate up to 1800 V.  The entire detector exists inside an environmentally controlled and light-tight "darkroom" on the FCAL platform. Individual PMTs respond differently to a fixed voltage, so each channel has an independent voltage setting determined from a gain balancing procedure.
−
** FCAL_2_0
+
−
** FCAL_0_2
+
−
** FCAL_-2_0
+
−
** FCAL_0_-2
+
  
−
* Row 2:
+
The FCAL expert page can be found [https://halldweb.jlab.org/hdops/wiki/index.php/Forward_Calorimeter_Expert here]. It contain more detailed information if you want a more detailed description of FCAL elements and operation.
−
** FCAL_16_0
+
−
** FCAL_0_16
+
−
** FCAL_-16_0
+
−
** FCAL_0_-16
+
  
−
* Row 3:
 
−
** FCAL_29_0
 
−
** FCAL_0_29
 
−
** FCAL_-29_0
 
−
** FCAL_0_-29
 
  
−
===Turning On The FCAL In Stages===
+
=Interlocks=
−
[[Image:OuterRing.png | thumb | left | 200px | Fig. 2. In red are the bases with a non-zero HV setpoint]]
+
−
[[Image:MiddleRing.png | thumb | 200px | Fig. 3. In red are the bases with a non-zero HV setpoint]]
+
−
[[Image:InnerRing.png | thumb | right | 200px | Fig. 4. In red are the bases with a non-zero HV setpoint]]
+
  
−
To avoid damaging PMTs from high rates near the center of the beam pipe the FCAL will be turned on in 3 stages. The steps are as follows:
+
[[Image:FCAL_darkroom.PNG | thumb | right| 400px | Fig. 2. FCAL Darkroom GUI page]]
  
−
1. Ensure the initial HV setpoint for all channels is 0 by clicking on the "SAVE/RESTORE" button in the FCAL GUI and selecting "Restore FCAL HV Setpoints" and selecting "SetAllTo0Volts.snap" file
+
A set of "interlocks" monitor the status inside the darkroom and are set to shut off power to bases and PMTs should these exceed set limits. This is mostly done to protect PMTs from potential damage from too much ambient light. There are sensors for the darkroom double doors and four sensor boxes in the darkroom. Each sensor box monitors temperature, humidity, and light levels.  
  
−
2. Ensure the HV is enabled by clicking on the "ALL FCAL HV" button and selecting "Turn On All Bases". <font color=red> This is an expert only operation</font>
+
Power supplies for the bases are interlocked and will be turned off when any of the following occurs:
 +
# both doors to the dark room are open simultaneously,
 +
# environmental sensors (light, temperature or humidity) in the dark room exceed preset limits or
 +
# there is a loss of power.
  
−
3. To turn on the Outer portion of the FCAL select the "Restore FCAL HV Setpoints" option in "SAVE/RESTORE" and select the snap file "OuterToEdgeVoltages.snap". Figure 2 shows the results of selecting tis snap file, the blue blocks have a 0 HV setpoint and the red block have a non-zero HV.
+
Should the interlock system trip, an expert should be called to reset the condition. Changes to the interlock settings are only allowed by an expert, or under his/her guidance.
  
−
4. To turn on the Middle portion of the FCAL select the "Restore FCAL HV Setpoints" option in "SAVE/RESTORE" and select the snap file "MiddleToEdgeVoltages.snap". Figure 3 shows the results of selecting tis snap file, the blue blocks have a 0 HV setpoint and the red block have a non-zero HV.
+
= Routine operation=
  
−
5. To turn on the Inner portion of the FCAL select the "Restore FCAL HV Setpoints" option in "SAVE/RESTORE" and select the snap file "InnerToEdgeVoltages.snap". Figure 4 shows the results of selecting tis snap file, the blue blocks have a 0 HV setpoint and the red block have a non-zero HV.
+
==FCAL Occupancies==
 +
FCAL occupancies will likely have a small number of features to them. Isolated channels may have no/low occupancy (or more rarely very high occupancy). This may vary run-to-run a little bit. This type of behavior doesn't require notifying FCAL experts.
  
 +
Any behavior other than the above should be reported to FCAL experts. A typical occupancy plot from Spring '17 is shown here:
 +
[[Image:Fcal_digOcc2D.png|thumb|center|500px]]
  
 +
==FCAL EPICS GUI Screens==
 +
* '''FCAL Voltages''': This screen gives information on FCAL bases providing power to PMTs. This gives an overview of the FCAL HV status, can be used to power channels on/off, and to navigate to individual channels. <font color=red>Red  </font>  channels have been turned off by FCAL experts. <font color=yellow>Yellow</font> channels indicate either voltage or communication issues. Problematic channels will most likely line up with holes in occupancy.
 +
*'''FCAL Darkroom''': This gives an overview of sensors monitoring the environment inside the FCAL darkroom. These should always be green unless someone is inside the darkroom.
 +
*'''FCAL LED pulser''': The long and short term behavior of the FCAL is monitored by an LED pulser and controlled here. This should always be running at 10 Hz and cycles between three different colors hourly. If no LED pulser appears to be going, contact an FCAL expert. Do not change anything here unless you know what you are doing!
 +
*'''FCAL Scalars''': This gives fADC scalar readouts from the FCAL for individual channels.
  
−
== Routine operation==
+
==FCAL Alarms==
  
−
A shift taker will need to monitor the environmental and base sensors through a GUI interface. The darkroom is equipped with 4 sensor boxes, each reads the temperature, humidity, and light conditions inside the darkroom. Each of the 2800 bases are equipped with temperature, high voltage, and current sensors. The following instructions are for non-experts during routine operation.<font color=red> Operations beyond what is specified here shall only be performed by the system expert or under his/her direction. </font> If the expert directs an operation outside the  ones described here, a specific note must be entered into the log book.
+
===Dark room===
  
−
Control and monitoring of the system is accomplished using the Hall D graphical user interfaces (GUI) to EPICS [http://www.epics.org] [http://www.aps.anl.gov/epics]. A description of how to
+
Potential FCAL alarms could come from environmental monitors inside the FCAL darkroom.  However, it is not expected that we should ever reach any alarm values. The FCAL Darkroom environmental variables can be viewed using MyaViewer and selecting the group 'HD_FCAL_ROOM'
−
bring up the GUI screens can be found in [[Slow Controls Shift| Section Slow Controls]].  We briefly describe the use of these interfaces:
+
  
−
===Checking the Darkroom environment===
+
===High Voltage===
−
[[Image:FCAL_Darkroom.png | thumb | 400px | Fig. 5. MyaViewer screen for monitoring the FCAL Darkroom Temperature, Humidity, and Light]]
+
−
Environmental sensors record temperatures and humidity conditions inside the dark room. Automatic alarm systems will be in place in the event an environmental sensor reads a value above what is considered safe or normal operating conditions. In the case an environmental alarm is triggered all power to the bases will be turned off. A non-expert user will call an expert user and do nothing else.
+
  
−
The FCAL Darkroom environmental variables can be viewed using MyaViewer and selecting the group 'HD_FCAL_ROOM'
+
* Problems with the FCAL voltage on a base will be indicated by a yellow box around a channel on the the FCAL voltage GUI.
−
 
+
* These are diagnosed by clicking on the channel to bring up the '''Single Channel''' screen.
−
===HV Alarms===
+
*# If the ''Voltage Setpoint'' and the ''Voltage Setpoint Readback'' are similar but the ''Measured Voltage'' is near to 0, this can be corrected as follows:
−
The FCAL will alarm when a base reads back a current greater than 14.9 mA for 30 consecutive minutes. In the case an alarm is tripped a shift taker can disable the HV to that bases from the alarm handler.
+
*## On the channel with issues begin lowering the voltage by clicking on the ''Voltage Setpoint'' arrows.  After about 20 V of change the ''Measured Voltage'' should jump to the correct value.
−
 
+
*## Do this for all channels with issues.
−
 
+
*## Restore the voltages using the "SAVE/RESTORE" button on the voltage GUI to fix your changes.
−
===Voltage control===
+
−
[[Image:FCAL_User_Screen.png | thumb | 400px | Fig. 6. User screen for the FCAL. This is the primary screen used for a non-expert shift taker. With this GUI a shift taker can save/restore HV setpoints for the 2800 bases, turn off/on the bases, or select 25 bases to display in a table (by selecting a quadrant then the strand on that quadrant, and finally the quarter to display)]]
+
−
The bases generate their own high voltage which can be set through the GUI interface.
+
−
A non-expert user will only be able to do the following from the non-expert shift taker GUI shown in Figure 2:
+
−
* Restore HV setpoints
+
−
** Click 'SAVE/RESTORE' and select 'Restore FCAL HV Setpoints'
+
−
* Select 25 bases to display in a table
+
−
** This is done by first selecting a quadrant, then selecting a particular strand in that quadrant, and finally choosing which quarter of that strand to display
+
−
** For convenience you can type the column and row number then use the right area of the FCAL GUI. Then press "Submit" and the chain starting with quadrant then strand and position in that strand is output.
+
  
 
Below is a brief explanation of the other functionality capable on the non-expert GUI, <font color=red>but a shift taker must be directed by an expert to use any of the following</font>:
 
Below is a brief explanation of the other functionality capable on the non-expert GUI, <font color=red>but a shift taker must be directed by an expert to use any of the following</font>:
Line 76: Line 56:
 
** Click 'All FCAL HV' and select 'Turn On All Bases'
 
** Click 'All FCAL HV' and select 'Turn On All Bases'
 
* Disable the HV on all bases
 
* Disable the HV on all bases
−
** Can be done in 2 ways: 1) At the top of the screen there is a red outline area with 'Turn OFF ALL' written, click the 'HV' button to power off all bases, 2) click 'All FCAL HV' and select 'Turn Off All Bases'
+
** Can be done in 2 ways:
 +
**# At the top of the screen there is a red outline area with 'Turn OFF ALL' written, click the 'HV' button to power off all bases,  
 +
**# click 'All FCAL HV' and select 'Turn Off All Bases'
 
* Save HV setpoints
 
* Save HV setpoints
 
** Click 'SAVE/RESTORE' and select 'Save FCAL HV Setpoints'
 
** Click 'SAVE/RESTORE' and select 'Save FCAL HV Setpoints'
  
−
===Correlating strands with FCAL coordinates===
+
==FCAL Base lockup==
−
Below is slow control map where each color represents a different strand. Each strand is daisy chained to 100 bases. This view is looking upstream at the FCAL and the strands are numbered from 1 to 28. Strand 1 to 14 is only the left half of the FCAL and starts in the upper left corner and ends at the lower left corner. Strand 15 to 28 starts in the upper right corner and ends at the lower right corner.
+
  
−
[[Image:FCAL_Slow_Control_Map_Strands.png ]]
+
Bases might lose communication and stop providing high voltage.  This is identified when the '''channel GUI (click on the channel in the voltage GUI) shows the alarm state as "INVALID, LINK ALARM"''' when the mouse is hovered over the yellow Chanel Status indicator.  If this instead shows "OK, OK" then the base is not locked up.
  
−
===FCAL features a shift taker should be aware of===
+
In addition
−
It is known that some bases are either not setting their HV properly or have other unwanted behavior so there will be holes in FCAL occupancy plots.  
+
# the voltage GUI has a yellow border for the channel indicating alarm
−
Below is an explanation of some of the reasons a hole may appear:
+
# The scalers show zero counts for the channel (the RootSpy occupancy may also show a hole.)
 +
This is corrected by the following sequence:
 +
# End the run.  We usually don't end a run early for this issue.
 +
# Turn off the high voltage to the whole FCAL.
 +
# Run the script. "/home/hdops/FCAL/tools/FCAL_basePowerReset.py"
 +
# Wait for 30 seconds.
 +
# Turn on the FCAL high voltage.
 +
# Start a new run (as appropriate).
 +
# '''MAKE A LOG ENTRY with the number of the locked up base.'''
 +
<gallery mode="packed" widths=300px heights=350px>
 +
File:base_lockup_V.png|Figure: FCAL voltage showing 2 alarm channels
 +
File:base_lockup_scalers.png|Figure: FCAL scalers showing 2 missing channels
 +
File:base_lockup_channel.png|Figure: FCAL channel showing a communication failure alarm
 +
File:base_lockup_RootSpy.png|Figure: FCAL occupancy in RootSpy showing 2 missing channels
 +
</gallery>
  
−
1. After an IOC reboots a few bases will disable it HV, ~ 4 bases per reboot. It is unknown why this happens but an expert will re-enable the HV for those bases every morning.
+
=Expert personnel =
−
 
+
−
2. Recently a few bases have been observed to read back a HV around 2200 Volts. It was verified that this is a real read back and the reason is still unknown. If a base reads over 2200 Volts that base will automatically have its HV disabled. The current solution is to recycle the power to that strand, this is an expert operation and only experts can will resolve these issue.
+
−
 
+
−
It's good to also be aware of the voltage setpoints in the EPICS GUI. It is known that the FCAL EPICS IOC crashes about every 4 hours. The IOC is setup to automatically reboot but when it does the HV setpoints will all be set to 0. Do not try to fix the issue or call an expert, this is a know problem but the setpoint seen in EPICS is not a real reading of the base. The base is essentially unaware of the IOC reboot and will continue enabling its HV with no interruption.
+
−
== Interlocks ==
+
−
Power supplies for the bases are interlocked and will be turned off when  a) both doors to the dark room are open simultaneously, b) environmental sensors (light, temperature or humidity) in the dark room exceed preset limits or c) there is a loss of power. Should the interlock system trip, an expert should be called to reset the condition. Changes to the interlock settings are only allowed by an expert, or under his/her guidance.
+
−
 
+
−
== Expert personnel ==
+
 
The individuals responsible for checking that the FCAL is ready to take data  and setting its operating parameters  are shown in following table.
 
The individuals responsible for checking that the FCAL is ready to take data  and setting its operating parameters  are shown in following table.
 
Problems with normal operation of the FCAL should be referred to those individuals and any changes to their settings must be
 
Problems with normal operation of the FCAL should be referred to those individuals and any changes to their settings must be
Line 107: Line 94:
 
! width=200px | Name    !!  width=100px | Number !! Date of qualification  
 
! width=200px | Name    !!  width=100px | Number !! Date of qualification  
 
|-
 
|-
−
| Manuel Lara     ||  align=center | (956) 622-8216  ||  align=center |  
+
| Calorimeter Expert || align=center | 354-9399 || align=center|
 +
|-
 +
| Mark Dalton     ||  align=center | Cell:757-849-2929 ||  align=center |  
 
|-
 
|-
−
| Adesh    ||  align=center |   |(757) 816-3649|  align=center |  
+
| Malte Albrecht  ||  align=center | Cell:757-316-4936 ||  align=center |  
 
  |}
 
  |}

Latest revision as of 09:03, 23 January 2023

Forward Calorimeter Summary

Fig. 1. FCAL during construction, view looking downstream

The hall D Forward Calorimeter (FCAL) is an electromagnetic calorimeter designed to detect neutrals in the forward direction from approximately 2-11 degrees in lab theta, measuring from the center of the target.

The Forward Calorimeter is a 2800 element lead glass detector consisting of 4cm x 4cm x 45cm bars. The light produced in each element is measured using FEU 84-3 PMTs. The PMTs are powered using custom Cockcroft-Walton bases and operate up to 1800 V. The entire detector exists inside an environmentally controlled and light-tight "darkroom" on the FCAL platform. Individual PMTs respond differently to a fixed voltage, so each channel has an independent voltage setting determined from a gain balancing procedure.

The FCAL expert page can be found here. It contain more detailed information if you want a more detailed description of FCAL elements and operation.


Interlocks

Fig. 2. FCAL Darkroom GUI page

A set of "interlocks" monitor the status inside the darkroom and are set to shut off power to bases and PMTs should these exceed set limits. This is mostly done to protect PMTs from potential damage from too much ambient light. There are sensors for the darkroom double doors and four sensor boxes in the darkroom. Each sensor box monitors temperature, humidity, and light levels.

Power supplies for the bases are interlocked and will be turned off when any of the following occurs:

  1. both doors to the dark room are open simultaneously,
  2. environmental sensors (light, temperature or humidity) in the dark room exceed preset limits or
  3. there is a loss of power.

Should the interlock system trip, an expert should be called to reset the condition. Changes to the interlock settings are only allowed by an expert, or under his/her guidance.

Routine operation

FCAL Occupancies

FCAL occupancies will likely have a small number of features to them. Isolated channels may have no/low occupancy (or more rarely very high occupancy). This may vary run-to-run a little bit. This type of behavior doesn't require notifying FCAL experts.

Any behavior other than the above should be reported to FCAL experts. A typical occupancy plot from Spring '17 is shown here:

Fcal digOcc2D.png

FCAL EPICS GUI Screens

  • FCAL Voltages: This screen gives information on FCAL bases providing power to PMTs. This gives an overview of the FCAL HV status, can be used to power channels on/off, and to navigate to individual channels. Red channels have been turned off by FCAL experts. Yellow channels indicate either voltage or communication issues. Problematic channels will most likely line up with holes in occupancy.
  • FCAL Darkroom: This gives an overview of sensors monitoring the environment inside the FCAL darkroom. These should always be green unless someone is inside the darkroom.
  • FCAL LED pulser: The long and short term behavior of the FCAL is monitored by an LED pulser and controlled here. This should always be running at 10 Hz and cycles between three different colors hourly. If no LED pulser appears to be going, contact an FCAL expert. Do not change anything here unless you know what you are doing!
  • FCAL Scalars: This gives fADC scalar readouts from the FCAL for individual channels.

FCAL Alarms

Dark room

Potential FCAL alarms could come from environmental monitors inside the FCAL darkroom. However, it is not expected that we should ever reach any alarm values. The FCAL Darkroom environmental variables can be viewed using MyaViewer and selecting the group 'HD_FCAL_ROOM'

High Voltage

  • Problems with the FCAL voltage on a base will be indicated by a yellow box around a channel on the the FCAL voltage GUI.
  • These are diagnosed by clicking on the channel to bring up the Single Channel screen.
    1. If the Voltage Setpoint and the Voltage Setpoint Readback are similar but the Measured Voltage is near to 0, this can be corrected as follows:
      1. On the channel with issues begin lowering the voltage by clicking on the Voltage Setpoint arrows. After about 20 V of change the Measured Voltage should jump to the correct value.
      2. Do this for all channels with issues.
      3. Restore the voltages using the "SAVE/RESTORE" button on the voltage GUI to fix your changes.

Below is a brief explanation of the other functionality capable on the non-expert GUI, but a shift taker must be directed by an expert to use any of the following:

  • Enable HV on all bases
    • Click 'All FCAL HV' and select 'Turn On All Bases'
  • Disable the HV on all bases
    • Can be done in 2 ways:
      1. At the top of the screen there is a red outline area with 'Turn OFF ALL' written, click the 'HV' button to power off all bases,
      2. click 'All FCAL HV' and select 'Turn Off All Bases'
  • Save HV setpoints
    • Click 'SAVE/RESTORE' and select 'Save FCAL HV Setpoints'

FCAL Base lockup

Bases might lose communication and stop providing high voltage. This is identified when the channel GUI (click on the channel in the voltage GUI) shows the alarm state as "INVALID, LINK ALARM" when the mouse is hovered over the yellow Chanel Status indicator. If this instead shows "OK, OK" then the base is not locked up.

In addition

  1. the voltage GUI has a yellow border for the channel indicating alarm
  2. The scalers show zero counts for the channel (the RootSpy occupancy may also show a hole.)

This is corrected by the following sequence:

  1. End the run. We usually don't end a run early for this issue.
  2. Turn off the high voltage to the whole FCAL.
  3. Run the script. "/home/hdops/FCAL/tools/FCAL_basePowerReset.py"
  4. Wait for 30 seconds.
  5. Turn on the FCAL high voltage.
  6. Start a new run (as appropriate).
  7. MAKE A LOG ENTRY with the number of the locked up base.

  • Figure: FCAL voltage showing 2 alarm channels

  • Figure: FCAL scalers showing 2 missing channels

  • Figure: FCAL channel showing a communication failure alarm

  • Figure: FCAL occupancy in RootSpy showing 2 missing channels

Expert personnel

The individuals responsible for checking that the FCAL is ready to take data and setting its operating parameters are shown in following table. Problems with normal operation of the FCAL should be referred to those individuals and any changes to their settings must be approved by them. Additional experts may be trained by the system owner and their name and date added to this table.

Table: Expert personnel for the FCAL system
Name Number Date of qualification
Calorimeter Expert 354-9399
Mark Dalton Cell:757-849-2929
Malte Albrecht Cell:757-316-4936
Retrieved from "https://halldweb.jlab.org/hdops/wiki/index.php?title=Forward_Calorimeter_Shift&oldid=4580"