Difference between revisions of "Forward Calorimeter Shift"

From Hall D Ops Wiki
Jump to: navigation, search
(FCAL Base lockup)
 
(107 intermediate revisions by 5 users not shown)
Line 1: Line 1:
== The Forward Calorimeter ==
+
= Forward Calorimeter Summary =
The Forward Calorimeter (FCAL) is a 2800 element lead glass detector consisting of 4cm x 4cm x 45cm bars.  The light produced in each element is measured using FEU 84-3 PMTs. The PMTs are readout using custom Cockcroft-Walton bases and operate up to 1800 V.  The entire detector exists inside an environmentally controlled and light-tight darkroom on the FCAL platform.
+
  
== Rates Monitoring==
+
[[Image:FCAL_construction.JPG | thumb |left| 300px | Fig. 1. FCAL during construction, view looking downstream]]
[[Image:FCAL_Crates_2D.png | thumb | 400px | Fig. 1. Location of crates on the FCAL with crate number in the z]]
+
The rates will be monitored using the rcm.sh script. There is a tab labeled "FCAL r" which shows the rates for each crate. Figure 1 shows the correspondence with crate and FCAL location where the colors represent the rocfcal number. There is also a tab labeled "FCAL r12" which gives a more detailed view of 12 specific channels symmetric about the beam pipe. The tab shows 3 rows and 4 columns as detailed below:
+
* 1st row
+
** 1st col: (2 , 0)  RocFCAL: 5  Slot: 3  Channel: 13
+
** 2nd col: (0 , 2)  RocFCAL: 2  Slot: 3  Channel: 4
+
** 3rd col: (-2, 0)  RocFCAL: 11  Slot: 3  Channel: 2
+
** 4th col: (0 ,-2)  RocFCAL: 8  Slot: 3  Channel: 11
+
  
* 2nd row
+
The hall D Forward Calorimeter (FCAL) is an electromagnetic calorimeter designed to detect neutrals in the forward direction from approximately 2-11 degrees in lab theta, measuring from the center of the target.
** 1st col: (16 , 0)   RocFCAL: 5  Slot: 9  Channel: 15
+
** 2nd col: (0  , 16)  RocFCAL: 2 Slot: 9  Channel: 12
+
** 3rd col: (-16, 0)    RocFCAL: 11 Slot: 9  Channel: 0
+
** 4th col: (0  ,-16)  RocFCAL: 8  Slot: 9  Channel: 3
+
  
* 3rd row
+
The Forward Calorimeter is a 2800 element lead glass detector consisting of 4cm x 4cm x 45cm bars. The light produced in each element is measured using FEU 84-3 PMTs. The PMTs are powered using custom Cockcroft-Walton bases and operate up to 1800 V. The entire detector exists inside an environmentally controlled and light-tight "darkroom" on the FCAL platform. Individual PMTs respond differently to a fixed voltage, so each channel has an independent voltage setting determined from a gain balancing procedure.
** 1st col: (29 , 0)    RocFCAL: 6 Slot: Channel: 2
+
** 2nd col: (0  , 29)  RocFCAL: 4  Slot: 3  Channel: 2
+
** 3rd col: (-29, 0)    RocFCAL: 7  Slot: 3  Channel: 2
+
** 4th col: (0  ,-29)  RocFCAL: 9  Slot: 3  Channel: 2
+
  
===Turning On The FCAL In Stages===
+
The FCAL expert page can be found [https://halldweb.jlab.org/hdops/wiki/index.php/Forward_Calorimeter_Expert here]. It contain more detailed information if you want a more detailed description of FCAL elements and operation.
[[Image:OuterRing.png | thumb | left | 200px | Fig. 1. Location of crates on the FCAL with crate number in the z]]
+
[[Image:MiddleRing.png | thumb | 200px | Fig. 1. Location of crates on the FCAL with crate number in the z]]
+
[[Image:InnerRing.png | thumb | right | 200px | Fig. 1. Location of crates on the FCAL with crate number in the z]]
+
  
To avoid damaging PMTs from high rates near the center of the beam pipe the FCAL will be turned on in 3 stages. The steps are as follows:
 
  
1. Ensure the initial HV setpoint for all channels is 0 by clicking on the "SAVE/RESTORE" button in the FCAL GUI and selecting "Restore FCAL HV Setpoints" and selecting "SetAllTo0Volts.snap" file
+
=Interlocks=
  
2. Ensure the HV is enabled by clicking on the "ALL FCAL HV" button and selecting "Turn On All Bases"
+
[[Image:FCAL_darkroom.PNG | thumb | right| 400px | Fig. 2. FCAL Darkroom GUI page]]
  
3. To turn on the Outer portion of the FCAL select the "Restore FCAL HV Setpoints" option in "SAVE/RESTORE" and select the snap file "OuterToEdgeVoltages.snap"
+
A set of "interlocks" monitor the status inside the darkroom and are set to shut off power to bases and PMTs should these exceed set limits. This is mostly done to protect PMTs from potential damage from too much ambient light. There are sensors for the darkroom double doors and four sensor boxes in the darkroom. Each sensor box monitors temperature, humidity, and light levels.  
  
4. To turn on the Middle portion of the FCAL select the "Restore FCAL HV Setpoints" option in "SAVE/RESTORE" and select the snap file "MiddleToEdgeVoltages.snap"
+
Power supplies for the bases are interlocked and will be turned off when any of the following occurs:
 +
# both doors to the dark room are open simultaneously,
 +
# environmental sensors (light, temperature or humidity) in the dark room exceed preset limits or
 +
# there is a loss of power.  
  
5. To turn on the Inner portion of the FCAL select the "Restore FCAL HV Setpoints" option in "SAVE/RESTORE" and select the snap file "InnerToEdgeVoltages.snap"
+
Should the interlock system trip, an expert should be called to reset the condition. Changes to the interlock settings are only allowed by an expert, or under his/her guidance.
  
 +
= Routine operation=
  
 +
==FCAL Occupancies==
 +
FCAL occupancies will likely have a small number of features to them. Isolated channels may have no/low occupancy (or more rarely very high occupancy). This may vary run-to-run a little bit. This type of behavior doesn't require notifying FCAL experts.
  
== Routine operation==
+
Any behavior other than the above should be reported to FCAL experts. A typical occupancy plot from Spring '17 is shown here:
 +
[[Image:Fcal_digOcc2D.png|thumb|center|500px]]
  
A shift taker will need to monitor the environmental and base sensors through a GUI interface. The darkroom is equipped with 4 sensor boxes, each reads the temperature, humidity, and light conditions inside the darkroom. Each of the 2800 bases are equipped with temperature, high voltage, and current sensors. The following instructions are for non-experts during routine operation.<font color=red> Operations beyond what is specified here shall only be performed by the system expert or under his/her direction. </font> If the expert directs an operation outside the ones described here, a specific note must be entered into the log book.
+
==FCAL EPICS GUI Screens==
 +
* '''FCAL Voltages''': This screen gives information on FCAL bases providing power to PMTs. This gives an overview of the FCAL HV status, can be used to power channels on/off, and to navigate to individual channels. <font color=red>Red  </font>  channels have been turned off by FCAL experts. <font color=yellow>Yellow</font> channels indicate either voltage or communication issues. Problematic channels will most likely line up with holes in occupancy.
 +
*'''FCAL Darkroom''': This gives an overview of sensors monitoring the environment inside the FCAL darkroom. These should always be green unless someone is inside the darkroom.
 +
*'''FCAL LED pulser''': The long and short term behavior of the FCAL is monitored by an LED pulser and controlled here. This should always be running at 10 Hz and cycles between three different colors hourly. If no LED pulser appears to be going, contact an FCAL expert. Do not change anything here unless you know what you are doing!
 +
*'''FCAL Scalars''': This gives fADC scalar readouts from the FCAL for individual channels.
  
Control and monitoring of the system is accomplished using the Hall D graphical user interfaces (GUI) to EPICS [http://www.epics.org] [http://www.aps.anl.gov/epics]. A description of how to
+
==FCAL Alarms==
bring up the GUI screens can be found in [[Slow Controls Shift| Section Slow Controls]].  We briefly describe the use of these interfaces:
+
  
===Checking the environment===
+
===Dark room===
[[Image:FCAL_Darkroom.png | thumb | 400px | Fig. 1. MyaViewer screen for monitoring the FCAL Darkroom Temperature, Humidity, and Light]]
+
Environmental sensors record temperatures and humidity conditions inside the dark room and in the photomultiplier tube bases. Automatic alarm systems will be in place in the event an environmental or base sensor reads a value above what is considered safe or normal operating conditions. In the case an alarm system is triggered problem areas of the FCAL will flash red and power to the bases will shutoff. A non-expert user will call an expert user and do nothing else.
+
The FCAL Darkroom environmental variables can be viewed using MyaViewer and selecting the group 'HD_FCAL_ROOM'
+
  
===Voltage control===
+
Potential FCAL alarms could come from environmental monitors inside the FCAL darkroom. However, it is not expected that we should ever reach any alarm values. The FCAL Darkroom environmental variables can be viewed using MyaViewer and selecting the group 'HD_FCAL_ROOM'
[[Image:FCAL_User_Screen.png | thumb | 400px | Fig. 2. User screen for the FCAL. This is the primary screen used for a non-expert shift taker. With this GUI a shift taker can save/restore HV setpoints for the 2800 bases, turn off/on the bases, or select 25 bases to display in a table (by selecting a quadrant then the strand on that quadrant, and finally the quarter to display)]]
+
The bases generate their own high voltage which can be set through the GUI interface.
+
A non-expert user will only be able to do the following from the non-expert shift taker GUI shown in Figure 2:
+
* Restore HV setpoints
+
** Click 'SAVE/RESTORE' and select 'Restore FCAL HV Setpoints'
+
  
Below is a brief explanation of the other functionality capable on the non-expert GUI, but a shift taker must be directed by an expert to use any of the following:
+
===High Voltage===
 +
 
 +
* Problems with the FCAL voltage on a base will be indicated by a yellow box around a channel on the the FCAL voltage GUI.
 +
* These are diagnosed by clicking on the channel to bring up the '''Single Channel''' screen.
 +
*# If the ''Voltage Setpoint'' and the ''Voltage Setpoint Readback'' are similar but the ''Measured Voltage'' is near to 0, this can be corrected as follows:
 +
*## On the channel with issues begin lowering the voltage by clicking on the ''Voltage Setpoint'' arrows.  After about 20 V of change the ''Measured Voltage'' should jump to the correct value.
 +
*## Do this for all channels with issues.
 +
*## Restore the voltages using the "SAVE/RESTORE" button on the voltage GUI to fix your changes.
 +
 
 +
Below is a brief explanation of the other functionality capable on the non-expert GUI, <font color=red>but a shift taker must be directed by an expert to use any of the following</font>:
 
* Enable HV on all bases
 
* Enable HV on all bases
 
** Click 'All FCAL HV' and select 'Turn On All Bases'
 
** Click 'All FCAL HV' and select 'Turn On All Bases'
 
* Disable the HV on all bases
 
* Disable the HV on all bases
** Can be done in 2 ways: 1) At the top of the screen there is a red outline area with 'Turn OFF ALL' written, click the 'HV' button to power off all bases, 2) click 'All FCAL HV' and select 'Turn Off All Bases'
+
** Can be done in 2 ways:
 +
**# At the top of the screen there is a red outline area with 'Turn OFF ALL' written, click the 'HV' button to power off all bases,  
 +
**# click 'All FCAL HV' and select 'Turn Off All Bases'
 
* Save HV setpoints
 
* Save HV setpoints
 
** Click 'SAVE/RESTORE' and select 'Save FCAL HV Setpoints'
 
** Click 'SAVE/RESTORE' and select 'Save FCAL HV Setpoints'
* Select 25 bases to display in a table
 
** This is done by first selecting a quadrant, then selecting a particular strand in that quadrant, and finally choosing which quarter of that strand to display
 
  
===Alarms===
+
==FCAL Base lockup==
The FCAL will alarm when a base reads back a current greater than 14.9 mA for 30 consecutive minutes. In the case an alarm is tripped a shift taker can disable the HV to that bases from the alarm handler.
+
  
===Correlating strands with FCAL coordinates===
+
Bases might lose communication and stop providing high voltage.  This is identified when the '''channel GUI (click on the channel in the voltage GUI) shows the alarm state as "INVALID, LINK ALARM"''' when the mouse is hovered over the yellow Chanel Status indicator.  If this instead shows "OK, OK" then the base is not locked up.
  
===FCAL features a shift taker should be aware of===
+
In addition
It is known that some bases are either not setting their HV properly or have other unwanted behavior so there will be holes in FCAL occupancy plots. Below is an explanation of some of the reasons a hole may appear:
+
# the voltage GUI has a yellow border for the channel indicating alarm
 +
# The scalers show zero counts for the channel (the RootSpy occupancy may also show a hole.)
 +
This is corrected by the following sequence:
 +
# End the run.  We usually don't end a run early for this issue.
 +
# Turn off the high voltage to the whole FCAL.
 +
# Run the script. "/home/hdops/FCAL/tools/FCAL_basePowerReset.py"
 +
# Wait for 30 seconds.
 +
# Turn on the FCAL high voltage.
 +
# Start a new run (as appropriate).
 +
# '''MAKE A LOG ENTRY with the number of the locked up base.'''
 +
<gallery mode="packed" widths=300px heights=350px>
 +
File:base_lockup_V.png|Figure: FCAL voltage showing 2 alarm channels
 +
File:base_lockup_scalers.png|Figure: FCAL scalers showing 2 missing channels
 +
File:base_lockup_channel.png|Figure: FCAL channel showing a communication failure alarm
 +
File:base_lockup_RootSpy.png|Figure: FCAL occupancy in RootSpy showing 2 missing channels
 +
</gallery>
  
1. It is known that the FCAL EPICS IOC crashes about every 4 hours. The IOC is setup to automatically reboot but when it does the HV setpoints will all be set to 0. Do not try to fix the issue or call an expert, this is a know problem but the setpoint seen in EPICS is not a real reading of the base. The base is essentially unaware of the IOC reboot and will continue enabling its HV with no interruption.
+
=Expert personnel =
 
+
2. After an IOC reboots a few bases will disable it HV, ~ 4 bases per reboot. It is unknown why this happens but an expert will re-enable the HV for those bases every morning.
+
 
+
3. Recently a few bases have been observed to read back a HV around 2200 Volts. It was verified that this is a real read back and the reason is still unknown. If a base reads over 2200 Volts that base will automatically have its HV disabled. The current solution is to recycle the power to that strand, this is an expert operation and only experts can will resolve these issue.
+
 
+
== Interlocks ==
+
Power supplies for the bases are interlocked and will be turned off when  a) both doors to the dark room are open simultaneously, b) environmental sensors (light, temperature or humidity) in the dark room exceed preset limits or c) there is a loss of power. Should the interlock system trip, an expert should be called to reset the condition. Changes to the interlock settings are only allowed by an expert, or under his/her guidance.
+
 
+
== Expert personnel ==
+
 
The individuals responsible for checking that the FCAL is ready to take data  and setting its operating parameters  are shown in following table.
 
The individuals responsible for checking that the FCAL is ready to take data  and setting its operating parameters  are shown in following table.
 
Problems with normal operation of the FCAL should be referred to those individuals and any changes to their settings must be
 
Problems with normal operation of the FCAL should be referred to those individuals and any changes to their settings must be
Line 98: Line 94:
 
! width=200px | Name    !!  width=100px | Number !! Date of qualification  
 
! width=200px | Name    !!  width=100px | Number !! Date of qualification  
 
|-
 
|-
| Manuel Lara     ||  align=center | (956) 622-8216  ||  align=center |  
+
| Calorimeter Expert || align=center | 354-9399 || align=center|
 +
|-
 +
| Mark Dalton     ||  align=center | Cell:757-849-2929 ||  align=center |
 +
|-
 +
| Malte Albrecht  ||  align=center | Cell:757-316-4936 ||  align=center |  
 
  |}
 
  |}

Latest revision as of 09:03, 23 January 2023

Forward Calorimeter Summary

Fig. 1. FCAL during construction, view looking downstream

The hall D Forward Calorimeter (FCAL) is an electromagnetic calorimeter designed to detect neutrals in the forward direction from approximately 2-11 degrees in lab theta, measuring from the center of the target.

The Forward Calorimeter is a 2800 element lead glass detector consisting of 4cm x 4cm x 45cm bars. The light produced in each element is measured using FEU 84-3 PMTs. The PMTs are powered using custom Cockcroft-Walton bases and operate up to 1800 V. The entire detector exists inside an environmentally controlled and light-tight "darkroom" on the FCAL platform. Individual PMTs respond differently to a fixed voltage, so each channel has an independent voltage setting determined from a gain balancing procedure.

The FCAL expert page can be found here. It contain more detailed information if you want a more detailed description of FCAL elements and operation.


Interlocks

Fig. 2. FCAL Darkroom GUI page

A set of "interlocks" monitor the status inside the darkroom and are set to shut off power to bases and PMTs should these exceed set limits. This is mostly done to protect PMTs from potential damage from too much ambient light. There are sensors for the darkroom double doors and four sensor boxes in the darkroom. Each sensor box monitors temperature, humidity, and light levels.

Power supplies for the bases are interlocked and will be turned off when any of the following occurs:

  1. both doors to the dark room are open simultaneously,
  2. environmental sensors (light, temperature or humidity) in the dark room exceed preset limits or
  3. there is a loss of power.

Should the interlock system trip, an expert should be called to reset the condition. Changes to the interlock settings are only allowed by an expert, or under his/her guidance.

Routine operation

FCAL Occupancies

FCAL occupancies will likely have a small number of features to them. Isolated channels may have no/low occupancy (or more rarely very high occupancy). This may vary run-to-run a little bit. This type of behavior doesn't require notifying FCAL experts.

Any behavior other than the above should be reported to FCAL experts. A typical occupancy plot from Spring '17 is shown here:

Fcal digOcc2D.png

FCAL EPICS GUI Screens

  • FCAL Voltages: This screen gives information on FCAL bases providing power to PMTs. This gives an overview of the FCAL HV status, can be used to power channels on/off, and to navigate to individual channels. Red channels have been turned off by FCAL experts. Yellow channels indicate either voltage or communication issues. Problematic channels will most likely line up with holes in occupancy.
  • FCAL Darkroom: This gives an overview of sensors monitoring the environment inside the FCAL darkroom. These should always be green unless someone is inside the darkroom.
  • FCAL LED pulser: The long and short term behavior of the FCAL is monitored by an LED pulser and controlled here. This should always be running at 10 Hz and cycles between three different colors hourly. If no LED pulser appears to be going, contact an FCAL expert. Do not change anything here unless you know what you are doing!
  • FCAL Scalars: This gives fADC scalar readouts from the FCAL for individual channels.

FCAL Alarms

Dark room

Potential FCAL alarms could come from environmental monitors inside the FCAL darkroom. However, it is not expected that we should ever reach any alarm values. The FCAL Darkroom environmental variables can be viewed using MyaViewer and selecting the group 'HD_FCAL_ROOM'

High Voltage

  • Problems with the FCAL voltage on a base will be indicated by a yellow box around a channel on the the FCAL voltage GUI.
  • These are diagnosed by clicking on the channel to bring up the Single Channel screen.
    1. If the Voltage Setpoint and the Voltage Setpoint Readback are similar but the Measured Voltage is near to 0, this can be corrected as follows:
      1. On the channel with issues begin lowering the voltage by clicking on the Voltage Setpoint arrows. After about 20 V of change the Measured Voltage should jump to the correct value.
      2. Do this for all channels with issues.
      3. Restore the voltages using the "SAVE/RESTORE" button on the voltage GUI to fix your changes.

Below is a brief explanation of the other functionality capable on the non-expert GUI, but a shift taker must be directed by an expert to use any of the following:

  • Enable HV on all bases
    • Click 'All FCAL HV' and select 'Turn On All Bases'
  • Disable the HV on all bases
    • Can be done in 2 ways:
      1. At the top of the screen there is a red outline area with 'Turn OFF ALL' written, click the 'HV' button to power off all bases,
      2. click 'All FCAL HV' and select 'Turn Off All Bases'
  • Save HV setpoints
    • Click 'SAVE/RESTORE' and select 'Save FCAL HV Setpoints'

FCAL Base lockup

Bases might lose communication and stop providing high voltage. This is identified when the channel GUI (click on the channel in the voltage GUI) shows the alarm state as "INVALID, LINK ALARM" when the mouse is hovered over the yellow Chanel Status indicator. If this instead shows "OK, OK" then the base is not locked up.

In addition

  1. the voltage GUI has a yellow border for the channel indicating alarm
  2. The scalers show zero counts for the channel (the RootSpy occupancy may also show a hole.)

This is corrected by the following sequence:

  1. End the run. We usually don't end a run early for this issue.
  2. Turn off the high voltage to the whole FCAL.
  3. Run the script. "/home/hdops/FCAL/tools/FCAL_basePowerReset.py"
  4. Wait for 30 seconds.
  5. Turn on the FCAL high voltage.
  6. Start a new run (as appropriate).
  7. MAKE A LOG ENTRY with the number of the locked up base.

Expert personnel

The individuals responsible for checking that the FCAL is ready to take data and setting its operating parameters are shown in following table. Problems with normal operation of the FCAL should be referred to those individuals and any changes to their settings must be approved by them. Additional experts may be trained by the system owner and their name and date added to this table.

Table: Expert personnel for the FCAL system
Name Number Date of qualification
Calorimeter Expert 354-9399
Mark Dalton Cell:757-849-2929
Malte Albrecht Cell:757-316-4936