Difference between revisions of "Forward Calorimeter Shift"

From Hall D Ops Wiki
Jump to: navigation, search
(→‎FCAL EPICS GUI Screens)
(→‎FCAL Base lockup procedure)
 
(49 intermediate revisions by 6 users not shown)
Line 1: Line 1:
−
== Forward Calorimeter Summary ==
+
= FCAL High Voltage channel with '''<span style="background: fuchsia;">purple border </span>''' =
  
−
The hall D Forward Calorimeter (FCAL) is an electromagnetic calorimeter designed to detect neutrals in the forward direction from approximately 2-11 degrees in lab theta, measuring from the center of the target.
+
* '''Communications''' problems with the FCAL voltage on a base will be indicated by a '''<span style="background: fuchsia;">purple border</span>'''  around a channel on the FCAL voltage GUI.
 +
** You might be able to fix the channel
 +
** Please look at FCAL voltage GUI every couple of hours or so.
  
−
The Forward Calorimeter is a 2800 element lead glass detector consisting of 4cm x 4cm x 45cm bars.  The light produced in each element is measured using FEU 84-3 PMTs. The PMTs are powered using custom Cockcroft-Walton bases and operate up to 1800 V.  The entire detector exists inside an environmentally controlled and light-tight "darkroom" on the FCAL platform. Individual PMTs respond differently to a fixed voltage, so each channel has an independent voltage setting determined from a gain balancing procedure.
+
The '''<span style="background: fuchsia;">purple border</span>''' might be caused by:
 +
# A '''base lockup'''
 +
#* Follow FCAL Base lockup procedure
 +
# A base that has a general '''communications issue'''
 +
#* Must be fixed by expert
  
−
In general, operations that can be performed from CSS GUI windows should probably only be done by FCAL experts.
+
==FCAL Base lockup procedure==
  
−
An older version of this page from the commissioning era can be found here [https://halldweb.jlab.org/hdops/wiki/index.php/Forward_Calorimeter_Shift_Comissioning_Era here].
+
Bases might lose communication and stop providing high voltage due to radiation.  
  
−
== FCAL Elements & Mappings==
+
Lockup diagnosis:
−
*The FCAL consists of the following elements:
+
# The scalers show zero counts for the channel (the RootSpy occupancy may also show a hole.)
−
**FCAL Darkroom:
+
# The voltage GUI has a '''<span style="background: fuchsia;">purple border</span>''' for the channel indicating alarm
−
*** 2,800 channels, each with a lead-glass block, PMT, base, and cable to read-out electronics
+
# ''' The alarm state is "INVALID, LINK ALARM" '''
−
*** 4 sets of light, temperature, and humidity monitors
+
## Click on the channel in the voltage screen to get the '''FCAL Table: Single Channel''' screen
−
** 12 Electronics Crates
+
## Hover over the yellow Chanel Status indicator
−
*** Up to 18 fADC slots per crate
+
## Mouse tooltip shows the alarm state as "INVALID, LINK ALARM"'''
−
**** Up to 16 channel inputs per slot, signals from FCAL
+
## If this instead shows "OK, OK" then the base is not locked up
−
** Base communication equipment:
+
This is corrected by the following sequence:
−
*** 7 Anagate Quattro devices that communicate with fcal bases
+
# End the run.  We usually don't end a run early for this issue.
−
**** 100 bases on "strand" to each anagate x 4 ports per anagate
+
# Count the number of channels with a yellow border.
−
*** 1 Anagate Uno to manage quattro devices
+
# '''MAKE A LOG ENTRY with the number of the locked up base.'''
−
*** 4 Power Chassis: provides power to all bases for two anagates (up to 800 bases)
+
# Turn off the high voltage to the whole FCAL.
−
** PDU equipment:
+
# Run the script. "/home/hdops/FCAL/tools/FCAL_basePowerReset.py"
−
*** Expert usage only
+
# Wait for 30 seconds.
−
*** 1 pdu to physically power cycle Anagate devices
+
# Reload the production HV settings by clicking on "Save/Restore" -> "Restore FCAL HV Setpoints", then select entry called "ProductionHV_CURRENT.snap" (it may not be the top entry). Click "Restore"
−
*** 1 pdu to physically power cycle Power Chassis
+
# Turn on the FCAL high voltage.
 +
# Start a new run (as appropriate).
 +
# Count the number of channels with a yellow border.  If there are more than in step 2 then follow the set point not received procedure below.
 +
<gallery mode="packed" widths=200px heights=350px>
 +
File:FCAL_voltages_6lockup.png|Figure: FCAL voltage showing 6 locked up channels
 +
File:FCAL_scalers_6lockup.png|Figure: FCAL scalers showing 6 locked up channels missing
 +
File:base_lockup_channel.png|Figure: FCAL channel showing a communication failure alarm
 +
</gallery>
  
−
[[Image:FCAL_construction.JPG | thumb | 400px | Fig. 1. FCAL during construction, view looking downstream]]
+
= FCAL High Voltage channel with '''<span style="background: yellow;">yellow border</span>''' =
−
[[Image:FCAL_Crates_2D.png | thumb | 400px | Fig. 2. FCAL arranged by inputs to crates, view looking upstream]]
+
−
[[Image:FCAL_Slow_Control_Map_Strands.png | thumb | 400px | Fig. 3. Arrangement of base communication "strands", view looking upstream]]
+
−
[[Image:FCAL_Voltages.PNG | thumb | 400px | Fig. 4. EPICS GUI of base Anagates, arranged as they are on the FCAL platform]]
+
−
[[Image:FCAL_Darkroom.png | thumb | 400px | Fig. 5. MyaViewer screen for monitoring the FCAL Darkroom Temperature, Humidity, and Light]]
+
−
[[Image:FCAL_darkroom.PNG | thumb | 400px | Fig. 6. FCAL Darkroom GUI page]]
+
  
 +
* '''Voltage''' problems with the FCAL voltage on a base will be indicated by a '''<span style="background: yellow;">yellow border </span>''' around a channel on the FCAL voltage GUI.
 +
** You might be able to fix the channel
  
−
== Routine operation==
+
The '''<span style="background: yellow;">yellow border</span>''' might be caused by:
 +
# A base that did ''not receive'' the voltage set point communication
 +
#* By nudging the voltage you can fix this (follow the set point not received procedure below)
 +
# A ''failing base'' (anything that is none of the above)
 +
#* Must be fixed by expert on a maintenance day
  
−
===FCAL Occupancies===
+
==Set point not received procedure==
−
FCAL occupancies will likely have a small number of features to them. Single channels may have no/low occupancy. Base HV issues viewable from the FCAL HV GUI will explain some but not all channels (other problems include bad ADC readout electronics and PMT damage). Some channels may also have very high occupancy, this is likely due to abnormally large "switching noise" from the channel's base. Channels may also turn on/off in adjacent runs. Normal behavior such as any of this doesn't require notifying FCAL experts.
+
  
−
Other behavior (e.g. a 4x4 block going missing in occupancy plots) should be reported to FCAL experts. A typical occupancy plot from Spring '17 is shown here:
+
Any time the voltages in the FCAL are reset there will be some bases that do not receive the voltage set point. Epics is slow, it might take 10 minutes before this becomes clear.
−
[[Image:Fcal_digOcc2D.png|thumb|center|500px]]
+
  
−
===FCAL EPICS GUI Screens===
+
* These are diagnosed by clicking on the channel to bring up the '''Single Channel''' screen.
−
* '''FCAL Voltages''': This screen gives information on FCAL bases providing power to pmts. This gives an overview of the FCAL HV status, can be used to power channels on/off, and to navigate to individual channels. <font color=red>Red </font>  channels have been turned off by FCAL experts. <font color=yellow>Yellow</font> channels indicate either voltage or communication issues. Problematic channels will most likely line up with holes in occupancy.
+
*# If the ''Voltage Setpoint'' and the ''Voltage Setpoint Readback'' are similar but the ''Measured Voltage'' is near to 0, this is likely to be a missed communication between the IOC and the base.  This can be corrected by forcing an updated voltage set point communication with that base :
−
*'''FCAL Darkroom''': This gives an overview of sensors monitoring the environment inside the FCAL darkroom. These should always be green unless someone is inside the darkroom..
+
*## On the channel with issues begin lowering the voltage by clicking on the ''Voltage Setpoint'' arrows.  After about 20 V of change the ''Measured Voltage'' should jump to the correct value.
−
*'''FCAL LED pulser''': The long and short term behavior of the FCAL is monitored by an LED pulser and controlled here. This should always be running at 10 Hz and cycles between three different colors hourly. If no LED pulser appears to be going, contact an FCAL expert. Do not change anything here unless you know what you are doing!
+
*## Do this for all channels with issues.
−
*'''FCAL Scalars''': This gives fADC scalar readouts from the FCAL for individual channels.
+
*## '''NB:''' Restore the voltages using CSS -> FCAL Voltages -> SAVE/RESTORE -> Restore FCAL HV Setpoints -> '''ProductionHV_CURRENT.snap'''
−
* '''Test Setup Voltages''': Old, can safely be ignored.
+
  
−
===FCAL Alarms===
 
−
Potential FCAL alarms could come from environmental monitors inside the FCAL darkroom. However, it is not expected that we should ever reach any alarm values. The FCAL Darkroom environmental variables can be viewed using MyaViewer and selecting the group 'HD_FCAL_ROOM'
 
  
−
Starting Fall 2016 running, channels with base HV issues or communication issues were no longer included in EPICS alarms.
 
  
−
===Correlating strands with FCAL coordinates===
 
−
Below is slow control map where each color represents a different strand. Each strand is daisy chained to 100 bases. This view is looking upstream at the FCAL and the strands are numbered from 1 to 28. Strand 1 to 14 is only the left half of the FCAL and starts in the upper left corner and ends at the lower left corner. Strand 15 to 28 starts in the upper right corner and ends at the lower right corner.
 
  
−
 
+
=High Voltage Expert=
−
==Voltage control==
+
−
The bases generate their own high voltage which can be set through the GUI interface.
+
−
A non-expert user will only be able to do the following from the non-expert shift taker GUI shown in Figure 2:
+
−
* Restore HV setpoints
+
−
** Click 'SAVE/RESTORE' and select 'Restore FCAL HV Setpoints'
+
  
 
Below is a brief explanation of the other functionality capable on the non-expert GUI, <font color=red>but a shift taker must be directed by an expert to use any of the following</font>:
 
Below is a brief explanation of the other functionality capable on the non-expert GUI, <font color=red>but a shift taker must be directed by an expert to use any of the following</font>:
Line 69: Line 70:
 
** Click 'All FCAL HV' and select 'Turn On All Bases'
 
** Click 'All FCAL HV' and select 'Turn On All Bases'
 
* Disable the HV on all bases
 
* Disable the HV on all bases
−
** Can be done in 2 ways: 1) At the top of the screen there is a red outline area with 'Turn OFF ALL' written, click the 'HV' button to power off all bases, 2) click 'All FCAL HV' and select 'Turn Off All Bases'
+
** Can be done in 2 ways:
 +
**# At the top of the screen there is a red outline area with 'Turn OFF ALL' written, click the 'HV' button to power off all bases,  
 +
**# click 'All FCAL HV' and select 'Turn Off All Bases'
 
* Save HV setpoints
 
* Save HV setpoints
 
** Click 'SAVE/RESTORE' and select 'Save FCAL HV Setpoints'
 
** Click 'SAVE/RESTORE' and select 'Save FCAL HV Setpoints'
  
−
== Interlocks ==
 
  
−
A set of "interlocks" monitor the status inside the darkroom and are set to shut off power to bases and PMTs should these exceed set limits. This is mostly done to protect PMTs from potential damage from too much ambient light. In all, there are sensors for the darkroom double doors and four sensor boxes. Each sensor box monitors temperature, humidity, and light levels.
 
  
  
−
Power supplies for the bases are interlocked and will be turned off when  a) both doors to the dark room are open simultaneously, b) environmental sensors (light, temperature or humidity) in the dark room exceed preset limits or c) there is a loss of power. Should the interlock system trip, an expert should be called to reset the condition. Changes to the interlock settings are only allowed by an expert, or under his/her guidance.
+
= Forward Calorimeter Summary =
  
−
== Expert personnel ==
+
[[Image:FCAL_construction.JPG | thumb |left| 300px | Fig. 1. FCAL during construction, view looking downstream]]
 +
 
 +
The hall D Forward Calorimeter (FCAL) is an electromagnetic calorimeter designed to detect neutrals in the forward direction from approximately 2-11 degrees in lab theta, measuring from the center of the target.
 +
 
 +
The Forward Calorimeter is a 2800 element lead glass detector consisting of 4cm x 4cm x 45cm bars.  The light produced in each element is measured using FEU 84-3 PMTs. The PMTs are powered using custom Cockcroft-Walton bases and operate up to 1800 V.  The entire detector exists inside an environmentally controlled and light-tight "darkroom" on the FCAL platform. Individual PMTs respond differently to a fixed voltage, so each channel has an independent voltage setting determined from a gain balancing procedure.
 +
 
 +
The FCAL expert page can be found [https://halldweb.jlab.org/hdops/wiki/index.php/Forward_Calorimeter_Expert here]. It contain more detailed information if you want a more detailed description of FCAL elements and operation.
 +
 
 +
=Interlocks=
 +
 
 +
[[Image:FCAL_darkroom.PNG | thumb | right| 400px | Fig. 2. FCAL Darkroom GUI page]]
 +
 
 +
A set of "interlocks" monitor the status inside the darkroom and are set to shut off power to bases and PMTs should these exceed set limits. This is mostly done to protect PMTs from potential damage from too much ambient light. There are sensors for the darkroom double doors and four sensor boxes in the darkroom. Each sensor box monitors temperature, humidity, and light levels.
 +
 
 +
Power supplies for the bases are interlocked and will be turned off when any of the following occurs:
 +
# both doors to the dark room are open simultaneously,
 +
# environmental sensors (light, temperature or humidity) in the dark room exceed preset limits or
 +
# there is a loss of power.
 +
 
 +
Should the interlock system trip, an expert should be called to reset the condition. Changes to the interlock settings are only allowed by an expert, or under his/her guidance.
 +
 
 +
 
 +
 
 +
 
 +
= Routine operation=
 +
 
 +
==FCAL Occupancies==
 +
FCAL occupancies will likely have a small number of features to them. Isolated channels may have no/low occupancy (or more rarely very high occupancy). This may vary run-to-run a little bit. This type of behavior doesn't require notifying FCAL experts.
 +
 
 +
Any behavior other than the above should be reported to FCAL experts. A typical occupancy plot from Spring '17 is shown here:
 +
[[Image:Fcal_digOcc2D.png|thumb|center|500px]]
 +
 
 +
==FCAL EPICS GUI Screens==
 +
* '''FCAL Voltages''': This screen gives information on FCAL bases providing power to PMTs. This gives an overview of the FCAL HV status, can be used to power channels on/off, and to navigate to individual channels. <font color=red>Red  </font>  channels have been turned off by FCAL experts. <font color=yellow>Yellow</font> channels indicate either voltage or communication issues. Problematic channels will most likely line up with holes in occupancy.
 +
*'''FCAL Darkroom''': This gives an overview of sensors monitoring the environment inside the FCAL darkroom. These should always be green unless someone is inside the darkroom.
 +
*'''FCAL LED pulser''': The long and short term behavior of the FCAL is monitored by an LED pulser and controlled here. This should always be running at 10 Hz and cycles between three different colors hourly. If no LED pulser appears to be going, contact an FCAL expert. Do not change anything here unless you know what you are doing!
 +
*'''FCAL Scalars''': This gives fADC scalar readouts from the FCAL for individual channels.
 +
 
 +
==FCAL Alarms==
 +
 
 +
===Dark room===
 +
 
 +
Potential FCAL alarms could come from environmental monitors inside the FCAL darkroom.  However, it is not expected that we should ever reach any alarm values.  The FCAL Darkroom environmental variables can be viewed using MyaViewer and selecting the group 'HD_FCAL_ROOM'
 +
 
 +
=Expert personnel =
 
The individuals responsible for checking that the FCAL is ready to take data  and setting its operating parameters  are shown in following table.
 
The individuals responsible for checking that the FCAL is ready to take data  and setting its operating parameters  are shown in following table.
 
Problems with normal operation of the FCAL should be referred to those individuals and any changes to their settings must be
 
Problems with normal operation of the FCAL should be referred to those individuals and any changes to their settings must be
Line 90: Line 135:
 
! width=200px | Name    !!  width=100px | Number !! Date of qualification  
 
! width=200px | Name    !!  width=100px | Number !! Date of qualification  
 
|-
 
|-
−
| Adesh Subedi     ||  align=center |   (757) 816-3649||  align=center |  
+
| Calorimeter Expert || align=center | 354-9399 || align=center|
 +
|-
 +
| Mark Dalton     ||  align=center | Cell:757-849-2929 ||  align=center |  
 
|-
 
|-
−
| Jon Zarling    ||  align=center | (763) 913-6155  ||  align=center |  
+
| Malte Albrecht  ||  align=center | Cell:757-316-4936 ||  align=center |  
 
  |}
 
  |}

Latest revision as of 11:53, 26 May 2025

FCAL High Voltage channel with purple border

  • Communications problems with the FCAL voltage on a base will be indicated by a purple border around a channel on the FCAL voltage GUI.
    • You might be able to fix the channel
    • Please look at FCAL voltage GUI every couple of hours or so.

The purple border might be caused by:

  1. A base lockup
    • Follow FCAL Base lockup procedure
  2. A base that has a general communications issue
    • Must be fixed by expert

FCAL Base lockup procedure

Bases might lose communication and stop providing high voltage due to radiation.

Lockup diagnosis:

  1. The scalers show zero counts for the channel (the RootSpy occupancy may also show a hole.)
  2. The voltage GUI has a purple border for the channel indicating alarm
  3. The alarm state is "INVALID, LINK ALARM"
    1. Click on the channel in the voltage screen to get the FCAL Table: Single Channel screen
    2. Hover over the yellow Chanel Status indicator
    3. Mouse tooltip shows the alarm state as "INVALID, LINK ALARM"
    4. If this instead shows "OK, OK" then the base is not locked up

This is corrected by the following sequence:

  1. End the run. We usually don't end a run early for this issue.
  2. Count the number of channels with a yellow border.
  3. MAKE A LOG ENTRY with the number of the locked up base.
  4. Turn off the high voltage to the whole FCAL.
  5. Run the script. "/home/hdops/FCAL/tools/FCAL_basePowerReset.py"
  6. Wait for 30 seconds.
  7. Reload the production HV settings by clicking on "Save/Restore" -> "Restore FCAL HV Setpoints", then select entry called "ProductionHV_CURRENT.snap" (it may not be the top entry). Click "Restore"
  8. Turn on the FCAL high voltage.
  9. Start a new run (as appropriate).
  10. Count the number of channels with a yellow border. If there are more than in step 2 then follow the set point not received procedure below.

  • Figure: FCAL voltage showing 6 locked up channels

  • Figure: FCAL scalers showing 6 locked up channels missing

  • Figure: FCAL channel showing a communication failure alarm

FCAL High Voltage channel with yellow border

  • Voltage problems with the FCAL voltage on a base will be indicated by a yellow border around a channel on the FCAL voltage GUI.
    • You might be able to fix the channel

The yellow border might be caused by:

  1. A base that did not receive the voltage set point communication
    • By nudging the voltage you can fix this (follow the set point not received procedure below)
  2. A failing base (anything that is none of the above)
    • Must be fixed by expert on a maintenance day

Set point not received procedure

Any time the voltages in the FCAL are reset there will be some bases that do not receive the voltage set point. Epics is slow, it might take 10 minutes before this becomes clear.

  • These are diagnosed by clicking on the channel to bring up the Single Channel screen.
    1. If the Voltage Setpoint and the Voltage Setpoint Readback are similar but the Measured Voltage is near to 0, this is likely to be a missed communication between the IOC and the base. This can be corrected by forcing an updated voltage set point communication with that base :
      1. On the channel with issues begin lowering the voltage by clicking on the Voltage Setpoint arrows. After about 20 V of change the Measured Voltage should jump to the correct value.
      2. Do this for all channels with issues.
      3. NB: Restore the voltages using CSS -> FCAL Voltages -> SAVE/RESTORE -> Restore FCAL HV Setpoints -> ProductionHV_CURRENT.snap



High Voltage Expert

Below is a brief explanation of the other functionality capable on the non-expert GUI, but a shift taker must be directed by an expert to use any of the following:

  • Enable HV on all bases
    • Click 'All FCAL HV' and select 'Turn On All Bases'
  • Disable the HV on all bases
    • Can be done in 2 ways:
      1. At the top of the screen there is a red outline area with 'Turn OFF ALL' written, click the 'HV' button to power off all bases,
      2. click 'All FCAL HV' and select 'Turn Off All Bases'
  • Save HV setpoints
    • Click 'SAVE/RESTORE' and select 'Save FCAL HV Setpoints'



Forward Calorimeter Summary

Fig. 1. FCAL during construction, view looking downstream

The hall D Forward Calorimeter (FCAL) is an electromagnetic calorimeter designed to detect neutrals in the forward direction from approximately 2-11 degrees in lab theta, measuring from the center of the target.

The Forward Calorimeter is a 2800 element lead glass detector consisting of 4cm x 4cm x 45cm bars. The light produced in each element is measured using FEU 84-3 PMTs. The PMTs are powered using custom Cockcroft-Walton bases and operate up to 1800 V. The entire detector exists inside an environmentally controlled and light-tight "darkroom" on the FCAL platform. Individual PMTs respond differently to a fixed voltage, so each channel has an independent voltage setting determined from a gain balancing procedure.

The FCAL expert page can be found here. It contain more detailed information if you want a more detailed description of FCAL elements and operation.

Interlocks

Fig. 2. FCAL Darkroom GUI page

A set of "interlocks" monitor the status inside the darkroom and are set to shut off power to bases and PMTs should these exceed set limits. This is mostly done to protect PMTs from potential damage from too much ambient light. There are sensors for the darkroom double doors and four sensor boxes in the darkroom. Each sensor box monitors temperature, humidity, and light levels.

Power supplies for the bases are interlocked and will be turned off when any of the following occurs:

  1. both doors to the dark room are open simultaneously,
  2. environmental sensors (light, temperature or humidity) in the dark room exceed preset limits or
  3. there is a loss of power.

Should the interlock system trip, an expert should be called to reset the condition. Changes to the interlock settings are only allowed by an expert, or under his/her guidance.



Routine operation

FCAL Occupancies

FCAL occupancies will likely have a small number of features to them. Isolated channels may have no/low occupancy (or more rarely very high occupancy). This may vary run-to-run a little bit. This type of behavior doesn't require notifying FCAL experts.

Any behavior other than the above should be reported to FCAL experts. A typical occupancy plot from Spring '17 is shown here:

Fcal digOcc2D.png

FCAL EPICS GUI Screens

  • FCAL Voltages: This screen gives information on FCAL bases providing power to PMTs. This gives an overview of the FCAL HV status, can be used to power channels on/off, and to navigate to individual channels. Red channels have been turned off by FCAL experts. Yellow channels indicate either voltage or communication issues. Problematic channels will most likely line up with holes in occupancy.
  • FCAL Darkroom: This gives an overview of sensors monitoring the environment inside the FCAL darkroom. These should always be green unless someone is inside the darkroom.
  • FCAL LED pulser: The long and short term behavior of the FCAL is monitored by an LED pulser and controlled here. This should always be running at 10 Hz and cycles between three different colors hourly. If no LED pulser appears to be going, contact an FCAL expert. Do not change anything here unless you know what you are doing!
  • FCAL Scalars: This gives fADC scalar readouts from the FCAL for individual channels.

FCAL Alarms

Dark room

Potential FCAL alarms could come from environmental monitors inside the FCAL darkroom. However, it is not expected that we should ever reach any alarm values. The FCAL Darkroom environmental variables can be viewed using MyaViewer and selecting the group 'HD_FCAL_ROOM'

Expert personnel

The individuals responsible for checking that the FCAL is ready to take data and setting its operating parameters are shown in following table. Problems with normal operation of the FCAL should be referred to those individuals and any changes to their settings must be approved by them. Additional experts may be trained by the system owner and their name and date added to this table.

Table: Expert personnel for the FCAL system
Name Number Date of qualification
Calorimeter Expert 354-9399
Mark Dalton Cell:757-849-2929
Malte Albrecht Cell:757-316-4936
Retrieved from "https://halldweb.jlab.org/hdops/wiki/index.php?title=Forward_Calorimeter_Shift&oldid=5052"