Difference between revisions of "Shift Guidelines for Shift Leader"

From Hall D Ops Wiki
Jump to: navigation, search
(→‎Run info logging)
 
(72 intermediate revisions by 11 users not shown)
Line 1: Line 1:
 +
See also [[Short_Term_Run_Plan | Short Term Run Plan]]
 +
 +
=== Hall Delta ===
 +
 +
It's good practice to refer to our hall as "Hall Delta" when talking to MCC, to make sure that there is no acoustic confusion between us and Hall B.
 +
 
=== Beam tuning ===
 
=== Beam tuning ===
  
 
# Make sure the collimator is in blocking position.
 
# Make sure the collimator is in blocking position.
 +
# Verify that amorphous radiator is RETRACTED and that the goniometer is in BLANK or RETRACTED positions.
  
−
=== Run info logging ===
+
=== Changing radiators ===
  
−
# Keep a list of the runs at the end of the shift summary log entry, eg https://logbooks.jlab.org/entry/3795057
+
# Request accelerator insert the dumplette before any radiator changes. See "[[Beam_Line_Detectors_Shift#Collimator_and_Radiators |Move radiator ]]".
−
# Make sure that the DAQ "RUN" field is set to PHYSICS_PRIMEX for production runs, and EXPERT for test runs.
+
# Each time the amorphous radiator is inserted, make a new AMO reference with beam on. See "[[Beam_Line_Detectors_Shift#Setting_Amorphous_Reference |Setting Amorphous Reference]]."
−
# Keep the runs to 2h real-time at most. Calibrations drifts are noticeable after about 2h. 
+
# Each time the diamond radiator is inserted or its orientation changed, use the nudge buttons to obtain the desired coherent edge fit value, save the new configurations as instructed in "[[Beam_Line_Detectors_Shift#Adjusting_the_coherent_peak_positions |Adjusting the peak positions]]" and make a screenshot log entry of the Co-BREM EPICS GUI, see [[Logbook_Screenshot_Tool | screenshot tool for making E-log entries]].
−
# Write run info into the DAQ run comments box at the start of the run and save it.
+
# Watch the goniometer movement in the image in the Goniometer EPICS screen <b>every time</b> the radiator mounted on the goniometer is changed to make sure that the desired radiator of the goniometer is actually inserted into the beam.
−
# Make sure that the paper list of runs is kept up to date.
+
  
−
=== Monitoring ===
+
=== The DAQ ===
  
−
# Half-way through each run, post the rootspy monitoring histograms to the logbook and reset them.  Post the new plots at the end of the run.
+
# When depressing each of the DAQ buttons (Configure, Download, Prestart) wait for each action to be fully done: monitor the progress in the Run Control rcGui (bottom monitor) as well as the blue, yellow and green hd_all.tsg windows (top monitor).  When things have settled you will see SUCCESSFUL or similar messages.
−
# Make a screenshot of the GlueX overview CSS logentry mid-shift.
+
# Wait to PRESTART the DAQ until you expect to hit GO as soon as Prestart has finished. The EPICS values are written into RCDB as part of the Prestart script, they are used for data analysis and calibration, so they need to be relevant to the data being taken (beam on, correct current, radiator in place, temperature and pressure when the run starts).
−
# Remind the shift worker to complete the shift checklist.  
+
# After hitting GO, write run info (radiator, beam current etc) into the DAQ run comments window and save it.  Do not close the window, in case you need to re-save new information.
−
# Watch the beam monitoring stripcharts, currently using beam_instability_monitoring configuration, & post to the logbook at the end of the shift
+
# Keep the runs to 2h real-time at most.  Calibration drifts are noticeable after about 2h.  
−
# New, maybe temporary: if PS coincidence rates drop by 30% or more, do harp scans (MCC & ours)
+
# If the DAQ lifetime drops to zero:
 +
## Make a separate logbook entry with the info that Sergey requested (see https://logbooks.jlab.org/entry/4351918) and add him (furletov) to the notifications.  There’s  no need to phone him unless the problem is recurrent.
 +
## Stop the run, start a new run.
 +
## If it happens again, then stop the run, restart the daq, start a new run.
 +
## If problems persist, call the expert.
  
−
=== Time to call expert ===
+
=== Run info logging ===
  
−
# Expert or RC should be called after 10-15 min of attempting to solve a problem.
+
# Keep a list of the runs at the end of the shift summary log entry.
 +
#* Please include the following: run number, start & stop times, beam current, target, radiator & orientation, daq configuration, daq event rate, daq livetime, number of events, production/test/junk/any_other_comment for example, see https://logbooks.jlab.org/entry/4369402:
 +
#* Two HTML scripts are available on the gluon3 desktop, called logbook_template.html – used in the above link -- or a fancier logbook_template_v2.html; either can be copied into your shift log entry and then change the Text Format to HTML within your elog entry. 
 +
#* Of course, plain ascii format is also ok as shown in these examples https://logbooks.jlab.org/entry/4350257 or https://logbooks.jlab.org/entry/4151847
 +
# Make sure that the paper list of runs is kept up to date.
  
−
=== Changing radiators ===
+
=== Monitoring ===
  
−
# Request accelerator insert the dumplette before any radiator changes.
+
# "Alt L" brings up the [[Logbook_Screenshot_Tool | screenshot tool for making E-log entries]].
−
# Each time the amorphous radiator is inserted, make a new AMO reference with beam on.  See "[[Beam_Line_Detectors_Shift#Setting_Amorphous_Reference |Setting Amorphous Reference]]."
+
# Watch the beam monitoring strip charts: from terminal execute MyaViewer and right click to "Load a Configuration" with the name "shift_leader"
−
# Each time the diamond radiator is inserted or its orientation changed, use the nudge buttons to obtain the desired coherent edge fit value, save the new configurations as instructed in "[[Beam_Line_Detectors_Shift#Adjusting_the_peak_positions | Adjusting the peak positions]]" and make a screenshot logentry of the cobrems gui.
+
#* Post screenshot of "shift_leader" configuration to the logbook at the end of the shift
 +
#* Watch the main trigger rate (HD:trig:rate:main), which has a habit of freezing. [https://halldweb.jlab.org/hdops/wiki/index.php/Level-1_Trigger_Shift_PrimEx#Trigger_Monitor How to revive the main trigger rate]
 +
#* Watch the PSC coincidence scaler rate (PSC:coinc:scaler:rate), which has a habit of freezing.  [https://halldweb.jlab.org/hdops/wiki/index.php/Pair_Spectrometer_Shift#Run_PS_Scalers How to revive the PS scalers]
 +
# [https://halldweb.jlab.org/hdops/wiki/index.php/Guidelines_for_Remote_Worker_Shift#Monitoring Shift Worker duties]
 +
# If the shift worker is remote: the shift leader gets to go outside to read the '''gas bottle meters'''. The values are needed for the Hall D Shift Check-list, see [https://halldweb.jlab.org/hdops/wiki/index.php/Primer_for_shift_takers Primer_for_shift_takers] for what you find there.
  
 
=== Alarms ===
 
=== Alarms ===
  
−
# When acknowledging alarms for the solenoid, mechanical and DIRC, acknowledge them at the lowest level of the tree.  This is to avoid sending 'OK' messages to the on-call phones for all the branches of the tree which were not alarming.
+
# '''When acknowledging alarms for the solenoid, mechanical and DIRC, acknowledge them at the lowest level of the tree.  This is to avoid sending 'OK' messages to the on-call phones for all the branches of the tree which were not alarming.'''
−
# Make separate logbook entries for worrying alarms & contact experts as needed.
+
# Make separate logbook entries for worrying alarms & contact experts as needed. The alarm's drop-down menu has an option to show guidance, this is often useful.
 +
# [https://logbooks.jlab.org/book/halld Logbook for Hall D] can be access through web browser. The web browser can be launched from <i>hdops</i> account on the <i>gluon</i> cluster using [[Web browser launching utility on gluon machines | web browser launch utility]].
 +
# The FCAL voltages are not included in the alarm system.  Please check the voltage screen once per run and if there are yellow borders around any channels follow the [[Forward_Calorimeter_Shift | procedure on the wiki ]].
  
−
=== Beam Time Accounting===
+
=== Problems ===
  
−
The following link gives extensive general information on filling out the Beam Time Accounting (BTA)
+
# Any common issues with detectors are described on their [https://halldweb.jlab.org/hdops/wiki/index.php/Main_Page shift instruction pages - under Operations - Systems]
−
[[BTA_guidelines | Beam-Time Accounting (BTA) Guidelines]]
+
# If the DAQ lifetime drops to zero, [https://halldweb.jlab.org/hdops/wiki/index.php/Shift_Guidelines_for_Shift_Leader#The_DAQ see above]
 +
# When you make a logbook entry about a detector, please add the detector expert(s) to the notify field.
 +
# Expert or RC should be called after at most 10-15 min of attempting to solve a problem.
  
−
====6 hour shifts====
+
=== Beam Time Accounting===
 +
 
 +
'''[[BTA_guidelines | Beam-Time Accounting (BTA) guidelines are here]]'''  Note that MCC crew chief will need the BTA sheet to be updated soon after the 7th hour of each Hall D shift ends.  MCC Crew Chief usually calls 1-2 min after the start of the 7th hour.  The Shift Leader must have the table filled (each hour click the pencil to edit each row as described at the above link, and then click the check mark next to each row.  All 7 rows need to be checked as such for the Crew Chief.  The Leader does not need to sign off after the 7th hour, but after the 8th hour.
  
−
The BTA form does not accommodate 6 hour shifts. Therefore we have to share accounts. When you arrive for shift the BTA system should already be logged in with the current shift-worker's credentials. Please verify that the numbers have been filled in and saved. At 8am, 4pm and midnight the BTA will need to be signed.  After signing the BTA, log out of the system and log in again with your own credentials.  This procedure ensures that only 2 people are involved in any BTA form at a time, which should simplify things.
+
===MCC busy times===
 +
* Please avoid 8:00-8:15am, 4:00-4:15pm, and 12:00-12:15am when making non-emergency requests for beam delivery modifications. ([https://www.jlab.org/sites/default/files/user-liaison/documents/Web20211122.1.pdf from page 8 of this])
 +
* Accelerator shift changes are nominally 7:30-8:30 am, 3:30-4:30pm, and 11:30pm-12:30am.
  
 
===Access ===
 
===Access ===
  
−
# We keep a list of requests for opportunistic access on the whiteboard. We also have to keep a list of who is actually in the hall during accesses.
+
* We have to keep a list of who is in the hall during accesses.
 +
* We also keep a list of requests for opportunistic access on the whiteboard.
 +
 
 +
=== Half-wave Plate (HWP) in Injector ===
 +
The polarization of the electron beam is reversed by inserting (or withdrawing) a half-wave plate (HPW) in the injector. Other halls will call to inform us when the HPW position changes, so that we can also keep track of the electron polarization. [Note: the linear polarization of our photon beam does not depend on the electron beam polarization.] If we are informed of a HPW change, stop the current run, make a separate log entry indicating new position of HWP (in/out), and start a new run.
  
 
===VNC Server===
 
===VNC Server===
  
 
Remote shift workers access the counting room via a VNC session. You are responsible for having the VNC server up and running. Find instructions for starting the server [https://logbooks.jlab.org/entry/3909587 here]. To view the desktop in the counting room, enter the command
 
Remote shift workers access the counting room via a VNC session. You are responsible for having the VNC server up and running. Find instructions for starting the server [https://logbooks.jlab.org/entry/3909587 here]. To view the desktop in the counting room, enter the command
−
  vncserver -shared localhost:33
+
  vncviewer -shared localhost:33
 
into a terminal on the gluon03 workstation to start the VNC client.
 
into a terminal on the gluon03 workstation to start the VNC client.
  
−
if you ever see the need to kill a vncserver use the command:
+
If you ever need to kill a vncserver use the command:
 
  vncserver -kill :XX
 
  vncserver -kill :XX
−
where XX is the port.
+
where XX is the port number. You need to be logged into gluon03 as hdops, of course.
 +
 
 +
And then you restart the server with the command:
 +
vncserver -geometry 1600x1080 :XX -bs
 +
where again XX is the port number.
 +
 
 +
===Zoom===
 +
 
 +
Be sure that the Zoom meeting ([https://halldweb.jlab.org/wiki-private/index.php/Connect_to_ZoomGov_Meetings#ZoomGov_Meeting_ID.2FPasscodes 'Shift Operation', near the end of this list]) is connected on the PC at the far right of the work stations (hdguest1).  The webcam connects by default, but you must set the microphone to "Chat 50 Analog Mono" under the settings menu of the Zoom window.  This will use the small microphone labelled "ClearOne" which should be between gluon03 and gluon04. You may need to use the 'Having issues with the Zoom Workplace app? Join from your browser' link under the 'Launch Meeting' Button to make sure the zoom connects using the Browser.
 +
 
 +
 
 +
===Confused?===
 +
 
 +
Please use the [https://forms.gle/N1cU1GWgXr81pWDc6 '''new hdops wiki feedback form'''] to report confusing or outdated shift instruction wiki pages. Thank you.

Latest revision as of 20:37, 8 May 2025

See also Short Term Run Plan

Hall Delta

It's good practice to refer to our hall as "Hall Delta" when talking to MCC, to make sure that there is no acoustic confusion between us and Hall B.

Beam tuning

  1. Make sure the collimator is in blocking position.
  2. Verify that amorphous radiator is RETRACTED and that the goniometer is in BLANK or RETRACTED positions.

Changing radiators

  1. Request accelerator insert the dumplette before any radiator changes. See "Move radiator ".
  2. Each time the amorphous radiator is inserted, make a new AMO reference with beam on. See "Setting Amorphous Reference."
  3. Each time the diamond radiator is inserted or its orientation changed, use the nudge buttons to obtain the desired coherent edge fit value, save the new configurations as instructed in "Adjusting the peak positions" and make a screenshot log entry of the Co-BREM EPICS GUI, see screenshot tool for making E-log entries.
  4. Watch the goniometer movement in the image in the Goniometer EPICS screen every time the radiator mounted on the goniometer is changed to make sure that the desired radiator of the goniometer is actually inserted into the beam.

The DAQ

  1. When depressing each of the DAQ buttons (Configure, Download, Prestart) wait for each action to be fully done: monitor the progress in the Run Control rcGui (bottom monitor) as well as the blue, yellow and green hd_all.tsg windows (top monitor). When things have settled you will see SUCCESSFUL or similar messages.
  2. Wait to PRESTART the DAQ until you expect to hit GO as soon as Prestart has finished. The EPICS values are written into RCDB as part of the Prestart script, they are used for data analysis and calibration, so they need to be relevant to the data being taken (beam on, correct current, radiator in place, temperature and pressure when the run starts).
  3. After hitting GO, write run info (radiator, beam current etc) into the DAQ run comments window and save it. Do not close the window, in case you need to re-save new information.
  4. Keep the runs to 2h real-time at most. Calibration drifts are noticeable after about 2h.
  5. If the DAQ lifetime drops to zero:
    1. Make a separate logbook entry with the info that Sergey requested (see https://logbooks.jlab.org/entry/4351918) and add him (furletov) to the notifications. There’s no need to phone him unless the problem is recurrent.
    2. Stop the run, start a new run.
    3. If it happens again, then stop the run, restart the daq, start a new run.
    4. If problems persist, call the expert.

Run info logging

  1. Keep a list of the runs at the end of the shift summary log entry.
    • Please include the following: run number, start & stop times, beam current, target, radiator & orientation, daq configuration, daq event rate, daq livetime, number of events, production/test/junk/any_other_comment for example, see https://logbooks.jlab.org/entry/4369402:
    • Two HTML scripts are available on the gluon3 desktop, called logbook_template.html – used in the above link -- or a fancier logbook_template_v2.html; either can be copied into your shift log entry and then change the Text Format to HTML within your elog entry.
    • Of course, plain ascii format is also ok as shown in these examples https://logbooks.jlab.org/entry/4350257 or https://logbooks.jlab.org/entry/4151847
  2. Make sure that the paper list of runs is kept up to date.

Monitoring

  1. "Alt L" brings up the screenshot tool for making E-log entries.
  2. Watch the beam monitoring strip charts: from terminal execute MyaViewer and right click to "Load a Configuration" with the name "shift_leader"
    • Post screenshot of "shift_leader" configuration to the logbook at the end of the shift
    • Watch the main trigger rate (HD:trig:rate:main), which has a habit of freezing. How to revive the main trigger rate
    • Watch the PSC coincidence scaler rate (PSC:coinc:scaler:rate), which has a habit of freezing. How to revive the PS scalers
  3. Shift Worker duties
  4. If the shift worker is remote: the shift leader gets to go outside to read the gas bottle meters. The values are needed for the Hall D Shift Check-list, see Primer_for_shift_takers for what you find there.

Alarms

  1. When acknowledging alarms for the solenoid, mechanical and DIRC, acknowledge them at the lowest level of the tree. This is to avoid sending 'OK' messages to the on-call phones for all the branches of the tree which were not alarming.
  2. Make separate logbook entries for worrying alarms & contact experts as needed. The alarm's drop-down menu has an option to show guidance, this is often useful.
  3. Logbook for Hall D can be access through web browser. The web browser can be launched from hdops account on the gluon cluster using web browser launch utility.
  4. The FCAL voltages are not included in the alarm system. Please check the voltage screen once per run and if there are yellow borders around any channels follow the procedure on the wiki .

Problems

  1. Any common issues with detectors are described on their shift instruction pages - under Operations - Systems
  2. If the DAQ lifetime drops to zero, see above
  3. When you make a logbook entry about a detector, please add the detector expert(s) to the notify field.
  4. Expert or RC should be called after at most 10-15 min of attempting to solve a problem.

Beam Time Accounting

Beam-Time Accounting (BTA) guidelines are here Note that MCC crew chief will need the BTA sheet to be updated soon after the 7th hour of each Hall D shift ends. MCC Crew Chief usually calls 1-2 min after the start of the 7th hour. The Shift Leader must have the table filled (each hour click the pencil to edit each row as described at the above link, and then click the check mark next to each row. All 7 rows need to be checked as such for the Crew Chief. The Leader does not need to sign off after the 7th hour, but after the 8th hour.

MCC busy times

  • Please avoid 8:00-8:15am, 4:00-4:15pm, and 12:00-12:15am when making non-emergency requests for beam delivery modifications. (from page 8 of this)
  • Accelerator shift changes are nominally 7:30-8:30 am, 3:30-4:30pm, and 11:30pm-12:30am.

Access

  • We have to keep a list of who is in the hall during accesses.
  • We also keep a list of requests for opportunistic access on the whiteboard.

Half-wave Plate (HWP) in Injector

The polarization of the electron beam is reversed by inserting (or withdrawing) a half-wave plate (HPW) in the injector. Other halls will call to inform us when the HPW position changes, so that we can also keep track of the electron polarization. [Note: the linear polarization of our photon beam does not depend on the electron beam polarization.] If we are informed of a HPW change, stop the current run, make a separate log entry indicating new position of HWP (in/out), and start a new run.

VNC Server

Remote shift workers access the counting room via a VNC session. You are responsible for having the VNC server up and running. Find instructions for starting the server here. To view the desktop in the counting room, enter the command 

vncviewer -shared localhost:33

into a terminal on the gluon03 workstation to start the VNC client.

If you ever need to kill a vncserver use the command: 

vncserver -kill :XX

where XX is the port number. You need to be logged into gluon03 as hdops, of course.

And then you restart the server with the command: 

vncserver -geometry 1600x1080 :XX -bs

where again XX is the port number.

Zoom

Be sure that the Zoom meeting ('Shift Operation', near the end of this list) is connected on the PC at the far right of the work stations (hdguest1). The webcam connects by default, but you must set the microphone to "Chat 50 Analog Mono" under the settings menu of the Zoom window. This will use the small microphone labelled "ClearOne" which should be between gluon03 and gluon04. You may need to use the 'Having issues with the Zoom Workplace app? Join from your browser' link under the 'Launch Meeting' Button to make sure the zoom connects using the Browser.


Confused?

Please use the new hdops wiki feedback form to report confusing or outdated shift instruction wiki pages. Thank you.

Retrieved from "https://halldweb.jlab.org/hdops/wiki/index.php?title=Shift_Guidelines_for_Shift_Leader&oldid=4992"