Difference between revisions of "Guidelines for Remote Worker Shift"

Latest revision as of 00:33, 4 March 2023

Monitoring

• Mid-run, post the rootspy monitoring histograms to the logbook and reset them. Post the new plots at the end of the run.

• Mid-shift, make a screenshot of the CSS GlueX overview.
  • "Alt L" brings up the screenshot tool for making E-log entries.

• End-shift, complete the shift checklist.

• If the shift worker is remote: the shift leader gets to go outside to read the gas bottle meters. The values are needed for the Hall D Shift Check-list, see Primer_for_shift_takers for what you find there.

Remote Shift Policy

During the 2022-2023 running we expect all shifters to be on site, however, we will allow remote participation for workers in exceptional circumstances. Remote shifters should notify the PDL at least 7 days prior to the shift so that we can be sure the remote shift connection is ready. The current PDL is shown in the left menu bar of the shift schedule. Also, all remote shifters should read and "sign" the documents linked here.

Responsibilities of the Remote Worker

The role of the remote worker is to support the data acquisition and monitoring that is being conducted by the shift leader that is physically located in the counting room. Remote workers will have access to a counting room desktop display via VNC and will be able to interact with it exactly as if they were in the counting room. The remote worker should monitor data quality and data acquisition as well is keep in constant live communication with the shift leader. The remote worker should never conduct an operation that will result in the change of hardware configuration in hall, e.g., manipulating high voltages, moving the diamond, etc.

Days prior to taking any remote shifts, shift crew members should:

• verify they able to establish a VNC connection to the counting room, which necessarily involves using two-factor authentication (see notes at the bottom of this page) to login to hallgw.jlab.org,
• read the Hall D Conduct of Operations (COO) document and other relevant information linked on the Hall D Safety Page,
• obtain, as necessary, proper training and orientation from an experienced shift worker.
• read and acknowledge (by submitting the form) the documentation that is posted here. This is a replacement for the usual signature in the yellow binder that is in the counting house for in-person shifts.

• A video providing a demonstration of the connection procedures and some discussion of remote shift responsibilities is available here in the document database.

Tasks for the remote shift person during a shift:

• Establish and maintain live communication with onsite shift person through Zoom using the meeting ID 1601759979 with passcode 641259 or this link https://jlab-org.zoomgov.com/j/1601759979
• Open a VNC session, according to the instructions below, and conduct shift duties. Remember that that the onsite person can both see and interact with the desktop that the remote worker is using. This enables, for example, pointing to features in plots and discussing potential issues.
• Continue doing any activities that would typically be done by the worker in the counting house, except conducting any operation that involves controlling hardware including acknowledging alarms. Control of hardware and DAQ interface should only be done by the shift leader that is physically present in the counting room.
• Establish and maintain an e-log entry for your shift like: "Remote Shift WorkerEarlyRiser" and update frequently (use local browser)
• the utility "slack" is another way to communicate with the onsite shift: "@Hall D Operations"
• Resources and Tasks for Shift Takers
  • Primer for Shift Takers
  • Hydra Monitoring
  • Monitoring data with Rootspy
  • Look for detector anomalies using the event viewer
  • Monitor the Coherent Edge for diamond radiators only. Do not adjust the edge. Ask the shift leader to perform this operation.
  • Monitor detector rates using EPICS slow control with CSS
  • Monitor DAQ rates with grafana (link only works online)

Initiating a Remote Connection to the Counting Room

The remote VNC session must be run through a ssh tunnel though hallgw.jlab.org. The instructions below describe how to establish this tunnel and then connect a VNC client which will be run locally on the workers machine to the tunneled port where the VNC server is running in the counting room. The shift leader is resposible for having the VNC server running. Shift leaders can find instructions for starting the server here.

Initial Setup Instructions

The setup instructions below will need to be completed and tested well before the first shift.

• Linux: the minimum requirement for remote shift taker is a vnc client referred to generically as vncviewer on linux in the instructions below. This client could in principle be TightVNC, TigerVNC, RealVNC, or any other. Many linux distributions have the executable vncviewer as distributed as part of the TigerVNC package: https://tigervnc.org
• Mac: likely the stock macOS is sufficient
• Windows:

1. install both PuTTY from https://www.putty.org and version 2.x of TightVNC for Windows from https://www.tightvnc.com
2. configure and test a PuTTY session for the VNC tunnel by opening PuTTY and entering the following customizations. The words underlined reference the side menu and italics are the fields to adjust. (replace XX with a user preferred number: 01 to 99)
   1. Session → Host Name: your_jlab_userid@hallgw.jlab.org
   2. Connection → SSH → Remote command: echo “Enter hdops password next…”; ssh -t -L59XX:localhost:5933 hdops@gluon03

NOTE: Until further notice please replace 5933 by 5901 in all instances due to some reset that happened 3/2.

1. 1. Connection → SSH → Tunnels → Add new forwarded port:
      1. Source port: 59XX
      2. Destination: localhost:59XX
      3. click Add
   2. Return to Session and in the Saved Sessions area name this session something like “Hall D VNC Tunnel” and click Save.

• Windows troubleshooting:
  • If there is any issue with your PuTTY configuration, you may find that the terminal will disappear just after successfully logging in with PIN + cryptokey. If so, remove the remote command and instead type the command ssh -t -L59XX:localhost:5933 hdops@gluon03 to your PuTTY terminal manually.
  • Under Connection → SSH → Remote command, be sure that options "Don't start a shell or command at all" and "Enable compression" are unchecked.

Establishing a Remote VNC Session

Each time you start a remote shift you will need to establish a VNC session to conduct the shift. Follow the instructions below.

Create an ssh Tunnel

In all cases below, you will end up with a terminal session open as hdops on gluon03. Leave this session active -- it acts as the ssh tunnel. Once the session is closed, the tunnel is gone.

• Linux: open a local terminal and run: ssh -t -L59XX:localhost:59XX your_jlab_userid@hallgw.jlab.org ssh -t -L59XX:localhost:5933 hdops@gluon03

1. replace XX with a user preferred number: 01 to 99

NOTE: Until further notice please replace 5933 by 5901 in all instances due to some reset that happened 3/2.

1. The first password required is the two-factor authentication code, the second password is the hdops account password. (If you do not know the hdops account password you may obtain it verbally from the shift leader through the zoom connection. Please do not email or post the password anywhere.)
2. verify you are logged in to gluon03 with account hdops (the vncserver on this account uses port 33)
3. minimize this window to avoid confusion and leave the session running until the end of your shift

• Mac: same as Linux above using the Terminal app
• Windows: select the appropriate pre-configured PuTTY session that you setup according to the instructions above and click “Load” and the “Open” to start the session

1. type in the 12-digit two-factor authentication code to the first password prompt and then the hdops password to the second password prompt
2. verify you are logged in to gluon03 with account hdops
3. minimize this window to avoid confusion and leave the session running until the end of your shift

Run a VNC Client

Now you need to connect a locally run VNC client to the port 59XX on your local machine. In all of the instructions below replace XX with the number chosen when you established the tunnel above.

• Linux: use the command line VNC client that is generically named vncviewer in the instructions -- replace with the client you installed. From a new, local terminal (not the hdops account on gluon03!) execute the command vncviewer -shared ::59XX

1. 59XX is the port
2. -shared option to keep other connections made by other users
3. enter the hdops password when prompted by the VNC client

• Mac:

1. in the Finder select "Go" and "Connect to Server" or use the shortcut command-K and then use the address: vnc://localhost:59XX
2. enter the hdops password when prompted by the Screen Sharing app
3. (Note: if unable to start a VNC session through Safari then download a VNC client like RealVNC and connect to localhost:59XX)

• Windows:

1. from Windows menu select TightVNC Viewer
2. in the Remote Host field enter: localhost::59XX (note the double colon)
3. in the “Options” box verify the bottom checkbox “Request shared session” is selected
4. click “Connect”
5. enter the hdops password when prompted by TightVNC

Troubleshooting

VNC App Not Connecting to localhost:XX

If your vncserver / TigerVNC / TightVNC program gives errors similar to "connection refused by computer or remote host" and you are confident you've set things up properly, the vncserver may need to be restarted. This is rare but has been observed. After informing the shift leader, follow the instructions here to kill and restart the vncserver session.

Other Issues

If you are unable to get past both password prompts, then it suggests you either have a problem with two-factor authentication or you are not correctly entering the hdops password. To try to understand which is the problem, test your two-factor authentication by executing ssh -t -L59XX:localhost:59XX your_jlab_userid@hallgw.jlab.org. If you are unable to successfully complete this connection, then contact the help desk for assistance in resetting your two-factor authentication capability.

If you can use 2FA, but unable to get to gluon03, it suggests a problem with the hdops password. Ask the expert shifter or a colleague to remind you of this.

If you successfully log in to gluon03 but then get mysterious errors in setting up the VNC session. First, double-check you are using the hdops password to authenticate the VNC session. Then look in the terminal window above the "splash" message on the login to gluon03. You may see errors like this:

Password:
bind [127.0.0.1]:5923: Address already in use
channel_setup_fwd_listener_tcpip: cannot listen to port: 5923
Could not request local forwarding.


If that is the case try a different choice for XX. Unfortunately, there is no good way to know which choices are available. Please terminate sessions once your shift is complete.

Two-Factor Authentication Notes

Access to the Hall online systems requires the use of two-factor authentication (2FA). Prior to the run period an attempt was made to enable two-factor authentication for most potential remote users through the MobilePASS system. For information about the two-factor authentication system at JLab please refer to the computer center KnowledgeBase article (JLab login required to view).

• if you want to test your 2FA capability, then try to ssh to hallgw.jlab.org -- remember that the 12-digit password is your PIN (setup at token registration) followed by the one time passcode generated by the app with no space in between
• if you are authorized to have 2FA but are unable to login, contact the help desk (757) 269-7155 during business hours and they can reissue the token immediately
• if you are not currently authorized to have 2FA (e.g., a new collaborator as of late summer 2021) initiate an incident ticket with the helpdesk and cc your JLab sponsor and/or Sergey Furletov on the request -- note that such a request may take a few days to process