Guidelines for Remote Worker Shift
Contents
Responsibilities of the Remote Worker
The role of the remote worker is to support the data acquisition and monitoring that is being conducted by the shift leader that is physically located in the counting room. Remote workers will have access to a counting room desktop display via VNC and will be able to interact with it exactly as if they were in the counting room. The remote worker should monitor data quality and data acquisition as well is keep in constant live communication with the shift leader. The remote worker should never conduct an operation that will result in the change of hardware configuration in hall, e.g., manipulating high voltages, moving the diamond, etc.
Days prior to taking any remote shifts, shift crew members should:
- verify they able to establish a VNC connection to the counting room, which necessarily involves using two-factor authentication (see notes at the bottom of this page) to login to hallgw.jlab.org,
- read the Hall D Conduct of Operations (COO) document and other relevant information linked on the Hall D Safety Page,
- obtain, as necessary, proper training and orientation from an experienced shift worker.
- read and acknowledge (by submitting the form) the documentation that is posted here. This is a replacement for the usual signature in the yellow binder that is in the counting house for in-person shifts.
- A video providing a demonstration of the connection procedures and some discussion of remote shift responsibilities is available here in the document database.
Tasks for the remote shift person during a shift:
- Establish and maintain live communication with onsite shift person through Zoom using the meeting ID 1601759979 with passcode 641259 or this link https://jlab-org.zoomgov.com/j/1601759979
- Open a VNC session, according to the instructions below, and conduct shift duties. Remember that that the onsite person can both see and interact with the desktop that the remote worker is using. This enables, for example, pointing to features in plots and discussing potential issues.
- Continue doing any activities that would typically be done by the worker in the counting house, except conducting any operation that involves controlling hardware including acknowledging alarms. Control of hardware and DAQ interface should only be done by the shift leader that is physically present in the counting room.
- Establish and maintain an e-log entry for your shift like: "Remote Shift WorkerEasyRiser" and update frequently (use local browser)
- the utility "slack" is another way to communicate with the onsite shift: "@Hall D Operations"
- Resources and Tasks for Shift Takers
- Primer for Shift Takers
- Hydra Monitoring
- Monitoring data with Rootspy
- Look for detector anomalies using the event viewer
- Monitor the Coherent Edge for diamond radiators only. Do not adjust the edge. Ask the shift leader to perform this operation.
- Monitor detector rates using EPICS slow control with CSS
- Monitor DAQ rates with grafana (link only works online)
Initiating a Remote Connection to the Counting Room
The remote VNC session must be run through a ssh tunnel though hallgw.jlab.org. The instructions below describe how to establish this tunnel and then connect a VNC client which will be run locally on the workers machine to the tunneled port where the VNC server is running in the counting room. The shift leader is resposible for having the VNC server running. Shift leaders can find instructions for starting the server here.
Initial Setup Instructions
The setup instructions below will need to be completed and tested well before the first shift.
- Linux: the minimum requirement for remote shift taker is a vnc client referred to generically as vncviewer on linux in the instructions below. This client could in principle be TightVNC, TigerVNC, RealVNC, or any other. Many linux distributions have the executable vncviewer as distributed as part of the TigerVNC package: https://tigervnc.org
- Mac: likely the stock macOS is sufficient
- Windows:
- install both PuTTY from https://www.putty.org and version 2.x of TightVNC for Windows from https://www.tightvnc.com
- configure and test a PuTTY session for the VNC tunnel by opening PuTTY and entering the following customizations. The words underlined reference the side menu and italics are the fields to adjust. (replace XX with a user preferred number: 01 to 99)
- Session → Host Name:
your_jlab_userid@hallgw.jlab.org
- Connection → SSH → Remote command:
echo “Enter hdops password next…”; ssh -t -L59XX:localhost:5933 hdops@gluon03
- Connection → SSH → Tunnels → Add new forwarded port:
- Source port:
59XX
- Destination:
localhost:59XX
- click Add
- Source port:
- Return to Session and in the Saved Sessions area name this session something like “Hall D VNC Tunnel” and click Save.
- Session → Host Name:
- Windows troubleshooting:
- If there is any issue with your PuTTY configuration, you may find that the terminal will disappear just after successfully logging in with PIN + cryptokey. If so, remove the remote command and instead type the command
ssh -t -L59XX:localhost:5933 hdops@gluon03
to your PuTTY terminal manually. - Under Connection → SSH → Remote command, be sure that options "Don't start a shell or command at all" and "Enable compression" are unchecked.
- If there is any issue with your PuTTY configuration, you may find that the terminal will disappear just after successfully logging in with PIN + cryptokey. If so, remove the remote command and instead type the command
Establishing a Remote VNC Session
Each time you start a remote shift you will need to establish a VNC session to conduct the shift. Follow the instructions below.
Create an ssh Tunnel
In all cases below, you will end up with a terminal session open as hdops on gluon03. Leave this session active -- it acts as the ssh tunnel. Once the session is closed, the tunnel is gone.
- Linux: open a local terminal and run:
ssh -t -L59XX:localhost:59XX your_jlab_userid@hallgw.jlab.org ssh -t -L59XX:localhost:5933 hdops@gluon03
- replace XX with a user preferred number: 01 to 99
- The first password required is the two-factor authentication code, the second password is the hdops account password. (If you do not know the hdops account password you may obtain it verbally from the shift leader through a BlueJeans connection. Please do not email or post the password anywhere.)
- verify you are logged in to gluon03 with account hdops (the vncserver on this account uses port 33)
- minimize this window to avoid confusion and leave the session running until the end of your shift
- Mac: same as Linux above using the Terminal app
- Windows: select the appropriate pre-configured PuTTY session that you setup according to the instructions above and click “Load” and the “Open” to start the session
- type in the two-factor authentication code to the first password prompt and then the hdops password to the second password prompt
- verify you are logged in to gluon03 with account hdops
- minimize this window to avoid confusion and leave the session running until the end of your shift
Run a VNC Client
Now you need to connect a locally run VNC client to the port 59XX on your local machine. In all of the instructions below replace XX with the number chosen when you established the tunnel above.
- Linux: use the command line VNC client that is generically named
vncviewer
in the instructions -- replace with the client you installed. From a new, local terminal (not the hdops account on gluon03!) execute the commandvncviewer -shared ::59XX
-
59XX
is the port -
-shared
option to keep other connections made by other users - enter the hdops password when prompted by the VNC client
- Mac:
- in the Finder select "Go" and "Connect to Server" or use the shortcut command-K and then use the address:
vnc://localhost:59XX
- enter the hdops password when prompted by the Screen Sharing app
- (Note: if unable to start a VNC session through Safari then download a VNC client like RealVNC and connect to localhost:59XX)
- Windows:
- from Windows menu select TightVNC Viewer
- in the Remote Host field enter:
localhost::59XX
(note the double colon) - in the “Options” box verify the bottom checkbox “Request shared session” is selected
- click “Connect”
- enter the hdops password when prompted by TightVNC
Troubleshooting
If you are unable to get past both password prompts, then it suggests you either have a problem with two-factor authentication or you are not correctly entering the hdops password. To try to understand which is the problem, test your two-factor authentication by executing ssh -t -L59XX:localhost:59XX your_jlab_userid@hallgw.jlab.org
. If you are unable to successfully complete this connection, then contact the help desk for assistance in resetting your two-factor authentication capability.
If you can use 2FA, but unable to get to gluon03, it suggests a problem with the hdops password. Ask the expert shifter or a colleague to remind you of this.
If you successfully log in to gluon03 but then get mysterious errors in setting up the VNC session. First, double-check you are using the hdops password to authenticate the VNC session. Then look in the terminal window above the "splash" message on the login to gluon03. You may see errors like this:
Password:
bind [127.0.0.1]:5923: Address already in use
channel_setup_fwd_listener_tcpip: cannot listen to port: 5923
Could not request local forwarding.
If that is the case try a different choice for XX
. Unfortunately, there is no good way to know which choices are available. Please terminate sessions once your shift is complete.
Two-Factor Authentication Notes
Access to the Hall online systems requires the use of two-factor authentication (2FA). Prior to the run period an attempt was made to enable two-factor authentication for most potential remote users through the MobilePASS system. For information about the two-factor authentication system at JLab please refer to the computer center KnowledgeBase article (JLab login required to view).
- if you want to test your 2FA capability, then try to ssh to hallgw.jlab.org -- remember that the password is your PIN (setup at token registration) followed by the one time passcode generated by the app with no space in between
- if you are authorized to have 2FA but are unable to login, contact the help desk (757) 269-7155 during business hours and they can reissue the token immediately
- if you are not currently authorized to have 2FA (e.g., a new collaborator as of late summer 2021) initiate an incident ticket with the helpdesk and cc your JLab sponsor and/or Mark Ito on the request -- note that such a request may take a few days to process