Guidelines for Remote Worker Shift

From Hall D Ops Wiki
Jump to: navigation, search

Responsibilities of the Remote Worker

The role of the remote worker is to support the data acquisition and monitoring that is being conducted by the shift leader that is physically located in the counting room. Remote workers will have access to a counting room desktop display via VNC and will be able to interact with it exactly as if they were in the counting room. The remote worker should monitor data quality and data acquisition as well is keep in constant live communication with the shift leader. The remote worker should never conduct an operation that will result in the change of hardware configuration in hall, e.g., manipulating high voltages, moving the diamond, etc.

Days prior to taking any remote shifts, shift crew members should:

  • verify they able to establish a VNC connection to the counting room, which necessarily involves using two-factor authentication (see notes at the bottom of this page) to login to hallgw.jlab.org,
  • read the Hall D Conduct of Operations (COO) document and other relevant information linked on the Hall D Safety Page,
  • obtain, as necessary, proper training and orientation from an experienced shift worker.


  • A video providing a demonstration of the connection procedures and some discussion of remote shift responsibilities is available here in the document database.


Tasks for the remote shift person during a shift:

  • Establish and maintain live communication with onsite shift person through BlueJeans using the room 348194990 with code 7936 or this link. If needed, the moderator number is 1111.
  • Open a VNC session, according to the instructions below, and conduct shift duties. Remember that that the onsite person can both see and interact with the desktop that the remote worker is using. This enables, for example, pointing to features in plots and discussing potential issues.
  • Continue doing any activities that would typically be done by the worker in the counting house, except conducting any operation that involves controlling hardware including acknowledging alarms. Control of hardware and DAQ interface should only be done by the shift leader that is physically present in the counting room.
  • Establish and maintain an e-log entry for your shift like: "Remote Shift WorkerEasyRiser" and update frequently (use local browser)
  • the utility "slack" is another way to communicate with the onsite shift: "@Hall D Operations"
  • Resources and Tasks for Shift Takers

Initiating a Remote Connection to the Counting Room

The remote VNC session must be run through a ssh tunnel though hallgw.jlab.org. The instructions below describe how to establish this tunnel and then connect a VNC client which will be run locally on the workers machine to the tunneled port where the VNC server is running in the counting room.

Initial Setup Instructions

The setup instructions below will need to be completed and tested well before the first shift.

  • Linux: the minimum requirement for remote shift taker is a vnc client referred to generically as vncviewer on linux in the instructions below. This client could in principle be TightVNC, TigerVNC, RealVNC, or any other. Many linux distributions have the executable vncviewer as distributed as part of the TigerVNC package: https://tigervnc.org
  • Mac: likely the stock macOS is sufficient
  • Windows:
  1. install both PuTTY from https://www.putty.org and version 2.x of TightVNC for Windows from https://www.tightvnc.com
  2. configure and test a PuTTY session for the VNC tunnel by opening PuTTY and entering the following customizations. The words underlined reference the side menu and italics are the fields to adjust. (replace XX with a user preferred number: 01 to 99)
    1. SessionHost Name: your_jlab_userid@hallgw.jlab.org
    2. Connection → SSHRemote command: echo “Enter hdops password next…”; ssh -t -L59XX:localhost:5933 hdops@gluon03
    3. Connection → SSH → TunnelsAdd new forwarded port:
      1. Source port: 59XX
      2. Destination: localhost:59XX
      3. click Add
    4. Return to Session and in the Saved Sessions area name this session something like “Hall D VNC Tunnel” and click Save.
  • Windows troubleshooting:
    • If there is any issue with your PuTTY configuration, you may find that the terminal will disappear just after successfully logging in with PIN + cryptokey. If so, remove the remote command and instead type the command ssh -t -L59XX:localhost:5933 hdops@gluon03 to your PuTTY terminal manually.
    • Under Connection → SSHRemote command, be sure that options "Don't start a shell or command at all" and "Enable compression" are unchecked.


Establishing a Remote VNC Session

Each time you start a remote shift you will need to establish a VNC session to conduct the shift. Follow the instructions below.

Create an ssh Tunnel

In all cases below, you will end up with a terminal session open as hdops on gluon03. Leave this session active -- it acts as the ssh tunnel. Once the session is closed, the tunnel is gone.

  • Linux: open a local terminal and run: ssh -t -L59XX:localhost:59XX your_jlab_userid@hallgw.jlab.org ssh -t -L59XX:localhost:5933 hdops@gluon03
  1. replace XX with a user preferred number: 01 to 99
  2. The first password required is the two-factor authentication code, the second password is the hdops account password. (If you do not know the hdops account password you may obtain it verbally from the shift leader through a BlueJeans connection. Please do not email or post the password anywhere.)
  3. verify you are logged in to gluon03 with account hdops (the vncserver on this account uses port 33)
  4. minimize this window to avoid confusion and leave the session running until the end of your shift
  • Mac: same as Linux above using the Terminal app
  • Windows: select the appropriate pre-configured PuTTY session that you setup according to the instructions above and click “Load” and the “Open” to start the session
  1. type in the two-factor authentication code to the first password prompt and then the hdops password to the second password prompt
  2. verify you are logged in to gluon03 with account hdops
  3. minimize this window to avoid confusion and leave the session running until the end of your shift

Run a VNC Client

Now you need to connect a locally run VNC client to the port 59XX on your local machine. In all of the instructions below replace XX with the number chosen when you established the tunnel above.

  • Linux: use the command line VNC client that is generically named vncviewer in the instructions -- replace with the client you installed. From a new, local terminal (not the hdops account on gluon03!) execute the command vncviewer -shared ::59XX
  1. 59XX is the port
  2. -shared option to keep other connections made by other users
  3. enter the hdops password when prompted by the VNC client
  • Mac:
  1. in the Finder select "Go" and "Connect to Server" or use the shortcut command-K and then use the address: vnc://localhost:59XX
  2. enter the hdops password when prompted by the Screen Sharing app
  3. (Note: if unable to start a VNC session through Safari then download a VNC client like RealVNC and connect to localhost:59XX)
  • Windows:
  1. from Windows menu select TightVNC Viewer
  2. in the Remote Host field enter: localhost::59XX (note the double colon)
  3. in the “Options” box verify the bottom checkbox “Request shared session” is selected
  4. click “Connect”
  5. enter the hdops password when prompted by TightVNC

Troubleshooting

If you are unable to get past both password prompts, then it suggests you either have a problem with two-factor authentication or you are not correctly entering the hdops password. To try to understand which is the problem, test your two-factor authentication by executing ssh -t -L59XX:localhost:59XX your_jlab_userid@hallgw.jlab.org. If you are unable to successfully complete this connection, then contact the help desk for assistance in resetting your two-factor authentication capability.

If you can use 2FA, but unable to get to gluon03, it suggests a problem with the hdops password. Ask the expert shifter or a colleague to remind you of this.

If you successfully log in to gluon03 but then get mysterious errors in setting up the VNC session. First, double-check you are using the hdops password to authenticate the VNC session. Then look in the terminal window above the "splash" message on the login to gluon03. You may see errors like this:

Password:
bind [127.0.0.1]:5923: Address already in use
channel_setup_fwd_listener_tcpip: cannot listen to port: 5923
Could not request local forwarding.

If that is the case try a different choice for XX. Unfortunately, there is no good way to know which choices are available. Please terminate sessions once your shift is complete.

Two-Factor Authentication Notes

Access to the Hall online systems requires the use of two-factor authentication (2FA). Prior to the run period an attempt was made to enable two-factor authentication for most potential remote users through the MobilePASS system. For information about the two-factor authentication system at JLab please refer to the computer center KnowledgeBase article (JLab login required to view).

  • if you want to test your 2FA capability, then try to ssh to hallgw.jlab.org -- remember that the password is your PIN (setup at token registration) followed by the one time passcode generated by the app with no space in between
  • if you are authorized to have 2FA but are unable to login, contact the help desk (757) 269-7155 during business hours and they can reissue the token immediately
  • if you are not currently authorized to have 2FA (e.g., a new collaborator as of late summer 2021) initiate an incident ticket with the helpdesk and cc your JLab sponsor and/or Mark Ito on the request -- note that such a request may take a few days to process

Video howto session