Fast Monitoring System - PXI
- 1 Overview
- 2 PXI DAQ Configuration and Channel List
- 3 Communication with EPICS
- 4 Communication with PLC
- 5 How to Run PXI-EPICS Control
- 6 How to Analyize ROOT Files
A PXI system is built and configured to monitor the vital signals of the Hall-D super-conducting magnet to ensure its safe operation, particularly to detector the turn-to-turn short faults. Currently, the system contains 4 ADC modules which can sample up to 32 channels simultaneously at a maximum rate of 250 kHz. A LabVIEW RT program is automatically loaded and executed when the system is booted as a stand alone turn-key system. Through its Ethernet connection, the system hosts an EPICS server so that the measurements can be easily accessed and archived remotely. It also serves as an Ethernet device to the magnet control system based on PLC to provides various readings including current, field and voltages.
- hostname: HallD-PXI.jlab.org
- IP address: 220.127.116.11
Reboot the PXI system
- Option I - local power cycling
- Climb the ladder and reach the top of the magnet where the PXI system is located.
- Reboot the PXI by pushing the power button in the front panel.
- Option II - using remote PDU
- The power to the PXI system is distributed by a CyberPower PDU unit, the PXI can be rebooted using the PDU's web interface.
- Go to http://halldcppdu01.jlab.org (IP: 18.104.22.168).
- log in using user name halld and corresponding password (ask Elliott for password).
- In the left panel, click Outlet Control
- Check the #1 box, then choose Reboot in the pull down menu for the Select Task.
- Click Apply button to reboot the PXI
- One will be directed back to the Outlet Status page, and will see the status of PXI change to OFF then ON.
- Option III - using a web browser
This method requires the web sever running properly on PXI.
- Go to http://halld-pxi.jlab.org (MS Silverlight plugin is required).
- Click Restart on the top right of the page to reboot the system remotely.
- Option IV - using NI-MAX
This method requires the Ethernet interface running on PXI.
- Use remote desktop to log onto halld2.jlab.org as user halldusers. Ask Elliott for password.
- Double click the NI-MAX short-cut on the desktop to open Measurement & Automation Explorer.
- Expand the device tree on the left panel and select HallD-PXI under Remote Systems
- Click the Restart button on the right panel and acknowledge the prompt to reboot the system.
PXI DAQ Configuration and Channel List
By turning on the PXI system, a LabVIEW-RT program will be automatically executed and a DAQ session will be started. One can also stop a DAQ session and start a new one through EPICS control GUI.
During the initialization of a DAQ session, a configuration file Channel_Config.ini will be loaded, which defines various DAQ parameters as well as properties of individual channels.
A spreadsheet of the channel list is routinely updated and can be found in M drive:
M:\halld\controls\Rockwell Software\HallD_Controls_Development\Reference Documentation\PXI_Channel_List_date.xlsx
There are two methods to access and modify the file:
- Option I - Web-Browser
The easiest way is to edit the file using a web-browser from a Windows machine (MS Silverlight plugin is required)
- Go to http://halld-pxi.jlab.org
- Click login on top-right of the page, type in user name 'halld' and password when prompted. Ask Elliott to get the password.
- Click the Remote File Browser from the left panel
- Expand the directory tree to open the folder called
- Choose Channel_Config.ini in the right panel, then click Edit File ... to open it
- WARNING! Modify the file content if you know what you are doing, click Save Changes or Cancel to exit the editor.
- Option II - FTP
- Use any ftp client to log on to halld-pxi.jlab.org as halld
- The configuration file can be found in directory
Description of Channel_Config.ini
An example of the Channel_Config.ini is attached.
Some sections of the configuration are shown here with some description after #:
[Global] Sample_Rate = 100000 # ADC sampling rate in Hz, maximum for 4300 module is 250kHz Report_Rate = 10 # EPICS report rate in Hz Data_Rate = 5000 # ROOT file/local binary data file recording rate in Hz Save_Length = 300 # Duration of a single local binary data file in seconds Save_History = 0 # Number of binary data files kept on local disk Fake_Signal = FALSE # Flag for fake data stream
[Modules] # Section for list of modules installed. Slot2 = "4300" # Module type in each slot. Slot4 = "4300" # Currently, only 4300 and 4300B are supported Slot6 = "4300" # Slot8 = "4300B" #
[Slot2_Ch0] # Channel Section. The name has to follow the 'Slot#_Ch#' convention Active = TRUE # Channel activated for DAQ: TRUE or FALSE Channel_Name = "VTT4" # Channel name, will also be used for EPICS PV variables, eg: HallD-PXI:Data:VTT4 and HallD-PXI:Array:VTT4 Voltage_Range = 10 # Voltage range for ADC channel (+-), available ranges are: 1, 5 and 10 for 4300; 30, 150 and 300 for 4300B Offset = -0.000169 # Offset applied for EPICS PV variables and PLC variables Slope = 1 # Scaling factor applied for EPICS PV variables and PLC variables Delay = 0 # Channel delay compensation in seconds
Communication with EPICS
NOTE: Make sure that gluon01.jlab.org(22.214.171.124) is included in
Types of EPICS variables
There are two types of EPICS variables served by this PXI system.
These variables are updated at the rate specified as
Report_Rate in the configuration file. The PXI system average all readings between two reports to improve the data quality. The PV names for these variables are
Channel_Name is defined in the configuration file.
The arrays are mainly used for the remote archiving of the high speed data. Each array has a fixed dimension of 5000, and the update rate depends on the specified
Data_Rate in the configuration. Same as the single element variables, these values are also taken from averages. The PV names for these variables are
NOTE: In order to read these arrays, the EPICS array size limit needs to be modified by changing the value of
EPICS_CA_MAX_ARRAY_BYTES to 100000
EPICS ProcServer for ROOT File Recording
An EPICS procserver is running on gluon01.jlab.org to save the high speed data to rootfiles when needed. All the rootfiles will be saved on gluon01 under directory
and period backed up on JLab silo tapes. To enable or disable the data writing, one can use the EPICS GUI
Reboot the ProcServer
- Log onto gluon01.jlab.org as user halld, ask Elliott for password,
- Connect to the PXI-EPICS server by typing
telnet localhost 26060or simply type
- Hit Ctrl+X to reboot the server, reboot it a couple more times to make sure no error messages coming out,
- Hit Ctrl+] to exit the server
- Type 'quit' to exit telnet
NOTE: Best to REBOOT procserver if PXI system is restarted.
Communication with PLC
The PXI system communicates with the PLC magnet control as an Ethernet device.
- Data from PXI to PLC
- Port: 100
- Format: Array of 36 integers in 32 bit (144 Bytes in total)
- Data from PLC to PXI
- Port: 200
- Format: Array of 1 integers in 32 bit (4 Bytes in total)
A spread sheet is kept on M drive for the latest list of channels:
M:\halld\controls\Rockwell Software\HallD_Controls_Development\Reference Documentation\PXI_Tag_List_date.xlsx
How to Run PXI-EPICS Control
The PXI system can be controlled and monitored through a set of EPICS PVs. A user interface was created using Control System Studio (CSS).
List of Control/Status EPICS PVs
- Control PVs
HallD-PXI:Control:Start: start DAQ
HallD-PXI:Control:Stop: stop DAQ
HallD-PXI:Control:Quit: quit the LabVIEW-RT program
HallD-PXI:Control:Zero: take 1-minuite data to calculate offsets of all channels
HallD-PXI:Control:Reset: reset offsets of all channels
HallD-PXI:Control:ForceAlarm: force the turn-to-turn alarm to be true
- Status PVs
HallD-PXI:Status:Beat: heart beat, updating at 1Hz
HallD-PXI:Status:Status: indicator of current machine state
HallD-PXI:Status:Error: error bit of the LabVIEW-RT program
HallD-PXI:Status:Message: error message of the LabVIEW-RT program
HallD-PXI:Status:CPU: CPU usage of the PXI system
HallD-PXI:Status:Interval: Time interval between two EPICS updates
HallD-PXI:Status:Jitter: Jitter of time intervals
HallD-PXI:Status:FIFO: Buffer depth for local data writing, depleted when reaches 1
HallD-PXI:Status:Alarm: turn-to-turn short alarm bit
Start EPICS Control
- Log onto gluon01.jlab.org using username halld from any Linux terminal, ask Elliott if you do not know the password.
- Type css & to start the Control System Studio.
- If PXI control is not already loaded, click File -> Open File in the top left menu, open
Use EPICS Control
NOTE: Make sure that the PXI's heart beat is blinking, and all the EPICS channels are connected.
The main screen gives you the access to PXI control, ROOT File Recording, strip-charts for CPU usage, solenoid current, field strength, pickup coil signals and buttons to open other strip-charts for sub coil voltages and voltage ratios.
To Start DAQ
- By default, the PXI-DAQ should be running, if not, hit Start to start the DAQ process, you should see that the status changes from System Standby to Initialize DAQ to DAQ Running
- Make sure magnet is in a state that all signals remain zero,
- Stop running DAQ proccess by hitting Stop,
- Click zero button to start zeroing process, the status will change from System Standby to Initizalize to Zeroing. After about 1 minute, the zeroing process should automatically stop and System Standby should be shown then.
- One needs to start a normal DAQ afterwards.
- Stop DAQ
- Click Reset Offset button
- Start DAQ
Generate Fake Turn-to-Turn Short Alarm
The feature only works during a normal DAQ process
- Click T-2-T Alarm to trigger an alarm
- Click T-2-T Alarm again to return to disable the fake alarm
Open Strip Charts
- Individual Voltage Taps
- Click Open Sub Coil Chart
- Total Voltage and Ratios of Voltages
- Click Open Ratio Chart button
- Quench Detector and Voltage Leads Signals
- Click Open QD/Leads Chart button
Start or Stop ROOT File Recording
- ROOT Files need to be recorded for offline analysis, start a new ROOT file everytime a new ramp up starts from 0 A.
- Check the status of ROOT File Recording, if it says Idle .. Click to start new file, then click the button, acknowledge the popup message to start a new ROOT file. The button should then say Writing Data .. Click to stop.
- Double check the existance of the new file.
- first log onto gluon01 from a Linux terminal,
- then type
ll -h /local/scratch/PXI_DATA/ROOT/*.rootor
checkto display the list of ROOT files, try to find the file with the latest time stamp, also repeat the command a couple of more times to make sure that the file size is increasing.
- If the ROOT file is neither found nor being updated, one needs to restart the EPICS ProcService.
- Stop ROOT file writing by clicking the button after the current ramps down to 0 A.
- Add the new file to the file list in elog
How to Analyize ROOT Files
List of ROOT Files
- HallD Elog entry of list of ROOT files
- all the ROOT files are on gluon01.jlab.org in directory
/local/scratch/PXI_DATA/ROOT, there is also a symbolic link
./datain the precompiled Analyzer directory on gluon01.
Structure of ROOT Files
- Channels in ROOT Files
b_field: Magnetic field reading from Hall probe
i_shunt: Voltage drop across the shunt resistor which is proportional to the current
i_sol: Magnetic current reading from ZFCT unit
qd_err: Error signal from the Quench Detector
v_mps: Total voltage from the magnet power supply (MPS)
v_pc4d: voltages from pickup coils
vtt21: voltages across voltage taps
- Raw Data Files
- As stated as raw, all the readings in these ROOT files does not include calibrations such as baseline adjustment and scaling factor.
- Data from different channels are saved in separate trees named as
- Each tree contains a single branch called
- The branch consists of the following leaves:
record_tsec: Long64_4, the second of the timestamp
record_tnes: Long64_4, the nanosecond of the timestamp
record_data: Float_t, a pack of 5000 readings
- Resampled Data Files
- Data files with reduced sampling rates can be generated using the ROOT File Analyzer.
- The readings include calibrations such as baseline adjustment and scaling factors.
- Each file contains a single tree called
- The tree consists of the following branches:
time: Double_t, time elapsed since the DAQ starts
Channel_Name: Float_t, readings of individual channels
Solenoid ROOT File Analysis Tool
Run the Analyzer
- Log onto gluon01 using username halld:
- Go to analyzer directory
- Or simply type
- Run the analyzer:
Checkout and Compile the Analyzer from Source Code
The source files of the Analyzer are stored in the Hall-D SVN repository.
- Check out the code
- Make sure that the ROOT library is included in
- Compile the code:
- Execute the compiled binary:
Use the Analyzer
The Analyzer's GUI is split into one top panel and one bottom panel. The top panel shows the magnet current history and the bottom panel is used to show plots for different analysis.
NOTE: The horizontal range of the current history plot is used to control the data range for all the analysis.
Usage of Buttons and Fields
- Open a ROOT file, the file can either be a raw data file created by the EPICS ProcServer, or a resampled data file created by the Analyzer.
- Calculate (Offset)
- Calculate offsets of all channels in selected range
- Save (Offset)
- Save calculated offset values into a txt file
- Load (Offset)
- Load offset values from a txt file
- Reset (Offset)
- Reset all offsets to zero
- Exit the Analyzer
- Hall Probe Delay
- Delay time (s) for the Hall probe readout, put 0 unless instructed by experts
- Resample Rreq.
- Resample frequency (Hz) for plots, can be 0 - 5 kHz, the higher the slower
- dI/dt Windows
- Windows size (s) used to calculate the current ramp rate
- Show legends in plots
- Calculate mean values of individual histograms shown in the bottom panel, the results are printed in terminal
- Do 2nd order polynomial fits on inductance, the results are printed in terminal as A B C where y = A + Bx +Cx2
- Save ROOT File
- Create a ROOT file with reduced sampling rate and offsets, the created file can be found in directory
- Plot current, magnetic field measured by the Hall probe and the ratio. The ratio can be used to identify turn-to-turn short.
- Pickup Coils
- Plot pickup coil signals, field calculated by integrating pickup coil outputs, and ratio of field to current. The ratio can be used to identify turn-to-turn short.
- Plot voltage reading of all sub-coils as well as the total voltages
- Quench Detector
- (only available in new version) Plot Quench detector error signal and voltage across leads
- Plot current change rate, inductance of sub-coils calculated based on L = V/(dI/dt) as a function of current.
- Plot current change rate and resistance calculated based on R = (V-L*dI/dt)/I
- Plot magnet current and temperature of sub-coils estimated based on their resistance values
- Print the bottom panel into a PDF file and a PNG file, the image files can be found in directory