Difference between revisions of "HOWTO use AmpTools on the JLab farm GPUs"

From GlueXWiki
Jump to: navigation, search
(AmpTools Compilation with CUDA)
m (AmpTools Compilation with CUDA)
Line 47: Line 47:
  
 
'''6)''' Edit the AmpTools Makefile to pass the appropriate GPU architecture for that machine to the cuda complier
 
'''6)''' Edit the AmpTools Makefile to pass the appropriate GPU architecture for that machine to the cuda complier
$PWD/AmpTools/Makefile:
 
 
  CUDA_FLAGS := -m64 -arch=sm_75
 
  CUDA_FLAGS := -m64 -arch=sm_75
  

Revision as of 11:51, 26 April 2021

Access through SLURM

JLab currently provides 4 NVidia Titan RTX cards. They can be accessed through SLURM, where N is the number of requested cards (1-4):

>salloc --gres gpu:TitanRTX:N --partition gpu --nodes 1

An interactive shell (e.g. bash) on the node with requested allocation can be opened with srun:

>srun --pty bash

Information about the cards, cuda version and usage is displayed with this command:

>nvidia-smi

+-----------------------------------------------------------------------------+
| NVIDIA-SMI 418.87.01    Driver Version: 418.87.01    CUDA Version: 10.1     |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|===============================+======================+======================|
|   0  TITAN RTX           Off  | 00000000:3E:00.0 Off |                  N/A |
| 41%   27C    P8     2W / 280W |      0MiB / 24190MiB |      0%      Default |
+-------------------------------+----------------------+----------------------+

+-----------------------------------------------------------------------------+
| Processes:                                                       GPU Memory |
|  GPU       PID   Type   Process name                             Usage      |
|=============================================================================|
|  No running processes found                                                 |
+-----------------------------------------------------------------------------+

AmpTools Compilation with CUDA

This example was done in csh for the Titan RTX cards on sciml1902.

1) Download latest AmpTools release

wget https://github.com/mashephe/AmpTools/archive/refs/tags/v0.11.0.tar.gz

2) Extract files

tar -xvf v0.11.0.tar.gz

3) Load cuda environment module

module add cuda
setenv CUDA_INSTALL_PATH /usr/local/cuda

4) Set AMPTOOLS directory

setenv AMPTOOLS $PWD/AmpTools

5) Put root-config in your path

setenv PATH $ROOTSYS/bin:$PATH

6) Edit the AmpTools Makefile to pass the appropriate GPU architecture for that machine to the cuda complier

CUDA_FLAGS := -m64 -arch=sm_75

7) Build main AmpTools library with GPU support

make GPU=1

Performing Fits Interactively

Submitting Batch Jobs