Users do not need to specify much information if they do not care where their job runs or on what hardware.  The less you specify, the sooner your job is likely to run because it will run on the first available node.  Slurm uses the default information from your account, the cluster, and the partition to run a job.  However, this may not work for some users.  You made need more than one node or core, more memory than the default, a longer run time (wall time), a specific CPU, or a high-speed network, just to name a few.


Users can request specific hardware in their batch jobs in several ways.  Here we list a few examples of common resources users request.  This list is not exhaustive, as there are many combinations of ways to request specific hardware for your jobs.  Here we show a combination of Slurm directives used in combination with the Slurm features we have set on the nodes.  


List of hardware currently installed in the Academic cluster 


Slurm features



Use the 'snodes' command on the command line for info on what resources are available right now.  More details



NOTE: The --constraint directive is used to specify Slurm features.  To specify more than one feature, use the & sign between them.  If you'd like to request one type of node OR another type, use the | symbol between the constraints (this is above the \ on your keyboard)


Requesting Specific Node Resources:

Node ResourcesSample Slurm Directives
Multi-node with InfiniBand or OmniPath

--nodes=2 --ntask-per-node=X --constraint=IB

or

--nodes=2 --ntask-per-node=X --constraint=OPA

or 

--nodes=2 --ntasks-per-node=X --constraint=IB&CPU-E5645

  
40 Core Nodes 
Any 40 core nodes--nodes=1 --ntasks-per-node=40
40 core large memory nodes--nodes=1 --ntasks-per-node=40 --mem=754000
  
32 Core Nodes 
512GB nodes--nodes=1 --ntasks-per-node=32 --mem=512000
256GB nodes (INTEL Processors)--nodes=1 --ntasks-per-node=32 --mem=256000 --constraint=CPU-E7-4830
256GB nodes (AMD Processors)--nodes=1 --ntasks-per-node=32 --mem=256000 --constraint=CPU-6132HE
or
--nodes=1 --ntasks-per-node=32 --mem=256000 --constraint=AMD
Any 32 core nodes--nodes=1 --ntasks-per-node=32  (you must specify CPU tag if you want to ensure you don't get on 40 core node)
  
16 Core Nodes 
Nodes with 16 cores--nodes=1 --ntasks-per-node=16 --constraint=CPU-E5-2660
Nodes with at least 16 cores--nodes=1 --ntasks-per-node=16
  
12 Core Nodes 
Nodes with 12 cores--nodes=1 --ntasks-per-node=12 --constraint=CPU-E5645
Any nodes with at least 12 cores--nodes=1 --ntasks-per-node=12
  
8 Core Nodes 
Nodes with 8 cores (IBM)--nodes=1 --ntasks-per-node=8 --constraint=CPU-L5520
Nodes with 8 cores (DELL)--nodes=1 --ntasks-per-node=8 --constraint=CPU-L5630
Any nodes with at least 8 cores--nodes=1 --ntasks-per-node=8
  
Memory 
Any nodes with at least X GB--nodes=1 --mem=X000
  
GPUs 
Nodes with GPUs (32 core)--nodes=1 --ntasks-per-node=32 --gres=gpu:1 (or gpu:2)
Nodes with GPUs (40 core)--nodes=1 --ntasks-per-node=40 --gres=gpu:1 (or gpu:2)
Specific GPUs (V100)--nodes=1 --gres=gpu:1 --constraint=V100
Specific GPUs (P100)
--nodes=1 --gres=gpu:1 --constraint=P100
Specific GPU memory config for V100 GPUs--gres=gpu:tesla_v100-pcie-32gb:1 
or
--gres=gpu:tesla_v100-pcie-16gb:1
or to specify which GPU slot
--gres=gpu:tesla_v100-pcie-16gb:1(S:0)
--gres=gpu:tesla_v100-pcie-32gb:1(S:1)
To use all cores on a node w/more than 1 GPU you must disable CPU binding--gres-flags=disable-binding



We encourage you to check out recordings from previous workshops that explain in detail about these topics:

1. Introduction to submitting jobs on the CCR clusters (Password: CCR-HPC2020) 

2. Job Submission Strategies & Utilizing Idle Nodes (Password: CCR-HPC2020)