HPC at the University of Sheffield and Beyond
Twin Karmakharm
Research Software Engineer
Research Software Engineering Sheffield
Helping to increase your research impact
through sustainble research sofware.
- Project consultancy and secondment
- Grant writing support
- Courses and bookable help sessions (Code Clinic)
- For more information:
Overview
- What is HPC and why would I use it?
- Available HPC systems
- Using HPC the right way
What is HPC and why would I use it?
Why use HPC?
When your machine is just not enough...
"It takes me X days to run my algorithm/analysis/simulation."
"I can't load X GBs of dataset into memory."
"There's not enough room to hold my X TBs of dataset."
What is HPC?
High Performance Computing (HPC) refers to a network (cluster) of connected computers (nodes)
HPC - Hardware
Like your PC but bigger...
HPC - Hardware (Bessemer)
And there's a lot of them...
HPC - Hardware (Bessemer)
With fast connection between nodes...
HPC - Hardware
Many have specialist compute hardware e.g. GPUs, FPGAs, etc.
HPC - Software
- Most run on a distribution of Linux
- Has a scheduler to allow multiple users to run tasks (jobs) on the cluster in a fair way
But things run faster on my machine?
- It can take time for your queued jobs to run
- A node can be used by multiple users at the same time
- Individual CPU cores on HPC may be slower than your machine
- Network storage is slower than SSD
You need to parallelise!
Take advantage of the fact that there's a lot of CPUs and Nodes:
You need to parallelise!
Break down your problem into small tasks and run them concurrently:
You need to parallelise!
Break down your problem into small tasks and run them concurrently:
- Multiple independent jobs can be submitted at the same time
- Schedulers have support for submitting an 'array' of jobs (task arrays/job arrays)
- Manage parallisation in your code using:
- Accelerators e.g. GPUs: CUDA, OpenCL
- Single node (multiple CPUs): Threads, MKL, OpenMP
- Between nodes: Message Passing Interface (MPI)
- Many scientific software has parallelisation support either built-in or through plugins
When to not use HPC?
- When your computation task is very serial and is impossible to parallelise
- Your task will probably run slower on the HPC
- Performing small amount of computation on large datasets
- It might take you longer just to upload the dataset to the cluster!
- Hosting websites/applications
What's HPC used for?
- Simulation: Climate, Cell, Molecular Dynamics, etc.
- Optimisation: Engineering, urban planning, etc.
- Machine Learning training and inferencing
- Data cleaning, pre/post-processing, and analysis
- And many others
Some example projects at Sheffield
- Deep Learning for segmenting lung images
- Classification and Predict Malignant Transformation of Oral Epithelial Dysplasia
- Computational models for analysing political advertisements
- Predicting turbulent flows through an array of obstacles using convolutional neural networks
- The Conformational Dynamics of Supercoiled DNA
- Robust ASR for Complex Acoustic Scenes
- And many more!
HPC resources at Sheffield and beyond
HPC Tiers
There are different tiers of HPC, with generally more computing resource the higher you go, from the level of local institutions (Tier-3) to multi-national (Tier-0).
University of Sheffield's HPC (Tier-3)
- ShARC (2024 CPU cores, 40 GPUs)
- Suitable for multi-node MPI jobs
- Bessemer (1040 CPU cores, 4 GPUs)
- Newest system, came on-line in 2019
- Suitable for running many independent jobs
or single-node MPI jobs - 8x GPU nodes with 4x NVIDIA Tesla V100 each (DCS only!)
- More public GPUs coming soon!
University of Sheffield's HPC (Tier-3) - Access
- HPC Driving licence test must be completed.
- Contact it-servicedesk@sheffield.ac.uk for staff access
- Students should get their academic supervisor to fill in the HPC Access request form
- Same account allows access to ShARC and Bessemer
University of Sheffield's HPC (Tier-3) - Support
- Many applications and libraries installed by default, e.g. conda, Gromacs, Matlab, etc.
- Documentation for the systems can be found at https://docs.hpc.shef.ac.uk
- Community support on the HPC google group/mailing list (bit.ly/uos-hpc-group), subscribe by sending an e-mail to hpc+subscribe@sheffield.ac.uk
- RSE's Code Clinic (rse.shef.ac.uk/support/code-clinic/) can also provide HPC advice
University of Sheffield's HPC (Tier-3) - Priority
If additional compute resource is needed...
- You can pay to reserve additional HPC resources
- Nodes can be purchased and added to the clusters
- Contact it-servicedesk@sheffield.ac.uk
Regional HPC Clusters (Tier-2) - UoS Associated
- Free at point-of-use and open to all departments!
- JADE 2 - Request access at https://bit.ly/jade2-access
- For training deep learning models and molecular dynamics simulation
- Single node compute only
- 63x Nvidia DGX MAX-Q nodes with 8 V100 (32GB) GPUs and 512GB RAM each.
- 70 TB DDN AI400 (NVMe storage)
- 1PB Lustre shared storage
Regional HPC Clusters (Tier-2) - UoS Associated
- Free at point-of-use and open to all departments!
- Bede
- No restrictions on the type of computation
- Multi-node MPI, fast CPU to GPU data transfer
- 32x IBM Power9 Dual-CPU nodes, each with 4 Nvidia V100 GPUs
- 6x Inferencing nodes (Nvidia T4s and FPGAs)
- To request access, go to n8cir.org.uk/supporting-research/facilities/bede/docs/bede_registrations/
Regional HPC Clusters (Tier-2) - Others
- Get access through the EPSRC Resource Allocation Panel (RAP) or collaborating with the local institutions
- Cirrus - General HPC compute
- CSD3 - General HPC compute
Regional HPC Clusters (Tier-2) - Others
- Isambard 2 - General HPC compute and contains a variety of HPC technology platforms for testing/benchmarking
- Kelvin2 - General HPC compute, North Ireland (Queen's University Belfast and Ulster University)
- Thomas - Molecular modelling/simulation focus
National HPC Clusters (Tier-1)
- Archer 2 (archer.ac.uk)
- Driving test gives you trial compute hours (archer2.ac.uk/training/driving_test.php).
- Various routes of access are available (archer2.ac.uk/support-access/access)
- DiRAC (dirac.ac.uk)
- Call for proposals can be found at (dirac.ac.uk/callforproposals)
Multi-national HPC Clusters (Tier-0)
- PRACE (prace-ri.eu) - A number of leading European-wide clusters are part of this group
Cloud Services
- It's also possible to get more compute through the cloud providers e.g.
- Processing time, individual machines or an entire HPC cluster can be provisioned
- You're charged based on time used
- No 'official' way of purchashing from IT Services at present
- contact it-servicedesk@sheffield.ac.uk
Using HPC the right way
A few tips to improve your HPC experience
Submit your jobs
- Running things 'live' in an interactive session is great for testing things out and running small tasks
- ... but you're time-limited to a few hours
- It can be difficult to get interactive sessions on popular resources e.g. GPUs
- So create a job script and submit your job to the queue!
- It's possible to submit multiple jobs have have them run concurrently!
- Look into using task arrays i.e. submiting multiple jobs from a single script
Request only what you need
More resource requested means it's harder for the scheduler to find a slot for your job.
Always set your job execution-time!
| SGE (Iceberg, ShARC) | SLURM (Bessemer, JADE) | |
|---|---|---|
| Execution-time | -l h_rt=hh:mm:ss |
-t [mins] -t [days-hh:mm:ss] |
| Memory | -l rmem=xxG |
--mem=xxG |
| No. CPU cores | -pe <env> <nn> |
-c <nn> |
Use package managers or containers
... to standardise your software environment across your local machine, HPCs and cloud.
- You do not have administrator's rights on the HPC
- Software package managers (e.g. apt, yum) need these rights
Use package managers or containers
- You can try to build/install sofware manually, but many software have a complex tree of depedencies
e.g. here's a depency tree for Tensorflow 2
tensorflow-gpu==2.1.0
- absl-py [required: >=0.7.0, installed: 0.9.0]
- six [required: Any, installed: 1.14.0]
- astor [required: >=0.6.0, installed: 0.8.1]
- gast [required: ==0.2.2, installed: 0.2.2]
- google-pasta [required: >=0.1.6, installed: 0.2.0]
- six [required: Any, installed: 1.14.0]
- grpcio [required: >=1.8.6, installed: 1.28.1]
- six [required: >=1.5.2, installed: 1.14.0]
- keras-applications [required: >=1.0.8, installed: 1.0.8]
- h5py [required: Any, installed: 2.10.0]
- numpy [required: >=1.7, installed: 1.18.2]
- six [required: Any, installed: 1.14.0]
- numpy [required: >=1.9.1, installed: 1.18.2]
- keras-preprocessing [required: >=1.1.0, installed: 1.1.0]
- numpy [required: >=1.9.1, installed: 1.18.2]
- six [required: >=1.9.0, installed: 1.14.0]
- numpy [required: >=1.16.0,<2.0, installed: 1.18.2]
- opt-einsum [required: >=2.3.2, installed: 3.2.0]
- numpy [required: >=1.7, installed: 1.18.2]
- protobuf [required: >=3.8.0, installed: 3.11.3]
- setuptools [required: Any, installed: 45.2.0.post20200210]
- six [required: >=1.9, installed: 1.14.0]
- scipy [required: ==1.4.1, installed: 1.4.1]
- numpy [required: >=1.13.3, installed: 1.18.2]
- six [required: >=1.12.0, installed: 1.14.0]
- tensorboard [required: >=2.1.0,<2.2.0, installed: 2.1.1]
- absl-py [required: >=0.4, installed: 0.9.0]
- six [required: Any, installed: 1.14.0]
- google-auth [required: >=1.6.3,<2, installed: 1.13.1]
- cachetools [required: >=2.0.0,<5.0, installed: 4.0.0]
- pyasn1-modules [required: >=0.2.1, installed: 0.2.8]
- pyasn1 [required: >=0.4.6,<0.5.0, installed: 0.4.8]
- rsa [required: >=3.1.4,<4.1, installed: 4.0]
- pyasn1 [required: >=0.1.3, installed: 0.4.8]
- setuptools [required: >=40.3.0, installed: 45.2.0.post20200210]
- six [required: >=1.9.0, installed: 1.14.0]
- google-auth-oauthlib [required: >=0.4.1,<0.5, installed: 0.4.1]
- google-auth [required: Any, installed: 1.13.1]
- cachetools [required: >=2.0.0,<5.0, installed: 4.0.0]
- pyasn1-modules [required: >=0.2.1, installed: 0.2.8]
- pyasn1 [required: >=0.4.6,<0.5.0, installed: 0.4.8]
- rsa [required: >=3.1.4,<4.1, installed: 4.0]
- pyasn1 [required: >=0.1.3, installed: 0.4.8]
- setuptools [required: >=40.3.0, installed: 45.2.0.post20200210]
- six [required: >=1.9.0, installed: 1.14.0]
- requests-oauthlib [required: >=0.7.0, installed: 1.3.0]
- oauthlib [required: >=3.0.0, installed: 3.1.0]
- requests [required: >=2.0.0, installed: 2.22.0]
- certifi [required: >=2017.4.17, installed: 2019.11.28]
- chardet [required: >=3.0.2,<3.1.0, installed: 3.0.4]
- idna [required: >=2.5,<2.9, installed: 2.8]
- urllib3 [required: >=1.21.1,<1.26,!=1.25.1,!=1.25.0, installed: 1.25.7]
- grpcio [required: >=1.24.3, installed: 1.28.1]
- six [required: >=1.5.2, installed: 1.14.0]
- markdown [required: >=2.6.8, installed: 3.2.1]
- setuptools [required: >=36, installed: 45.2.0.post20200210]
- numpy [required: >=1.12.0, installed: 1.18.2]
- protobuf [required: >=3.6.0, installed: 3.11.3]
- setuptools [required: Any, installed: 45.2.0.post20200210]
- six [required: >=1.9, installed: 1.14.0]
- requests [required: >=2.21.0,<3, installed: 2.22.0]
- certifi [required: >=2017.4.17, installed: 2019.11.28]
- chardet [required: >=3.0.2,<3.1.0, installed: 3.0.4]
- idna [required: >=2.5,<2.9, installed: 2.8]
- urllib3 [required: >=1.21.1,<1.26,!=1.25.1,!=1.25.0, installed: 1.25.7]
- setuptools [required: >=41.0.0, installed: 45.2.0.post20200210]
- six [required: >=1.10.0, installed: 1.14.0]
- werkzeug [required: >=0.11.15, installed: 0.15.4]
- wheel [required: >=0.26, installed: 0.34.2]Use package managers or containers
- Package managers
- Automates installation and removal of software, and reproduce software environments
- conda (conda.io)
- Pre-built packages for Pythons, R, etc.
- If you're using Python or R, try conda first
- Spack (spack.io)
- Optimised for HPCs, software is built locally and optimised to hardware
Use package managers or containers
Virtual machines
Containers
Use package managers or containers
- Containers
- A way to package your software and all its dependencies in a self-contained unit (image)
- Containers requires less resource to run and are faster to launch than virtual machines (VMs) as it does not emulate the entire operating system
- Possible to package data, and even entire workflows within the image
- Possible to archive the entire image
Use package managers or containers
- Containers
- Singularity (Now apptainer)
- Designed specifically to for the HPC, does not need administrator rights to
run - Still need administrator rights to build images
- Only runs on Linux (can run within a Linux VM or Windows Subsystem for Linux v2)
- Designed specifically to for the HPC, does not need administrator rights to
- Singularity (Now apptainer)
Use package managers or containers
- Containers
- Docker
- Used mostly in Enterprise where it's normal to own an entire (virtual) machine
- Needs administrator rights to build and run
- However, Singularity can be used to run docker images!
- Runs on Windows, mac and Linux
- Docker
A lot more detail on using HPC the easier way
Conclusion
Conclusion
- You need to parallelise to get the most out of HPC
- UoS provides access to many HPC systems free at point-of-use
- ShARC, Bessemer, JADE 2, and Bede
- Submit your tasks as batch jobs
- Request only the resources you need to get your job run faster!
- If you're stuck, the RSE's Code Clinic (rse.shef.ac.uk/support/code-clinic/) may be able to help you!
Code Clinic
Help with programming problems and general advice on best practice related to research software.
HPC at the University of Sheffield and Beyond
Twin Karmakharm
Research Software Engineer