Crop Diversity HPC Help

Hello! Here you’ll find the documentation for the UK’s Crop Diversity Bioinformatics High Performance Computing (HPC) Linux cluster - gruffalo - and its associated data storage and services.

Important

Run by the James Hutton Institute’s Information & Computational Sciences (ICS) Research Computing team, and the Scientific Computing section of NIAB’s IT team, this resource supports the computational needs of researchers from six UK institutions. See Supported Organisations for details.

Note

Compute clusters contain a lot of CPUs, so it’s worth seeing how we’re Counting CPUs.

Introduction

• Frequently Asked Questions
• System Overview
• Performance

Getting started

• Requesting an Account
• Getting Connected
• Guest Accounts

High performance computing

• Slurm - Overview
  • Jobs
  • Queues (partitions)
  • Interactive jobs
  • Batch jobs
  • Array jobs
  • Allocating resources
  • Cancelling a job
  • Scratch space
  • Job summaries
  • Automatic requeuing
  • Other helpful parameters
• Slurm - Queue Policies & Advice
  • Specifying queues
  • Fair share policy
  • Additional advice and guidance
• Slurm - Shortcuts and Aliases
  • srsh
  • snodes
  • sjobacct
  • sjobstat
• Bioconda
  • Installing Bioconda
  • Finding packages
  • Installing packages
  • Listing packages
  • Removing packages
  • Environments
  • Removing Bioconda
  • Bioconda and Slurm
• Compiling Software
  • Compiler versions (a.k.a. changing gcc)
• Working with Compressed Files
  • Compressing files
  • Decompressing files
  • Leaving files compressed
• Apptainer (Singularity)
  • Running containers
  • Using aliases
  • Building containers
• GPU Processing
• Machine Learning
• Tools & Applications
  • Aspera
  • Bioconda
  • Databases
  • Java
• Database Mirrors
  • NCBI BLAST
  • Pfam
  • Uniprot
• OpenMPI
  • How to submit MPI jobs
• Green Computing
  • How to tell if a node is powered down
  • When will nodes power up/down?
  • How much energy will this save?
  • Our commitment to Green Computing
  • What can I do to help?

Data storage

• Data Storage & Management Policy
  • Storage at-a-glance
  • Data management
  • BeeGFS storage
  • Local scratch
• Backups
  • Retention policies
  • Restoring backups
• File Transfers
  • Command line options
  • Graphical options
• Samba Access
  • Account details
  • Connection details
  • Shared folders
  • Access from macOS

Miscellaneous

• Contact Us
• Monitoring & Tracking
  • Real-time metrics
  • Disk-usage summaries
  • Login attempts
  • Remote monitoring
• Linux Basics
  • About Linux
  • The terminal/shell
• Tips & Tricks
  • Special keys
  • History
  • Clearing the screen
  • Running multiple commands
  • Following log files
  • Interactive programming and data analysis using a JupyterLab server
• Publications
• Autumn 2024 Upgrades
  • There’s a new data centre?
  • When will the cluster move?
  • What happens to the old cluster?
  • What new equipment will we have?
  • What’s the plan for the new storage?
  • Any other changes I should be aware of?
  • Who is funding this?

The legal stuff

• Supported Organisations
• Service Level Agreement
• Acceptable Use Policy
• GDPR Privacy Notice

Note

Use the Edit on GitHub links to make or suggest changes or additions to this help; new or updated pages are synced every five minutes.