Crop Diversity HPC Help

Hello! Here you’ll find the documentation for the UK’s Crop Diversity Bioinformatics High Performance Computing (HPC) Linux cluster - gruffalo - and its associated data storage and services.

Important

Run by the James Hutton Institute’s Information & Computational Sciences (ICS) Research Computing team, and the Scientific Computing section of NIAB’s IT team, this resource supports the computational needs of researchers from six UK institutions. See Supported Organisations for details.

Note

Compute clusters contain a lot of CPUs, so it’s worth seeing how we’re Counting CPUs.

Introduction

• Frequently Asked Questions
• System Overview
• Performance

Getting started

• Requesting an Account
• Getting Connected

High performance computing

• Slurm - Overview
  • Jobs
  • Queues (partitions)
  • Interactive jobs
  • Batch jobs
  • Array jobs
  • Allocating resources
  • Cancelling a job
  • Scratch space
  • Job summaries
  • Automatic requeuing
  • Other helpful parameters
• Slurm - Queue Policies & Advice
  • Specifying queues
  • Fair share policy
  • Additional advice and guidance
• Slurm - Shortcuts and Aliases
  • srsh
  • snodes
  • sjobacct
  • sjobstat
• Bioconda
  • Installing Bioconda
  • Finding packages
  • Installing packages
  • Listing packages
  • Removing packages
  • Environments
  • Removing Bioconda
  • Bioconda and Slurm
• Compiling Software
• Working with Compressed Files
  • Compressing files
  • Decompressing files
  • Leaving files compressed
• Apptainer (Singularity)
  • Running containers
  • Using aliases
  • Building containers
• GPU Processing
• Machine Learning
• Tools & Applications
  • Aspera
  • Bioconda
  • Databases
• Database Mirrors
  • NCBI BLAST
  • Pfam
  • Uniprot
• OpenMPI
  • How to submit MPI jobs
• Green Computing
  • How to tell if a node is powered down
  • When will nodes power up/down?
  • How much energy will this save?
  • Our commitment to Green Computing
  • What can I do to help?

Data storage

• Data Storage & Management Policy
  • Storage at-a-glance
  • Data management
  • BeeGFS storage
  • Local scratch
• Backups
  • Retention policies
  • Restoring backups
• File Transfers
  • Command line options
  • Graphical options
• Samba Access
  • Account details
  • Connection details
  • Shared folders
  • Access from macOS

Miscellaneous

• Contact Us
• Monitoring & Tracking
  • Real-time metrics
  • Disk-usage summaries
  • Login attempts
• Linux Basics
  • About Linux
  • The terminal/shell
• Tips & Tricks
  • Special keys
  • History
  • Clearing the screen
  • Running multiple commands
  • Following log files
  • Interactive programming and data analysis using a JupyterLab server
• Publications
• Cluster Changes
  • A new network
  • Debian Linux
  • New compute hardware
  • New storage
  • Fair share policies
  • Scratch auto-purge
  • Scratch…is gone?
  • A new location for bioconda (and apps)
  • Updates to SLURM
  • Queue changes
  • GPU drivers and CUDA toolkit
  • Missing compute hardware
  • Better UPS coverage
  • JHI archive

The legal stuff

• Supported Organisations
• Service Level Agreement
• Acceptable Use Policy
• GDPR Privacy Notice

Note

Use the Edit on GitHub links to make or suggest changes or additions to this help; new or updated pages are synced every five minutes.