Crop Diversity HPC Help

Hello! Here you’ll find the documentation for the UK’s Crop Diversity Bioinformatics High Performance Computing (HPC) Linux cluster - gruffalo - and its associated data storage and services.

Important

Run by the James Hutton Institute’s Information & Computational Sciences (ICS) Research Computing team, this resource supports the computational needs of researchers from seven UK institutions. See Supported Organisations for details.

Note

Compute clusters contain a lot of CPUs, so it’s worth seeing how we’re Counting CPUs.

Introduction

• Frequently Asked Questions
• System Overview
• Performance

Getting started

• Requesting an Account
• Getting Connected
• Password Reset

High performance computing

• Slurm - Overview
  • Jobs
  • Queues (partitions)
  • Interactive jobs
  • Batch jobs
  • Array jobs
  • Allocating resources
  • Cancelling a job
  • Scratch space
  • Job summaries
  • Automatic requeuing
  • Other helpful parameters
• Slurm - Queue Policies & Advice
  • Specifying queues
  • Fair share policy
  • Additional advice and guidance
• Slurm - Shortcuts and Aliases
  • srsh
  • snodes
  • sjobacct
  • sjobstat
• Bioconda
  • Installing Bioconda
  • Finding packages
  • Installing packages
  • Listing packages
  • Removing packages
  • Environments
  • Removing Bioconda
  • Bioconda and Slurm
• Bioconda & R
• Compiling Software
• Working with Compressed Files
  • Compressing files
  • Decompressing files
  • Leaving files compressed
• Apptainer (Singularity)
  • Running containers
  • Using aliases
  • Building containers
• GPU Processing
• Machine Learning
• Tools & Applications
  • Aspera
  • Bioconda
  • Databases
• Database Mirrors
  • NCBI BLAST
  • Pfam
  • Uniprot
• OpenMPI
  • How to submit MPI jobs
• Green Computing
  • How to tell if a node is powered down
  • When will nodes power up/down?
  • How much energy will this save?
  • Our commitment to Green Computing
  • What can I do to help?

Data storage

• Data Storage & Management Policy
  • Storage at-a-glance
  • Data management
  • NFS storage
  • BeeGFS storage
  • Local scratch
• Backups
  • Retention policies
  • Restoring backups
  • Archived data
• File Transfers
  • Command line options
  • Graphical options
• Samba Access
  • Account details
  • Connection details
  • Shared folders
  • Access from macOS

Miscellaneous

• Contact Us
• Monitoring & Tracking
  • Real-time metrics
  • Disk-usage summaries
  • Login attempts
• Linux Basics
  • About Linux
  • The terminal/shell
• Tips & Tricks
  • Special keys
  • History
  • Clearing the screen
  • Running multiple commands
  • Following log files
  • Interactive programming and data analysis using a JupyterLab server
• Publications
• Cluster Changes
  • A new network
  • Debian Linux
  • New compute hardware
  • New storage
  • Fair share policies
  • Scratch auto-purge
  • Scratch…is gone?
  • A new location for bioconda (and apps)
  • Updates to SLURM
  • Queue changes
  • GPU drivers and CUDA toolkit
  • Missing compute hardware
  • Better UPS coverage
  • JHI archive

The legal stuff

• Supported Organisations
• Service Level Agreement
• Acceptable Use Policy
• Commercial Access T’s & C’s
• GDPR Privacy Notice

Note

Use the Edit on GitHub links to make or suggest changes or additions to this help; new or updated pages are synced every five minutes.