Video: https://youtu.be/qxF7a246t_I
Description: Discover the power of High Performance Computing (HPC) with CoCalc, offering an integrated environment to harness high-performance compute servers effectively. This tutorial guides you through using CoCalc's HPC capabilities, featuring tools like Slurm, the Intel HPC Toolkit, Clang, and GFortran. Learn how to select and configure the appropriate compute server through a detailed step-by-step process, write and compile Fortran code, and execute parallel processes with ease. Suitable for both beginners exploring cost-effective options and seasoned researchers needing robust computing power, this demo provides practical insights into setting up a virtual machine on Google Cloud, optimizing costs, and streamlining your HPC tasks using familiar interfaces such as VS Code and JupyterLab.
CoCalc compute servers have a High Performance Computing (HPC) image, which includes Slurm, the Intel HPC Toolkit, clang, gfortran, and other compilers.
To use it, click the "HPC/Fortran" template:
Or select the HPC image on Google Cloud:
The HPC template by default uses the h3-standard-88 instance type which has 88 CPU's, 352GB of RAM, and each vCPU is a true CPU. This machine type is highly optimized for HPC tasks. It is also not cheap, so if you're on a budget or just exporing, select any other machine type. E.g., if you type "cpu:8" into th Machine Type box, you'll see all options sorted by price with 8 vCPU, which should be enough for a quick tutorial:
I'm selecting n2d-highcpu-8, and also up higher, switching the Provisioning to Spot, since we're just doing a tutorial:
Now this will only cost $0.09/hour! You can also switch the region to us-east5, and it's even cheaper at $0.07/hour!
I'm also enabling DNS so that we can easily use VS Code and JupyterLab directly on the compute server:
Optional: It can be nice for HPC work to avoid using a networked filesystem at all. Thus we'll edit code and compile it in a directory that only exists on the compute server. Add the directory "hpc-demo" to the "Fast Local Directories":
Then hit enter:
Finally, let's start it up, which should take about 2 minutes:
And here it goes:
You can click "Serial" on the left to watch the boot process if you want:
Once it is fully running and all the progress bars are at 100%, let's write some Fortran code:
In the File exploer set the server in the upper left to our compute server:
See
Then navigate to hpc-demo, and click +New --> binary_tree.f90 and paste this Fortran code in:
To compile our code, click "+New --> Linux Terminal", and set the terminal to run on the compute server:
Now use the Intel Fortran Compiler by typing ifx binary_tree.f90 :
And run the code:
This compute server has all the standard Intel HPC tools installed, and also Clang and GNU compilers. We can also use gfortran to compile the same code by typing gfortran binary_tree.f90 :
Also, the Slurm Workload Manager is installed and configured to run as many processes in parallel as vCPU's that your compute server has. Let's run our Fortran program 8 times via srun -n 8 ./a.out
Below we verify that it indeed did run 8 times.
As of May 2024 CoCalc doesn't explicitly support clustering. However, Google Cloud VM's offer over 400 vCPU per machine, so for a large number of applications using a cluster isn't necessary. Spin up a VM for $5-$10/hour, run it for a day, and get your HPC research done now!
Finally, if you're more used to VS Code than CoCalc as environment, it's easy to run directly on the compute server, by clicking the VS Code button to the left of the compute server or via the menu in the upper right. Similar for JupyterLab.