Running OpenFOAM

Firstly follow the instructions in ‘3. Create a HPC cluster’ to SSH to the cluster:

 pcluster ssh cfd -i cfd_ireland.pem

Move simulation file from S3 bucket to the working directory

First of all lets copy an OpenFOAM case you may have created previously. An example 4M motorbike cell case can be found here for x86 (e.g Intel/AMD build) and an Arm (Graviton) version, both together with sample submission scripts if you don’t have a case. It’s unmeshed so less than 1MB, so we therefore need to mesh and run it. Follow the instructions in ‘Creating a S3 Bucket’ to upload the case to S3.

 cd /fsx

Next lets copy from S3 an OpenFOAM setup (e.g motorBikeDemo.tgz) that you’ve uploaded.

 aws s3 cp s3://yourbucketname/motorBikeDemo.tgz .

Lets untar the case:

 tar -xf motorBikeDemo.tgz

Finally, move to the motorBikeDemo folder

cd motorBikeDemo

You’ll see we have a complete OpenFOAM case together with an example submission script. The test-case is a modified version of the standard OpenFOAM motorbike tutorial which will create a mesh of approximately 4M cells.

As you have a small head node of just 2 vCPUs and 4GB RAM, you want to do the meshing and solving via the compute or mesh queue. Lets explore how to do this.

Create a Slurm submit script

We have created a HPC cluster using AWS ParallelCluster that is based upon the Slurm scheduler. You can read more about Slurm here but in essence it allows us to submit the case to an arbitary number of cores and Slurm/AWS ParallelCluster will do the scheduling for you.

The first step is to run a submission script. An example setup script is provided in submit.sh

Please note that to use the Arm cluster, please just change the ntasks to be 128 and the instance type to be c6gn.16xlarge. Making sure to also change the decomposePar dictionaries to match.

 vi submit.sh
#!/bin/bash
#SBATCH --job-name=foam-108
#SBATCH --ntasks=108
#SBATCH --output=%x_%j.out
#SBATCH --partition=compute
#SBATCH --constraint=c5n.18xlarge

module load openmpi
source /fsx/OpenFOAM/OpenFOAM-v2012/etc/bashrc

cp $FOAM_TUTORIALS/resources/geometry/motorBike.obj.gz constant/triSurface/
surfaceFeatureExtract  > ./log/surfaceFeatureExtract.log 2>&1
blockMesh  > ./log/blockMesh.log 2>&1
decomposePar -decomposeParDict system/decomposeParDict.hierarchical  > ./log/decomposePar.log 2>&1
mpirun -np $SLURM_NTASKS snappyHexMesh -parallel -overwrite -decomposeParDict system/decomposeParDict.hierarchical   > ./log/snappyHexMesh.log 2>&1
mpirun -np $SLURM_NTASKS checkMesh -parallel -allGeometry -constant -allTopology -decomposeParDict system/decomposeParDict.hierarchical > ./log/checkMesh.log 2>&1
mpirun -np $SLURM_NTASKS redistributePar -parallel -overwrite -decomposeParDict system/decomposeParDict.ptscotch > ./log/decomposePar2.log 2>&1
mpirun -np $SLURM_NTASKS renumberMesh -parallel -overwrite -constant -decomposeParDict system/decomposeParDict.ptscotch > ./log/renumberMesh.log 2>&1
mpirun -np $SLURM_NTASKS patchSummary -parallel -decomposeParDict system/decomposeParDict.ptscotch > ./log/patchSummary.log 2>&1
ls -d processor* | xargs -i rm -rf ./{}/0
ls -d processor* | xargs -i cp -r 0.orig ./{}/0
mpirun -np $SLURM_NTASKS potentialFoam -parallel -noFunctionObjects -initialiseUBCs -decomposeParDict system/decomposeParDict.ptscotch > ./log/potentialFoam.log 2>&1s
mpirun -np $SLURM_NTASKS simpleFoam -parallel  -decomposeParDict system/decomposeParDict.ptscotch > ./log/simpleFoam.log 2>&1

In this script, we specify the number of cores e.g tasks to be 108 (x3 c5n.18xlarge). The partition line refers to the queues we created previously e.g compute and mesh. The constraint part is only needed if you have multiple compute options for each queue, however we could remove it given we have only one per queue.

The rest of the script is just to load the MPI version and source the OpenFOAM installation. Finally we have typically OpenFOAM commands where the number of cores is taken from the SBATCH lines at the top of the script.

Submitting jobs to the scheduler

Use sbatch command to submit the first script.

sbatch submit.sh

Other useful Slurm commands:

  • squeue – shows the status of all running jobs in the queue.
  • sinfo – shows partition and node information for a system
  • srun – run an interactive job
  • scancel jobid – kill a Slurm job

If you type squeue you should initially see the following which states that it’s in a queue.

[ec2-user@ip-10-0-0-22 motorBikeDemo]$ squeue
             JOBID PARTITION     NAME     USER ST       TIME  NODES NODELIST(REASON) 
                 7   compute  foam-108 ec2-user CF       0:03      3 compute-dy-c5n18xlarge-[1-3]

In the back-end this is now sending a signal for EC2 instances to be created (which you can see via your EC2 console). It should take around 4-5 minutes for these to be launched. If you do not see this move to ‘R’ for running within 5 minutes, check your EC2 console and make sure you have requested an increase to your Service Quotas as described in previous sections.

You should now see:

[ec2-user@ip-10-0-0-22 motorBikeDemo]$ squeue
             JOBID PARTITION     NAME     USER ST       TIME  NODES NODELIST(REASON)
                 7   compute  foam-108 ec2-user R       0:05      3 compute-dy-c5n18xlarge-[1-3]

The case is setup to write out the various logs in the ‘log’ folder and it should take approximately 5 minutes to go through the meshing and solving process.

The mesh is approximately 4M cells in this example and therefore running across 2 nodes is probably already at the limit of scalability. You can see more information about the scaling of OpenFOAM on AWS in the benchmark section.

You can tar up these results (e.g postProcessing section) and bring them back to your machine. This is a typical workflow where end-users only need to bring back the end post-processing results rather than the whole simulation files.

 tar -czvf results.tgz postProcessing

We can then copy it to S3:

 aws s3 cp results.tgz s3://bucketname/

When you are finished make sure you then do the following, where the JOBID, can be seen by firstly typing ‘squeue’. If its empty then nothing is running:

 scancel JOBID

This was just a basic example of OpenFOAM but hopefully it gives you an example of how it can be used on AWS.