OpenFOAM Installation

The first step is to upload the install files by following the steps in ‘Creating a S3 bucket’ (or alternatively you can use Git or wget to pull installation files from online). The following example is assuming OpenFOAM v2012 which was downloaded from here, however a similar process could be followed for other versions of OpenFOAM.

Please note that the same steps can be taken whether on an Intel/AMD cluster or if you followed the Arm cluster creation installations as OpenFOAM v2012 supports Arm natively.

Assuming you’ve done that, lets connect to the cluster:

 pcluster ssh cfd -i ~/cfd_ireland.pem

Then lets check what’s in our S3 bucket:

 aws s3 ls bucketname

You should see the contents of the bucket listed which should include the installation files. Proceed to type the following commands which will download the installation file from your S3 bucket to the cluster.

Lets firstly move to our FSx disk by typing:

 cd /fsx/

Then lets create an OpenFOAM folder

 mkdir /fsx/OpenFOAM/
 cd /fsx/OpenFOAM/

Then we can copy the installation file from our S3 bucket there:

 aws s3 cp s3://bucketname/OpenFOAM-v2012.tgz .
 aws s3 cp s3://bucketname/ThirdParty-v2012.tgz .

Alternatively for OpenFOAM v2012 you could do:

 wget https://sourceforge.net/projects/openfoam/files/v2012/OpenFOAM-v2012.tgz 
 wget https://sourceforge.net/projects/openfoam/files/v2012/ThirdParty-v2012.tgz

We then need to untar them both:

 tar -xf OpenFOAM-v2012.tgz
 tar -xf ThirdParty-v2012.tgz

Finally we will navigate to the installation folder:

 cd OpenFOAM-v2012

Next we’ll change the installation location as we’ll install it in the /fsx/OpenFOAM folder rather than the default $HOME directory.

 cd /fsx/OpenFOAM/OpenFOAM-v2012/etc

Now edit the bashrc file and on line 115 change from:

 projectDir="$HOME/OpenFOAM/OpenFOAM-$WM_PROJECT_VERSION"

to

 projectDir="/fsx/OpenFOAM/OpenFOAM-$WM_PROJECT_VERSION"

Compilation

The compilation will take more than 5 hours on just 2 CPUs on the headnode so instead we can compile on the compute nodes using 36 or 64 cores depending on whether you use x86 (Intel/AMD) or Arm (Graviton 2) cluster.

We will cover more about the cluster submission system in the next section so for now just create the following file using a text editor of your choice. Note that we are using OpenMPI 4.1, which is the default for ParallelCluster. It is possible to use IntelMPI, however OpenFOAM requires modifications to various compiler options so for simplicity we will use OpenMPI for this workshop.

 vi compile.sh

Arm cluster:

#!/bin/bash
#SBATCH --job-name=foam-64
#SBATCH --ntasks=64
#SBATCH --output=%x_%j.out
#SBATCH --partition=compute
#SBATCH --constraint=c6gn.16xlarge
module load openmpi
source /fsx/OpenFOAM/OpenFOAM-v2012/etc/bashrc
export WM_NCOMPPROCS=64
cd /fsx/OpenFOAM/OpenFOAM-v2012/
./Allwmake > log.dat

x86 cluster

#!/bin/bash
#SBATCH --job-name=foam-36
#SBATCH --ntasks=36
#SBATCH --output=%x_%j.out
#SBATCH --partition=compute
#SBATCH --constraint=c5n.18xlarge
module load openmpi
source /fsx/OpenFOAM/OpenFOAM-v2012/etc/bashrc
export WM_NCOMPPROCS=36
cd /fsx/OpenFOAM/OpenFOAM-v2012/
./Allwmake > log.dat

Just submit this to the cluster using the following command which should take approximately 30 minutes:

 sbatch compile.sh

You should see something like what is shown below at the end of the ‘log’ script

2021-01-06 08:36:05 +0000
========================================
Finished compile of visualization with OpenFOAM-v2012
  Gcc system compiler
  linuxARM64GccDPInt32Opt, with SYSTEMOPENMPI sys-openmpi


2021-01-06 08:36:05 +0000
========================================
  OpenFOAM-v2012
  Gcc system compiler
  linuxARM64GccDPInt32Opt, with SYSTEMOPENMPI sys-openmpi

  api   = 2012
  patch = 0
  bin   = 308 entries
  lib   = 148 entries

========================================