The first step is to upload the install files by following the steps in ‘Creating a S3 bucket’ (or alternatively you can use Git or wget to pull installation files from online). The following example is assuming OpenFOAM v2012 which was downloaded from here, however a similar process could be followed for other versions of OpenFOAM.
Please note that the same steps can be taken whether on an Intel/AMD cluster or if you followed the Arm cluster creation installations as OpenFOAM v2012 supports Arm natively.
Assuming you’ve done that, lets connect to the cluster:
pcluster ssh cfd -i ~/cfd_ireland.pem
Then lets check what’s in our S3 bucket:
aws s3 ls bucketname
You should see the contents of the bucket listed which should include the installation files. Proceed to type the following commands which will download the installation file from your S3 bucket to the cluster.
Lets firstly move to our FSx disk by typing:
Then lets create an OpenFOAM folder
mkdir /fsx/OpenFOAM/ cd /fsx/OpenFOAM/
Then we can copy the installation file from our S3 bucket there:
aws s3 cp s3://bucketname/OpenFOAM-v2012.tgz . aws s3 cp s3://bucketname/ThirdParty-v2012.tgz .
Alternatively for OpenFOAM v2012 you could do:
wget https://sourceforge.net/projects/openfoam/files/v2012/OpenFOAM-v2012.tgz wget https://sourceforge.net/projects/openfoam/files/v2012/ThirdParty-v2012.tgz
We then need to untar them both:
tar -xf OpenFOAM-v2012.tgz tar -xf ThirdParty-v2012.tgz
Finally we will navigate to the installation folder:
Next we’ll change the installation location as we’ll install it in the /fsx/OpenFOAM folder rather than the default $HOME directory.
Now edit the bashrc file and on line 115 change from:
The compilation will take more than 5 hours on just 2 CPUs on the headnode so instead we can compile on the compute nodes using 36 or 64 cores depending on whether you use x86 (Intel/AMD) or Arm (Graviton 2) cluster.
We will cover more about the cluster submission system in the next section so for now just create the following file using a text editor of your choice. Note that we are using OpenMPI 4.1, which is the default for ParallelCluster. It is possible to use IntelMPI, however OpenFOAM requires modifications to various compiler options so for simplicity we will use OpenMPI for this workshop.
#!/bin/bash #SBATCH --job-name=foam-64 #SBATCH --ntasks=64 #SBATCH --output=%x_%j.out #SBATCH --partition=compute #SBATCH --constraint=c6gn.16xlarge module load openmpi source /fsx/OpenFOAM/OpenFOAM-v2012/etc/bashrc export WM_NCOMPPROCS=64 cd /fsx/OpenFOAM/OpenFOAM-v2012/ ./Allwmake > log.dat
#!/bin/bash #SBATCH --job-name=foam-36 #SBATCH --ntasks=36 #SBATCH --output=%x_%j.out #SBATCH --partition=compute #SBATCH --constraint=c5n.18xlarge module load openmpi source /fsx/OpenFOAM/OpenFOAM-v2012/etc/bashrc export WM_NCOMPPROCS=36 cd /fsx/OpenFOAM/OpenFOAM-v2012/ ./Allwmake > log.dat
Just submit this to the cluster using the following command which should take approximately 30 minutes:
You should see something like what is shown below at the end of the ‘log’ script
2021-01-06 08:36:05 +0000 ======================================== Finished compile of visualization with OpenFOAM-v2012 Gcc system compiler linuxARM64GccDPInt32Opt, with SYSTEMOPENMPI sys-openmpi 2021-01-06 08:36:05 +0000 ======================================== OpenFOAM-v2012 Gcc system compiler linuxARM64GccDPInt32Opt, with SYSTEMOPENMPI sys-openmpi api = 2012 patch = 0 bin = 308 entries lib = 148 entries ========================================