Running examples

Go to one of the subdirectories of the Example folders, for example, BinaryBH and type:

make all

Assuming this compiles with no errors, an executable will have been produced with a name which depends on your compiler options but will look something like:

GRChombo_BinaryBH3d.Linux.64.mpiicpc.ifort.DEBUG.OPT.MPI.OPENMPCC.ex

Normally you will not want to output the resulting hdf5 files to your home directory (where you would normally store the code), so you should amend the following lines in the params.txt file to give the (full, not relative) path to your scratch directory, where large temporary output files are stored, e.g.

chk_prefix = /scratch/myname/BinaryBHchk_
plt_prefix = /scratch/myname/BinaryBHplt_

The executable can be run in parallel (e.g. on 4 processors) using a command like

mpirun -np 4 ./GRChombo_BinaryBH3d.Linux.64.mpiicpc.ifort.DEBUG.OPT.MPI.OPENMPCC.ex params.txt

However, usually on a cluster you are required to submit a batch job, so that the simulation is queued and run on the compute nodes. So you would write a jobscript and submit that to the queue. There should be examples for MPI and hybrid OpenMP/MPI jobs in your cluster documentation. Note that you may also need to reload the module files within the jobscript, depending on how your cluster works. Some examples for systems we use can be found here. The number of cores you need will depend on the job and the memory per core on your cluster. If you run out of memory, you may want to switch to using several OpenMP threads per task, rather than running a pure MPI job. You submit the job with a command which depends on the scheduler, again for example:

sbatch jobscript.txt

This should produce a lot of hdf5 files, which you can then view to check what has been simulated (see Visualising outputs).

Note that your job will output files like pout.0 for each processor. Looking at these files during the run allows you to see how fast it is going. More output can be produced by setting

verbose = 1

in the params.txt file. These files will also tell you how many boxes are being processed by each processor - for good load balancing and to avoid using more processors than you need, ideally all the processors should have one (or several) boxes each, and there should be at least one per processor.

Note that Chombo does not adjust the box sizes automatically based on the number of processes you choose so you almost certainly need to adjust them for your specific application and job configuration to avoid wasting resources and/or achieve good speeds.

More information on load balancing is given in the section on performance optimisation. This is highly recommended reading.

For a more specific example of setting up a run and the outputs, see Running the BinaryBH example.