Use and install PETSc

This is a miscellaneous page of tips and advice on using and installing PETSc for use with the AH finder.

Dial3 (E. de Jong)

I've installed PETSc on dial3 in our shared scratch space, so it should be ready to use for anyone that's on our project. For future reference, I'll just describe how you link it to Chombo and compile your example.

1. You should add the following lines to ~/.bashrc to signal where the PETSc installation is:

export PETSC_DIR=/scratch/dp092/shared/pe-scripts/petsc-3.14.4
export PETSC_ARCH=x86-rome
export PETSC_INSTALL=/scratch/dp092/shared/petsc_install

2. You can compile Chombo with the following Make.defs.local: most important change are the explicit links to petsc, its dependencies and the mpifort library:

DIM           = 3
CXX           = CC
FC            = ftn
OPT           = HIGH
DEBUG         = FALSE
MPI           = TRUE
OPENMPCC      = TRUE
USE_64         = TRUE
USE_HDF        = TRUE
MPICXX         = CC
HDFINCFLAGS    = -I/opt/cray/pe/hdf5-parallel/1.12.0.2/CRAYCLANG/10.0/include
HDFLIBFLAGS    = -L/opt/cray/pe/hdf5-parallel/1.12.0.2/CRAYCLANG/10.0/lib -lhdf5 -lz
HDFMPIINCFLAGS = -I/opt/cray/pe/hdf5-parallel/1.12.0.2/CRAYCLANG/10.0/include
HDFMPILIBFLAGS = -L/opt/cray/pe/hdf5-parallel/1.12.0.2/CRAYCLANG/10.0/lib -lhdf5 -lz
USE_MT         = FALSE
cxxoptflags    = -g -O3
foptflags      = -O3 -hfp3 -h omp
XTRACXXFLAGS   = -craype-verbose -Wno-inconsistent-missing-override
syslibflags    = -L/opt/cray/pe/libsci/21.04.1.1/CRAY/9.0/x86_64/lib -lsci_cray -lmpifort -L${PETSC_INSTALL}/lib -lmetis -lparmetis -lsuperlu -lsuperlu_dist -lptscotch -lptscotcherr -lptscotcherrexit -lptscotchparmetis -lscotch -lscotcherr -lscotcherrexit -lscotchmetis -lpord -lcmumps -ldmumps -lesmumps -lmumps_common -lptesmumps -lsmumps -lzmumps -lpetsc
cxxcppflags   := ${cxxcppflags} -DUSE_AHFINDER

3. Finally, you can compile your example with the following Makefile (make sure you've set CHOMBO_HOME in your ~/.bashrc as usual): most important change is addition of PETSc include directory to src_dirs:

# -*- Mode: Makefile -*-

### This makefile produces an executable for each name in the `ebase'
###  variable using the libraries named in the `LibNames' variable.

# Included makefiles need an absolute path to the Chombo installation
# CHOMBO_HOME := Please set the CHOMBO_HOME locally (e.g. export CHOMBO_HOME=... in bash)

GRCHOMBO_SOURCE = ../../Source

ebase := Main_ScalarField

LibNames := AMRTimeDependent AMRTools BoxTools

src_dirs := $(GRCHOMBO_SOURCE)/utils \
            $(GRCHOMBO_SOURCE)/simd  \
            $(GRCHOMBO_SOURCE)/CCZ4  \
            $(GRCHOMBO_SOURCE)/Matter  \
            $(GRCHOMBO_SOURCE)/BoxUtils  \
            $(GRCHOMBO_SOURCE)/GRChomboCore  \
            $(GRCHOMBO_SOURCE)/TaggingCriteria  \
            $(GRCHOMBO_SOURCE)/AMRInterpolator  \
            $(GRCHOMBO_SOURCE)/ApparentHorizonFinder  \
            $(GRCHOMBO_SOURCE)/InitialConditions/BlackHoles \
            $(GRCHOMBO_SOURCE)/BlackHoles \
	    $(PETSC_INSTALL)/include

include $(CHOMBO_HOME)/mk/Make.test

Using PETSc with Mac (R. Croft)

This was done using macOS BigSur but may work for other macOS. Official PETSc install website.

Download PETSc from github

git clone -b release https://gitlab.com/petsc/petsc.git petsc
git pull # obtain new release fixes (since a prior clone or pull)

Configure PETSc (from the home of PETSc installation)

./configure --with-cc=<mpi wrapper to C compiler> --with-cxx=<mpi wrapper to C++ compiler> --with-fc=<mpi wrapper to Fortran compiler> --with-hdf5-dir="my_hdf5_path"

for example

./configure --with-cc=mpicc --with-cxx=mpicxx --with-fc=mpifort --with-hdf5-dir=/usr/local/Cellar/hdf5/1.12.2_2

Successful configuration should give PETSC_DIR and PETSC_ARCH paths. PETSC_DIR is the home directory of the PETSc repository. Set both PETSC_DIR and PETSC_ARCH as environment variables in one of ~/.bashrc, ~/.bash_profile, ~/.zshrc or ~/.zsh_profile depending which shell you use (zshell is defualt on macOS Catalina / 10.15 or newer)

Install PETSc with make

make all 

or

make PETSC_DIR=${PETSC_DIR} PETSC_ARCH=${PETSC_ARCH} all

where the second option is only required if PETSC_DIR and PETSC_ARCH have not been set as environment variables.

Check PETSc libraries are working with

make check 

or

make PETSC_DIR=${PETSC_DIR} PETSC_ARCH=${PETSC_ARCH} check

where the second option is only required if PETSC_DIR and PETSC_ARCH have not been set as environment variables.

Change the Make.defs.local file in $CHOMBO_HOME ( = .../Chombo/lib). Make sure to include

• syslibflags = ${MPI_LINKFLAGS} -lblas -llapack -L${PETSC_DIR}/${PETSC_ARCH}/lib -lpetsc -framework Accelerate
• cxxcppflags := ${cxxcppflags}
• XTRACPPFLAGS = -DUSE_AHFINDER -I${PETSC_DIR}/include -I${PETSC_DIR}/${PETSC_ARCH}/include

Instead of adding the -I include flags here the directories could be included in the GNUmakefile in the GRChombo Example directory in the usual way.

Now compile compile Chombo in the usual way from $CHOMBO_HOME ( = .../Chombo/lib)

make realclean
make lib -j 8

Make GRChombo the usual way (again using the make realclean option before make all).