Modifying and Rebuilding PETSc and petsc4py in Firedrake

By default, every firedrake installation comes with a clone of the firedrake project fork of petsc in [firedrake-install-dir]/src/petsc on the firedrake branch.

Rebuilding PETSc

It is possible to make experimental changes to the firedrake installation of PETSc, though PETSc must be rebuilt for the changes to have the desired effect.

The time consuming way to do this is to run firedrake-update --rebuild. The PETSc mirror will be updated and completely rebuilt - often taking over an hour.

Note: If the PETSc ABI has changed (in principal whenever the PETSc release version's second number has changed) firedrake-update --rebuild must be done.

Most of the time though, you can speed things up a lot with the below procedure

1. Make desired modifications
2. Activate the firedrake venv
3. Navigate to [firedrake-install-dir]/src/petsc
4. Run make PETSC_DIR=[firedrake-install-dir]/src/petsc PETSC_ARCH=default all. This should take less than 10 minutes.
5. Run the suggested make PETSC_DIR=[firedrake-install-dir]/src/petsc PETSC_ARCH=default check check (or similar - the log text from successfully performing step 4 should tell you the exact command) to make sure the libraries are working. This should take less that 2 minutes.

Troubleshooting Notes

If the desired modifications (step 1) are on a separate branch

The branch used must have a working remote (such that git pull will work) or else firedrake-update will fail.

If make (step 4.) fails

Try the following:

1. Remove the regularly troublesome PASTIX headers with rm -rf default/include/pastix*.h in the [firedrake-install-dir]/src/petsc directory
2. Run make reconfigure - warning! This may take up to an hour
3. Run the make ... command recommended in the log text from make reconfigure (it will probably be make PETSC_DIR=[firedrake-install-dir]/src/petsc PETSC_ARCH=default all)

If you have modified the API and therefore need to update petsc4py

See below

Updating petsc4py

If local changes are made to petsc4py the following will update firedrake to reflect those changes:

1. Activate the firedrake venv
2. Navigate to [firedrake-install-dir]/src/petsc.
3. Set the PETSC_DIR and PETSC_ARCH environment variables with the export PETSC_DIR=[firedrake-install-dir]/src/petsc and export PETSC_ARCH=default commands
4. Run pip install -U --no-deps src/binding/petsc4py petsc4py should now be updated.

Notes

You may find that pip uses a cached petsc4py build instead of rebuilding from scratch. To resolve this you can run rm -rf $VIRTUAL_ENV/src/petsc/src/binding/petsc4py/build prior to pip install <args>.