MPI documentation

Need a section on using MPI

• Recommend exclusive node use
• Using --contraint=ib and export FI_PROVIDER=verbs
• Using export I_MPI_PMI_LIBRARY=/usr/lib/x86_64-linux-gnu/libpmi2.so.0 for Intel MPI
• Using srun/mpirun/mpiexec

Also gathered wisdom on other implementations.

from @gstuart :

conda installs its own MPI which is not built with Slurm integration. If this is a single node job, you can use mpirun  instead of srun and just use the conda mpi