PGI MPI Questions

mpirun -np 4 results in the master node and not all the slave nodes running

When we run the test programs from the bench directory using mpirun -np 4, the master node takes one of the processes and one of the workers takes the other 3 processes. The other 2 workers never take on part of the load.

How can we fix this?

If a master node is not used in computation, then to use mpirun make sure the following file: 

$PGI/linux86/util/machines/machines.LINUX

contains the names of all the worker nodes, but not the name of the master node.

Then, if starting a process from the master, add the '-nolocal' switch to the 'mpirun' command line to direct mpirun to not run the executable on the local machine.

How do I use my own version of mpi libs with pghpf?

Obtain the file mpi.c from ftp.pgroup.com/x86/3.1/linux86-patches/mpi/mpi.c

Compile mpi.c -c with pgcc, making sure the correct mpi.h file is used. The PGI version of mpi.h is found in $PGI/linux86/include on the CDK product.

If linking in mpi.o with your pghpf application works for you, you can use ar to replace mpi.o in libpghpf_mpi.a and libpghpf_mpi_p.a.

Could you go through the steps to build MPICH 1.2.0?

Here is the detailed directions to build MPI 1.2.0 and install it in a $PGI directory: Building MPICH 1.2.0 for Linux system with glibc version 212 or 213, and PGI release 3.2 (with the -comm=shared option)
  1. Untar mpich 1.2.0 into a directory which we will call $MPICH Distribution can be obtained from www.mcs.anl.gov/mpi/mpich. These instructions would need to be modified for the latest 1.2.1 version, which we did not build for the 3.2 PGI compiler release.

  2. Make a pgiconfig file in $MPICH which contains: 

    ./configure -c++=pgCC -cc=pgcc -fc=pgf77 -cflags="-Msignextend -tp px 
    -DUSE_U_INT_FOR_XDR -DHAVE_RPC_RPC_H=1" -opt=-fast
-fflags="-tp px" -c++flags="-tp px" -f90flags="-tp px" -f90=pgf90 
    -prefix=/usr/local/mpich -comm=shared sed -e 's@MPIR_HOME   
    = .*$@MPIR_HOME   = $$\\PGI/linux86@' Makefile > Makefile.pgi

  3. Run pgiconfig% >& pgiconfig.log

  4. Build MPICH using the following commands: 

            % make clean
        % make mpi >& Make.log

  5. Edit the file $MPICH/bin/mpirun and change the definition of MPIR_HOME on line 22 to "MPIR_HOME=$PGI/linux86", and delete lines 34, 33, 32, and 30; edit $MPICH/bin/mpirun.args and change the definition of EXECER on line 8 to "EXECER=$PGI/linux86/util/execer", and delete lines 13, 12, 11 and 9, and then add the following lines after line 36 (i.e. line 36 after the prior deletions): 

       if [ "$PBS_NODEFILE" != "" ]
   then
       machineFile=$PBS_NODEFILE
   fi

    Edit $MPICH/bin/mpirun.ch_p4 and change the definition of MPIR_HOME on line 5 to "MPIR_HOME=$PGI/linux86", and delete lines 14, 13, 12, and 10.

    Edit $MPICH/bin/mpirun.pg: line 78 to "machineDir=${MPIR_HOME}/util/machines" line 91 to "echo directory ${MPIR_HOME}/util/machines."


  6. You can re-install MPICH in your $PGI install tree manually by following these instructions:
    To install MPICH in the $PGI directory tree, the following files must be pulled from the $MPICH/bin directory into $PGI/linux86/bin: 

            mpirun
        mpirun.args
        mpirun.ch_p4
        mpirun.pg
        tarch
        tdevice

    the following files from the $MPICH/mpe directory into $PGI/linux86/include: 

    basex11.h
        mpe.h
        mpe_graphics.h
        mpe_log.h
        mpeconf.h
        mpeexten.h
        mpetools.h

    the following files from the $MPICH/include include into $PGI/linux86/include:

            binding.h
        mpi.h
        mpi_errno.h
        mpidefs.h
        mpif.h
        mpio.h
        patchlevel.h
        protofix.h

    the following from the $MPICH/lib directory into $PGI/linux86/lib: 

           libampi.a
        libfmpich.a
        liblmpi.a
        libmpe.a
        libmpe_nompi.a
        libmpich++.a
        libmpich.a
        libpmpich.a
        libtmpi.a
        mpe_prof.o
        mpe_proff.o

    and the file $MPICH/util/execer must be installed in of $PGI/linux86/util.

What configuration does PGI use to build MPICH 1.2.1, 1.2.2, 1.2.3, 1.2.4, 1.2.5, and 1.2.6?

These configurations work for us, but building MPICH is not always easy, and if you are having trouble, consult the MPICH organization, or purchase the CDK product, which conatins an integrated MPICH in the installation.

For MPICH 1.2.1, we use 

       export CC=pgcc
       export FC=pgf77
       export F90=pgf90
       ./configure --enable-debug -c++=pgCC \
       -cflags="-Msignextend -tp px" -opt=-fast -fflags="-tp px" \
       -c++flags="-tp px"  -f90flags="-tp px" \
       -prefix=/usr/local/mpich

For MPICH 1.2.2, 1.2.3 and 1.2.4 

    ./configure  --enable-debug -c++=pgCC -cc=pgcc -fc=pgf77 \
    -cflags="-Msignextend -tp px \
    -DUSE_U_INT_FOR_XDR -DHAVE_RPC_RPC_H=1" -opt=-fast \
    -fflags="-tp px" -c++flags="-tp px" -f90flags="-tp px" -f90=pgf90 \
    -prefix=/usr/local/mpich

For MPICH 1.2.5 

    env CC=pgcc CFLAGS='-Msignextend -tp px' \
        FC=pgf77 FFLAGS='-tp px' \
        CXX=pgCC CCFLAGS='-tp px' \
        F90=pgf90 F90FLAGS='-tp px' \
        ./configure --enable-debug -opt=-fast \
        -prefix=/usr/local/mpich-1.2.5.2
    make
    make install

For MPICH 1.2.6, with 32-bit systems 

    env CC=pgcc CFLAGS='-Msignextend -tp px' \
         FC=pgf77 FFLAGS='-tp px' \
         CXX=pgCC CCFLAGS='-tp px' \
         F90=pgf90 F90FLAGS='-tp px' MAKE=gmake\
         ./configure --enable-debug -opt=-fast \
         -prefix=/usr/lib/mpich-1.2.6
    gmake -f Makefile

For MPICH 1.2.6, with 64-bit systems 

    env CC=pgcc CFLAGS='-Msignextend ' \
         FC=pgf77 FFLAGS= \
         CXX=pgCC CCFLAGS= \
         F90=pgf90 F90FLAGS= MAKE=gmake \
         ./configure --enable-debug -opt=-fast -prefix=/opt/mpi
    gmake -f Makefile
Click me