Copyright 1998-2001, University of Notre Dame.
Authors: Jeffrey M. Squyres and Arun Rodrigues with Brian Barrett,
         Kinis L. Meyer, M. D. McNally, and Andrew Lumsdaine

This file is part of the Notre Dame LAM implementation of MPI.

You should have received a copy of the License Agreement for the Notre
Dame LAM implementation of MPI along with the software; see the file
LICENSE.  If not, contact Office of Research, University of Notre
Dame, Notre Dame, IN 46556.

Permission to modify the code and to distribute modified code is
granted, provided the text of this NOTICE is retained, a notice that
the code was modified is included with the above COPYRIGHT NOTICE and
with the COPYRIGHT NOTICE in the LICENSE file, and that the LICENSE
file is distributed with the modified code.

LICENSOR MAKES NO REPRESENTATIONS OR WARRANTIES, EXPRESS OR IMPLIED.
By way of example, but not limitation, Licensor MAKES NO
REPRESENTATIONS OR WARRANTIES OF MERCHANTABILITY OR FITNESS FOR ANY
PARTICULAR PURPOSE OR THAT THE USE OF THE LICENSED SOFTWARE COMPONENTS
OR DOCUMENTATION WILL NOT INFRINGE ANY PATENTS, COPYRIGHTS, TRADEMARKS
OR OTHER RIGHTS.

Additional copyrights may follow.


The alltoall program is a canonical example of MPI usage.  It sends a
message from each MPI rank to each other MPI rank.

Note that we carefully distinguish between the order of sending and
receiving.  It is *not* sufficient to have a single loop where each
rank does a simple "MPI_Send(...); MPI_Recv(...);" -- doing so can
lead to deadlock.  This is a subtlety in the MPI standard --
MPI_Send() *may* or *may not* block.  In LAM, MPI_Send() will block if
a message is too long; it will wait until the message has been started
to be received on the target rank.  Under that size, MPI_Send() will
[most likely] return immediately regardless of what the target rank is
doing.

Note that this is not a LAM-specific issue; every MPI implementation
is coded in this way -- that MPI_Send tries to complete immediately,
but above a certain threshold (usually related a combination of
message size and the cumulative size of unreceived messages to that
target), MPI_Send may block until the target rank starts receiving. */

Finally, notice that this program is actually a poor example of the
all-to-all communication pattern -- there is some degree of
serialization in the sending of messages.  For example, rank N has to
wait for rank 0 to send to it before it can continue.  That is, rank 0
causes a daisy-chain of reactions that allow the rest of the ranks to
continue -- each rank will not "start" until rank 0 contacts it.

Use "make" to compile this example.  Make will use mpicc to compile
the program:

        mpicc -o alltoall alltoall.c 

This program must be run on 2 or more ranks.
