MPI derived datatype works for floats, but not for doubles. Is it an alignment issue?

问题

I have a weird issue related to a C-structure that is communicated with the help of an MPI derived datatype. The example below works; it simply sends a message consisting of one integer plus 4 float values.

Minmum working example:

#include <mpi.h>
#include <stdio.h>

int main(int argc, char *argv[]) {
    MPI_Init(&argc, &argv);

    int i, rank, tag = 1;
    MPI_Status status;
    MPI_Comm_rank(MPI_COMM_WORLD, &rank);

    // Array of doubles plus element count
    typedef struct {
        int row;
        float elements[4];
    } My_array;

    // Derived datatype for an array of doubles plus element count
    MPI_Datatype MY_ARRAY_TYPE;
    const int nr_blocks = 2;
    int blocklengths[2] = {1, 4};
    MPI_Datatype oldtypes[2] = {MPI_INT, MPI_FLOAT};
    MPI_Aint extent, lb;
    MPI_Type_get_extent(MPI_INT, &lb, &extent);
    MPI_Aint displacements[2] = {0, extent};
    MPI_Type_create_struct(nr_blocks, blocklengths, displacements,
                       oldtypes, &MY_ARRAY_TYPE);
    MPI_Type_commit(&MY_ARRAY_TYPE);

   if(rank == 0) {
        My_array array1  = {3, 3.1, 3.2, 3.3, 3.4};
        MPI_Send(&array1, 1, MY_ARRAY_TYPE, 1, tag, MPI_COMM_WORLD);
   }
   if(rank == 1) {
        My_array array2;
        MPI_Recv(&array2, 1, MY_ARRAY_TYPE, 0, tag, MPI_COMM_WORLD, &status);
        printf("Rank %d received elements of row %d:\n", rank, array2.row);
        for(i = 0; i < 4; i++)
            printf("\t%.1f\n", array2.elements[i]);
    }
    MPI_Type_free(&MY_ARRAY_TYPE);
    MPI_Finalize();
}

If you have access to an MPI installation, the example can be compiled by mpicc -o example example.c and run by mpirun -np 2 example. The output should be

Rank 1 received elements of row 3:
    3.1
    3.2
    3.3
    3.4

The problem: Now when the array of floats is changed to an array of doubles, and accordingly MPI_FLOATto MPI_DOUBLE, I get a wrong result.

This code:

#include <mpi.h>
#include <stdio.h>

int main(int argc, char *argv[]) {
    MPI_Init(&argc, &argv);

    int i, rank, tag = 1;
    MPI_Status status;
    MPI_Comm_rank(MPI_COMM_WORLD, &rank);

    // Array of doubles plus element count
    typedef struct {
        int row;
        double elements[4];
    } My_array;

    // Derived datatype for an array of doubles plus element count
    MPI_Datatype MY_ARRAY_TYPE;
    const int nr_blocks = 2;
    int blocklengths[2] = {1, 4};
    MPI_Datatype oldtypes[2] = {MPI_INT, MPI_DOUBLE};
    MPI_Aint extent, lb;
    MPI_Type_get_extent(MPI_INT, &lb, &extent);
    MPI_Aint displacements[2] = {0, extent};
    MPI_Type_create_struct(nr_blocks, blocklengths, displacements,
                       oldtypes, &MY_ARRAY_TYPE);
    MPI_Type_commit(&MY_ARRAY_TYPE);

   if(rank == 0) {
        My_array array1  = {3, 3.1, 3.2, 3.3, 3.4};
        MPI_Send(&array1, 1, MY_ARRAY_TYPE, 1, tag, MPI_COMM_WORLD);
   }
   if(rank == 1) {
        My_array array2;
        MPI_Recv(&array2, 1, MY_ARRAY_TYPE, 0, tag, MPI_COMM_WORLD, &status);
        printf("Rank %d received elements of row %d:\n", rank, array2.row);
        for(i = 0; i < 4; i++)
            printf("\t%.1f\n", array2.elements[i]);
    }
    MPI_Type_free(&MY_ARRAY_TYPE);
    MPI_Finalize();
}

produces:

Rank 1 received elements of row 3:
    3.1
    3.2
    3.3
    0.0

I tried around a bit, using other data in the struct and the derived datatype (e.g., an array of integers instead of just one, int/MPI_INT instead of float/MPI_FLOAT, etc.) and saw that the problem arises only when doubles are used. Which makes me suspect that this might be an alignment issue of sorts - but I am stuck there. MPI should take care of alignments automatically.

Question: Why does the above example work with float/MPI_FLOAT, but not with double/MPI_DOUBLE and how can I fix it?

Some machine specifics that might be relevant:

CPU: AMD Opteron 6134
Address sizes: 48 bit
Alignment: 64
Compiler: gcc 4.4.7
MPI library: (Unfortunately) Vendor specific

Edit: as suggested in the comments by Vladimir F, I added the code that does not work.

回答1:

I just found out what the problem is: it is indeed alignment. That the second code listing produces the first 3 doubles correctly is nothing but a weird coincidence...By using the extend of MPI_INT as the offset of the following value, I assumed that there would be no padding. It is better to compute the offsets like this:

#include <stddef.c> 
...
MPI_Datatype MY_ARRAY_TYPE;
const int nr_blocks = 2;
int blocklengths[2] = {1, 4};
MPI_Datatype oldtypes[2] = {MPI_INT, MPI_DOUBLE};
MPI_Aint displacements[2];
displacements[0] = offsetof(My_array, row);
displacements[1] = offsetof(My_array, elements);
MPI_Type_create_struct(nr_blocks, blocklengths, displacements,
                   oldtypes, &MY_ARRAY_TYPE);
MPI_Type_commit(&MY_ARRAY_TYPE);
...

I'd really be interested to see how it can work out this way...why do we get 3 correct values and one 0.0? Since the alignment was off by 4 bytes and doubles are represented by 8 bytes on my platform, why didn't I get some random numbers? How can the first 3 have been decoded correctly if they each took the lower 4 bytes of one double plus the upper 4 of the next double?

来源：https://stackoverflow.com/questions/31135173/mpi-derived-datatype-works-for-floats-but-not-for-doubles-is-it-an-alignment-i

标签

struct

mpi

memory-alignment