I would normally agree with Shawn Chin about using existing libraries to do the file reading; in this case I might disagree because the file format is so simple and it's so important for MPI to know how the data is laid out in memory. A 2d nxm array allocated as a contiguous 1-d array of nxm is very different from rows scattered all over memory! As always, this is C's fault for not having real multi-d arrays. On the other hand, you could check out the libnetpbm libraries and see how it's allocated, or as Shawn suggests, copy the whole thing into contiguous memory after reading it in.
Note too that this would actually be easier with the (binary) P5 format, as one could use MPI-IO to read in the data in parallel right at the beginning, rather than having one processor doing all the reading and using scatter/gather to do the data distribution. With ascii files, you never really know how long a record is going to be, which makes coordinated I/O very difficult.
Also note that this really isn't a 2d problem - you are just doing an elementwise operation on every piece of the array. So you can greatly simplify things by just treating the data as a 1d array and ignoring the geometry. This wouldn't be the case if you were (say) applying a 2d filter to the image, as there the geometry matters and you'd have to partition data accordingly; but here we don't care.
Finally, even in this simple case you have to use scatterv and gatherv because the number of cells in the image might not evenly divide by the number of MPI tasks. You could simplify the logic here just by padding the array to make it evenly divide; then you could
avoid some of the extra steps here.
So if you have a read_pgm() and write_pgm() that you know return pointers into a single contiguous block of memory, you can do something like this:
int main(int argc, char **argv) {
int ierr;
int rank, size;
int **greys;
int rows, cols, maxval;
int ncells;
int mystart, myend, myncells;
const int IONODE=0;
int *disps, *counts, *mydata;
int *data;
ierr = MPI_Init(&argc, &argv);
if (argc != 3) {
fprintf(stderr,"Usage: %s infile outfile\n",argv[0]);
fprintf(stderr," outputs the negative of the input file.\n");
return -1;
}
ierr = MPI_Comm_rank(MPI_COMM_WORLD, &rank);
ierr |= MPI_Comm_size(MPI_COMM_WORLD, &size);
if (ierr) {
fprintf(stderr,"Catastrophic MPI problem; exiting\n");
MPI_Abort(MPI_COMM_WORLD,1);
}
if (rank == IONODE) {
if (read_pgm(argv[1], &greys, &rows, &cols, &maxval)) {
fprintf(stderr,"Could not read file; exiting\n");
MPI_Abort(MPI_COMM_WORLD,2);
}
ncells = rows*cols;
disps = (int *)malloc(size * sizeof(int));
counts= (int *)malloc(size * sizeof(int));
data = &(greys[0][0]); /* we know all the data is contiguous */
}
/* everyone calculate their number of cells */
ierr = MPI_Bcast(&ncells, 1, MPI_INT, IONODE, MPI_COMM_WORLD);
myncells = ncells/size;
mystart = rank*myncells;
myend = mystart + myncells - 1;
if (rank == size-1) myend = ncells-1;
myncells = (myend-mystart)+1;
mydata = (int *)malloc(myncells * sizeof(int));
/* assemble the list of counts. Might not be equal if don't divide evenly. */
ierr = MPI_Gather(&myncells, 1, MPI_INT, counts, 1, MPI_INT, IONODE, MPI_COMM_WORLD);
if (rank == IONODE) {
disps[0] = 0;
for (int i=1; i<size; i++) {
disps[i] = disps[i-1] + counts[i-1];
}
}
/* scatter the data */
ierr = MPI_Scatterv(data, counts, disps, MPI_INT, mydata, myncells,
MPI_INT, IONODE, MPI_COMM_WORLD);
/* everyone has to know maxval */
ierr = MPI_Bcast(&maxval, 1, MPI_INT, IONODE, MPI_COMM_WORLD);
for (int i=0; i<myncells; i++)
mydata[i] = maxval-mydata[i];
/* Gather the data */
ierr = MPI_Gatherv(mydata, myncells, MPI_INT, data, counts, disps,
MPI_INT, IONODE, MPI_COMM_WORLD);
if (rank == IONODE) {
write_pgm(argv[2], greys, rows, cols, maxval);
}
free(mydata);
if (rank == IONODE) {
free(counts);
free(disps);
free(&(greys[0][0]));
free(greys);
}
MPI_Finalize();
return 0;
}
