SHMEM - Lesson 2b: Memory Layout

Lesson 2b: Memory Layout Introduction

One important concept we have not yet covered is the memory layout in the SHMEM world. Unlike other message passing interfaces, SHMEM has decided to have a portion of memory designated to be the symmetric memory. What does this mean? Well, in essence, each PE that has as a portion of memory that looks the same as all others. This is what allows us to just grab and put elements onto other PEs.

For example, say PE0 is to put elements onto an array in PE1. Well, then that target array (not the source) must be in the symmetric portion of the memory. This can be done a few different ways:

Declaring a static array
Calling shmem_malloc to place data in the symmetric heap

We have already seen an example of using the static section of symmetric memory (see Lesson 2). We will now take a dive and see an example of how to use the symmetric heap.

//heap_ex1.c

#include <stdio.h>
#include <shmem.h>

int main(){
	int my_pe, num_pe;          //declare variables for both pe id of processor and the number of pes
	
	shmem_init();
	num_pe = shmem_n_pes();     //obtain the number of pes that can be used
	my_pe  = shmem_my_pe();     //obtain the pe id number

    double cols = 5;

    double *target;
    target = (double*) shmem_malloc(sizeof(double)*rows);

    if (me == 0){
        double *A;
        A = (double *) malloc(sizeof(double)*rows);
        for (int i = 0; i < cols; i ++) {
            A[i] = i;        
        }
        shmem_double_put(target, A, cols, 1); // target is in symmetric heap
    }

    shmem_barrier_all(); //making sure the put has been finished
    if (me == 1){
        for (int i = 0; i < cols; i++) {
            printf("%f ", target[i]);
        }
        printf("\n");
    }
	shmem_finalize();
	return 0;
}

To compile:
oshcc heap_ex1.c -o heap_ex1
To run:
oshrun -np 2 heap_ex1

We can do something similar with double arrays as well.

//heap_ex2.c

#include <stdio.h>
#include <shmem.h>

int main(){
	int my_pe, num_pe;          //declare variables for both pe id of processor and the number of pes
	
	shmem_init();
	num_pe = shmem_n_pes();     //obtain the number of pes that can be used
	my_pe  = shmem_my_pe();     //obtain the pe id number

    double rows = 3, cols = 5;

    double **target;
    target = (double**) shmem_malloc(sizeof(double*)*rows);

    for (int i = 0; i < rows; i++) 
        target[i] = (double*) shmem_malloc(sizeof(double)*rows)


    if (me == 0){
        double **A;
        A = (double **) malloc(sizeof(double*)*rows);
        for (int i = 0; i < rows; i ++) {
            A[i] = (double*) calloc(sizeof(double),cols);
            for (int j = 0; j < cols; j++) {
                A[i][j] = i*cols + j;
            }
            shmem_double_put(target[i], A[i], cols, 1);
        }
    }


    shmem_barrier_all(); //making sure all of the puts have been done
    if (me == 1){
        for (int i = 0; i < rows; i++) {
            for (int j = 0; j < rows; j++) {
                printf("%f ", target[i][j]);
            }
            printf("\n");
        }
    }

	shmem_finalize();
	return 0;
}

To compile:
oshcc heap_ex2.c -o heap_ex2
To run:
oshrun -np 2 heap_ex2

My Journey To Understanding and Using SHMEM:

Lesson 2b: Memory Layout Introduction