SHMEMmer Wanna Be

My Journey To Understanding and Using SHMEM:

Lesson 1: shmem_get

Scenario: Suppose you and your neighbor Alice are pretty good friends, good enough friends to share tools. Your new home project, putting together a table, requires you to use a hammer, which so happens to be at Alice's. You run over to her house and knock on the door. Alice, however, is just finishing up doing the dishes and is not able to come to the door right away. So, you wait. A few minutes later, she shows up and finds out what you need and grabs it for you. You then run back to your home and are able to finish the project.

In the programming world, the waiting game can really put a damper on accessing data. Similar to the above scenario, there are some situations that call for splitting the data up between multiple processors rather than each having all of the data. In these scenarios, it would be nice to just have access to the data being stored on other PEs and grab it when necessary (and do so without asking). And this is what shmem allows you to do and does so beautifully. BUT, with such freedom comes much responsibility on the programmer.

The ability to grab data from another processor's memory without asking is one instance where openMPI and shmem differ.

We will cover two basic get commands:
  1. grabbing only one element
  2. grabbing an array of elements
To grab one element, we can use the following command:
        TYPE shmem_<TYPENAME>_g(const TYPE *source, int pe);
where TYPE is one of the following:
TYPE TYPENAME
short short
int int
long long
long long longlong
unsigned short ushort
unsigned int uint
unsigned long ulong
unsigned long long ulonglong
int32_t int32
int64_t int64
uint32_t uint32
uint64_t uint64
size_t size
ptrdiff_t ptrdiff


Notice that this command actually returns the item itself; whereas when it comes to grabbing an array, we will have to pass in the pointer to the location of the destination. 

//grab_data_ex0.c

#include <stdio.h>
#include <shmem.h>

int main(){
	int my_pe, num_pe;  //declare variables for both pe id of processor
                        //and the number of pes
	
	shmem_init();
	num_pe = shmem_n_pes();     //obtain the number of pes that can be used
	my_pe  = shmem_my_pe();     //obtain the pe id number

	int pe_source = 0;          //pe where the desirable data is residing
	
	static double source = 1;
	double target = 0;          //element to be filled
	
	//if not main node, grab data from the main node
	if (my_pe != 0){
		target = shmem_double_g(&source, pe_source);
	}
	printf("%d: Data at target is %f\n", my_pe, target);	
	shmem_finalize();
	return 0;
}


To compile:
        oshcc grab_data_ex0.c -o grab_data_ex0
To run:
        oshrun -np 10 grab_data_ex0


Unlike lesson 0, we are not going to go through the code line by line, but we will just look at the middle portion: Let's first look at the shmem command

To grab an array of elements, we can use one of the two following commands: where the former can only be used with your basic C types listed in the above table. 

//grab_data_ex1.c

#include <stdio.h>
#include <shmem.h>

int main(){
	int my_pe, num_pe;          //declare variables for both pe id of processor 
                                //and the number of pes
	
	shmem_init();
	num_pe = shmem_n_pes();     //obtain the number of pes that can be used
	my_pe  = shmem_my_pe();     //obtain the pe id number

	int pe_source = 0;          //pe where the desirable data is residing
	
	static double source[10] = {1,2,3,4,5,6,7,8,9,10};
	double target[10];          //array to be filled
	
	//if not main node, grab data from the main node
	if (my_pe != 0){
		shmem_double_get(target, source, 10, pe_source);
	}
	printf("%d: Data at target[0] is %f\n", my_pe, target[0]);	
	shmem_finalize();
	return 0;
}


To compile:
        oshcc grab_data_ex1.c -o grab_data_ex1
To run:
        oshrun -np 10 grab_data_ex1