Basic Concepts in Parallel Programming

If you are interested in learning more about the paradigms mentioned above, Gustav Meglicki has some excellent notes explaining parallel programming paradigms in the I590 section of his website.

• Each processor has its own private memory, as shown in the following diagram.



• A cluster of stand-alone workstations, interconnected by a high-speed network, constitutes a distributed-memory parallel computer. The AVIDD cluster, which includes hundreds of processors reserved for distributed parallel computing, is a good example. More information on AVIDD is given in the Low-down on AVIDD - Technical Details sub-section.

• Processors communicate only via formal message passing.

• This workshop will focus on programming for distributed memory systems.

• All processors have a common memory module, i.e., they share the memory.

  

• Processors may communicate by alternately accessing the same memory location. And they may also communicate via formal message passing like distributed-memory processors.

• Each node on the IBM SP at IU uses the shared-memory architecture. Within the SP, the Orion complex includes a 32-CPU Regatta node with 192 GB memory, four 16-CPU nodes; the Aries complex consists of 127 4-CPU nodes, each with 1-2 GB memory.

  Note: Well, theoritically even on AVIDD, each node is a 2-CPU SMP, but let us not bother about that too much :-) You may be familiar with so-called SMP (symmetric multiprocessor) workstations. The processors have a symmetric relationship with each other. I.e., the processors equally share the workstation's resources, including memory. Therefore, an SMP workstation is a shared-memory machine.

• The Research Database Complex (RDC) system at IU, which is used exclusively for database based research (Oracle), is another example of a shared-memory machine; it incorporates 2 identical Sun V1280 systems with 12 CPUs each and 96 GB of shared memory. Do not request for any account on the RDC unless you have a project involving a big database component.

• This tutorial will not consider programming for systems that use a shared memory architecture.