parallel processing

The next improvement was multiprogramming. In a multiprogramming system, multiple programs submitted by users were each allowed to use the processor for a short time. To users it appeared that all of the programs were executing at the same time. Problems of resource contention first arose in these systems. Explicit requests for resources led to the problem of the deadlock. Competition for resources on machines with no tie-breaking instructions lead to the critical section routine.

Vector processing was another attempt to increase performance by doing more than one thing at a time. In this case, capabilities were added to machines to allow a single instruction to add (or subtract, or multiply, or otherwise manipulate) two arrays of numbers. This was valuable in certain engineering applications where data naturally occurred in the form of vectors or matrices. In applications with less well-formed data, vector processing was not so valuable.

The next step in parallel processing was the introduction of multiprocessing. In these systems, two or more processors shared the work to be done. The earliest versions had a master/slave configuration. One processor (the master) was programmed to be responsible for all of the work in the system; the other (the slave) performed only those tasks it was assigned by the master. This arrangement was necessary because it was not then understood how to program the machines so they could cooperate in managing the resources of the system.

Solving these problems led to the symmetric multiprocessing system (SMP). In an SMP system, each processor is equally capable and responsible for managing the flow of work through the system. Initially, the goal was to make SMP systems appear to programmers to be exactly the same as single processor, multiprogramming systems. (This standard of behavior is known as sequential consistency). However, engineers found that system performance could be increased by someplace in the range of 10-20% by executing some instructions out of order and requiring programmers to deal with the increased complexity. (The problem can become visible only when two or more programs simultaneously read and write the same operands; thus the burden of dealing with the increased complexity falls on only a very few programmers and then only in very specialized circumstances.) The question of how SMP machines should behave on shared data is not yet resolved.

As the number of processors in SMP systems increases, the time it takes for data to propagate from one part of the system to all other parts grows also. When the number of processors is somewhere in the range of several dozen, the performance benefit of adding more processors to the system is too small to justify the additional expense. To get around the problem of long propagation times, message passing systems were created. In these systems, programs that share data send messages to each other to announce that particular operands have been assigned a new value. Instead of a broadcast of an operand's new value to all parts of a system, the new value is communicated only to those programs that need to know the new value. Instead of a shared memory, there is a network to support the transfer of messages between programs. This simplification allows hundreds, even thousands, of processors to work together efficiently in one system. (In the vernacular of systems architecture, these systems "scale well.") Hence such systems have been given the name of massively parallel processing (MPP) systems.

The most successful MPP applications have been for problems that can be broken down into many separate, independent operations on vast quantities of data. In data mining, there is a need to perform multiple searches of a static database. In artificial intelligence, there is the need to analyze multiple alternatives, as in a chess game. Often MPP systems are structured as clusters of processors. Within each cluster the processors interact as in a SMP system. It is only between the clusters that messages are passed. Because operands may be addressed either via messages or via memory addresses, some MPP systems are called NUMA machines, for Non-Uniform Memory Addressing.

SMP machines are relatively simple to program; MPP machines are not. SMP machines do well on all types of problems, providing the amount of data involved is not too large. For certain problems, such as data mining of vast data bases, only MPP systems will serve.

Learn more about Cloud Computing, grid and utility computing

Do you have something to add to this definition? Let us know. Send your comments to techterms@whatis.com

More resources from around the web: - Carnegie-Mellon University hosts a listing of supercomputing and parallel processing research terms and links. - At the University of Wisconsin, Doug Burger and Mark Hill have created The WWW Computer Architecture Home Page . - David A. Bader provides an IEEE listing of parallel computing sites .

FILE EXTENSION AND FILE FORMAT LIST File Extension and File Format List: A B C D E F G H I J K L M N O P Q R S T U V W X Y Z #

RELATED CONTENT The hidden costs of scale-out supercomputing Since the mid-1990s, supercomputer users have moved from scale-up big-box servers to scale-out clusters. But some say beware the hidden costs. Liquid Web opens 'cloud'-enabling data center A Web hoster will run Storm, it own cloud computing platform. at its new data center in Michigan. Storm will compete with Amazon Elastic Compute... Application performance monitoring firm targets cloud computing Nimsoft's new application performance monitoring software extends monitoring to the cloud and offers an alternative to the costly Big Four systems...

RELATED GLOSSARY TERMS Terms from Whatis.com − the technology online dictionary Blue Gene  (SearchDataCenter.com) Blue Gene is an experimental parallel processing supercomputer developed by IBM that employs thousands of processors, each of which demands minimal... cluster computing  (SearchDataCenter.com) In computers, clustering is the use of multiple computers, typically PCs or UNIX workstations, multiple storage devices, and redundant...