Parallel Languages Programming Directory, Links. Development, Programmers, Programmer Reference, Resources @ eIT.in

Add Links: If you have a web site that you wish to include in this database, do let us know the details by sending a note about your URL to narsi@esource.in. We’ll quickly review the web site, and if found relevant, add it to the database. We look forward to web site owners and link exchange partners to submit URL. Thanks!

Looking for an outsourcing partner for software programming & development? Have you talked to us?

Other IT Web Sites from eIT.in

Content derived from Wikipedia article on Parallel Computing

Parallel computing

From Wikipedia, the free encyclopedia

Parallel computing is the simultaneous execution of the same task (split up and specially adapted) on multiple processors in order to obtain results faster. The idea is based on the fact that the process of solving a problem usually can be divided into smaller tasks, which may be carried out simultaneously with some coordination.

Contents

1 Parallel computing systems

2 Theory and practice

2.1 Performance vs. cost

3 Terminology in parallel computing

4 Algorithms

5 Parallel problems

6 Parallel programming

6.1 Parallel programming models

7 Topics in parallel computing

8 See also

9 References

10 External links

Parallel computing systems

A parallel computing system is a computer with more than one processor for parallel processing. In the past, each processor of a multiprocessing system always came in its own processor packaging, but recently introduced multicore processors contain multiple logical processors in a single package.

There are many different kinds of parallel computers. They are distinguished by the kind of interconnection between processors (known as "processing elements" or PEs) and memory.

Flynn's taxonomy, one of the most accepted taxonomies of parallel architectures, classifies parallel (and serial) computers according to

whether all processors execute the same instructions at the same time (single instruction/multiple data -- SIMD) or

each processor executes different instructions (multiple instruction/multiple data -- MIMD).

One major way to classify parallel computers is based on their memory architectures. Shared memory parallel computers have multiple processors accessing all available memory as global address space. They can be further divided into two main classes based on memory access times: Uniform memory access (UMA), in which access times to all parts of memory are equal, or Non-Uniform memory access (NUMA), in which they are not. Distributed memory parallel computers also have multiple processors, but each of the processors can only access its own local memory; no global memory address space exists across them.

Parallel computing systems can also be categorized by the numbers of processors in them. Systems with thousands of such processors are known as massively parallel. Subsequently there are what are referred to as "Large scale" vs "Small scale" parallel processors. This depends on the size of the processor, eg. a PC based parallel system would generally be considered a small scale system.

Parallel processor machines are also divided into symmetric and asymmetric multiprocessors, depending on whether all the processors are the same or not (for instance if only one is capable of running the operating system code and others are less privileged).

A variety of architectures have been developed for parallel processing. For example a Ring architecture has processors linked by a ring structure. Other architectures include Hypercubes, Fat trees, systolic arrays, and so on.

Theory and practice

Parallel computers can be modelled as Parallel Random Access Machines (PRAMs). The PRAM model ignores the cost of interconnection between the constituent computing units, but is nevertheless very useful in providing upper bounds on the parallel solvability of many problems. In reality the interconnection plays a significant role.

The processors may communicate and cooperate in solving a problem or they may run independently, often under the control of another processor which distributes work to and collects results from them (a "processor farm").

Processors in a parallel computer may communicate with each other in a number of ways, including shared (either multiported or multiplexed) memory, a crossbar, a shared bus or an interconnect network of a myriad of topologies including star, ring, tree, hypercube, fat hypercube (a hypercube with more than one processor at a node), an n-dimensional mesh, etc. Parallel computers based on interconnect network need to employ some kind of routing to enable passing of messages between nodes that are not directly connected. The communication medium used for communication between the processors is likely to be hierarchical in large multiprocessor machines. Similarly, memory may be either private to the processor, shared between a number of processors, or globally shared. Systolic array is an example of a multiprocessor with fixed function nodes, local-only memory and no message routing.

Approaches to parallel computers include:

Multiprocessing

Computer cluster

Parallel supercomputers

Distributed computing

NUMA vs. SMP vs. massively parallel computer systems

Grid computing

Performance vs. cost

While a system of n parallel processors is less efficient than one n-times-faster processor, the parallel system is often cheaper to build. Parallel computation is used for tasks which require very large amounts of computation, take a lot of time, and can be divided into n independent subtasks. In recent years, most high performance computing systems, also known as supercomputers, have parallel architectures.

Terminology in parallel computing

Some frequently used terms in parallel computing are:

Task

a logically high level, discrete, independent section of computational work. A task is typically executed by a processor as a program

Synchronization

the coordination of simultaneous tasks to ensure correctness and avoid unexpected race conditions.

Speedup

also called parallel speedup, which is defined as wall-clock time of best serial execution divided by wall-clock time of parallel execution.

Parallel overhead

the extra work associated with parallel version compared to its sequential code, mostly the extra CPU time and memory space requirements from synchronization, data communications, parallel environment creation and cancellation, etc.

Scalability

a parallel system's ability to gain proportionate increase in parallel speedup with the addition of more processors.

Algorithms

Parallel algorithms can be constructed by redesigning serial algorithms to make effective use of parallel hardware. However, not all algorithms can be parallelized. This is summed up in a famous saying:

One woman can have a baby in nine months, but nine women can't have a baby in one month.

In practice, linear speedup (i.e., speedup proportional to the number of processors) is very difficult to achieve. This is because many algorithms are essentially sequential in nature (Amdahl's law states this more formally).

Certain workloads can benefit from pipeline parallelism when extra processors are added. This uses a factory assembly line approach to divide the work. If the work can be divided into n stages where a discrete deliverable is passed from stage to stage, then up to n processors can be used. However, the slowest stage will hold up the other stages so it is rare to be able to fully use n processors.

Parallel problems

Well known parallel software problem sets include embarrassingly parallel and Grand Challenge problems.

Parallel programming

Parallel programming is the design, implementation, and tuning of parallel computer programs which take advantage of parallel computing systems. It also refers to the application of parallel programming methods to existing serial programs (parallelization).

Parallel programming focuses on partitioning the overall problem into separate tasks, allocating tasks to processors and synchronizing the tasks to get meaningful results. Parallel programming can only be applied to problems that are inherently parallelizable, mostly without data dependence. A problem can be partitioned based on domain decomposition or functional decomposition, or a combination.

There are two major approaches to parallel programming.

implicit parallelism -- the system (the compiler or some other program) partitions the problem and allocates tasks to processors automatically (also called automatic parallelizing compilers) -- or

explicit parallelism where the programmer must annotate his program to show how it is to be partitioned.

Many factors and techniques impact the performance of parallel programming:

Load balancing attempts to keep all processors busy by moving tasks from heavily loaded processors to less loaded ones.

Some people consider parallel programming to be synonymous with concurrent programming. Others draw a distinction between parallel programming, which uses well-defined and structured patterns of communications between processes and focuses on parallel execution of processes to enhance throughput, and concurrent programming, which typically involves defining new patterns of communication between processes that may have been made concurrent for reasons other than performance. In either case, communication between processes is performed either via shared memory or with message passing, either of which may be implemented in terms of the other.

Programs which work correctly in a single CPU system may not do so in a parallel environment. This is because multiple copies of the same program may interfere with each other, for instance by accessing the same memory location at the same time. Therefore, careful programming (synchronization) is required in a parallel system.

Parallel programming models

Main article: Parallel programming model

A parallel programming model is a set of software technologies to express parallel algorithms and match applications with the underlying parallel systems. It encloses the areas of applications, languages, compilers, libraries, communication systems, and parallel I/O. People have to choose a proper parallel programming model or a form of mixture of them to develop their parallel applications on a particular platform.

Parallel models are implemented in several ways: as libraries invoked from traditional sequential languages, as language extensions, or complete new execution models. They are also roughly categorized for two kinds of systems: shared memory systems and distributed memory systems, though the lines between them are largely blurred nowadays.

Topics in parallel computing

Generic:

Automatic parallelization

Parallel algorithm

Cellular automaton

Grand Challenge problems

Computer science topics:

Lazy evaluation vs strict evaluation

Complexity class NC

Communicating sequential processes

Dataflow architecture

Parallel graph reduction

Practical problems:

Parallel computer interconnects

Parallel computer I/O

Reliability problems in large systems

Programming languages/models:

OpenMP

Message Passing Interface/MPICH

Occam

Linda

Cilk

Specific:

Atari Transputer Workstation

BBN Butterfly computers

Beowulf cluster

Blue Gene

Deep Blue

Fifth generation computer systems project

ILLIAC III

ILLIAC IV

Parallel Element Processing Ensemble

Meiko Computing Surface

NCUBE

Teramac

Transputer

Parallel computing to increase fault tolerance:

Master-checker

Companies (largely historical):

Thinking Machines

Convex Computer Corporation

Meiko

Control Data Corporation

Myrias Research Corporation

See also

Stream processing

Computer cluster

Concurrent computing

DNA computing

Grid computing

Important publications in parallel computing

Parallel rendering

Degree of parallelism

Symmetric multiprocessing

Concurrency

Threads

References

This article was originally based on material from the Free On-line Dictionary of Computing, which is licensed under the GFDL.

http://www.llnl.gov/computing/tutorials/parallel_comp/ Introduction to Parallel Computing

External links

The Parawiki - a wiki on Parallel Computing

Introduction to Parallel Computing

"Multiprocessor Optimizations: Fine-Tuning Concurrent Access to Large Data Collections" by Ian Emmons

The Center for Programming Models for Scalable Parallel Computing

Internet Parallel Computing Archive

National HPCC Software Exchange

Parallel processing topic area at IEEE Distributed Computing Online

"The Free Lunch Is Over: A Fundamental Turn Toward Concurrency in Software" by Herb Sutter

Parallel programming citations from CiteSeer

Retrieved from http://en.wikipedia.org/wiki/Parallel_computing

End of Wikipedia content, http://en.wikipedia.org/wiki/Parallel_computing

Web Resources for Parallel Programming Languages

More eIT.in References

o Programming Languages

§ The A-Z of Programming Languages

§ A, B, C, D, E, F, G, H, I, J, K, L, M, N, O, P, Q, R, S, T, U, V, W, X, Y, Z

§ P

· Pascal

· Perl

· PHP Programming

· Phantom

· Pike

· PiXCL

· PL/B

· PL/I

· Plankalkül

· Pliant

· Parallel Languages

· Pop

· PostScript

· Prolog

· Python

Main Sections @ eIT.in

Software

· Free & Open Source Software

· Operating Systems

o Mainframe & Legacy Operating Systems

· Midrange

· Mainframe Operating Systems

· Databases

· Programming & Development Directory

o Programming Languages

§ The A-Z of Programming Languages

§ A, B, C, D, E, F, G, H, I, J, K, L, M, N, O, P, Q, R, S, T, U, V, W, X, Y, Z

· Internet & Web

§ Web Servers

§ Application Servers

§ Server Side Scripting

§ Web Services

Computer Hardware
Networking & Communications Technologies
IT Strategy & Design
IT in Industries & Businesses
IT in Functions & Business Processes

IT Infrastructure
Tech Support
IT, Software Industry, Trade & Business Network
IT, Software Industry News & Updates

About eIT.in

eIT.in is a comprehensive directory for everything IT & Software. It contains over 500 categories, and well over 10,000 web resources

eIT.in provides directory and web links resources for the IT, software, programming & software development domains. It is intended to be useful for application, applications programmers & developers, software technology programmer & developer, databases software development, administrators & DBAs, application developers, strategy architect, design specialists and architects, migration, integration, customization consultants and customisation analysts, administration, maintenance & support professionals, outsourcing consultant, bespoke solutions programming developers & coders, project management & functional analyst, and for system administrators, testing & quality control engineers. It will make an effort to provide resources on tutorial/tutorials, guide, guides, tips, faq, faqs on these topics.

eIT.in content is available under GPL: All directory content at mainframe.in is under the General Public License (GPL). Under this license, anyone is free to copy & use any amount of directory content @ eIT.in, make changes to it and use it in any way they wish, as long as they also allow the same rights to anyone else for this content. The concept of GPL has been adapted from the GNU GPL of the Free Software Movement. To those who wish to use content from eIT.in, our only request is that they acknowledge the source and provide a link back to eIT.in. This is only a request!

Countries & Cities Where eIT.in Provides Assistance

eIT Cities: Bangalore, Chennai, Mumbai, Bhubaneswar, Mysore, Kolkaka, Delhi, Pune, Trivandrum, Hyderabad

You are the $$$ Section of eIT.in

Reference

GeoDig – Get Local!

Have you checked out the GeoDig directories for over 30 countries? GeoDig provides useful local and regional web resources for over 200 cities around the world. See the list of cities and countries for which GeoDig provides locality-specific web resources.

North America

USA - Alabama (AL) > Birmingham; Alaska; Arkansas (AR) > Little Rock; Arizona (AZ) > Phoenix, Las Vegas, Tucson; California (CA) > Los Angeles, San Francisco, Sacramento, Fresno, Bakersfield; Colorado, CO > Denver; Connecticut, CT > Hartford; District of Columbia, DC > Washington DC; Delaware (DE) > Wilmington; Florida > Miami, Orlando, Tampa, Orlando, Sarasota, West Palm Beach, Jacksonville; Georgia > Atlanta; Hawaii > Honolulu; Idaho; Illinois > Chicago; Indiana > Indianapolis; Iowa; Kansas (KS); Kentucky (KY) > Louisville; Louisiana (LA) > New Orleans, Baton Rouge; Maine; Maryland (MD) > Baltimore; Massachusetts > Boston, Springfield; Michigan > Detroit, Grand Rapids; Minnesota > Minneapolis-St. Paul; Mississippi (MS); Missouri (MO) > Kansas City, St. Louis; Montana; Nebraska (NE) > Omaha; Nevada (NV) > Las Vegas; New Hampshire; New Jersey (NJ) > Jersey City, Newark; New Mexico (NM) > Albuquerque; New York > New York, Buffalo, Rochester, Albany, Syracuse; North Carolina (NC) > Raleigh-Durham, Charlotte, Greensboro; North Dakota; Ohio> Columbus, Cincinnati, Cleveland, Toledo, Youngstown, Dayton; Oklahoma (OK) > Oklahoma City, Tulsa; Oregon > Portland; Pennsylvania > Philadelphia, Allentown, Pittsburgh, Harrisburg, Scranton, ; Rhode Island (RI) > Providence; South Carolina (SC) > Greenville; South Dakota; Tennessee (TN) > Knoxville, Memphis, Nashville; Texas > Austin, Dallas, Houston, San Antonio, El Paso, Austin, McAllen; Utah (UT) > Salt Lake City; Vermont; Virginia (VA) > Norfolk, Richmond; Washington > Seattle; West Virginia; Wisconsin (WI) > Milwaukee; Wyoming

Canada - Vancouver, Montreal, Toronto, Calgary, Ottawa-Gatineau, Edmonton, Quebec City, Winnipeg, Hamilton, London

You are the $$$ Section of eIT.in

Europe - UK - London, Glasgow, Manchester, Birmingham, Liverpool, Sheffield, Leeds, Bristol, Edinburgh, Leicester; France - Paris, Marseille, Lyon, Toulouse, Nice, Nantes, Strasbourg, Montpellier, Bordeaux; Germany - Frankfurt (Frankfurt am Main), Munich (München), Berlin, Düsseldorf, Hamburg, Cologne (Köln), Essen, Dortmund, Stuttgart, Bremen, Duisburg, Hannover, Nürnberg (Nuremberg), Dresden, Leipzig; Italy - Milan (Milano), Rome (Roma), Napoli (Naples), Torino (Turin), Palermo, Bologna, Firenze (Florence), Genova (Genoa); Spain - Madrid, Barcelona, Valencia, Sevilla, Zaragoza, Malaga, Murcia, Las Palmas, Bilbao; Scandinavia - Finland - Helsinki (Helsingin), Espoo, Tampere (Tampereen), Vantaa, Turku, Oulu, Sweden - Stockholm, Goteborg (Göteborg), Malmo (Malmö), Uppsala, Vasteras (Västerås), Denmark - Copenhagen (Københavns), Aarhus (Århus), Odense, Aalborg (Ålborg), Norway - Oslo, Bergen, Stavanger, Trondheim; Benelux - Belgium - Brussels (Brussel), Antwerp (Antwerpen), Ghent (Gent, Gand), Charleroi, Liège (Liege), Netherlands - Amsterdam, Rotterdam, Utrecht, Eindhoven, Tilburg, ‘s-Gravenhage (sGravenhage), Groningen, Luxembourg - Luxembourg City; Portugal – Lisbon; Greece – Athens; Hungary – Budapest; Poland – Warsaw; Switzerland - Zürich (Zurich), Geneva (Geneve, Genève), Basel, Bern (Berne), Lausanne; Austria - Linz, Vienna (Wien), Graz, Linz, Salzburg, Innsbruck; Ireland – Dublin

Asia - India - Mumbai, New Delhi, Bangalore; China & Hong Kong - Hong Kong, Beijing, Shanghai, Tianjin, Wuhan, Shenyang, Guangzhou, Harbin, Xian; Japan - Tokyo, Osaka, Yokohama, Nagoya, Sapporo, Kyoto, Kobe, Fukuoka, Kawasaki, Hiroshima; South Korea - Seoul, Pusa, Taegu, Incheon, Taejeon, Taiwan - Taipei; Malaysia - Kuala Lumpur; Singapore; Russia - Moscow, St Petersburg

You are the $$$ Section of eIT.in

Middle East - Turkey - Istanbul, Israel - Tel Aviv

Oceania - Australia - Sydney, Melbourne, Brisbane, Perth, Adelaide

Africa - South Africa - Johannesburg, Cape Town, Durban

eIT.in is a product of eSource India & Sourcing India

Other eSource & Sourcing sites: IT & Software (Dir, SAP), BPO, Chemicals, Textiles, Plant Oils, dotMobi, Billion Dollar Questions,

Biodiesel Encyclopedia, Linens, ideOS, Follars – Free, Open-source Dollars, Quali5.com – Own A Keyword Forever, AntiSE, Serkai, Leather & Hide, GeoDig