Prepare for your exams
Get points
Guidelines and tips

Prepare for your exams

Study with the several resources on Docsity

Earn points to download

Earn points by helping other students or get them with a premium plan

Guidelines and tips

Sell on Docsity

Prepare for your exams

Study with the several resources on Docsity

Find documents

Prepare for your exams with the study notes shared by other students like you on Docsity

Search Store documents

The best documents sold by students who completed their studies

Search through all study resources

Docsity AINEW

Summarize your documents, ask them questions, convert them into quizzes and concept maps

Explore questions

Clear up your doubts by reading the answers to questions asked by your fellow students

Earn points to download

Earn points by helping other students or get them with a premium plan

Share documents

20 Points

For each uploaded document

Answer questions

5 Points

For each given answer (max 1 per day)

All the ways to get free points

Get points immediately

Choose a premium plan with all the points you need

Study Opportunities

Choose your next study program

Get in touch with the best universities in the world. Search through thousands of universities and official partners

Community

Ask the community

Ask the community for help and clear up your study doubts

University Rankings

Discover the best universities in your country according to Docsity users

Free resources

Our save-the-student-ebooks!

Download our free guides on studying techniques, anxiety management strategies, and thesis advice from Docsity tutors

From our blog

Exams and Study

Go to the blog

Lecture Notes on Advanced Computer Architecture II, Lecture notes of Computer Architecture and Organization

University of Wisconsin (UW) - Whitewater Computer Architecture and Organization

Lecture notes on Advanced Computer Architecture II, a course offered at the University of Wisconsin-Madison in Spring 2017. The notes cover topics such as sources of parallelism, programming models, major abstractions, shared memory, message passing, functional parallelism, automatic extraction, data parallelism, and coordinating work. The document also discusses programming model elements, MP interfaces, application binary interface (ABI), and instruction set architecture (ISA).

Typology: Lecture notes

2016/2017

Uploaded on 05/11/2023

houhou 🇺🇸

(7)

269 documents

1 / 86

This page cannot be seen from the preview

Don't miss anything!

ECE/CS 757: Advanced

Computer Architecture II

Instructor:Mikko H Lipasti

Spring 2017

University of Wisconsin-Madison

Lecture notes based on slides created by John Shen,

Mark Hill, David Wood, Guri Sohi, Jim Smith, Natalie

Enright Jerger, Michel Dubois, Murali Annavaram,

Per Stenström and probably others

Partial preview of the text

Download Lecture Notes on Advanced Computer Architecture II and more Lecture notes Computer Architecture and Organization in PDF only on Docsity!

ECE/CS 757: Advanced

Computer Architecture II

Instructor:Mikko H Lipasti

Spring 2017

University of Wisconsin-Madison

Lecture notes based on slides created by John Shen,

Mark Hill, David Wood, Guri Sohi, Jim Smith, Natalie

Enright Jerger, Michel Dubois, Murali Annavaram,

Per Stenström and probably others

Lecture Outline

Introduction to Parallel Software
- Sources of parallelism
- Expressing parallelism
Programming Models
Major Abstractions
- Processes & threads
- Communication
- Synchronization
Shared Memory
- API description
- Implementation at ABI, ISA levels
- ISA support
Message Passing
- API description
- Implementation at ABI, ISA levels
- ISA support Mikko Lipasti-University of Wisconsin 2

Finding Parallelism

1. Functional parallelism

– Car: {engine, brakes, entertain, nav, …}

– Game: {physics, logic, UI, render, …}

– Signal processing: {transform, filter, scaling, …}

2. Automatic extraction

– Decompose serial programs

3. Data parallelism

– Vector, matrix, db table, pixels, …

4. Request parallelism

– Web, shared database, telephony, …

Mikko Lipasti-University of Wisconsin 4

1. Functional Parallelism

1. Functional parallelism

– Car: {engine, brakes, entertain, nav, …}

– Game: {physics, logic, UI, render, …}

– Signal processing: {transform, filter, scaling, …}

• Relatively easy to identify and utilize

• Provides small-scale parallelism

– 3x-10x

• Balancing stages/functions is difficult

Mikko Lipasti-University of Wisconsin 5

3. Data Parallelism

3. Data parallelism

– Vector, matrix, db table, pixels, web pages,…

• Large data => significant parallelism

• Many ways to express parallelism

– Vector/SIMD

– Threads, processes, shared memory

– Message-passing

• Challenges:

– Balancing & coordinating work

– Communication vs. computation at scale

Mikko Lipasti-University of Wisconsin 7

4. Request Parallelism

• Multiple users => significant parallelism

• Challenges

Synchronization, communication, balancing work Web Browsing Users Web Server(s) Database Server(s) Mikko Lipasti-University of Wisconsin 8

Coordinating Work

• Synchronization

Some data somewhere is shared
Coordinate/order updates and reads
Otherwise  chaos

• Traditionally: locks and mutual exclusion

Hard to get right, even harder to tune for perf.

• Research to reality: Transactional Memory

Programmer: Declare potential conflict
Hardware and/or software: speculate & check
Commit or roll back and retry
IBM, Intel, others, now support in HW Mikko Lipasti-University of Wisconsin 10

Expressing Parallelism

• SIMD – introduced by Cray-1 vector supercomputer

MMX, SSE/SSE2/SSE3/SSE4, AVX at small scale

• SPMD or SIMT – GPGPU model (later)

All processors execute same program on disjoint data
Loose synchronization vs. rigid lockstep of SIMD

• MIMD – most general (this lecture)

Each processor executes its own program

• Expressed through standard interfaces

– API, ABI, ISA

Mikko Lipasti-University of Wisconsin 11

Programming Models

High level paradigm for expressing an algorithm
- Examples:
  - Functional
  - Sequential, procedural
  - Shared memory
  - Message Passing
Embodied in high level languages that support concurrent execution - Incorporated into HLL constructs - Incorporated as libraries added to existing sequential language
Top level features:
- For conventional models – shared memory, message passing
- Multiple threads are conceptually visible to programmer
- Communication/synchronization are visible to programmer (c) 2007 Jim Smith 13

Application Programming

Interface (API)

Interface where HLL programmer works
High level language plus libraries
- Individual libraries are sometimes referred to as an “API”
User level runtime software is often part of API implementation - Executes procedures - Manages user-level state
Examples:
- C and pthreads
- FORTRAN and MPI (c) 2007 Jim Smith 14

Instruction Set Architecture (ISA)

Interface between hardware and software
- What the hardware implements
Architected state
- Registers
- Memory architecture
All instructions
- May include parallel (SIMD) operations
- Both non-privileged and privileged
Exceptions (traps, interrupts) (c) 2007 Jim Smith 16

Programming Model Elements

For both Shared Memory and Message Passing
Processes and threads
- Process: A shared address space and one or more threads of control
- Thread: A program sequencer and private address space
- Task : Less formal term – part of an overall job
- Created, terminated, scheduled, etc.
Communication
- Passing of data
Synchronization
- Communicating control information
- To assure reliable, deterministic communication (c) 2007 Jim Smith 17

Shared Memory

Flat shared memory or object heap
- Synchronization via memory variables enables reliable sharing
Single process
Multiple threads per process
- Private memory per thread
Typically built on shared memory hardware system T h r e a d 1 P r i v a t e V a r i a b l e s T h r e a d 1 T h r e a d 2 T h r e a d^ N

Lecture Notes on Advanced Computer Architecture II, Lecture notes of Computer Architecture and Organization

Related documents

Partial preview of the text

Download Lecture Notes on Advanced Computer Architecture II and more Lecture notes Computer Architecture and Organization in PDF only on Docsity!

ECE/CS 757: Advanced

Computer Architecture II

Instructor:Mikko H Lipasti

Spring 2017

University of Wisconsin-Madison

Lecture notes based on slides created by John Shen,

Mark Hill, David Wood, Guri Sohi, Jim Smith, Natalie

Enright Jerger, Michel Dubois, Murali Annavaram,

Per Stenström and probably others

Finding Parallelism

1. Functional parallelism

– Car: {engine, brakes, entertain, nav, …}

– Game: {physics, logic, UI, render, …}

– Signal processing: {transform, filter, scaling, …}

2. Automatic extraction

– Decompose serial programs

3. Data parallelism

– Vector, matrix, db table, pixels, …

4. Request parallelism

– Web, shared database, telephony, …

1. Functional Parallelism

1. Functional parallelism

– Car: {engine, brakes, entertain, nav, …}

– Game: {physics, logic, UI, render, …}

– Signal processing: {transform, filter, scaling, …}

• Relatively easy to identify and utilize

• Provides small-scale parallelism

– 3x-10x

• Balancing stages/functions is difficult

3. Data Parallelism

3. Data parallelism

– Vector, matrix, db table, pixels, web pages,…

• Large data => significant parallelism

• Many ways to express parallelism

– Vector/SIMD

– Threads, processes, shared memory

– Message-passing

• Challenges:

– Balancing & coordinating work

– Communication vs. computation at scale

4. Request Parallelism

• Multiple users => significant parallelism

• Challenges

Coordinating Work

• Synchronization

• Traditionally: locks and mutual exclusion

• Research to reality: Transactional Memory

Expressing Parallelism

• SIMD – introduced by Cray-1 vector supercomputer

• SPMD or SIMT – GPGPU model (later)

• MIMD – most general (this lecture)

• Expressed through standard interfaces

– API, ABI, ISA

Programming Models

Application Programming

Interface (API)

Instruction Set Architecture (ISA)

Programming Model Elements

Shared Memory

Threads and Processes

• Creation

– generic -- Fork

– pthread_create(….*thread_function….)

• Termination

– pthread_exit

– pthread_kill