Introduction to probability george g. roussas 2e, Lecture notes of Social Statistics and Data Analysis

Download Introduction to probability george g. roussas 2e and more Lecture notes Social Statistics and Data Analysis in PDF only on Docsity!

This page is intentionally left blank

Second Edition

George G. Roussas

Introduction to

Probability

AMSTERDAM • BOSTON • HEIDELBERG • LONDON • NEW YORK OXFORD • PARIS • SAN DIEGO • SAN FRANCISCO • SINGAPORE SYDNEY • TOKYO Academic Press is an imprint of Elsevier

Department of Statistics

University of California, Davis

To the memory of David Blackwell and Lucien Le Cam

This page is intentionally left blank

viii Contents

CHAPTER 7� Joint Probability Density Function of Two Random

Variables and Related Quantities �������������������������������� 137

7 � 1 Joint d.f. and Joint p.d.f. of Two Random Variables ................... 137 7 � 2 Marginal and Conditional p.d.f.'s, Conditional Expectation and Variance ................................................................................ 149

CHAPTER 8� Joint Moment-Generating Function, Covariance,

and Correlation Coefficient of Two Random

Variables ������������������������������������������������������������������������� 163

8 � 1 The Joint m.g.f. of Two Random Variables ................................ 163 8 � 2 Covariance and Correlation Coefficient of Two Random Variables ....................................................................... 167 8 � 3 Proof of Theorem 1, Some Further Results ................................ 174

CHAPTER 9� Some Generalizations to k Random Variables,

and Three Multivariate Distributions �������������������������� 179

9 � 1 Joint Distribution of k Random Variables and Related Quantities ................................................................ 179 9 � 2 Multinomial Distribution ............................................................ 183 9 � 3 Bivariate Normal Distribution .................................................... 189 9 � 4 Multivariate Normal Distribution ............................................... 198

CHAPTER 10� Independence of Random Variables and Some

Applications������������������������������������������������������������������ 201

10 � 1 Independence of Random Variables and Criteria of Independence .............................................................................. 201 10 � 2 The Reproductive Property of Certain Distributions .................. 213 10 � 3 Distribution of the Sample Variance under Normality................ 222

CHAPTER 11� Transformation of Random Variables������������������������ 225

11 � 1 Transforming a Single Random Variable .................................... 225 11 � 2 Transforming Two or More Random Variables .......................... 231 11 � 3 Linear Transformations ............................................................... 245 11 � 4 The Probability Integral Transform ............................................ 253 11 � 5 Order Statistics............................................................................ 255

CHAPTER 12� Two Modes of Convergence, the Weak Law

of Large Numbers, the Central Limit Theorem,

and Further Results ����������������������������������������������������� 265

12 � 1 Convergence in Distribution and in Probability ......................... 265 12 � 2 The Weak Law of Large Numbers and the Central Limit Theorem ............................................................................ 271

Contents ix

• CHAPTER 1� Some Motivating Examples ���������������������������������������������� Preface ��������������������������������������������������������������������������������������������������������xi
• CHAPTER 2� Some Fundamental Concepts �������������������������������������������
  • 2 � 1 Some Fundamental Concepts
  • 2 � 2 Some Fundamental Results...........................................................
  • 2 � 3 Random Variables
  • 2 � 4 Basic Concepts and Results in Counting
• CHAPTER 3� The Concept of Probability and Basic Results ������������
  • 3 � 1 Definition of Probability
  • 3 � 2 Some Basic Properties and Results...............................................
  • 3 � 3 Distribution of a Random Variable
• CHAPTER 4� Conditional Probability and Independence ������������������
  • 4 � 1 Conditional Probability and Related Results
  • 4 � 2 Independent Events and Related Results
• CHAPTER 5� Numerical Characteristics of a Random Variable������� - Random Variable........................................................................... 5 � 1 Expectation, Variance, and Moment-Generating Function of a
  • 5 � 2 Some Probability Inequalities
  • 5 � 3 Median and Mode of a Random Variable
• CHAPTER 6� Some Special Distributions ��������������������������������������������
  • 6 � 1 Some Special Discrete Distributions
    • 6 � 1 � 1 Binomial Distribution
    • 6 � 1 � 2 Geometric Distribution
    • 6 � 1 � 3 Poisson Distribution
    • 6 � 1 � 4 Hypergeometric Distribution
  • 6 � 2 Some Special Continuous Distributions
    • 6 � 2 � 1 Gamma Distribution
    • 6 � 2 � 2 Negative Exponential Distribution
    • 6 � 2 � 3 Chi-Square Distribution
    • 6 � 2 � 4 Normal Distribution
    • 6 � 2 � 5 Uniform (or Rectangular) Distribution
    • 6 � 2 � 6 The basics of the Central Limit Theorem(CLT)
    • 12 � 2 � 1 Applications of the WLLN
    • 12 � 2 � 2 Applications of the CLT
    • 12 � 2 � 3 The Continuity Correction
  • 12 � 3 Further Limit Theorems
• CHAPTER 13� An Overview of Statistical Inference ������������������������
  • 13 � 1 The Basics of Point Estimation...................................................
  • 13 � 2 The Basics of Interval Estimation
  • 13 � 3 The Basics of Testing Hypotheses
  • 13 � 4 The Basics of Regression Analysis
  • 13 � 5 The Basics of Analysis of Variance
  • 13 � 6 The Basics of Nonparametric Inference
• Appendix �������������������������������������������������������������������������������������������������
• Some Notation and Abbreviations�������������������������������������������������������
• Answers to Even-Numbered Exercises �����������������������������������������������
• Index ��������������������������������������������������������������������������������������������������������

xi

Overview

This book is an introductory textbook in probability. No prior knowledge in prob- ability is required; however, previous exposure to an elementary precalculus course in probability would prove beneficial in that the student would not see the basic con- cepts discussed here for the first time. The mathematical prerequisite is a year of calculus. Familiarity with the basic concepts of linear algebra would also be helpful in certain instances. Often students are exposed to such basic concepts within the calculus framework. Elementary dif- ferential and integral calculus will suffice for the majority of the book. In some parts of Chapters 7–11 the concept of a multiple integral is used. In Chapter 11, the student is expected to be at least vaguely familiar with the basic techniques of changing vari- ables in a single or a multiple integral.

Chapter Descriptions

The material discussed in this book is enough for a one-semester course in introduc- tory probability. This would include a rather detailed discussion of Chapters 1– except, perhaps, for the derivations of the probability density functions following Definitions 1 and 2 in Chapter 11. It could also include a cursory discussion of Chapter 13. Most of the material in Chapters 1–12—with a quick description of the basic concepts in Chapter 13—can also be covered in a one-quarter course in introductory probability. In such a case, the instructor would omit the derivations of the probabil- ity density functions mentioned above, as well as Sections 9.4, 10.3, 11.3, and 12.2.3. A chapter-by-chapter description follows. Chapter 1 consists of 16 examples selected from a broad variety of applications. Their purpose is to impress upon the student the breadth of applications of probability, and draw attention to the wide range of situations in which probability questions are pertinent. At this stage, one could not possibly provide answers to the questions posed without the methodology developed in the subsequent chapters. Answers to most of these questions are given in the form of examples and exercises throughout the remaining chapters. In Chapter 2, the concept of a random experiment is introduced, along with related concepts and some fundamental results. The concept of a random variable is also introduced here, along with the basics in counting. Chapter 3 is devoted to the introduction of the con- cept of probability and the discussion of some basic properties and results, including the distribution of a random variable. Conditional probability, related results, and independence are covered in Chapter

4. The quantities of expectation, variance, moment-generating function, median, and

Preface

xii Preface

mode of a random variable are discussed in Chapter 5, along with some basic prob- ability inequalities. The next chapter, Chapter 6, is devoted to the discussion of some of the com- monly used discrete and continuous distributions. When two random variables are involved, one talks about their joint distribution, as well as marginal and conditional probability density functions and conditional expectation and variance. The relevant material is discussed in Chapter 7. The dis- cussion is pursued in Chapter 8 with the introduction of the concepts of covariance and correlation coefficient of two random variables. The generalization of concepts in Chapter 7, when more than two random vari- ables are involved, is taken up in Chapter 9, which concludes with the discussion of two popular multivariate distributions and the citation of a third such distribution. Independence of events is suitably carried over to random variables. This is done in Chapter 10, in which some consequences of independence are also discussed. In addition, this chapter includes a result, Theorem 6 in Section 10.3, of significant importance in statistics. The next chapter, Chapter 11, concerns itself with the problem of determining the distribution of a random variable into which a given random variable is transformed. The same problem is considered when two or more random variables are transformed into a set of new random variables. The relevant results are mostly simply stated, as their justification is based on the change of variables in a single or a multiple integral, which is a calculus problem. The last three sections of the chapter are concerned with three classes of special but important transformations. The book is essentially concluded with Chapter 12, in which two of the most important results in probability are studied, namely, the weak law of large numbers and the central limit theorem. Some applications of these theorems are presented, and the chapter is concluded with further results that are basically a combination of the weak law of large numbers and the central limit theorem. Not only are these additional results of probabilistic interest, they are also of substantial statistical importance. As previously mentioned, the last chapter of the book provides an overview of statistical inference.

Features This book has a number of features that may be summarized as follows. It starts with a brief chapter consisting exclusively of examples that are meant to provide motiva- tion for studying probability. It lays out a substantial amount of material—organized in twelve chapters—in a logical and consistent manner. Before entering into the discussion of the concept of probability, it gathers together all needed fundamental concepts and results, including the basics in counting. The concept of a random variable and its distribution, along with the usual

xiv Preface

natural logarithm of x (the logarithm of x with base e ). The rule for the use of decimal numbers is that we retain three decimal digits, the last of which is rounded up to the next higher number (if the fourth decimal is greater or equal to 5). An exemption to this rule is made when the division is exact, and when the numbers are read out of tables. On several occasions, the reader is referred to proofs for more comprehensive treatment of some topics in the book A Course in Mathematical Statistics , 2nd edition (1997), Academic Press, by G.G. Roussas. Thanks are due to my project assistant, Carol Ramirez, for preparing a beautiful set of typed chapters out of a collection of messy manuscripts.

Preface to the Second Edition This is a revised version of the first edition of the book, copyrighted by Elsevier Inc, 2007. The revision consists in correcting typographical errors, some factual oversights, rearranging some of the material, and adding a handful of new section exercises. Also, a substantial number of hints is added to exercises to facilitate their solu- tions, and extensive cross-references are made as to where several of the examples mentioned are discussed in detail. Furthermore, a brief discussion of the central limit theorem is included early on in Chapter 6, Section 6.2.3. It is done so, because of its great importance in probability theory, and on the suggestion of a reviewer of the book. Finally, a substantial number of exercises have been added at the end of each chapter. However, because of inadequate time to have them included in the book itself, these exercises will be posted (booksite.elsevier.com/9780128000410), and their answers will be made available to those instructors using this book as a textbook. In all other respects, the book is unchanged, and the overview, chapter descrip- tion, features, and guiding comments in the preface of the first edition remain the same. The revision was skillfully implemented by my associates Michael McAssey and Chu Shing (Randy) Lai, to whom I express herewith my sincere appreciation. George Roussas Davis, California June 2013

CHAPTER

Some Motivating Examples

This chapter consists of a single section that is devoted to presenting a number of examples (16 to be precise), drawn from a broad spectrum of human activities. Their purpose is to demonstrate the wide applicability of probability (and statistics). In each case, several relevant questions are posed, which, however, cannot be answered here. Most of these questions are dealt with in subsequent chapters. In the formulation of these examples, certain terms, such as at random, average, data fit by a line, event, probability (estimated probability, probability model), rate of success, sample, and sampling (sample size), are used. These terms are presently to be understood in their everyday sense, and will be defined precisely later on.

Example 1. In a certain state of the Union, n landfills are classified according to their concentration of three hazardous chemicals: arsenic, barium, and mercury. Suppose that the concentration of each one of the three chemicals is characterized as either high or low. Then some of the questions that can be posed are as follows:

(i) If a landfill is chosen at random from among the n , what is the probability it is of a specific configuration? In particular, what is the probability that it has: (a) High concentration of barium? (b) High concentration of mercury and low concentration of both arsenic and barium? (c) High concentration of any two of the chemicals and low concentration of the third? (d) High concentration of any one of the chemicals and low concentration of the other two?

(ii) How can one check whether the proportions of the landfills falling into each one of the eight possible configurations (regarding the levels of concentration) agree with a priori stipulated numbers?

(See some brief comments in Chapter 2, and Example 1 in Chapter 3.)

Example 2. Suppose a disease is present in 100 p 1 % ( 0 < p 1 < 1 ) of a population. A diagnostic test is available but is yet to be perfected. The test shows 100 p 2 % false positives ( 0 < p 2 < 1 ) and 100 p 3 % false negatives ( 0 < p 3 < 1 ). That is, for a patient not having the disease, the test shows positive (+) with probability p 2 and negative (−) with probability 1 − p 2. For a patient having the disease, the test shows

Introduction to Probability, Second Edition. http://dx.doi.org/10.1016/B978-0-12-800041-0.00001-

Copyright © 2014 Elsevier Inc. All rights reserved.^1

Introduction to Probability 3

(ii) What is the probability that at most n /5 responses are correct? (iii) What is the probability that at least n /2 responses are correct?

(See a brief discussion in Chapter 2, and Theorem 1 in Chapter 6.)

Example 6. A government agency wishes to assess the prevailing rate of unem- ployment in a particular county. It is felt that this assessment can be done quickly and effectively by sampling a small fraction n , say, of the labor force in the county. The obvious questions to be considered here are:

(i) What is a suitable probability model describing the number of unemployed? (ii) What is an estimate of the rate of unemployment?

Example 7. Suppose that, for a particular cancer, chemotherapy provides a 5-year survival rate of 75% if the disease could be detected at an early stage. Suppose further that n patients, diagnosed to have this form of cancer at an early stage, are just starting the chemotherapy. Finally, let X be the number of patients among the n who survive 5 years. Then the following are some of the relevant questions that can be asked:

(i) What are the possible values of X , and what are the probabilities that each one of these values is taken on? (ii) What is the probability that X takes values between two specified numbers a and b , say? (iii) What is the average number of patients to survive 5 years, and what is the variation around this average?

Example 8. An advertisement manager for a radio station claims that over 100 p % ( 0 < p < 1 ) of all young adults in the city listen to a weekend music program. To establish this conjecture, a random sample of size n is taken from among the target population and those who listen to the weekend music program are counted.

(i) Decide on a suitable probability model describing the number of young adults who listen to the weekend music program. (ii) On the basis of the collected data, check whether the claim made is supported or not. (iii) How large a sample size n should be taken to ensure that the estimated average and the true proportion do not differ in absolute value by more than a specified number with prescribed (high) probability?

Example 9. When the output of a production process is stable at an acceptable standard, it is said to be “in control.” Suppose that a production process has been in control for some time and that the proportion of defectives has been p. As a means of monitoring the process, the production staff will sample n items. Occurrence of k or more defectives will be considered strong evidence for “out of control.”

4 CHAPTER 1 Some Motivating Examples

(i) Decide on a suitable probability model describing the number X of defectives; what are the possible values of X , and what is the probability that each of these values is taken on? (ii) On the basis of the data collected, check whether or not the process is out of control. (iii) How large a sample size n should be taken to ensure that the estimated propor- tion of defectives will not differ in absolute value from the true proportion of defectives by more than a specified quantity with prescribed (high) probability?

Example 10. At a given road intersection, suppose that X is the number of cars passing by until an observer spots a particular make of a car (e.g., a Mercedes). Then some of the questions one may ask are as follows:

(i) What are the possible values of X? (ii) What is the probability that each one of these values is taken on? (iii) How many cars would the observer expect to observe until the first Mercedes appears?

(See a brief discussion in Chapter 2, and Section 6.1.2 in Chapter 6.) Example 11. A city health department wishes to determine whether the mean bacteria count per unit volume of water at a lake beach is within the safety level of

200. A researcher collected n water samples of unit volume and recorded the bacteria counts. Relevant questions here are:

(i) What is the appropriate probability model describing the number X of bacteria in a unit volume of water; what are the possible values of X , and what is the probability that each one of these values is taken on? (ii) Do the data collected indicate that there is no cause for concern?

(See a brief discussion in Chapter 2, and Section 6.1.3 in Chapter 6.) Example 12. Measurements of the acidity (pH) of rain samples were recorded at n sites in an industrial region.

(i) Decide on a suitable probability model describing the number X of the acidity of rain measured. (ii) On the basis of the measurements taken, provide an estimate of the average acidity of rain in that region.

(See a brief discussion in Chapter 2, and Section 6.2.4 in Chapter 6.) Example 13. To study the growth of pine trees at an early state, a nursery worker records n measurements of the heights of 1-year-old red pine seedlings.

(i) Decide on a suitable probability model describing the heights X of the pine seedlings.