Download 1 Probability, Conditional Probability and Bayes Formula and more Exercises Algebra in PDF only on Docsity!
ISyE8843A, Brani Vidakovic Handout 1
1 Probability, Conditional Probability and Bayes Formula
The intuition of chance and probability develops at very early ages.^1 However, a formal, precise definition of the probability is elusive. If the experiment can be repeated potentially infinitely many times, then the probability of an event can be defined through relative frequencies. For instance, if we rolled a die repeatedly, we could construct a frequency distribution table showing how many times each face came up. These frequencies (ni) can be expressed as proportions or relative frequencies by dividing them by the total number of tosses n : fi = ni/n. If we saw six dots showing on 107 out of 600 tosses, that face’s proportion or relative frequency is f 6 = 107/600 = 0. 178 As more tosses are made, we “expect” the proportion of sixes to stabilize around 16.
Famous Coin Tosses: Buffon tossed a coin 4040 times. Heads appeared 2048 times. K. Pearson tossed a coin 12000 times and 24000 times. The heads appeared 6019 times and 12012, respectively. For these three tosses the relative frequencies of heads are 0.5049, 0.5016,and 0.5005.
What if the experiments can not be repeated? For example what is probability that Squiki the guinea pig survives its first treatment by a particular drug. Or “the experiment” of you taking ISyE8843 course in Fall 2004. It is legitimate to ask for the probability of getting a grade of an A. In such cases we can define probability subjectively as a measure of strength of belief.
Figure 1: A gem proof condition 1913 Liberty Head nickel, one of only five known and the finest of the five. Collector Jay Parrino of Kansas City bought the elusive nickel for a record $1,485,000, the first and only time an American coin has sold for over $1 million.
Tutubalin’s Problem. In a desk drawer in the house of Mr Jay Parrino of Kansas City there is a coin, 1913 Liberty Head nickel. What is the probability that the coin is heads up?
The symmetry properties of the experiment lead to the classical definition of probability. An ideal die is symmetric. All sides are “equiprobable”. The probability of 6, in our example is a ratio of the number of favorable outcomes (in our example only one favorable outcome, namely, 6 itself) and the number of all possible outcomes, 1/6.^2
(^1) Piaget, J. and Inhelder B. The Origin of the Idea of Chance in Children, W. W. Norton & Comp., N.Y. (^2) This definition is attacked by philosophers because of the fallacy called circulus vitiosus. One defines the notion of probability supposing that outcomes are equiprobable.
(Frequentist) An event’s probability is the proportion of times that we expect the event to occur, if the experiment were repeated a large number of times.
(Subjectivist) A subjective probability is an individual’s degree of belief in the occurrence of an event.
(Classical) An event’s probability is the ratio of the number of favorable outcomes and possible outcomes in a (symmetric) experiment.
Term Description Example
Experiment Phenomenon where outcomes are un- certain
Single throws of a six-sided die
Sample space Set of all outcomes of the experiment
S =
{ 1 , 2 , 3 , 4 , 5 , 6 }, (1, 2 , 3 , 4 , 5 , or 6 dots show)
Event A collection of outcomes; a subset of S
A = { 3 } (3 dots show), B = { 3 , 4 , 5 , or 6 } (3, 4, 5, or 6 dots show) or ’at least three dots show’
Probability A number between 0 and 1 assigned to an event.
P (A) = 16. P (B) = 46.
Sure event occurs every time an experiment is repeated and has the probability 1. Sure event is in fact the sample space S. An event that never occurs when an experiment is performed is called impossible event. The probability of an impossible event, denoted usually by ∅ is 0.
For any event A, the probability that A will occur is a number between 0 and 1, inclusive: 0 ≤ P (A) ≤ 1 ,
P (∅) = 0, P (S) = 1.
The intersection (product) A · B of two events A and B is an event that occurs if both events A and B occur. The key word in the definition of the intersection is and. In the case when the events A and B are independent the probability of the intersection is the product of probabilities: P (A · B) = P (A)P (B). Example: The outcomes of two consecutive flips of a fair coin are independent events. Events are said to be mutually exclusive if they have no outcomes in common. In other words, it is impossible that both could occur in a single trial of the experiment. For mutually exclusive events holds P (A · B) = P (∅) = 0.
These equations simplify solutions of some probability problems. If P (Ac) is easier to calculate than P (A), then P (Ac) and equations above let us obtain P (A) indirectly. This and some other properties of probability are summarized in table below.
Property Notation
If event S will always occur, its probability is 1. P (S) = 1
If event ∅ will never occur, its probability is 0. P (∅) = 0
Probabilities are always between 0 and 1, inclusive 0 ≤ P (A) ≤ 1
If A, B, C,... are all mutually exclusive then P (A ∪ B ∪ C... ) can be found by addition.
P (A ∪ B ∪ C... ) = P (A) +
P (B) + P (C) +...
If A and B are mutually exclusive then P (A ∪ B) can be found by addition. P^ (A^ ∪^ B) =^ P^ (A) +^ P^ (B)
Addition rule: The general addition rule for probabilities
P (A ∪ B) = P (A) + P (B) −
P (A · B)
Since A and Ac^ are mutually exclusive and between them include all possible outcomes, P (A ∪ Ac) is 1.
P (A ∪ Ac) = P (A) + P (Ac) = P (S) = 1, and P (Ac) = 1 − P (A)
2 Conditional Probability and Independence
A conditional probability is the probability of one event if another event occurred. In the “die-toss” example, the probability of event A, three dots showing, is P (A) = 16 on a single toss. But what if we know that event B, at least three dots showing, occurred? Then there are only four possible outcomes, one of which is A. The probability of A = { 3 } is 14 , given that B = { 3 , 4 , 5 , 6 } occurred. The conditional probability of A given B is written P (A|B).
P (A|B) =
P (A · B)
P (B)
Event A is independent of B if the conditional probability of A given B is the same as the unconditional probability of A. That is, they are independent if
P (A|B) = P (A)
In the die-toss example, P (A) = 16 and P (A|B) = 14 , so the events A and B are not independent.
The probability that two events A and B will both occur is obtained by applying the multiplication rule:
P (A · B) = P (A)P (B|A) = P (B)P (A|B)
where P (A|B) (P (B|A)) means the probability of A given B (B given A). For independent events only, the equation in the box simplifies to P (A · B) = P (A)P (B).
- Prove P (A 1 A 2... An) = P (A 1 |A 2... An) P (A 2 |A 3... An)... P (An− 1 |An) P (An). Example: Let the experiment involves a random draw from a standard deck of 52 playing cards. Define events A and B to be ”the card is ♠ and “the card is queen”. Are the events A and B independent? By definition, P (A&B) = P (Q♠) = 521. This is the product of P (♠) = 1352 and P (Q) = 524 , and A and B in question are independent. The intuition barely helps here. Pretend that from the original deck of cards 2 ♥ is excluded prior to the experiment. Now the events A and B become dependent since
P (A) · P (B) =
= P (A&B).
The multiplication rule tells us how to find probabilities for composite event (A · B). The probability of (A · B) is used in the general addition rule for finding the probability of (A ∪ B).
Rule Notation
Definitions The conditional probability of A given B is the probability of event A, if event B occurred.
P (A|B)
A is independent of B if the conditional probability of A given B is the same as the unconditional probability of A.
P (A|B) = P (A)
Multiplication rule: The general multiplication rule for probabilities
P (A · B) = P (A)P (B|A) =
P (B)P (A|B)
For independent events only, the multiplication rule is sim- plified. P^ (A^ ·^ B) =^ P^ (A)P^ (B)
3 Pairwise and Global Independence
If three events A, B, and C are such that any of the pairs are exclusive, i.e., AB = ∅, AC = ∅ or BC = ∅, then the events are mutually exclusive, i.e, ABC = ∅. In terms of independence the picture is different. Even if the events are pairwise independent for all three pairs A, B; A, C; and B, C, i.e., P (AB) = P (A)P (B), P (AC) = P (A)P (C), and P (BC) = P (B)P (C), they may be dependent in the totality, P (ABC) 6 = P (A)P (B)P (C). Here is one example.
(b) second (c) third? Solution: Denote with A the event that Stanley draws a favorable card (and consequently gets an A). (i) If he draws the card first, then clearly P (A) = 8/20 = 2/ 5. (ii) If Stanley draws the second, then one card was taken by the student before him. That first card taken might have been favorable (hypothesis H 1 ) or unfavorable (hypothesis H 2 ). Obviously, the hypotheses H 1 and H 2 partition the sample space since no other type of cards is possible, in this context. Also, the probabilities of H 1 and H 2 are 8/20 and 12/20, respectively. Now, after one card has been taken Stanley draws the second. If H 1 had happened, probability of A is 7/19, and if H 2 had happened, the probability of A is 8/19. Thus, P (A|H 1 ) = 7/ 19 and P (A|H 2 ) = 8/ 19. By the total probability formula, P (A) = 7 / 19 · 8 /20 + 8/ 19 · 12 /20 = 8/20 = 2/ 5. (iii) Stanley has the same probability of getting an A after two cards have been already taken. The hypotheses are H 1 ={ both cards taken favorable }, H 2 ={ exactly one card favorable }, and H 3 ={ none of the cards taken favorable }. P (H 1 ) = 8/ 20 · 7 / 19 , P (H 3 ) = 12/ 20 · 11 / 19. and P (H 2 ) = 1−P (H 1 )−P (H 3 ). Next, P (A|H 1 ) = 6/ 18 , P (A|H 2 ) = 7/ 18 , and P (A|H 3 ) = 8/ 18. Therefore, P (A) = 6/ 18 · 7 / 19 · 8 /20+ 7 / 18 · ... + 8/ 18 · 11 / 19 · 12 /20 = 12/ 20. Moral: Stanley’s lack on the exam does not depend on the order in drawing the examination card.
Two-headed coin Out of 100 coins one has heads on both sides. One coin is chosen at random and flipped two times. What is the probability to get (a) two heads? (b) two tails? Solution: (a) Let A be the event that two heads are obtained. Denote by H 1 the event (hypothesis) that a fair coin was chosen. The hypothesis H 2 = H 1 c is the event that the two-headed coin was chosen.
P (A) = P (A|H 1 )P (H 1 ) + P (A|H 2 )P (H 2 ) = 1/ 4 · 99 /100 + 1 · 1 /100 = 103/400 = 0. 2575.
(b) Exercise. [Ans. 0.2475]
5 Bayes’ Formula
Recall that multiplication rule claims:
P (AH) = P (A)P (H|A) = P (H)P (A|H).
This simple identity is the essence of Bayes’ Formula.
Bayes Formula. Let the event of interest A happens under any of hypotheses Hi with a known (conditional) probability P (A|Hi). Assume, in addition, that the probabili- ties of hypotheses H 1 ,... , Hn are known (prior probabilities). Then the conditional (posterior) probability of the hypothesis Hi, i = 1, 2 ,... , n, given that event A happened, is
P (Hi|A) = P (A|Hi)P (Hi) P (A)
where
P (A) = P (A|H 1 )P (H 1 ) + · · · + P (A|Hn)P (Hn).
Assume that out of N coins in a box, one has heads at both sides. Such “two-headed” coin can be purchased in Spencer stores. Assume that a coin is selected at random from the box, and without inspecting it, flipped k times. All k times the coin landed up heads. What is the probability that two headed coin was selected?
Denote with Ak the event that randomly selected coin lands heads up k times. The hypotheses are H 1 -the coin is two headed, and H 2 the coin is fair. It is easy to see that P (H 1 ) = 1/N and P (H 2 ) = (N − 1)/N. The conditional probabilities are P (Ak|H 1 ) = 1 for any k, and P (Ak|H 2 ) = 1/ 2 k. By total probability formula,
P (Ak) = 2 k^ + N − 1 2 kN
and
P (H 1 |Ak) = 2 k 2 k^ + N − 1
For N = 1, 000 , 000 and k = 1, 2 ,... , 30 the graph of posterior probabilities is given in Figure 2 It is interesting that our prior probability P (H 1 ) = 0. 000001 jumps to posterior probability of 0.9991, after observing 30 heads in a row. The matlab code bayes1 1.m producing Figure 2 is given in the Programs/Codes on the GTBayes Page.
Prosecutor’s Fallacy The prosecutor’s fallacy is a fallacy commonly occurring in criminal trials but also in other various arguments involving rare events. It consists of subtle exchanging of P (A|B) for P (B|A). A zealous prosecutor has collected an evidence, say fingerprint match, and has an expert testify that the probability of finding this evidence if the accused were innocent is tiny. The fallacy is committed the prosecutor proceeds to claim that the probability of the accused being innocent is comparably tiny. Why is this incorrect? Suppose there is a one-in-a-million chance of a match given that the accused is innocent. The prosector deduces that means there is only a one-in-a-million chance of innocence. But in a community of 10 million people, one expects 10 matches, and the accused is just one of those ten. That would indicate only a one-in-ten chance of guilt, if no other evidence is available.
Two Masked Robbers. Two masked robbers try to rob a crowded bank during the lunch hour but the teller presses a button that sets off an alarm and locks the front door. The robbers realizing they are trapped, throw away their masks and disappear into the chaotic crowd. Confronted with 40 people claiming they are innocent, the police gives everyone a lie detector test. Suppose that guilty people are detected with probability 0.85 and innocent people appear to be guilty with probability 0.08. What is the probability that Mr. Smith was one of the robbers given that the lie detector says he is?
Guessing. Subject in an experiment are told that either a red or a green light will flash. Each subject is to guess which light will flash. The subject is told that the probability of a red light is 0.7, independent of guesses. Assume that the subject is a probability matcher- that is , guesses red with probability .70 and green with probability .30. (i) What is the probability that the subject guesses correctly? (ii) Given that a subject guesses correctly, what is the probability that the light flashed red?
False Positives. False positives are a problem in any kind of test: no test is perfect, and sometimes the test will incorrectly report a positive result. For example, if a test for a particular disease is performed on a patient, then there is a chance (usually small) that the test will return a positive result even if the patient does not have the disease. The problem lies, however, not just in the chance of a false positive prior to testing, but determining the chance that a positive result is in fact a false positive. As we will demonstrate, using Bayes’ theorem, if a condition is rare, then the majority of positive results may be false positives, even if the test for that condition is (otherwise) reasonably accurate. Suppose that a test for a particular disease has a very high success rate:
- if a tested patient has the disease, the test accurately reports this, a ’positive’, 99% of the time (or, with probability 0.99), and
- if a tested patient does not have the disease, the test accurately reports that, a ’negative’, 95% of the time (i.e. with probability 0.95). Suppose also, however, that only 0.1% of the population have that disease (i.e. with probability 0.001). We now have all the information required to calculate the probability that, given the test was positive, that it is a false positive. Let D be the event that the patient has the disease, and P be the event that the test returns a positive result. The probability of a true positive is
P (D|P ) =
0. 99 × 0. 001
0. 99 × 0 .001 + 0. 05 × 0. 999
and hence the probability of a false positive is about (1 - 0.019) = 0.981. Despite the apparent high accuracy of the test, the incidence of the disease is so low (one in a thousand) that the vast majority of patients who test positive (98 in a hundred) do not have the disease. Nonetheless, this is 20 times the proportion before we knew the outcome of the test! The test is not useless, and re- testing may improve the reliability of the result. In particular, a test must be very reliable in reporting a negative result when the patient does not have the disease, if it is to avoid the problem of false positives. In mathematical terms, this would ensure that the second term in the denominator of the above calculation is small, relative to the first term. For example, if the test reported a negative result in patients without the disease with probability 0.999, then using this value in the calculation yields a probability of a false positive of roughly 0.5.
Multiple Choice. A student answers a multiple choice examination question that has 4 possible answers. Suppose that the probability that the student knows the answer to the question is 0.80 and the probability that the student guesses is 0.20. If student guesses, probability of correct answer is 0.25. (i) What is the probability that the fixed question is answered correctly? (ii) If it is answered correctly what is the probability that the student really knew the correct answer.
Manufacturing Bayes. Factory has three types of machines producing an item. Probabilities that the item is I quality f it is produced on i-th machine are given in the following table:
machine probability of I quality 1 0. 2 0. 3 0.
The total production is done 30% on type I machine, 50% on type II, and 20% on type III. One item is selected at random from the production. (i) What is the probability that it is of I quality? (ii) If it is of first quality, what is the probability that it was produced on the machine I?
Two-headed coin. 4 One out of 1000 coins has two tails. The coin is selected at random out of these 1000 coins and flipped 5 times. If tails appeared all 5 times, what is the probability that the selected coin was ‘two-tailed’?
Kokomo, Indiana. In Kokomo, IN, 65% are conservatives, 20% are liberals and 15% are independents. Records show that in a particular election 82% of conservatives voted, 65% of liberals voted and 50% of independents voted. If the person from the city is selected at random and it is learned that he/she did not vote, what is the probability that the person is liberal?
Inflation and Unemployment. Businesses commonly project revenues under alternative economic scenar- ios. For a stylized example, inflation could be high or low and unemployment could be high or low. There are four possible scenarios, with the assumed probabilities:
Scenario Inflation Unemployment Probability 1 high high 0. 2 high low 0. 3 low high 0. 4 low low 0.
(i) What is the probability of high inflation? (ii) What is the probability of high inflation if unemployment is high? (iii) Are inflation and unemployment independent?
References
[1] Batt, J. Stolen Innocence: A Mother’s Fight for Justice. The Authorised Story of Sally Clark, Ebury Press.
[2] Barbeau, E. (1993). The Problem of the Car and Goats, CMJ, 24:2, p. 149
[3] Casscells, W., Schoenberger, A., and Grayboys, T. (1978). Interpretation by physicians of clinical laboratory results. New England Journal of Medicine, 299 , 999-1000.
[4] Gillman, L. (1992). The Car and the Goats,AMM 99:1, p. 3
[5] Selvin, S. (1975). A Problem in Probability, American Statistician, 29:1, p. 67
