RANDOM EXPERIMENTS Definition: For a specific random experiment, its sample space consists of a list (set) of all possible outcomes. The individual (complete) outcomes are called simple events. If we are lucky, all simple events are equally likely. A few EXAMPLES: 1. Rolling a die {1, 2, 3, 4, 5, 6} 2. Rolling a die n times (rolling n dice at once). The sample space is the set of all 6n n-digit numbers, made up of 1, 2, ....6. For n = 2, this yields 11 12 13 14 15 16 21 22 23 24 25 26 31 32 33 34 35 36 41 42 43 44 45 46 51 52 53 54 55 56 61 62 63 64 65 66 Note that the order is important (12 and 21 are considered diﬀerent). This makes the simple events equally likely. 3. Selecting k objects out of n (2 people out of 4, 5 cards out of 52, etc.). Here, the SS consists of Ckn unordered sections (all equally likely), such as {AB, AC, AD, BC, BD, CD}. 4. Flipping a coin until a head appears. This time, the SS is (countably) infinite: {H, T H, T T H, T T T H, T T T T H, T T T T T H, ....}, which implies that the outcomes can no longer be equally likely. 1 5. Rotating a wheel with a pointer. The SS consists of all angles (at which the pointer stops) from 0 to 2π (in radians). The set of all such real numbers is infinity, yet they are still equally likely. This implies that the probability of any of them is 0, and we have to introduce the concept of probability density function instead (second half of the course). 6. Flipping a tack (⊥). The sample space is very simple: {⊥, i}, but how do we assign probabilities? This can be done only empirically (based on the observed frequencies of occurrence) - until this is done, we have to assume that the probabilities are p and 1 − p. An event is a subset of the sample space, usually denoted by a capital letter from the beginning of the alphabet. EXAMPLES (rolling a die twice): 1. A: the total number of dots equals 8. 2. B: neither of the two numbers is a six. 3. C: first number smaller than second. Thus, we need a few basics of Set Theory, even though we may occasionally use diﬀerent terminology: Sample Space instead of Universal Set, Simple Event instead of Element, Event instead of Subset, and Null Event instead of Empty Set. But we do use the same intersection (notation: A ∩ B, also A and B), union A ∪ B (A or B - this is the nonexclusive or) and complement A (not A). When dealing with 2 or 3 events, a lot of useful 2 information can be gathered from a Venn diagram, but beyond that, we need the full power of Boolean Algebra: Both ∩ and ∪ (individually) are commutative and associative. Intersection is distributive over union: A ∩ (B ∪ C ∪ ...) = (A ∩ B) ∪ (A ∩ C) ∪ ... Similarly, union is distributive over intersection: A ∪ (B ∩ C ∩ ...) = (A ∪ B) ∩ (A ∪ C) ∩ ... A few trivial rules: A ∩ Ω = A, A ∩ ∅ = ∅, A ∩ A = A, A ∪ Ω = Ω, A ∪ ∅ = A, A ∪ A = A, A ∩ A = ∅, A ∪ A = Ω, A¯ = A. Also, A ⊂ B implies that A ∩ B = A and A ∪ B = B. De Morgan’s Laws: A ∩ B ∩ C ∩ ... = A∪B∪C ∪..., and A ∪ B ∪ C ∪ ... = A∩B∩C ∩... A and B are called (mutually) exclusive (or disjoint) when A ∩ B = ∅. Rules of probability Probability of a simple event is defined as the relative frequency of its occurrence in an ‘infinite’ independent repetition of the corresponding experiment. In many cases, it can be established based on the symmetry of the experiment. Probability of any other event is the sum of probabilities of the simple events it consists of. This immediately implies: Pr(A) ≥ 0 Pr(∅) = 0 Pr(Ω) = 1 Note that Pr(A) = 0 does not necessarily imply that A = ∅, some non-empty events may have a zero probability (we have already seen an example). 3 A less trivial formula is Pr(A ∪ B) = Pr(A) + Pr(B) when A ∩ B = ∅. This implies that Pr(A) = 1 − Pr(A) and Pr(A ∩ B) = Pr(A) − Pr(A ∩ B) as two special cases. When A and B are not exclusive, we have to use Pr(A ∪ B) = Pr(A) + Pr(B) − Pr(A ∩ B) which can be extended to Pr(A ∪ B ∪ C) = Pr(A) + Pr(B) + Pr(C) − Pr(A ∩ B) − Pr(A ∩ C) − Pr(B ∩ C) + Pr(A ∩ B ∩ C) and, in general: Pr(A1 ∪ A2 ∪ A3 ∪ ... ∪ Ak ) = k X i=1 Pr(Ai ) − k X i<j Pr(Ai ∩ Aj ) + k X i<j< Pr(Ai ∩ Aj ∩ A ) −... ± Pr(A1 ∩ A2 ∩ A3 ∩ ... ∩ Ak ) (the plus sign for k odd, the minus sign for k even). The formula computes the probability that at least one of the Ai events happens. The probability of getting exactly one of the Ai events would be similarly computed by: k X i=1 Pr(Ai ) − 2 k X i<j Pr(Ai ∩ Aj ) + 3 k X i<j< Pr(Ai ∩ Aj ∩ A ) −... ± k Pr(A1 ∩ A2 ∩ A3 ∩ ... ∩ Ak ) EXAMPLE: The first k integers are ‘shuﬄed’. What is the probability that they all end up in their ‘natural’ place? Answer: 1 . k! At least one number will be misplaced: 1 − All will be misplaced: Pr(A1 ∩ A2 ∩ ... ∩ Ak ) = 4 1 k! (complement). Pr(A1 ∪ A2 ∪ ... ∪ Ak ) = 1 − Pr(A1 ∪ A2 ∪ ... ∪ Ak ) = 1− k X Pr(Ai ) + i=1 k X i<j Pr(Ai ∩ Aj )− ... ∓ Pr(A1 ∩ A2 ∩ ... ∩ Ak ) = µ ¶ k 1 1 1−k· + · k k(k − 1) 2 µ ¶ 1 k 1 + ... ∓ = − k! 3 k(k − 1)(k − 2) 1 1 1 1 − 1 + − + ... ∓ ≡ Qk 2! 3! k! Exactly i of them will end up in their original position (implying k − i will be misplaced): ¡k¢ i × (k − i)! k! Qk−i 1 × × Qk−i = × Qk−i = k i!(k − i)! k! i! Pi Thus, for example, when k = 5, the probability of 0, 1, 2, 3, 4 and 5 matches is 0 and 1 120 11 3 1 1 , , , , 30 8 6 12 respectively. ¥ The main point of these rules is: Probability of any (Boolean) expression involving A, B, C, ... can be always converted to probabilities involving the individual events and their simple (non-complemented) intersections. EXAMPLES: Pr[(A ∩ B) ∪ B ∩ C] = Pr(A ∩ B) + Pr(B ∩ C) − Pr(A ∩ B ∩ B ∩ C) = Pr(A ∩ B) + 1 − Pr(B ∩ C) − Pr(A ∩ B) + Pr(A ∩ B ∩ B ∩ C) = 1 − Pr(B ∩ C) + Pr(A ∩ B ∩ C) 5 ¥ Pr[(A ∩ B) ∪ C ∪ D] = Pr(A ∩ B) + Pr(C ∪ D) − Pr(A ∩ B ∩ C ∪ D) = Pr(A ∩ B) + 1 − Pr(C ∪ D) − Pr(A ∩ B) + Pr[A ∩ B ∩ (C ∪ D)] = 1 − Pr(C ∪ D) + Pr[(A ∩ B ∩ C) ∪ (A ∩ B ∩ D)] = 1 − Pr(C) − Pr(D) + Pr(C ∩ D) + Pr(A ∩ B ∩ C) + Pr(A ∩ B ∩ D) − Pr(A ∩ B ∩ C ∩ D) ¥ Back to: Four players are dealt 5 cards each. What is the probability that at least one player gets exactly 2 aces? Pr(A1 ∪ A2 ∪ A3 ∪ A4 ) = 4 X i=1 Pr(Ai ) − 4 X i< j Pr(Ai ∩ Aj ) + 0 − 0 = ¡ 4 ¢ ¡ 48 ¢ ¡4¢ ¡48¢ × × 2,2,0 ¡ 52 3,3,42 ¢ 4 × 2 ¡52¢ 3 − 6 × = 15.75% 5 ¥ 5,5,42 There are 100,000 lottery tickets marked 00000 to 99999. One is selected at random. What is the probability that the number on it contains 84 (consecutive, in that order) at least once? Let’s introduce four events: A means that the first two digits of the ticket are 84, B: 84 is found in the second and third position, C: 84 in position three and four, and D: 84 in the 6 last two positions. Pr(A ∪ B ∪ C ∪ D) = Pr(A) + Pr(B) + Pr(C) + Pr(D) − Pr(A ∩ C) − Pr(A ∩ D) − Pr(B ∩ D) + 0 = 4× 10 1, 000 −3× = 0.04 − 0.0003 = 3.97% 100, 000 100, 000 ¥ Tom, Frank Jim and 5 other boys are randomly seated at a round table. What is the probability that neither Tom nor Frank will sit next to Jim? Now we introduce: A: Tom sits next to Jim, and B: Frank sits next to Jim. Then, clearly ¯ = Pr(A ∪ B) = 1 − Pr(A ∪ B) = Pr(A¯ ∩ B) 1 − Pr(A) − Pr(B) + Pr(A ∩ B) = 1− 2 × 6! 2 × 6! 2 × 5! 10 − + = = 47.62% 7! 7! 7! 21 Probability tree and conditional probabilities Consider a random experiment which is done in several stages such as, for example, selecting 3 marbles (one by one, without replacement), from a box containing (originally) 3 red and 5 blue marbles. The easiest way to display possible outcomes of this experiment is 7 to draw a so called probability tree (a graphical representation of a sample space): Each path through this graph represents one simple event. It is easy to assign probabilities to individual branches of this tree (note that the probabilities at each ’fork’ have to add up to one). Let us try to answer a few questions: Which subset represents the event R1 (first marble red); B2 (second marble blue), etc. What are the probabilities of individual branches? 8 3 8 is obviously Pr(R1 ). But how about 5 ; 7 it’s definitely not Pr(B2 ).To give it its proper meaning, we have to introduce the notion of conditional probability of an event (B2 , in this case) given that another event (R1 ) has already happened. The notation: Pr(B2 |R1 ). Similarly 2 6 is Pr(R3 |R1 ∩ B2 ). How do we compute probabilities of simple events (and thus events in general)? Clearly, the probability that the corresponding random experiment will take the rbr path is 38 × 57 × 26 = 5 , 56 etc. Formally, this reads: Pr(R1 ∩ B2 ∩ R3 ) = Pr(R1 ) · Pr(B2 |R1 ) · Pr(R3 |R1 ∩ B2 ) This is the so called product rule, and can be generalized to any number of events: Pr(A ∩ B) = Pr(A) · Pr(B|A) Pr(A ∩ B ∩ C ∩ D) = Pr(A) · Pr(B|A) · Pr(C|A ∩ B) · Pr(D|A ∩ B ∩ C) .. . ¥ EXAMPLE: 4 players are dealt 13 cards each from an ordinary deck (of 52 cards). What is the probability that each player will get exactly one ace? Utilizing the product rule: ) ( 31 )( 36 ) ( 21 )( 24 ) ( 11 )( 12 ) ( 41 )( 48 12 12 12 · · · 1312 = 10.55% 52 39 26 ( 13 ) ( 13 ) ( 13 ) ( 13 ) ¥ In general (i.e. for any random experiment, and any two events), we define the conditional probability of B given A by: Pr(B|A) ≡ Pr(A ∩ B) Pr(A) ¥ 9 EXAMPLE: The experiment consists of rolling two dice (red and green), A is: ’the total number of dots equals 6’ (°), B is: ’the red die shows an even number’ (×). Compute Pr(B|A). Answer: 2/36 5/36 • • • • ° • × × × ⊗ × × • • ° • • • × ⊗ × × × × ° • • • • • × × × × × × = 25 . Note that Pr(A|B) = 2/36 18/36 = 1 9 is diﬀerent (there is no reason to expect them to be related in any way). The meaning of any such conditional probability is as follows: In many such cases we have to assume that the (whole) experiment has already happened, but we get only an incomplete information about its outcome (someone who observed it all can tell us that A has happened, but refuses to tell us anything else). This means that the whole sample space has shrunk to A, and the probabilities of its simple events have to be readjusted accordingly. Using this general definition of conditional probability, we can now compute Pr(B1 |R3 ) = 5+10 1+5+5+10 = 57 . Note that all formulas of probability are valid if we (consistently) insert the same condition, e.g. Pr(B|A) = 1 − Pr(B|A) Pr(A ∪ B|C) = Pr(A|C) + Pr(B|C) − Pr(A ∩ B|C) ¥ 10 Definition: Partition of a sample space is a collection of k (any integer) events, say A1 , A2 , A3 , ...., Ak , so that they (i) are mutually exclusive (no ‘overlaps’): Ai ∩ Aj = ∅ for any i and j, and (ii) cover the whole sample space (no ‘gaps’): A1 ∪ A2 ∪ A3 ∪ ... ∪ Ak = Ω. Obviously, the ‘finest’ partition is the collection of all simple events, and the ’crudest’ partition is Ω itself. The most interesting partitions will of course be the in-between cases. A common example is A and A (where A is an arbitrary event). Partitions (introduced by us) come useful when computing probability of another event B, by utilizing the Formula of Total Probability: Pr(B) = Pr(A1 ) Pr(B|A1 ) + Pr(A2 ) Pr(B|A2 ) + ... + Pr(Ak ) Pr(B|Ak ) ¥ EXAMPLE: Two players are dealt 5 cards each. What is the probability that they will have the same number of aces? We partition the sample space according to how many aces the first player gets, calling the events A0 , A1 , ..., A4 .(using a convenient index) Answer: (4)(48) (31)(444) (42)(483) (22)(453) (43)(482) + 52 · 47 + 52 · 0+ 4 52 0 · 0 = 49.33% 47 (5) (5) (5) (5) (5) (40)(485) (40)(435) (41)(484) · 47 + 52 · (525) (5) (5) bBayes’ rule Let’s start with an EXAMPLE: Consider four ’black’ boxes: two of them (Type I) have 1 green and 2 red marbles inside, one (Type II) has 1 green and 1 red marble, and one (Type III) has 2 green and 1 red 11 marble. One of these boxes is selected at random, and a marble drawn from it. Compute Pr(R): Pr(R|I) · Pr(I) + Pr(R|II) · Pr(II) + Pr(R|III) · Pr(III) = 8 24 + 3 24 + 2 24 = 54.17%. Similarly, Pr(G) = 11 . 24 12 How about Pr(I |R): Pr(I ∩ R) = Pr(R) 8 8/24 = = 61.54% > 50% 13/24 13 Pr(I|R) = The procedure for computing Pr(I |R) can be generalized as follows: check oﬀ (X) all simple events contributing to R, out of these check oﬀ (using a diﬀerent symbol, ° in our case) those which also contribute to I Then divide the probability of the latter set by the total probability of the former. Another EXAMPLE: Let 0.5% of a population in a certain area have TB. There is a medical test which can detect this condition in 95% of all (infected) cases, but at the same time the test is (falsely) positive for 10% of the healthy people. A person is selected randomly and tested. The test 13 is positive. What is the probability that the person actually has it? Using the Bayes’ rule, this is equal to 0.00475 0.00475+0.09950 = 4.556% (almost 10 times bigger than before the test)! Independence of two events is a very natural notion: if the experiment is done in such a manner that A (happening or not) cannot influence the probability of B, B is independent of A. Formally, this means that Pr(B|A) = Pr(B). Mathematically, this is equivalent to Pr(A ∩ B) = 14 Pr(A) · Pr(B), and also to Pr(A|B) = Pr(A), meaning that A is independent of B. The same condition is also equivalent to Pr(A ∩ B) = Pr(A) · Pr(B), etc. Example: An outcome of one die cannot influence the outcome of another die; but also: an outcome of a die cannot influence any of its future outcomes (no ‘memory’). I should mention that the condition of independence may sometimes be met ‘accidently’ (not of much use to us). Don’t confuse with 2 events being exclusive!! Three events are mutually independent if they are pairwise independent, plus: Pr(A ∩ B ∩ C) = Pr(A) · Pr(B) · Pr(C) (the natural independence is of the mutual type). Mutual independence of A, B and C ¯ but also this: any event build from A and B (e.g. A ∪ B ) must implies that of A, B and C, be independent of C. Mutual independence of k events means that the probability of intersection of any number (i.e. 2, 3, ... ) of these equals the product of the individual probabilities. This implies that any event build out of A, B, ... must be independent of any event build out of C, D, ... ,as long as the two sets are distinct. The most important implication of independence is this: the probability of a Boolean expression involving independent events can be expressed in terms of the individual probabilities. 15 EXAMPLES: Pr[(A ∪ B) ∩ C ∪ D] = Pr(A ∪ B) · [1 − Pr(C ∪ D)] = [Pr(A) + Pr(B) − Pr(A) Pr(B)] · [1 − Pr(C) − Pr(D) + Pr(C) Pr(D)] ¥ Pr[(A ∪ C ∩ D) ∩ B ∪ D] = Pr[(A ∪ C ∪ D) ∩ B ∩ D] = Pr[(A ∪ C ∪ D) ∩ D] · [1 − Pr(B)] = Pr[(A ∩ D) ∪ (C ∩ D) ∪ ∅] · [1 − Pr(B)] = {Pr(A) Pr(D) + Pr(D) − Pr(C) Pr(D) − Pr(A) · [1 − Pr(C)] Pr(D)} · [1 − Pr(B)] To make it easier, I usually give you specific values for Pr(A), Pr(B), ... END-OF-CHAPTER EXAMPLES: • Let us return to the lottery with 100,000 tickets, and compute the probability that a randomly selected ticket has an 8 and a 4 on it (each at least once, in any order, and not necessarily consecutive). This is a roll-of-a-die type of experiment! Define A: no 8 at any place, B: no 4. We need Pr(A ∩ B) = Pr(A ∪ B) = 1 − Pr(A ∪ B) = 1 − Pr(A) − Pr(B) + Pr(A ∩ B) 16 Clearly A ≡ A1 ∩ A2 ∩ ... ∩ A5 , where A1 : ‘no 8 in the first place’, A2 : ‘no 8 in the second place’, etc. A1 , A2 , ..., A5 are mutually independent, thus Pr(A) = Pr(A1 ) · 9 5 9 5 Pr(A2 ) · ... · Pr(A5 ) = ( 10 ) . Similarly, Pr(B) = ( 10 ) . Now, A ∩ B ≡ C1 ∩ C2 ∩ ... ∩ C5 where C1 : not 8 nor 4 in the first spot, C2 : not 8 nor 4 in the second, etc. These of 8 5 course are also independent, which implies Pr(A ∩ B) = ( 10 ). 9 5 8 5 ) + ( 10 ) = 14.67%. Answer: 1 − 2( 10 • The same question, but this time we want at least one 8 followed (sooner or later) by a 4 (at least once). What makes this diﬀerent from the original question is that 8 and 4 now don’t have to be consecutive. We partition the sample space according to the position at which 8 appears for the first time: B1 , B2 , ..., B5 , plus B0 (which means there is no 8). Now, if A is the event of our question (8 followed by a 4), we apply the formula of total probability: Pr(A) = Pr(B1 ) Pr(A|B1 ) + ... + Pr(B5 ) Pr(A|B5 ) 1 9 · [1 − ( )4 ]+ 10 10 9 1 9 1 9 9 · · [1 − ( )3 ] + ( )2 · · [1 − ( )2 ]+ 10 10 10 10 10 10 9 9 1 ( )3 · · [1 − ] + 0 + 0 = 8.146% 10 10 10 + Pr(B0 ) Pr(A|B0 ) = • Out of 10 dice, 9 of which are regular but one is ’crooked’ (6 has a probability of 0.5), 17 a die is selected at random and rolled twice. Given that the first roll resulted in a six, what is the (conditional) probability of getting a six again in the second roll? 9+9 = 25% 9 + 9 + 45 + 9 ? Are S1 and S2 independent? We have to check: Pr(S1 ∩ S2 ) = Pr(S1 ) · Pr(S2 ). ( 1 18 72 72 1 =) 6= · (= ) 20 360 360 360 25 18 Given that both rolls resulted in a six, what is the (conditional) probability of having selected the crooked die? 9 = 50% 9+9 • Ten people have been arrested as suspects in a crime one of them must have committed. A lie detector will (incorrectly) incriminate an innocent person with a 5% probability, it can (correctly) detect a guilty person with a 90% probability. One person has been randomly selected and tested; the lie detector has its red light flashing. What is the probability that he is the criminal? 2 0.090 = 0.090 + 0.045 3 19 All 10 people have been tested and exactly one incriminated. What is the probability of having the criminal now? The conditional sample space consists of 10 simple events (out of the original 210 = 1024): rggggggggg 0.9 × 0.959 grgggggggg 0.1 × 0.958 × 0.05 ........ 0.1 × 0.958 × 0.05 gggggggggr Answer: 0.9 × 0.959 = 95%. 0.9 × 0.959 + 9 × 0.1 × 0.958 × 0.05 • Two men take one shot each at a target. Mr. A can hit it with the probability of 14 , Mr. B’s chances are 25 . What is the probability that the target is hit (at least once)? Here, we have to, on our own, assume independence of the two shots. Pr(A ∪ B) = Pr(A) + Pr(B) − Pr(A ∩ B) = 1 − Pr(A ∪ B) = 1 − Pr(A ∩ B) = 1 − 34 · 3 5 1 4 + 2 5 − 1 10 = 55%, or: Pr(A ∪ B) = = 55%. • What is more likely, getting at least one 6 in four rolls of a die, or getting at least one double 6 in twenty four rolls of a pair of dice? )24 = The first probability is 1−( 56 )4 = 51.77%. The second one, similarly, equals 1−( 35 36 49.14%. • Four people are dealt 13 cards each. You (one of the players) got one ace. What is the probability that your partner has the other three aces? This is a very simple (natural) conditional probability, if we assume that we are the first player to be dealt his 13 cards, and our partner is the next one: 20 ( 33 )( 36 ) 10 ( 39 ) 13 = 3.129%. • A, B, C are mutually independent, having (the individual) probabilities of 0.25, 0.35 and 0.45, respectively. Compute Pr[(A ∩ B) ∪ C]. Pr(A ∩ B) + Pr(C) − Pr(A ∩ B ∩ C) = 0.25 × 0.65 + 0.45 − 0.25 × 0.65 × 0.45 = 53.94% • Two coins are flipped, followed by rolling a die as many times as the number of heads shown. What is the probability of getting fewer than 5 dots in total? Introduce a partition A0 , A1 , A2 according to how many heads are obtained. If B stands for ’getting fewer than 5 dots’, the total-probability formula yields: Pr(B) = Pr(A0 ) Pr(B|A0 ) + Pr(A1 ) Pr(B|A1 ) + Pr(A2 ) Pr(B|A2 ) = 2 4 1 6 1 ×1+ × + × = 62.5% 4 4 6 4 36 Given that there were exactly 3 dots in total, what is the conditional probability that 21 the coins showed exactly one head? Answer: 1/12 1/12+1/72 = 85.71%. • Jim, Joe, Tom and six other boys are randomly seated in a row. What is the probability that at least two of the three friends will sit next to each other? Let’s introduce A: Jim and Joe sit together, B: Jim and Tom sit together, C: Joe and Tom sit together. Pr(A ∪ B ∪ C) = Pr(A) + Pr(B) + Pr(C) 22 − Pr(A ∩ B) − Pr(A ∩ C) − Pr(B ∩ C) + Pr(A ∩ B ∩ C) There is 9! random arrangements of the boys, 2 × 8! will meet condition A (same with B and C), 2 × 7! will meet both A and B (same for A ∩ C and B ∩ C), none will meet all three. Answer: 3 × 2×8! 9! −3× 2×7! 9! = 58.33% (try this with circular arrangement). • There are 10 people at a party (no twins). Assuming that all 365 days of a year are equally likely to be someone’s birth date, and also ignoring leap years, what is the probability of all these ten people having diﬀerent birth dates? Hint: This is a roll-of-a-die type of experiment. 365 P10 = 88.31% 36510 Exactly two people having the same birth date (the rest all distinct)? 365 × ¡10¢ 2 × P8364 36510 = 11.16% These two answers account for 99.47% of the total probability. Two or three duplicates, and perhaps one triplicate would most likely take care of the rest; try it! 23