http://www.aast.edu/~khedr
Sample Space

The sample space S of a random experiment
is defined as the set of all possible outcomes.
An outcome or sample point ζ of a random
experiment is a result that cannot be
ζ∈S
decomposed into other results.
One and only one outcome occurs when a
random experiment is performed.

Outcomes are mutually exclusive – they cannot
occur simultaneously
BEEX – ECE 5605, Virginia Tech
26
Sample Spaces – example experiments

E1: select a ball from an urn containing balls numbered 1 to 50. Note the
number of the ball.
S1 = {1, 2,...,50}

E2: Select a ball from an urn containing balls numbered 1 to 4. Suppose
that balls 1 and 2 are black and that balls 3 and 4 are white. Note the
number and color of the ball you select.
S 2 = {(1, b), (2, b), (3, w), (4, w)}

E3: Toss a coin three times and note the sequence of heads and tails.
S3 = { HHH , HHT , HTH , THH , TTH , THT , HTT , TTT }

E4: Toss a coin three times and note the number of heads.
S 4 = {0,1, 2,3}
BEEX – ECE 5605, Virginia Tech
sample spaces are sets
27
Sample Spaces

E5: Count the number of voice packets containing only silence produced
from a group of N speakers in a 10ms period.
S5 = {0,1, 2,..., N }

E6: A block of information is transmitted repeatedly over a noisy channel
until an error-free block arrives at the receiver. Count the number of
transmissions required.
S6 = {1, 2,3,...}

E7: Pick a number at random between zero and one.
S7 = { x : 0 ≤ x ≤ 1} = [0,1]

[
0
S7
E8: Measure the time between two message arrivals at a message
center.
S8 = {t : t ≥ 0} = [0, ∞)
BEEX – ECE 5605, Virginia Tech
[
0
S8
]→
x
1
→t
sample spaces visualized as intervals on real line
28
y
Sample Spaces
↑
1
S12
0

1
E12: Pick two numbers at random between zero and one.
→x
S12 = {( x, y ) : 0 ≤ x ≤ 1 and 0 ≤ y ≤ 1} y

↑
1 one,
E13: Pick a number X at random between zero and
then pick a number Y at random between zero and X.
S13
S13 = {( x, y ) : 0 ≤ y ≤ x ≤ 1}
0

1 →x
E14: A system component is installed at time t=0. For t≥0 let
X(t)=1 as long as the component is functioning, and let
X(t)=0 after the component fails.
⎧⎪
⎫⎪
⎧1 0 ≤ t < t 0
, time of component failure t0 ⎬
S14 = ⎨ X (t ) : X (t ) = ⎨
t ≥ t0
⎪⎩
⎪⎭
⎩0
BEEX – ECE 5605, Virginia Tech
sample spaces visualized as regions of the plane
30
Sample spaces

Finite, countably infinite, uncountably infinite

Discrete sample space

If S is countable

Continuous sample space

Outcomes can be put in 1-to-1 correspondence with the
positive integers
If S is not countable
Can be multi-dimensional

Can sometimes be written as Cartesian product of other sets
BEEX – ECE 5605, Virginia Tech
31
Events

A subset of S

Does the outcome satisfy certain conditions?

E10: Determine the value of a voltage waveform at
time t1.
S10 = {v : −∞ < v < ∞} = (−∞, ∞ )

Is the voltage negative?
Event A occurs iff the outcome of the
experiment ζ is in the subset
iff = if and only if
A = {ζ : −∞ < ζ < 0}
BEEX – ECE 5605, Virginia Tech
36
Special events

Certain event S

Consists of all outcomes, hence occurs always
Impossible or null event ¯

Contains no outcomes, hence never occurs
BEEX – ECE 5605, Virginia Tech
37
Events – example experiments

E1: select a ball from an urn containing balls numbered 1 to 50. Note the
number of the ball.
“An even-numbered ball is selected”

E2: Select a ball from an urn containing balls numbered 1 to 4. Suppose
that balls 1 and 2 are black and that balls 3 and 4 are white. Note the
number and color of the ball you select.
“The ball is white and even-numbered”

A1 = {2, 4,..., 48,50}
A2 = {(4, w)}
E3: Toss a coin three times and note the sequence of heads and tails.
“The three tosses give the same outcome”

A3 = { HHH , TTT }
E4: Toss a coin three times and note the number of heads.
“The number of heads equals the number of tails”
BEEX – ECE 5605, Virginia Tech
A4 = ∅
38
Set (event) operations
A∪ B

Union

Intersection

Mutually exclusive

Complement

Implies

A∩ B
A∩ B = ∅
Ac such that Ac ∪ A = S and Ac ∩ A = ∅
all outcomes in A are also outcomes in B
Equal

A and B contain the same outcomes
BEEX – ECE 5605, Virginia Tech
A⊂ B
A=B
44
Venn diagrams
A
B
A
A∪ B
A
B
A∩ B
A
Ac
B
A∩ B = ∅
A B
A⊂ B
BEEX – ECE 5605, Virginia Tech
45
properties

Commutative A ∪ B = B ∪ A

Associative
A∩ B = B ∩ A
A ∪ ( B ∪ C ) = ( A ∪ B) ∪ C
A ∩ ( B ∩ C ) = ( A ∩ B) ∩ C

Distributive
A ∪ ( B ∩ C ) = ( A ∪ B) ∩ ( A ∪ C )
A ∩ ( B ∪ C ) = ( A ∩ B) ∪ ( A ∩ C )

deMorgan’s rules
( A ∩ B)c = Ac ∪ B c
( A ∪ B)c = Ac ∩ B c
BEEX – ECE 5605, Virginia Tech
46
Repeated union/intersection operations
n
∪A
k
= A1 ∪ A2 ∪
k =1
occurs if one or more of the events Ak occur
n
∩A
k
k =1
∪ An
= A1 ∩ A2 ∩ ∩ An
occurs if each of the events Ak occur
∞
∪A
k
k =1
BEEX – ECE 5605, Virginia Tech
∞
∩A
k
k =1
51
Conditional probability

Are two events A and B related, in the sense
that one tells us something about the other?

We observe/measure something to learn about
something else that’s not directly measurable
1
0

Conditional probability of event A given that
event B has occurred
P [ A | B]
BEEX – ECE 5605, Virginia Tech
P [ A ∩ B]
P [ B]
for P [ B ] > 0
27
P [ A | B]
P [ A ∩ B]
P [ B]
for P [ B ] > 0
event B has occurred ζ ∈ B
S
S.|B = B
B
A∩ B
A
P [ A | B ] deals with ζ ∈ A ∩ B
a renormalization of probability for the reduced sample space
BEEX – ECE 5605, Virginia Tech
28
Ex 2.49
P [ A | B]
P [ A ∩ B]
for P [ B ] > 0
P [ B]
satisfies the Axioms of probability
0 ≤ P [ A ∩ B], 0 < P [ B] ⇒ 0 ≤ P [ A | B]
Axiom I
A ∩ B ⊂ B ⇒ P [ A ∩ B] ≤ P [ B] ⇒ P [ A | B] ≤ 1
B⊂S
⇒ P [ S | B] =
P [ S ∩ B]
P [ B]
=
P [ B]
P [ B]
=1
Axiom II
( A ∩ B ) ∩ (C ∩ B ) = ∅
P ⎡⎣( A ∪ C ) ∩ B ⎤⎦ P ⎡⎣( A ∩ B ) ∪ ( C ∩ B ) ⎤⎦
then P [ A ∪ C | B ] =
=
P [ B]
P [ B]
Axiom III
P ⎡⎣( A ∩ B ) ⎤⎦ + P ⎡⎣( C ∩ B ) ⎤⎦
=
= P [ A | B ] + P [C | B ]
P [ B]
If A ∩ C = ∅ ⇒
BEEX – ECE 5605, Virginia Tech
29
Ex 2.21

Select a ball from an urn containing 2 black balls, labeled 1 and
2, and 2 white balls, labeled 3 and 4
S = {(1, b ) , ( 2, b ) , ( 3, w ) , ( 4, w )}

Assuming equi-probable outcomes, find P[A|B] and P[A|C],
where
A={(1,b),(2,b)} “black ball selected”
B={(2,b),(4,w)} “even-numbered ball selected”
C={(3,w),(4,w)} “number of ball is >2”
P [ A ∩ B ] = P ⎡⎣( 2, b ) ⎤⎦
P [ A ∩ C ] = P [∅ ] = 0
BEEX – ECE 5605, Virginia Tech
P [ A | B] =
P[ A | C] =
P [ A ∩ B]
P [ B]
P[ A ∩ C]
P [C ]
.25
=
= .5 = P [ A]
.5
0
= = 0 ≠ P [ A]
.5
knowing B doesn’t help, knowing C does
6
P [ A ∩ B]
P [ A | B]
Ex 2.23
P [ B]
⇓
P [ A ∩ B] = P [ A | B] P [ B]
binary communications channel
= P [ B | A] P [ A]
P [1sent ] = p
P [ random decision error ] = ε
input 0
1
1− p p
output 0
1− ε
ε
1
0
ε
1
1− ε
P [T0 ∩ R0 ] = P [ R0 | T0 ] P [T0 ] = (1 − ε )(1 − p )
P [T0 ∩ R1 ] = P [ R1 | T0 ] P [T0 ] = ε (1 − p )
P [T1 ∩ R0 ] = P [ R0 | T1 ] P [T1 ] = εp
P [T1 ∩ R1 ] = P [ R1 | T1 ] P [T1 ] = (1 − ε ) p
BEEX – ECE 5605, Virginia Tech
7
Total probability theorem
a partition of S: { B1 , B2 ,
n
, Bn } ∋ ∪ Bi = S and Bi ∩ B j = ∅∀i ≠ j
i =1
B2
B1
Bn
B3
B2
B1
mutually exclusive
B3
BEEX – ECE 5605, Virginia Tech
A
Bn
⎛ n ⎞ n
A = A ∩ S = A ∩ ⎜ ∪ Bi ⎟ = ∪ ( A ∩ Bi )
⎝ i =1 ⎠ i =1
C4
n
n
i =1
i =1
P [ A] = ∑ P [ A ∩ Bi ] = ∑ P [ A | Bi ] P [ Bi ]
8
Ex 2.24 (but based on Ex 2.23)
n
P [ A] = ∑ P [ A | Bi ] P [ Bi ]
i =1
Very useful when experiment consists of a sequence of 2 subexperiments
input 0
1
T0 and T1 partition S
1− p p
output 0
1− ε
ε
1
0
ε
1
1− ε
P [ R1 ] = P [ R1 | T1 ] P [T1 ] + P [ R1 | T0 ] P [T0 ]
= (1 − ε ) p + ε (1 − p )
P [ R0 ] = P [ R0 | T1 ] P [T1 ] + P [ R0 | T0 ] P [T0 ]
= εp + (1 − ε )(1 − p )
BEEX – ECE 5605, Virginia Tech
9
Bayes’ Rule
Let { B1 , B2 ,
, Bn } be a partition of S then
P ⎡⎣ B j ∩ A⎤⎦
P ⎡⎣ A | B j ⎤⎦ P ⎡⎣ B j ⎤⎦
P ⎡⎣ B j | A⎤⎦ =
= n
P [ A]
∑ P [ A | Bk ] P [ Bk ]
k =1
a posteriori probability
the partition corresponds to a priori events (of interest)
A corresponds to a measurement/observation
BEEX – ECE 5605, Virginia Tech
11
Ex 2.26 (modified to fit our 2.23, 2.24)

A 1 is received; what is the probability that a 1
was transmitted?
P [T1 | R1 ] =
P [ R1 | T1 ] P [T1 ]
Bayes’ Rule
P [ R1 ]
p = 0.5
(1 − ε ) p
(1 − ε ) / 2
=
=
= (1 − ε )
1/ 2
(1 − ε ) p + ε (1 − p )
total probability P [ R1 ] = P [ R1 | T1 ] P [T1 ] + P [ R1 | T0 ] P [T0 ]
= (1 − ε ) p + ε (1 − p )
P [T0 | R1 ] =
P [ R1 | T0 ] P [T0 ]
BEEX – ECE 5605, Virginia Tech
P [ R1 ]
ε (1 − p )
p = 0.5
ε/2
=
=
=ε
(1 − ε ) p + ε (1 − p ) 1/ 2
12
Independence of events

Knowledge of the occurrence of event B does
not alter the probability of some other event A

A does not depend on B
P [ A] = P [ A | B ] =
events A and B are independent if
P [ A ∩ B ] = P [ A] P [ B ]
⇓⇑ P [ B ] ≠ 0
P [ A | B ] = P [ A]
BEEX – ECE 5605, Virginia Tech
P [ A ∩ B]
P [ B]
problematic if P [ B ] = 0
⇓⇑ P [ A] ≠ 0
P [ B | A] = P [ B ]
14
Ex 2.29 2-D continuous
two numbers x and y are selected at random between 0 and 1
events A = { x > 0.5} , B = { y > 0.5} , C = { x > y}
y
1
B
P [ A ∩ B ] 1/ 4 1
1
P [ A | B] =
=
= = P [ A]
2
A
1/ 2 2
P [ B]
1
A and B are independent
1
0
2
x ratio of proportions has remained the same
y
1
A
P[ A | C] =
1
2
0
C
BEEX – ECE 5605, Virginia Tech
1
2
1
x
P[ A ∩ C]
P [C ]
3/8 3 1
=
= ≠ = P [ A]
1/ 2 4 2
ratio of proportions has increased
i.e. we gained “knowledge” from measuring C
16
Independence of A, B, and C
P [ A ∩ B ] = P [ A] P [ B ]
1. pairwise independence:
P [ A ∩ C ] = P [ A] P [ C ]
P [ B ∩ C ] = P [ B ] P [C ]
2. knowledge of joint occurrence, of any two,
does not affect the third: P [C | A ∩ B ] = P [C ]
A, B, C are independent
if the probability of the intersection
of any pair or triplet of events
equals the product of the probabilities
of the individual events
BEEX – ECE 5605, Virginia Tech
P[ A ∩ B ∩ C]
P [ A ∩ B]
= P [C ]
P [ A ∩ B ∩ C ] = P [ A ∩ B ] P [C ]
= P [ A] P [ B ] P [ C ]
17
Ex 2.30 pairwise independence is not enough
two numbers x and y are selected at random between 0 and 1
events: B = { y > 0.5} , D = { x < 0.5}
y
F = { x < 0.5; y < 0.5} ∪ { x > 0.5; y > 0.5}
1
1
2
B
D
1
2
0
y
1
1
2
1
x
F
P [ B ∩ D ∩ F ] = P [∅ ] = 0
F
0
BEEX – ECE 5605, Virginia Tech
1
2
1 11
= P [ B] P [ D]
P [ B ∩ D] = =
4 22
1 11
= P [ B] P [ F ]
P[B ∩ F ] = =
4 22
1 11
= P [ D] P [ F ]
P[D ∩ F ] = =
4 22
1
x
111 1
P [ B] P [ D] P [ F ] =
=
222 8
pairwise
independent
violates
2nd condition
18
Independence of n events

Probability of an event is not affected by the joint
occurrence of any subset of the other events
events B1 , B2 ,
, Bn
are said to be independent if for k=2,…,n
P ⎡⎣ Bi1 ∩ Bi2 ∩
∩ Bik ⎤⎦ = P ⎡⎣ Bi1 ⎤⎦ P ⎡⎣ Bi2 ⎤⎦
where 1 ≤ i1 < i2 <
P ⎡⎣ Bik ⎤⎦
< ik ≤ n
2n − n − 1 possible intersections to evaluate! Δ

Assuming independence of the events of
separate experiments is more common

E.g. one coin toss is independent of any before/after
independent experiments
BEEX – ECE 5605, Virginia Tech
19
Sequential experiments

Many random experiments can be viewed as
sequential experiments consisting of a
sequence of simpler subexperiments
The subexperiments may, or may not, be
independent
BEEX – ECE 5605, Virginia Tech
20
Sequences of independent experiments

Random experiment consisting of performing experiments E1,
E2,…,En
Outcome is an n-tuple s=(s1,s2,…,sn) where sk is the outcome of
the k-th subexperiment
Sample space of sequential experiment contains n-tuples; it is
denoted by the Cartesian product of the individual sample
spaces S1xS2x…xSn
Physical considerations will often indicate that the outcome of
any given subexperiment cannot affect the outcomes of the
other subexperiments; it is then reasonable to assume that the
events A1,A2,…,An – where Ak concerns only the k-th
subexperiment – are independent:
P [ A1 ∩ A2 ∩
∩ An ] = P [ A1 ] P [ A2 ]
P [ An ]
facilitates computing all probabilities of events of the sequential experiment
BEEX – ECE 5605, Virginia Tech
21
Ex 2.33

Select 10 numbers at random from [0,1]

Find P[first 5 #’s <1/4, last 5 #’s >1/2]
1⎫
⎧
Ak = ⎨ x < ⎬ for k = 1,...,5
4⎭
⎩

1⎫
⎧
Ak = ⎨ x > ⎬ for k = 6,...,10
2⎭
⎩
Assuming selection of a number is
independent of other selections
P [ A1 ∩ A2 ∩
BEEX – ECE 5605, Virginia Tech
∩ An ] = P [ A1 ] P [ A2 ]
5
⎛1⎞ ⎛1⎞
P [ An ] = ⎜ ⎟ ⎜ ⎟
⎝4⎠ ⎝2⎠
5
22
Bernoulli trial

Perform an experiment once; note whether
event A occurs (if so, it’s called a “success,” if
not, it’s called a “failure”)
Find probability of k successes in n
independent repetitions of a Bernoulli trial

It’s like coin tosses, with an unfair coin
P [ A] = p
BEEX – ECE 5605, Virginia Tech
23
Binomial probability law

Probability of k successes in n independent
Bernoulli trials
⎛n⎞ k
n−k
for k = 0,
pn (k ) = ⎜ ⎟ p (1 − p )
⎝k ⎠
,n
“success in k particular positions
& failure elsewhere”
# of distinct ways, Nn(k), that
such sequences can be chosen
⎛n⎞
n!
“n choose k” ⎜ ⎟
binomial coefficient
k
k
!
n
−
k
!
(
)
⎝ ⎠
BEEX – ECE 5605, Virginia Tech
24