Outline
Review & Motivation
Single Decision Problem
Sequential Decision Problem
Examining the Effect of Uncertain
Preferences upon Value of Sample
Information
Brett Houlding, Frank Coolen
Department of Mathematical Sciences, Durham University, UK
Brett Houlding, Frank Coolen
Examining the Effect of Uncertain Preferences upon Value of Sample Information
Outline
Review & Motivation
Single Decision Problem
Outline:
Review & Motivation
Expected Utility Theory
Value of Information
Adaptive Utility
Single Decision Problem
Effect of Uncertain Preferences
Example
Information over Preferences
Sequential Decision Problem
Understanding the Value of a Decision
Example
Brett Houlding, Frank Coolen
Examining the Effect of Uncertain Preferences upon Value of Sample Information
Sequential Decision Problem
Outline
Review & Motivation
Single Decision Problem
Sequential Decision Problem
Expected Utility Theory
Expected Utility Theory
I
Let Ω be the set of possible states of nature ω.
I
Let D be a finite set of decisions mapping Ω to a reward set R.
I
Assume true state of nature ω uncertain and denote beliefs by Pω .
I
Let u be a cardinal utility function mapping the set R to R.
I
Under axioms of expected utility theory, a Decision Maker’s (DM’s)
objective is to maximise expected utility return.
I
The DM should thus select decision d maximising Eω [u(d, ω)].
Brett Houlding, Frank Coolen
Examining the Effect of Uncertain Preferences upon Value of Sample Information
Outline
Review & Motivation
Single Decision Problem
Sequential Decision Problem
Value of Information
Value of Information
I
Let X be unknown information about ω.
I
Without X DM picks d to maximise Eω [u(d, ω)].
I
With X DM updates beliefs to Pω|X and picks d to maximise
Eω|X [u(d, ω)].
I
Not knowing X means Eω|X [u(d, ω)] is a random variable.
I
Let PX represent beliefs over what X will say.
I
Before X is known, DM expects that knowing X will permit a max
expected utility return of EX [maxd Eω|X [u(d, ω)]].
Brett Houlding, Frank Coolen
Examining the Effect of Uncertain Preferences upon Value of Sample Information
Outline
Review & Motivation
Single Decision Problem
Sequential Decision Problem
Value of Information
Continued...
I
Denote the value of X to the DM by Iω (X ).
I
Iω (X ) represents the fair price (in utility currency) to the DM for
knowledge of X .
I
Iω (X ) is the max expected return knowing X minus the max
expected return not knowing X .
I
Iω (X ) is an unknown random variable, its expectation, EX [Iω (X )], is
the DM’s expected value of X .
I
EX [Iω (X )] = EX [maxd Eω|X [u(d, ω)]] − maxd Eω [u(d, ω)]
Brett Houlding, Frank Coolen
Examining the Effect of Uncertain Preferences upon Value of Sample Information
Outline
Review & Motivation
Single Decision Problem
Sequential Decision Problem
Value of Information
Example
I
A DM has decision d to either Buy (B) or Not Buy (NB) stock.
I
The state of nature ω is whether the stock will Rise (R) or Fall (F ),
and prior beliefs are that Pω (R) = 0.5.
I
Utility function is such that u(B, R) = 1, u(B, F ) = −1 and
u(NB, R) = u(NB, F ) = 0.
I
Max expected utility is thus 0.
Brett Houlding, Frank Coolen
Examining the Effect of Uncertain Preferences upon Value of Sample Information
Outline
Review & Motivation
Single Decision Problem
Sequential Decision Problem
Value of Information
Example Continued...
I
An expert offers to sell information X on how stock will perform.
I
X can be either Good (G ) or Poorly (P).
I
You believe the expert is right with chance 0.9, e.g.,
Pω|X (R|G ) = 0.9.
I
If X = G , max expected return is 0.8, and if X = P, then this is 0.
I
Calculate predictive to find PX (G ) = 0.5.
I
Thus EX [Iω (X )] = (0.5 × 0.8 + 0.5 × 0) − 0 = 0.4.
Brett Houlding, Frank Coolen
Examining the Effect of Uncertain Preferences upon Value of Sample Information
Outline
Review & Motivation
Single Decision Problem
Sequential Decision Problem
Adaptive Utility
Adaptive Utility
I
Assume DM uncertain of preference ranking over elements in R.
I
Let θ be the DM’s state of mind representing actual preferences.
I
A classical utility function is hence conditional on a value of θ,
denoted u(r |θ).
I
An adaptive utility function is a function of both reward and θ,
denoted u a (r , θ).
I
Once classical utility functions have been scaled to be
commensurable, then u a (r , θ) = u(r |θ).
I
The DM’s objective is to maximse her adaptive utility function.
Brett Houlding, Frank Coolen
Examining the Effect of Uncertain Preferences upon Value of Sample Information
Outline
Review & Motivation
Single Decision Problem
Sequential Decision Problem
Questions
Questions
If we start to permit uncertain preferences, then we may be interested in:
I
How to generalise EV X for uncertain utility?
I
How does EV of X informative of ω depend on beliefs over θ?
I
Can one consider EV of X informative of θ?
I
What happens when X only arises following decision selection in
sequential problems?
Brett Houlding, Frank Coolen
Examining the Effect of Uncertain Preferences upon Value of Sample Information
Outline
Review & Motivation
Single Decision Problem
Sequential Decision Problem
Effect of Uncertain Preferences
Effect of Uncertain Preferences
Assume beliefs over θ and ω independent. If θ uncertain, and if the
objective is to max expected adaptive utility, then the value of X
informative of ω depends on:
I
The set D of possible decisions.
I
The adaptive utility function u a .
I
Beliefs over ω as represented by Pω .
I
The likelihood PX |ω .
I
But also now on beliefs over θ as represented by Pθ .
Brett Houlding, Frank Coolen
Examining the Effect of Uncertain Preferences upon Value of Sample Information
Outline
Review & Motivation
Single Decision Problem
Sequential Decision Problem
Effect of Uncertain Preferences
Expected Value of Information
I
Now the DM is assumed to wish to max her expected adaptive
utility function, Eω Eθ [u(d, ω, θ)].
I
If X informative of ω is known then max Eω|X Eθ [u(d, ω, θ)].
I
As before, the expected value of X is the expected max expected
utility obtainable with X , minus the max expected utility without X .
I
To demonstrate use of adaptive utility, replace Iy (X ) notation by
Iya (X ), where subscript y is parameter that X is informative of.
I
EX [Iωa (X )] = EX [maxd Eω|X Eθ [u(d, ω, θ)]] − maxd Eω Eθ [u(d, ω, θ)]
Brett Houlding, Frank Coolen
Examining the Effect of Uncertain Preferences upon Value of Sample Information
Outline
Review & Motivation
Single Decision Problem
Sequential Decision Problem
Effect of Uncertain Preferences
Properties
Similar to case of classical value of information, EX [Iωa (X )] satisfies:
I
Non-Negativity: EX [Iωa (X )] ≥ 0
I
Additivity: EX1 ,X2 [Iωa (X1 , X2 )] = EX1 ,X2 [Iωa (X2 |X1 )] + EX1 [Iωa (X1 )]
Brett Houlding, Frank Coolen
Examining the Effect of Uncertain Preferences upon Value of Sample Information
Outline
Review & Motivation
Single Decision Problem
Sequential Decision Problem
Effect of Uncertain Preferences
Relationship between Uncertainty and Value
Less uncertainty over θ means the DM is more sure over preferences, so
one may guess that more uncertainty decreases EX [Iωa (X )]. However,
there appears to be no simple relationship between uncertainty over θ
and EX [Iωa (X )]. In particular:
I
EX [Iωa (X )] is not necessarily increased as less values of θ are
possible.
I
EX [Iωa (X )] is not necessarily increased as less probability is
concentrated on a particular value of θ.
I
EX [Iωa (X )] is not necessarily increased as Vθ [θ] decreases.
I
EX [Iωa (X )] is not necessarily minimised in the case of equiprobable
values of θ.
Brett Houlding, Frank Coolen
Examining the Effect of Uncertain Preferences upon Value of Sample Information
Outline
Review & Motivation
Single Decision Problem
Sequential Decision Problem
Example
Example
The previous statements can be shown to be a result of the following
counter example:
I
θ ∈ {θ1 , θ2 }, Pθ (θ1 ) = p
I
ω ∈ {ω1 , ω2 }, Pω (ω1 ) = 0.5
I
X ∈ {x1 , x2 }, PX |ω (xj |ω1 ) = δij
I
d ∈ {d1 , d2 }, d1 (ω1 ) = d2 (ω2 ) = r1 , d1 (ω2 ) = d2 (ω1 ) = r2
I
u a (r1 , θ1 ) = u a (r2 , θ1 ) = 1, u a (r1 , θ2 ) = 2, u a (r2 , θ2 ) = 0
Brett Houlding, Frank Coolen
Examining the Effect of Uncertain Preferences upon Value of Sample Information
Outline
Review & Motivation
Single Decision Problem
Sequential Decision Problem
Example
Continued...
We find that:
I
maxd Eω Eθ [u a (d, ω, θ)] = 1, ∀p ∈ [0, 1]
I
EX [maxd Eω|X Eθ [u a (d, ω, θ)]] = 2 − p
I
EX [Iωa (X )] = 1 − p
I
Setting θ1 = v1 and θ2 = v2 gives Vθ [θ] = p(1 − p)(v1 − v2 )2 .
For a change in Pθ to increase EX [Iωa (X )] it must increase
EX [maxd Eω|X Eθ [u a (d, ω, θ)]] more than maxd Eω Eθ [u a (d, ω, θ)].
Brett Houlding, Frank Coolen
Examining the Effect of Uncertain Preferences upon Value of Sample Information
Outline
Review & Motivation
Single Decision Problem
Sequential Decision Problem
Information over Preferences
Dependencies & Information over θ
I
No problem in dealing with dependency of beliefs over θ and ω, and
similar results and properties hold.
I
Information over θ could arise from interview with experts.
I
It could also arise through trial of rewards, e.g., test drive or sample
tasters, or from observation of Advertisements.
I
Usually, however, we envisage X informative of θ will be observed
following decision selection in a sequential problem.
Brett Houlding, Frank Coolen
Examining the Effect of Uncertain Preferences upon Value of Sample Information
Outline
Review & Motivation
Single Decision Problem
Sequential Decision Problem
Understanding the Value of a Decision
Understanding the Value of a Decision
I
Now consider period i that X is observed: Iya (X ; i).
I
Iya (X ; i) can not increase if i is increased.
I
If d i selected in period i influences X , then must consider
EX [Iωa (X ; i)] in connection with d i .
I
Now decompose total expected value of decision d i into expected
‘pure’ value (resulting from the reward it entails), and expected
‘information’ value (resulting from value of X ).
I
This is why a DM in a sequential problems may select decisions not
optimal under prior beliefs, and that such experimentation should
occur at the earliest opportunity.
Brett Houlding, Frank Coolen
Examining the Effect of Uncertain Preferences upon Value of Sample Information
Outline
Review & Motivation
Single Decision Problem
Sequential Decision Problem
Example
Example
I
Must choose an Apple (A) or Banana (B) in each of 3 periods.
I
I
Assume ω redundant (a decision to choose A will result in A).
P3
P3
u a (d 1 , d 2 , d 3 , θ) = j=1 I{d i =A} + θ j=1 I{d i =B}
I
Thus choosing A increases utility by 1, B increases this by θ.
I
Assume θ ∈ {0.5, 1.5} and Pθ (0.5) = 0.6.
I
Let X = 1 if DM chooses A, whilst X = θ if DM chooses B.
I
Consider value of X observed after d 1 .
Brett Houlding, Frank Coolen
Examining the Effect of Uncertain Preferences upon Value of Sample Information
Outline
Review & Motivation
Single Decision Problem
Sequential Decision Problem
Example
Continued...
I
EX [Iθa (X ; 1)] = 0 for d 1 = A. No information can be gained!
I
EX [Iθa (X ; 1)] = 0.3 for d 1 = B. We will learn θ for sure!
I
Thus, even though selecting Apple gives sure reward of value 1,
whilst prior expectation of value resulting from a Banana is
Eθ [θ] = 0.9, d 1 = B has extra value of 0.3 because of its expected
information.
I
Hence why a DM does best in expectation by first selecting d 1 = B.
Brett Houlding, Frank Coolen
Examining the Effect of Uncertain Preferences upon Value of Sample Information