EE376A: Homeworks #3 Due on Thursday, February 5, 2015 You can hand in the homework either after class or deposit it, before 5 PM, in the EE376A drawer of the class file cabinet on the second floor of the Packard Building. 1. Shannon code. Consider the following method for generating a code for a random variable X which takes on m values {1, 2, . . . , m} with probabilities p1 , p2 , . . . , pm . Assume that the probabilities are ordered so that p1 ≥ p2 ≥ · · · ≥ pm . Define Fi = i−1 X pk , (1) k=1 the sum of the probabilities of all symbols less than i. Then the codeword for i is the number Fi ∈ [0, 1] rounded off to li bits, where li = dlog p1i e. (a) Show that the code constructed by this process is prefix-free and the average length satisfies H(X) ≤ L < H(X) + 1. (2) (b) Construct the code for the probability distribution (0.5, 0.25, 0.125, 0.125). 2. Sequence length. How much information does the length of a sequence give about the content of a sequence? Suppose we consider a i.i.d. Bernoulli(1/2) process {Xi }. Stop the process when the first 1 appears. Let N designate this stopping time. Thus X N is an element of the set of all finite length binary sequences {0, 1}∗ = {0, 1, 00, 01, 10, 11, 000, · · · }. (a) Find I(N ; X N ). (b) Find H(X N |N ). (c) Find H(X N ). Let’s now consider a different stopping time. For this part, again assume Xi ∼Bernoulli(1/2) but stop at time N = 6, with probability 1/3 and stop at time N = 12 with probability 2/3. Let this stopping time be independent of the sequence X1 X2 · · · X12 . (d) Find I(N ; X N ). (e) Find H(X N |N ). (f) Find H(X N ). Homework 3 Page 1 of 3 3. Bad codes. Which of these codes cannot be Huffman codes for any probability assignment? (a) {0, 10, 11}. (b) {00, 01, 10, 110}. (c) {01, 10}. 4. Huffman Code. Let X ∼ p(x), where X = {1, 2, 3, 4, 5, 6, 7, 8} 1 1 1 1 1 1 1 1 p = ( , , , , , , , ) 4 4 8 8 16 16 16 16 (a) Find the Huffman binary code for p. (b) Find its expected length and the entropy H(X). (c) Find the Huffman 3-ary code for p. 5. Fair to bent. Let X ∼ p(x), where X = {1, 2, 3, 4, 5, 6} 1 1 1 1 1 1 p = ( , , , , , ) 4 4 4 8 16 16 (a) Describe how to use fair coin flips Z1 , Z2 , Z3 , . . . to generate X so as to minimize the expected number of flips required. (b) What is the expected number of flips required? (c) What is H(X)? 6. Bad wine. One is given 6 bottles of wine. It is known that precisely one bottle has gone bad (tastes very bad). From inspection of the bottles it is determined that the probability 8 6 4 2 2 1 pi that the ith bottle is bad is given by (p1 , p2 , . . . , p6 ) = ( 23 , 23 , 23 , 23 , 23 , 23 ). Tasting will determine the bad wine. Suppose you taste the wines one at a time. Choose the order of tasting to minimize the expected number of tastings required to determine the bad bottle. Remember, if the first 5 wines pass the test you don’t have to taste the last. Homework 3 Page 2 of 3 (a) What is the expected number of tastings required? (b) Which bottle should be tasted first? Now you get smart. For the first sample, you mix some of the wines in a fresh glass and sample the mixture. You proceed, mixing and tasting, stopping when the bad bottle has been determined. (c) What is the minimum expected number of tastings required to determine the bad wine? (d) What mixture should be tasted first? 7. Optimal codeword lengths. Although the codeword lengths of an optimal variable length code are complicated functions of the message probabilities {p1 , p2 , . . . , pm }, it can be said that less probable symbols are encoded into longer codewords. Suppose that the message probabilities are given in decreasing order p1 > p2 ≥ · · · ≥ pm . (a) Prove that for any binary Huffman code, if the most probable message symbol has probability p1 > 2/5, then that symbol must be assigned a codeword of length 1. (b) Prove that for any binary Huffman code, if the most probable message symbol has probability p1 < 1/3, then that symbol must be assigned a codeword of length ≥ 2. 8. Prefix and Uniquely Decodable codes Consider the following code and answer the following questions: u Codeword a 10 b 00 c 11 d 110 (a) Is this a Prefix code? (b) Show that this code is uniquely decodable. Homework 3 Page 3 of 3