ee376a: Information Theory winter 2015 Tsachy Weissman today • what is information theory (high level)? • main topics • course goals and expectations • EE376a in the curriculum • course info • what is information theory, really? what is information theory? science of ! compression, ! storage, and ! transmission of information main problems and course structure • lossless compression • communication • lossy compression • joint source-channel coding problem i: lossless compression problem ii: (reliable) communication problem iii: lossy compression ? Decoder (estimator) Ed() ⇤ D d : X ⇥ Xˆ ⇧ R+ ⌃ problem iv: lossy communication (aka joint source channel coding) mathematical objects and tools • entropy • relative entropy • mutual information • chain rules • method of types goals and expectations acquaintance with: • information measures: entropy, relative entropy, mutual information • compression, storage, communication • “fundamental limits” • schemes for compression • “random coding” • “typical sequences” and interplay between information, probability, and statistics advanced information theory courses: audio, image, video compression - network information theory - universal schemes communication: information theory 376A statistical signal processing, denoising, compressed sensing -digital -optical -wireless coding: -algebraic -codes on graphs course info • what you’ll be held “accountable” for • references • midterm + final • course staff “accountability” • lectures: Tue&Thu 11:00am-12:15pm, Art 4 • weekly homework sets: Handed out every Thursday, and due the following Thursday at 5pm (at 2nd floor of Packard dropbox). First set will be handed out on Jan 15th. main textbook supporting texts (for further reading) • R. B. Ash, “Information Theory”, Dover, 1990! • T. Berger, “Rate Distortion Theory: A Mathematical Basis for Data Compression”, Prentice Hall, 1971.! • I. Csiszar and J. Korner, “Information Theory: Coding Theorems for Discrete Memoryless Systems”, Academic Press, 1981.! • A. El Gamal and Y.H. Kim, “Network Information Theory”, Cambridge University Press, 2011.! • R.G. Gallager, “Information Theory and Reliable Communication”, Wiley, 1968. exams • midterm: Tuesday, Feb 10, 7pm-9pm • final: Monday, March 16, 3:30pm-6:30pm • material: see “accountability” • final grade: hw 20%, midterm 35%, final 45% prerequisite • solid first course in probability ! • maturity and motivation to cope with a few abstract concepts staff • Instructor: Tsachy Weissman, Packard 256 office hours: Thursdays 2:00 to 4:00pm or by appointment [email protected] TAs Idoia Ochoa [email protected] Jiantao Jiao [email protected] supporting role: Albert No [email protected] office hours: to be given shortly Kartik Venkat [email protected] ! • Admin: Doug Chaffee, Packard 258 [email protected] • Grader: Youngsuk Park [email protected] website http://web.stanford.edu/class/ee376a/index.html (please give us a couple more days before perusing) questions? what is information theory, really? 1948 C. E. Shannon, “A Mathematical Theory of Communication,” Bell Syst. Tech. J., vol. 27, pp. 379–423, 623–656, Jul.–Oct. 1948. characterization of the “fundamental limits” focus is on the “what” more than the “how” 2 types of results: • “converse theorems” • “direct theorems” example I: lossless compression of a ternary source example II: binary source + channel example III: lossy compression of a Gaussian example IV: communication over the AWGN channel