15 PROBABILISTIC REASONING OVER TIME In which we try to interpret the present, understand the past, and perhaps predict the future, even when very little is crystal clear. Agents in partially observable environments must be able to keep track of the current state, to the extent that their sensors allow. In Section 4.4 we showed a methodology for doing that: an agent maintains a belief state that represents which states of the world are currently possible. From the belief state and a transition model, the agent can predict how the world might evolve in the next time step. From the percepts observed and a sensor model, the agent can update the belief state. This is a pervasive idea: in Chapter 4 belief states were represented by explicitly enumerated sets of states, whereas in Chapters 7 and 1 I they were represented by logical formulas. Those approaches defined belief states in terms of which world states were possible, but could say nothing about which states were likely or unlikely. In this chapter, we use probability theory to quantify the degree of belief in elements of the belief state. As we show in Section 15.1, time itself is handled in the same way as in Chapter 7: a changing world is modeled using a variable for each aspect of the world state at each point in time. The transition and sensor models may be uncertain: the transition model describes the probability distribution of the variables at time t, given the state of the world at past times. while the sensor model describes the probability of each percept at time t, given the current state of the world. Section 15.2 defines the basic inference tasks and describes the general structure of inference algorithms for temporal models. Then we describe three specific kinds of models: hidden Markov models, Kalman filters, and dynamic Bayesian networks (which include hidden Marker models and Kalman filters as special cases). Finally. Section 15.6 examines the problems faced when keeping track of name than one thing 15.1 TIME AND UNCERTAINTY We have developed our techniques for probabilistic reasoning in the context of static worlds, in which each random variable has a single fixed value. For example, when repairing a eat we assume that whatever is broken remains broken during the process of diagnosis; our job is to infer the state of the car from observed evidence, which also remains fixed. 566 Section 15.1. Time and Uncertainty 567 Now consider a slightly different problem: treating a diabetic patient. As in the case of car repair, we have evidence such as recent insulin doses, food intake, blood sugar measurements, and other physical signs. The task is to assess the current state of the patient, including the actual blood sugar level and insulin level. Given this information, we can make a decision about the patient's food intake and insulin dose. Unlike the case of car repair, here the dynamic aspects of the problem are essential. Blood sugar levels and measurements thereof can change rapidly over time, depending on recent food intake and insulin doses, metabolic activity, the time of day, and so on. To assess the current state from the history of evidence and to predict the outcomes of treatment actions, we must model these changes. The same considerations arise in many other contexts, such as tracking the location of a robot, tracking the economic activity of a nation, and making sense of a spoken or written sequence of words. How can dynamic situations like these be modeled? 15.1.1 TIME SLICE States and observations We view the world as a series of snapshots, or time slices, each of which contains a set of random variables, some observable and some not.' For simplicity, we will assume that the same subset of variables is observable in each time slice (although this is not strictly necessary in anything that follows). We will use X t to denote the set of state variables at time t, which are assumed to be unobservable, and E t to denote the set of observable evidence variables. The observation at time t is E t —e t for some set of values e t . Consider the following example: You are the security guard stationed at a secret underpound installation. You want to know whether it's raining today, but your only access to the outside world occurs each morning when you see the director coming in with, or without, an umbrella For each day t, the set E t thus contains a single evidence variable Umbrella, or (.1t for short (whether the umbrella appears), and the set X t contains a single state variable Rain t or Rt for short (whether it is raining). Other problems can involve larger sets of variables. In the diabetes example, we might have evidence variables, such as MeasuredBloadSugar t and PutseRate,, and state variables, such as BloodSugar, and Stomachamtenis t . (Notice that BloodSugar t and MeasuredBloodthigar t are not the same variable; this is how we deal with noisy measurements of actual quantities.) The interval between time slices also depends on the problem. For diabetes monitoring, a suitable interval might be an hour rather than a day. In this chapter we assume the interval between slices is fixed, so we can label times by integers. We will assume that the state sequence starts at I = 0; for various uninteresting reasons, we will assume that evidence starts arriving at t =1 rather than t = 0. Hence, our umbrella world is represented by state variables Ro, RI, R2,... and evidence variables Eri. , Ua, .... We will use the notation a:b to denote the sequence of integers from a to Et (inclusive), and the notation Xa ,b to denote the set of variables from X„ to Xb_ For example, th, a corresponds to the variables Ul U2, T.T3. , Uncertainty over cotifinzious time can be modeled by stochastic differential equations (SDEs). The models studied in this chapter can be viewed as discrete-time approximations to SDEs.