Are the Past and the Future Really Out There?