Psychophysical studies of change blindness indicate that, at any instant, human observers are aware of detail in few parts of the visual field. Such results suggest, to some theorists, that human vision reconstructs only a few portions of the visual scene and that, to bridge the resulting representational gaps, it often lets physical objects serve as their own short-term memory. We propose that human vision reconstructs no portion of the visual scene, and that it never lets physical objects serve as their own short-term memory.

Upshot: Formal models lead beyond ordinary experience to abstractions such as black holes and quantum entanglement. Applying such models to experience itself makes it seem unfamiliar and even paradoxical. We suggest, however, that doing so also leads to insights. It shows, in particular, that the “view from nowhere” employed by the theorist is both essential and deeply paradoxical, and it suggests that experience has an unrecorded, non-reportable component in addition to its remembered, reportable component.

Context: The evolution of perceptual systems and hence of observers remains largely disconnected from the question of the emergence of classical objects and spacetime. This disconnection between the biosciences and physics impedes progress toward understanding the role of the “observer” in physical theory. Problem: In this article we consider the problem of how to understand objects and spacetime in observer-relative evolutionary terms. Method: We rely on a comparative analysis using multiple formal frameworks. Results: The eigenform construct of von Foerster is compared to other formal representations of observer-environment interactions. Eigenforms are shown to be encoded on observer-environment interfaces and to encode fitness consequences of actions. Space and time are components of observational outcomes in this framework; it is suggested that spacetime constitutes an error-correcting code for fitness consequences. Implications: Our results contribute to an understanding of the world in which neither objects nor spacetime are observer-independent. Constructivist content: The eigenform concept of von Foerster is linked to the concepts of decoherence and holographic encoding from physics and the concept of fitness from evolutionary biology.

Key words: Active inference, boundary, conscious agent, icon, Markov blanket, redundancy.

A goal of perception is to estimate true properties of the world. A goal of categorization is to classify its structure. Aeons of evolution have shaped our senses to this end. These three assumptions motivate much work on human perception. I here argue, on evolutionary grounds, that all three are false. Instead, our perceptions constitute a species-specific user interface that guides behavior in a niche. Just as the icons of a PC’s interface hide the complexity of the computer, so our perceptions usefully hide the complexity of the world, and guide adaptive behavior. This interface theory of perception offers a framework, motivated by evolution, to guide research in object categorization. This framework informs a new class of evolutionary games, called interface games, in which pithy perceptions often drive true perceptions to extinction.

Excerpt: This standard account of visual illusions naturally raises the question as to why our perceptions should be fallible. What is wrong with our visual system that allows false perceptions to occur?

We propose that selection favors nonveridical perceptions that are tuned to fitness. Current textbooks assert, to the contrary, that perception is useful because, in the normal case, it is veridical. Intuition, both lay and expert, clearly sides with the textbooks. We thus expected that some commentators would reject our proposal and provide counterarguments that could stimulate a productive debate. We are pleased that several commentators did indeed rise to the occasion and have argued against our proposal. We are also pleased that several others found our proposal worth exploring and have offered ways to test it, develop it, and link it more deeply to the history of ideas in the science and philosophy of perception. To both groups of commentators: thank you. Point and counterpoint, backed by data and theory, is the essence of science. We hope that the exchange recorded here will advance the scientific understanding of perception and its evolution. In what follows, we respond to the commentaries in alphabetical order.

Key words: bayesian inference, categorization, parameter estimation, perceptual categorization and identification, visual perception.

Perception is a product of evolution. Our perceptual systems, like our limbs and livers, have been shaped by natural selection. The effects of selection on perception can be studied using evolutionary games and genetic algorithms. To this end, we define and classify perceptual strategies and allow them to compete in evolutionary games in a variety of worlds with a variety of fitness functions. We find that veridical perceptions – strategies tuned to the true structure of the world – are routinely dominated by nonveridical strategies tuned to fitness. Veridical perceptions escape extinction only if fitness varies monotonically with truth. Thus, a perceptual strategy favored by selection is best thought of not as a window on truth but as akin to a windows interface of a PC. Just as the color and shape of an icon for a text file do not entail that the text file itself has a color or shape, so also our perceptions of space-time and objects do not entail (by the Invention of Space-Time Theorem) that objective reality has the structure of space-time and objects. An interface serves to guide useful actions, not to resemble truth. Indeed, an interface hides the truth; for someone editing a paper or photo, seeing transistors and firmware is an irrelevant hindrance. For the perceptions of H. sapiens, space-time is the desktop and physical objects are the icons. Our perceptions of space-time and objects have been shaped by natural selection to hide the truth and guide adaptive behaviors. Perception is an adaptive interface.

Key words: bayesian inference and parameter estimation, perceptual learning, perceptual categorization and identification, spatial cognition, evolution.