Psychophysical studies of change blindness indicate that, at any instant, human observers are aware of detail in few parts of the visual field. Such results suggest, to some theorists, that human vision reconstructs only a few portions of the visual scene and that, to bridge the resulting representational gaps, it often lets physical objects serve as their own short-term memory. We propose that human vision reconstructs no portion of the visual scene, and that it never lets physical objects serve as their own short-term memory.

Upshot: Formal models lead beyond ordinary experience to abstractions such as black holes and quantum entanglement. Applying such models to experience itself makes it seem unfamiliar and even paradoxical. We suggest, however, that doing so also leads to insights. It shows, in particular, that the “view from nowhere” employed by the theorist is both essential and deeply paradoxical, and it suggests that experience has an unrecorded, non-reportable component in addition to its remembered, reportable component.

Context: The evolution of perceptual systems and hence of observers remains largely disconnected from the question of the emergence of classical objects and spacetime. This disconnection between the biosciences and physics impedes progress toward understanding the role of the “observer” in physical theory. Problem: In this article we consider the problem of how to understand objects and spacetime in observer-relative evolutionary terms. Method: We rely on a comparative analysis using multiple formal frameworks. Results: The eigenform construct of von Foerster is compared to other formal representations of observer-environment interactions. Eigenforms are shown to be encoded on observer-environment interfaces and to encode fitness consequences of actions. Space and time are components of observational outcomes in this framework; it is suggested that spacetime constitutes an error-correcting code for fitness consequences. Implications: Our results contribute to an understanding of the world in which neither objects nor spacetime are observer-independent. Constructivist content: The eigenform concept of von Foerster is linked to the concepts of decoherence and holographic encoding from physics and the concept of fitness from evolutionary biology.

Key words: Active inference, boundary, conscious agent, icon, Markov blanket, redundancy.

A goal of perception is to estimate true properties of the world. A goal of categorization is to classify its structure. Aeons of evolution have shaped our senses to this end. These three assumptions motivate much work on human perception. I here argue, on evolutionary grounds, that all three are false. Instead, our perceptions constitute a species-specific user interface that guides behavior in a niche. Just as the icons of a PC’s interface hide the complexity of the computer, so our perceptions usefully hide the complexity of the world, and guide adaptive behavior. This interface theory of perception offers a framework, motivated by evolution, to guide research in object categorization. This framework informs a new class of evolutionary games, called interface games, in which pithy perceptions often drive true perceptions to extinction.

Excerpt: This standard account of visual illusions naturally raises the question as to why our perceptions should be fallible. What is wrong with our visual system that allows false perceptions to occur?

Our perceptual systems are products of evolution and have been shaped, in part, by natural selection. It is widely assumed that natural selection favors veridical perceptions – that is, perceptions that accurately describe aspects of the objective world relevant to fitness. This assumption has been tested using the mathematics of evolutionary game theory. It is false. Monte Carlo simulations reveal that veridical perceptions are never more fit, and generically are less fit, than nonveridical perceptions of equal complexity that are tuned to fitness. Veridical perceptions go extinct, and their extinction rate increases as complexity increases. These results motivate a new theory of perceptual systems – as species-specific interfaces shaped by natural selection to hide objective reality and guide adaptive behavior. For Homo sapiens, space-time is the desktop of the interface and physical objects are icons on the desktop. The shapes and colors of physical objects no more resemble objective reality than the shapes and colors of desktop icons resemble files in a computer.

Key words: evolution, natural selection, evolutionary game theory, perception, bayesian inference, veridical perception, illusion.

Marr proposed that human vision constructs “a true description of what is there”. He argued that to understand human vision one must discover the features of the world it recovers and the constraints it uses in the process. Bayesian decision theory (BDT) is used in modern vision research as a probabilistic framework for understanding human vision along the lines laid out by Marr. Marr’s contribution to vision research is substantial and justly influential. We propose, however, that evolution by natural selection does not, in general, favor perceptions that are true descriptions of the objective world. Instead, research with evolutionary games shows that perceptual systems tuned solely to fitness routinely outcompete those tuned to truth. Fitness functions depend not just on the true state of the world, but also on the organism, its state, and the type of action. Thus, fitness and truth are distinct. Natural selection depends only on expected fitness. It shapes perceptual systems to guide fitter behavior, not to estimate truth. To study perception in an evolutionary context, we introduce the framework of Computational Evolutionary Perception (CEP). We show that CEP subsumes BDT, and reinterprets BDT as evaluating expected fitness rather than estimating truth.

Key words: evolution, fitness, natural selection, bayesian decision theory, vision, evolutionary game theory.

We propose that selection favors nonveridical perceptions that are tuned to fitness. Current textbooks assert, to the contrary, that perception is useful because, in the normal case, it is veridical. Intuition, both lay and expert, clearly sides with the textbooks. We thus expected that some commentators would reject our proposal and provide counterarguments that could stimulate a productive debate. We are pleased that several commentators did indeed rise to the occasion and have argued against our proposal. We are also pleased that several others found our proposal worth exploring and have offered ways to test it, develop it, and link it more deeply to the history of ideas in the science and philosophy of perception. To both groups of commentators: thank you. Point and counterpoint, backed by data and theory, is the essence of science. We hope that the exchange recorded here will advance the scientific understanding of perception and its evolution. In what follows, we respond to the commentaries in alphabetical order.

Key words: bayesian inference, categorization, parameter estimation, perceptual categorization and identification, visual perception.

Perception is a product of evolution. Our perceptual systems, like our limbs and livers, have been shaped by natural selection. The effects of selection on perception can be studied using evolutionary games and genetic algorithms. To this end, we define and classify perceptual strategies and allow them to compete in evolutionary games in a variety of worlds with a variety of fitness functions. We find that veridical perceptions – strategies tuned to the true structure of the world – are routinely dominated by nonveridical strategies tuned to fitness. Veridical perceptions escape extinction only if fitness varies monotonically with truth. Thus, a perceptual strategy favored by selection is best thought of not as a window on truth but as akin to a windows interface of a PC. Just as the color and shape of an icon for a text file do not entail that the text file itself has a color or shape, so also our perceptions of space-time and objects do not entail (by the Invention of Space-Time Theorem) that objective reality has the structure of space-time and objects. An interface serves to guide useful actions, not to resemble truth. Indeed, an interface hides the truth; for someone editing a paper or photo, seeing transistors and firmware is an irrelevant hindrance. For the perceptions of H. sapiens, space-time is the desktop and physical objects are the icons. Our perceptions of space-time and objects have been shaped by natural selection to hide the truth and guide adaptive behaviors. Perception is an adaptive interface.

Key words: bayesian inference and parameter estimation, perceptual learning, perceptual categorization and identification, spatial cognition, evolution.

Does natural selection favor veridical perceptions, those that more accurately depict the objective environment? Students of perception often claim that it does. But this claim, though influential, has not been adequately tested. Here we formalize the claim and a few alternatives. To test them, we introduce “interface games,” a class of evolutionary games in which perceptual strategies compete. We explore, in closed-form solutions and Monte Carlo simulations, some simpler games that assume frequency-dependent selection and complete mixing in infinite populations. We find that veridical perceptions can be driven to extinction by non-veridical strategies that are tuned to utility rather than objective reality. This suggests that natural selection need not favor veridical perceptions, and that the effects of selection on sensory perception deserve further study.

Key words: evolution, replicator dynamics, interface games, sensation, game theory.