Computation and Human Experience

Chapter 1. Introduction

(Please do not quote from this version, which changed slightly in proof. This version is also missing footnotes.)

1. Activity

Computational inquiry into human nature originated in the years after World War II. Scientists mobilized into wartime research had developed a series of technologies that lent themselves to anthropomorphic description, and once the war ended these technologies inspired novel forms of psychological theorizing. A servomechanism, for example, could aim a gun by continually sensing the target's location and pushing the gun in the direction needed to intercept it. Technologically sophisticated psychologists and mathematicians observed that this feedback cycle could be described in human-like terms as pursuing a purpose based on awareness of its environment and anticipation of the future. New methods of signal detection could likewise be described as making perceptual discriminations, and the analytical tools of information theory soon provided mathematical ways to talk about communication.

In the decades after the war, these technical ideas provided the intellectual license for a counterrevolution against behaviorism and a restoration of scientific status to human mental life. The explanatory power of these ideas lay in a suggestive confluence of metaphor, mathematics, and machinery. Metaphorical attributions of purpose were associated with the mathematics of servocontrol and realized in servomechanisms; metaphorical attributions of discrimination were associated with the mathematics of signal and noise and realized in communications equipment; and metaphorical attributions of communication were associated with the mathematics of information theory and realized in coding devices. The new psychology sought to describe human beings using vocabulary that could be metaphorically associated with technologically realizable mathematics.

The development of the stored-program digital computer put this project into high gear. It is a commonplace that the computer contributed a potent stock of metaphors to modern psychology, but it is important to understand just how these metaphors informed the new research. The outlines of the project were the same as with servocontrol, signal detection, coding theory: a bit of metaphor attached to a bit of mathematics and realized in a machine whose operation could then be narrated using intentional vocabulary. But the digital computer both generalized and circumscribed this project. By writing computer programs, one could physically realize absolutely any bit of finite mathematics one wished. The inside of the computer thus became an imaginative landscape in which programmers could physically realize an enormous variety of ideas about the nature of thought. Fertile as this project was, it was also circumscribed precisely by the boundaries of the computer. The feats of physics and chemistry that supported the digital abstraction operated inside the computer, and not outside.

In this way, a powerful dynamic of mutual reinforcement took hold between the technology of computation and a Cartesian view of human nature, with computational processes inside computers corresponding to thought processes inside minds. But the founders of computational psychology, while mostly avowed Cartesians, actually transformed Descartes's ideas in a complex and original way. They retained the radical experiential inwardness that Descartes, building on a long tradition, had painted as the human condition. And they retained the Cartesian understanding of human bodies and brains as physical objects, extended in space and subject to physical laws. Their innovation lay in a subversive reinterpretation of Descartes's ontological dualism (Gallistel 1980: 6-7). In The Passions of the Soul, Descartes had described the mind as an extensionless res cogitans that simultaneously participated in and transcended physical reality. The mind, in other words, interacted causally with the body, but was not itself a causal phenomenon. Sequestered in this nether region with its problematic relationship to the physical world, the mind's privileged object of contemplation was mathematics. The "clear and distinct ideas" that formed the basis of Descartes's epistemology in the Meditations were in the first instance mathematical ideas (Rorty 1979: 57-62; cf. Heidegger 1961 [1927]: 128-134). Of course, generations of mechanists beginning with Hobbes, and arguably from antiquity, had described human thought in monistic terms as the workings of machinery (Haugeland 1985: 23). But these theorists were always constrained by the primitive ideas about machinery that were available to them. Descartes's physiology suffered in this way, but not his psychology. Although they paid little heed to the prescriptive analysis of thought that Descartes had offered, the founders of computational psychology nonetheless consciously adopted and reworked the broader framework of Descartes's theory, starting with a single brilliant stroke. The mind does not simply contemplate mathematics, they asserted; the mind is itself mathematical, and the mathematics of mind is precisely a technical specification for the causally explicable operation of the brain.

This remarkable proposal set off what is justly called a "revolution" in philosophy and psychology as well as in technology. Technology is in large measure a cultural phenomenon, and never has it been more plainly so than in the 1950s. Computational studies in that decade were studies of faculties of intelligence and processes of thought, as part of a kind of cult of cognition whose icons were the rocket scientist, the symbolism of mathematics, and the computer itself. The images now strike us as dated and even camp, but we are still affected by the technical practice and the interpretation of human experience around which artificial intelligence, or AI, was first organized.

I wish to investigate this confluence of technology and human experience. The philosophical underside of technology has been deeply bound up with larger cultural movements, yet technical practitioners have generally understood themselves as responding to discrete instrumental "problems" and producing technologies that have "effects" upon the world. In this book I would like to contribute to a critical technical practice in which rigorous reflection upon technical ideas and practices becomes an integral part of day-to-day technical work itself.

I will proceed through a study in the intellectual history of research in AI. The point is not to exhaust the territory but to focus on certain chapters of AI's history that help illuminate the internal logic of its development as a technical practice. Although it will be necessary to examine a broad range of ideas about thought, perception, knowledge, and their physical realization in digital circuitry, I will focus centrally on computational theories of action. This choice is strategic, inasmuch as action has been a structurally marginal and problematic topic in AI; the recurring difficulties in this computational research on action, carefully interpreted, motivate critiques that strike to the heart of the field as it has historically been constituted. I aim to reorient research in AI away from cognition -- abstract processes in the head -- and toward activity -- concrete undertakings in the world. This is not a different subject, but a different approach to the same subject: different metaphors, methods, technologies, prototypes, and criteria of evaluation. Effecting such a reorientation will require technical innovation, but it will also require an awareness of the structure of ideas in AI and how these ideas are bound up with the language, the methodology, and the value systems of the field.

Roughly speaking, computational research into activity seeks technical ideas about action and representation that are well suited to the special requirements of situated, embodied agents living in the physical world. The "agents" could be robots we would like to build or creatures we would like to understand. The word agent though common in AI, does not appeal to everyone. Its advantage is its ambiguity -- robots, insects, cats, and people are all agents. Such vocabulary tacitly promises, of course, that computation provides useful ways of talking about robots, insects, cats, and people at the same time without reducing all of them to a bloodless technical order. In any event, I will have little to say about insects and cats. To say that an agent is situated is to emphasize that its actions make little sense outside of the particular situation in which it finds itself in the physical and social world; it is always provided with particular materials and involved with particular other agents. To say that an agent is embodied is simply to say that it has a body. Even better, following Merleau-Ponty (1962 [1945]), it is a body or exists as a body. As a physical being, it has a definite location, limited experience, and finite abilities. It is in the world, among the world's materials, and with other agents. The claim is not simply that these things are true (hardly anybody would deny them), but also that taking them seriously requires an overhaul of basic ideas about both computation and activity.

My project is both critical and constructive. By painting computational ideas in a larger philosophical context, I wish to ease critical dialogue between technology and the humanities and social sciences (Bolter 1984; Guzeldere and Franchi 1995). The field of AI could certainly benefit from a more sophisticated understanding of itself as a form of inquiry into human nature. In exchange, it offers a powerful mode of investigation into the practicalities and consequences of physical realization.

My recommendation of a shift of focus from cognition to activity converges with a number of other intellectual trends, each of which is also founded in a critique of Cartesianism. These include the otherwise disparate traditions that descend from Heidegger's phenomenological analysis of routine activity, Vygotsky's theory of human development, and Garfinkel's studies of the interactional construction of social reality. Each of these schools of thought has attempted to replace the philosophical opposition between a self-contained perceiving subject and an independent external object by describing our relationships to things as fundamentally bound up with their role in our ongoing projects, which in turn are defined by our cultures, located in forms of embodied activity, and acquired through socialization into a system of cultural practices. As AI reorients itself toward the study of activity, it will be able to engage in mutually beneficial dialogue with these traditions of research. This process begins with computational ways of thinking about routine activity.

2. Planning

Although the AI tradition has placed its principal emphasis on processes it conceives of as occurring entirely within the mind, there does exist a more or less conventional computational account of action. The early formulation of this account that had the most pervasive influence was George Miller, Eugene Galanter, and Karl Pribram's book, Plans and the Structure of Behavior (1960). These authors rejected the extreme behaviorist view that the organized nature of activity results from isolated responses to isolated stimuli. Instead, they adopted the opposite extreme view that the organization of human activity results from the execution of mental structures they called Plans. Plans were hierarchical in the sense that a typical Plan consisted of a series of smaller sub-Plans, each of which consisted of yet smaller sub-Plans, and so forth, down to the primitive Plan steps, which one imagines to correspond to individual muscle movements. (Miller, Galanter, and Pribram capitalized the word "Plan" to distinguish their special use of it, especially in regard to the hierarchical nature of Plans, from vernacular usage. Subsequent authors have not followed this convention. I will follow it when I mean to refer specifically to Miller, Galanter, and Pribram's concept.)

What is a Plan? "A Plan is any hierarchical process in the organism that can control the order in which a sequence of operations is to be performed" (Miller et al. 1960: 16). They state, as a "scientific hypothesis" about which they are "reasonably confident," that a Plan is "essentially the same as a program for a computer," a connotation the term has carried to the present day. Shortly thereafter, though, they state that "we shall also use the term `Plan' to designate a rough sketch of some course of action, just the major topic headings in the outline, as well as the completely detailed specification of every detailed operation" (Miller et al. 1960: 17). Thus a new Plan's hierarchical structure need not initially reach down to the most primitive actions, though the hierarchy must be constructed in full detail by the time any given step of it is executed. They define execution by saying that "a creature is executing a particular Plan when in fact that Plan is controlling the sequence of operations he is carrying out" (Miller et al. 1960: 17).

Miller, Galanter, and Pribram applied the term "Plan" as broadly as they could. In considering various aspects of everyday life, they focused everywhere on elements of intentionality, regularity, and goal-directedness and interpreted each one as the manifestation of a Plan. As with the servos, radars, and codes that first inspired Miller and his contemporaries in the 1940s, the concept of a Plan combined the rhetoric of structured behavior with the formalisms of programming and proposed that the latter serve as models of biological systems. A great difficulty in evaluating this proposal is the imprecise way in which Miller, Galanter, and Pribram used words like "Plan". They demonstrated that one can find aspects of apparent planfulness in absolutely any phenomenon of human life. But in order to carry out this policy of systematic assimilation, important aspects of activity had to be consigned to peripheral vision. These marginalized aspects of activity were exactly those which the language of Plans and their execution tends to deemphasize.

These ideas had an enormous influence on AI, but with some differences of emphasis. Although they occasionally employ the term "planning", Miller, Galanter, and Pribram provide no detailed theory of the construction of new Plans. The AI tradition, by contrast, has conducted extensive research on plan-construction but has generally assumed that execution is little more than a simple matter of running a computer program. What has remained is a definite view of human activity that has continued, whether implicitly or explicitly, to suffuse the rhetoric and technology of computational theories of action. In place of this view, I would like to substitute another, one that follows the anthropologically motivated theoretical orientations of Suchman (1987) and Lave (1988) in emphasizing the situated nature of human action. Let me contrast the old view and the new point by point:

* Why does activity appear to be organized?

Planning view: If someone's activity has a certain organization, that is because the person has constructed and executed a representation of that activity, namely a plan.

Alternative: Everyday life has an orderliness, a coherence, and patterns of change that are emergent attributes of people's interactions with their worlds. Forms of activity might be influenced by representations but are by no means mechanically determined by them.

* How do people engage in activity?

Planning view: Activity is fundamentally planned; contingency is a marginal phenomenon. People conduct their activity by constructing and executing plans.

Alternative: Activity is fundamentally improvised; contingency is the central phenomenon. People conduct their activity by continually redeciding what to do.

* How does the world influence activity?

Planning view: The world is fundamentally hostile, in the sense that rational action requires extensive, even exhaustive, attempts to anticipate difficulties. Life is difficult and complicated, a series of problems to be solved.

Alternative: The world is fundamentally benign, in the sense that our cultural environment and personal experiences provide sufficient support for our cognition that, as long as we keep our eyes open, we need not take account of potential difficulties without specific grounds for concern. Life is almost wholly routine, a fabric of familiar activities.

The alternative view of human activity that I have sketched here contains a seeming tension: how can activity be both improvised and routine? The answer is that the routine of everyday life is not a matter of performing precisely the same actions every day, as if one were a clockwork device executing a plan. Instead, the routine of everyday life is an emergent phenomenon of moment-to-moment interactions that work out in much the same way from day to day because of the relative stability of our relationships with our environments.

My sketched alternative also denies a central role to the use of plans. People certainly use plans. But real plans are nothing like computer programs. Sensibly organized goal-directed activity need not result from the use of a plan. And plans never serve as direct specifications of action. Instead, a plan is merely one resource among many that someone might use in deciding what to do (Suchman 1987). Before and beneath any use of plans is a continual process of moment-to-moment improvisation. "Improvisation", as I will employ the term, might involve ideas about the future and it might employ plans, but it is always a matter of deciding what to do now. Indeed, the use of plans is a relatively peripheral phenomenon and not a principal focus here.

To speak of a "planning view" is misleading in one respect: hardly anybody is aware of having committed themselves to such a view. Future chapters will explain more precisely the sense in which the planning view has governed research in AI. For the time being, it will be helpful to consider Heidegger's (1961 [1927]) account of why the emergence of something like the planning view is nearly inevitable. Most of us, Heidegger observes, spend our days immersed in practical concerns. We are concerned with the traffic, the paperwork, the dust, the celery -- with the objects that we encounter as we pursue our goals and enact our identities. We find it natural, therefore, to see the world as a constellation of objects. Moreover, the occasions on which particular objects really come to our attention are not representative of activity as a whole. Sometimes we momentarily detach ourselves from our daily concerns to contemplate an object in a special way -- as, for example, a work of art. And sometimes an object simply becomes obstinate; perhaps it is broken, or missing, or not the right size. In these situations, we confront the object as a stranger -- as something very much separate from us. It is *problems* that attract our attention, and problems play a wildly disproportionate role in the stories we tell about our lives. We hardly notice the vast background of ordinary, routine, unproblematic activities from which are lives are largely made. Even when a problem does arise, the detection and resolution of the problem both consist of concrete activities that are mostly routine. Because this unproblematic background of routine activity goes largely unnoticed, we can succumb to the illusion that life is basically a series of problems, that situations in life typically require thinking and planning, and that our normal way of relating to objects involves detached contemplation. This illusion does not simply arise in individual experience; it is also handed down through metaphors, narrative conventions, philosophical systems, and other cultural constructs. It is this illusory view of life -- the planning view -- that first crystallized in its modern form in Descartes and that originally defined the tacit agenda for research in AI. Yet, I will argue, the planning view is inadequate both as an account of human life and as an approach to computational modeling.

It is hard to know, of course, how to evaluate Heidegger's argument. Perhaps we should treat it as a just-so story; Heidegger, in any case, presented it as a phenomenological description and a reconstruction of the history of philosophy, not a logical deduction from premises or a scientific inference from evidence. For our purposes here, though, that is enough. Heidegger's story is useful in several ways. It confers an overall sense on the more detailed analyses of Descartes and other theorists of mechanism. It also directs our attention heuristically to technical difficulties that might otherwise have gone undiagnosed or misunderstood. Above all, it helps us cultivate an awareness of our own experience as human beings. Heidegger's crucial insight is that philosophical ideas tend to formalize the ways we experience our lives; if we experience our lives in superficial ways then our philosophies will be correspondingly superficial. The same reasoning applies to computational models, which (whether the model-builders realize it or not) are derived from philosophical theories and guided in their development by experiences of everyday life. Better descriptions of everyday life do not disprove technical ideas, but they do motivate different intuitions, and they also help evaluate the appeals to everyday intuition that are found throughout AI research. AI's pervasive focus on problems, for example, aligns with the unreflective emphasis on problems that Heidegger finds in the modern experience of everyday life. By failing to place problems in the context of an unproblematic background, AI may fall prey to a mistaken conception of them and an excessive emphasis on attempts to solve them. The point in each case is not to debunk AI or technology in general, but to gain what Heidegger would call a "free relation" to it, so that technological modes of thinking do not colonize our awareness of our lives (Dreyfus 1995). Let us turn to the methodological issues that arise as AI research is rethought in this way.

3. Why build things?

Every discipline has its distinctive ways of knowing, which it identifies with the activities it regards as its own: anthropologists do fieldwork, architects design buildings, monks meditate, and carpenters make things out of wood. Each discipline wears its defining activity as a badge of pride in a craftworker's embodied competence. It will be said, "You can read books all your life, but you don't really know about it until you do it". Disciplinary boundaries are often defined in such ways -- you are not an anthropologist unless you have spent a couple years in the field; you are not an architect unless you have built a building; and so forth -- and neighboring disciplines may be treated with condescension or contempt for their inferior methods. Each discipline's practitioners carry on what Schon (1983: 78) would call "reflective conversations" with their customary materials, and all of their professional interactions with one another presuppose this shared background of sustained practical engagement with a more or less standard set of tools, sites, and hassles. Defining a discipline through its own special activity carries risks. If a disciplinary community cultivates invidious contrasts between its own methods and those of other fields, it will surely become inbred and insular, emptied by hubris and intolerance. If it is guided by critical reflection on its practices and presuppositions, however, it has at least a chance of continually deepening its self-awareness, renewing itself through interdisciplinary dialogue without losing its distinctive advantages. The culture of any particular discipline will presumably be found somewhere between these extremes.

The discipline in question here is computational modeling, and specifically AI. Although I will criticize certain computational ideas and practices at great length, my enterprise is computational nonetheless. AI's distinctive activity is building things, specifically computers and computer programs. Building things, like fieldwork and meditation and design, is a way of knowing that cannot be reduced to the reading and writing of books (Chapman 1991: 216-217). To the contrary, it is an enterprise grounded in a routine daily practice. Sitting in the lab and working on gadgets or circuits or programs, it is an inescapable fact that some things can be built and other things cannot. Likewise, some techniques scale up to large tasks and others do not; and some devices operate robustly as environmental conditions fluctuate, whereas others break down. The AI community learns things by cultivating what Keller (1983) calls a "feeling for the organism", gradually making sense of the resulting patterns of what works and what does not. Edwards (1996: 250, italics in the original) rightly emphasizes that much of AI's practitioners' technical framework "emerged not abstractly but in their experiences with actual machines". And Simon (1969: 20) speaks of computing as an "empirical science" -- a science of design.

I take an unusual position on the nature of computation and computational research. For my purposes, computation relates to the analysis and synthesis of especially complicated things. These analytic and synthetic practices are best understood as nothing less grand or more specific than an inquiry into physical realization as such. This fact can be lost beneath ideologies and institutions that define computation in some other way, whether in terms of Turing machines, mathematical abstraction, intentionality, symbolic reasoning, or formal logic. Nonetheless, what truly founds computational work is the practitioner's evolving sense of what can be built and what cannot. This sense, at least on good days, is a glimpse of reality itself. Of course, we finite creatures never encounter this "reality" except through the mediation of a historically specific ensemble of institutions, practices, genres, ideologies, tools, career paths, divisions of labor, understandings of "problems" and "solutions", and so forth. These mediating systems vary historically through both practical experience and broader shifts of social climate. Nonetheless, at each point the technologist is pushing up against the limits of a given epoch's technology, against the limits of physical reality conceived and acted upon in a specific way. These limits are entirely real. But they are not simply a product of reality-in-itself; nor are they simply internal consequences of the idea-systems on their own, considered in abstraction from particular attempts to get things to work.

This is the sense in which people engaged in technical work are -- and, I think, must be -- philosophical realists. The something-or-other that stands behind each individual encounter with the limits of physical realization I would like to call practical reality. Practical reality is something beyond any particular model or ontology or theory. A given model might seem like the final word for years or centuries, but ultimately the limit-pushing of technical work will reveal its margins. The resulting period of stumbling and improvisation will make the previously taken-for-granted model seem contingent: good enough perhaps for some purposes, but no longer regarded (if it ever has been) as a transparent description of reality. Much of the sophistication of technical work in mechanical engineering and semiconductor physics, for example, lies in the astute choice of models for each purpose.

Technical communities negotiate ceaselessly with the practical reality of their work, but when their conceptions of that reality are mistaken, these negotiations do not necessarily suffice to set them straight. Computational research, for its part, has invested an enormous amount of effort in the development of a single model of computation: the dual scheme of abstraction and implementation that I will describe in Chapter 4. This framework has motivated a multitude of technical proposals, but it has also given rise to recurring patterns of technical trouble. Although computationalists do possess a certain degree of critical insight into the patterns of trouble that arise in their work, they also take a great deal for granted. Beneath the everyday practices of computational work and the everyday forms of reasoning by which computationalists reflect on their work, a vast array of tacit commitments lies unexamined. Each of these commitments has its margins, and the field's continual inadvertent encounters with these margins have accumulated, each compounding the others, to produce a dull sense of existential chaos. Nobody complains about this, for the simple reason that nobody has words to identify it. As successive manifestations of the difficulty have been misinterpreted and acted upon, the process has become increasingly difficult to disentangle or reverse.

In trying to set things right, a good place to start is with AI researchers' understanding of their own distinctive activity: building computer systems. AI people, by and large, insist that nothing is understood until it has been made into a working computer system. One reason to examine this insistence critically is its association with research values that disrupt interdisciplinary communication. This disruption goes in two directions -- from the inside out (i.e., from AI to the noncomputational world) and from the outside in (i.e., the other way round) -- and it is worth considering these two directions separately.

Research based on computer modeling of human life often strikes people from other fields as absurd. AI studies regularly oversimplify things, make radically counterfactual assumptions, and focus excessive attention on easy cases. In a sense this is nothing unusual: every field has to start somewhere, and it is usually easier to see your neighbor's leading assumptions than your own. But in another sense things really are different in AI than elsewhere. Computer people only believe what they can build, and this policy imposes a strong intellectual conservatism on the field. Intellectual trends might run in all directions at any speed, but computationalists mistrust anything unless they can nail down all four corners of it; they would, by and large, rather get it precise and wrong than vague and right. They often disagree about how much precision is required, and what kind of precision, but they require ideas that can be assimilated to computational demonstrations that actually get built. This is sometimes called the work ethic: it has to work. To get anything nailed down in enough detail to run on a computer requires considerable effort; in particular, it requires that one make all manner of arbitrary commitments on issues that may be tangential to the current focus of theoretical interest. It is no wonder, then, that AI work can seem outrageous to people whose training has instilled different priorities -- for example, conceptual coherence, ethnographic adequacy, political relevance, mathematical depth, or experimental support. And indeed it is often totally mysterious to outsiders what canons of progress and good research do govern such a seemingly disheveled enterprise. The answer is that good computational research is an evolving conversation with its own practical reality; a new result gets the pulse of this practical reality by suggesting the outlines of a computational explanation of some aspect of human life. The computationalist's sense of bumping up against reality itself -- of being compelled to some unexpected outcome by the facts of physical realizability as they manifest themselves in the lab late at night -- is deeply impressive to those who have gotten hold of it. Other details -- conceptual, empirical, political, and so forth -- can wait. That, at least, is how it feels.

How, then, can we keep what is good about AI's methods without falling into disciplinary chauvinism? We can start by rejecting the idea, derived from the institutions of engineering, that the sole test of computational research is whether it produces solutions to problems. These terms presuppose and conflate two unfortunate doctrines: Turing's model of computation (problems as input-output mappings) and the instrumentalism of engineering (problems as the unquestioned goals of one's employer). On this view, the work ethic is both right and wrong. It is right in the sense that building things is a valuable way of knowing. It is wrong, however, in that "working" is too narrow as a criterion of success. It is well known that a technical method can be perfectly well-defined and perform exactly according to its specification, while at the same time being as wrong-headed as you like (Jirotka and Goguen 1994). The point of a critical technical practice is that learning from technical experience takes sustained, sophisticated thought. Technical methods do not simply "work" or "fail to work". The picture is always mixed. Every method has its strengths and weaknesses, its elegance and its clumsiness, its subtle patterns of success and failure. These things ought to be the subject of intellectual discussion and dispute, both in particular cases and as general matters of methodology -- and not as a separate category of research but as something continuous with the presentation of technical projects.

The work ethic needs to be qualified in another way as well. To understand what is implied in a claim that a given computer model "works", one must distinguish two senses of "working". The first, narrow sense is "conforms to spec" -- that is, it works if its behavior conforms to a pregiven formal-mathematical specification. Since everything is defined mathematically, it does not matter what words we use to describe the system; we could use words like "plan", "learn", and "understand"; or we could use words like "foo", "bar", and "baz". In fact, programmers frequently employ nonsense terms like these when testing or demonstrating the logical behavior of a procedure. Local programming cultures will frequently invent their own sets of commonly used nonsense terms; where I went to school, the customary nonsense terms also included "blort", "quux", and "eep". But nonsense terms are not adequate for the second, broad sense of "working", which depends on specific words of natural language. As I mentioned at the very beginning, an AI system is only truly regarded as "working" when its operation can be narrated in intentional vocabulary, using words whose meanings go beyond the mathematical structures. When an AI system "works" in this broader sense, it is clearly a discursive construction, not just a mathematical fact, and the discursive construction only succeeds if the community assents. Critics of the field have frequently complained that AI people water down the meanings of the vernacular terms they employ, and they have sought to recover the original force of those terms, for example through the methods of ordinary language philosophy (Button, Coulter, Lee, and Sharrock 1995). But these critics have had little influence on the AI community's own internal standards of semantic probity. The community is certainly aware of the issue; McDermott (1981: 144), for example, forcefully warns against "wishful mnemonics" that lead to inflated claims. But these warnings have had little practical effect, and the reward systems of the field still depend solely on the production of technical schemata -- mathematically specified mechanisms and conventions for narrating their operation in natural language. The point, in any case, is that the practical reality with which AI people struggle in their work is not just "the world", considered as something objective and external to the research. It is much more complicated than this, a hybrid of physical reality and discursive construction. The trajectory of AI research can be shaped by the limitations of the physical world -- the speed of light, the three dimensions of space, cosmic rays that disrupt small memory chips -- and it can also be shaped by the limitations of the discursive world -- the available stock of vocabulary, metaphors, and narrative conventions. Technical tradition consists largely of intuitions, slogans, and lore about these hybrids, which AI people call "techniques", "methods", and "approaches"; and technical progress consists largely in the growth and transformation of this body of esoteric tradition. This is the sense in which computers are "language machines" (e.g., Edwards 1996: 28). Critical reflection on computer work is reflection upon both its material and semiotic dimensions, both synchronically and historically.

More specifically, the object of critical reflection is not computer programs as such but rather the process of technical work. Industrial software engineering is governed by rigid scripts that are dictated more by bureaucratic control imperatives than by the spirit of intellectual inquiry (Kraft 1977), but research programming is very much an improvisation -- a reflective conversation with the materials of computational practice. As it expands its collective understanding of this process, AI will become aware of itself as an intellectual enterprise whose concerns are continuous with those of numerous other disciplines. We are a long way from that goal, but any journey begins with wanting to go somewhere, and above all it is that desire itself that I hope to cultivate here.

To sum up, programming is a distinctive and valuable way of knowing. Doing it well requires both attunement to practical reality and acuity of critical reflection. Each of these criteria provides an indispensable guide and reinforcement to the other. Research always starts somewhere: within the whole background of concepts, methods, and values one learned in school. If our existing procedures are inadequate, practical reality will refuse to comply with our attempts to build things. And when technical work stalls, practical reality is trying to tell us something. Listening to it requires that we understand our technical exercises in the spirit of reductio ad absurdum, as the deconstruction of an inevitably inadequate system of ideas. Technique, in this sense, always contains an element of hubris. This is not shameful; it is simply the human condition. As the successive chapters of this book lay out some technical exercises of my own, the attentive reader will be able to draw up an extensive intellectual indictment of them, consisting of all the bogus assumptions that were required to put forth some proposal for evaluation. But the point of these technical exercises does not lie in their detailed empirical adequacy, or in their practical applicability; they do not provide canned techniques to take down from a shelf and apply in other cases. Instead, each exercise should be understood in the past tense as a case study, an attempt in good faith to evolve technical practice toward new ways of exploring human life. What matters, again, is the process. I hope simply to illustrate a kind of research, a way of learning through critical reflection on computational modeling projects. Others are welcome to form their own interpretations, provided that the practice of interpretation is taken seriously as a crucial component of the discipline itself.

4. How computation explains

Artificial intelligence, then, is a potentially valuable enterprise in need of considerable reform. It remains, however, to make precise the sense in which computation explains anything. A crucial conceptual difficulty is the complex manner in which AI straddles the boundary between science and engineering. Even though science and engineering have different goals and methods and vocabularies, the AI literature has drawn from both of them, often without any clear distinction. This is understandable enough, given the cross-fertilization that has contributed to the progress of computational science and engineering. (It is also understandable in the institutional context of early AI research, which was conducted largely by psychologists who were funded by instrumentally oriented military research agencies.) Still, it is sometimes important to distinguish clearly between the two projects, at least as ideal types. Science explains things that exist, aiming to learn the truth; engineering builds new things, aiming to serve instrumental goals. When a word like "planning" or "knowledge" or "reasoning" becomes a technical term, science and engineering pull it in different directions: science toward human planning and knowledge and reasoning; engineering toward whatever phenomena can profitably be realized within the existing technology (Woolgar 1987: 320). Yet the term artificial intelligence refers to a tradition of research that includes both scientific and engineering projects. Despite its ambiguity, I will use the term in the conventional way to refer to this tradition, and I will use the term computational psychology to name the scientific project of using computational methods to explain human phenomena. The term cognitive science will refer to the interdisciplinary scientific movement that began in the 1950s and gained steam in the 1980s, not all of which uses computational methods. The adjective technical will relate to any discipline of design that employs mathematical formalization (including any sort of digital design or computer software), whether for engineering design or for scientific modeling.

My own intention is to do science, or at least to learn something about human nature, and not to solve industrial problems. But I would also like to benefit from the powerful modes of reasoning that go into an engineering design rationale. One way to reconcile the claims of science and engineering is to posit that people are, in some sense, well engineered. If this is actually the case, then the engineering task of building synthetic intelligence might discover truths of "human design" as well. This is a tricky proposition, given that engineering is geared to the construction of devices with well-defined instrumental purposes. People, of course, have no such well-defined purposes and can be spoken of as "engineered" only in some special and limited sense. In particular, it is not necessary to embrace the dangerous notion that people are "optimal" from the point of view of some instrumental criterion. It is enough to understand how the existence of creatures such as ourselves might be possible at all.

How, then, can computation explain things about people? The most common proposal is that human beings instantiate some mechanism, psychology tries to discover what that mechanism is, and success is judged by matching the input-output behavior of hypothesized mechanisms to the input-output behavior of human beings (Fodor 1968: 121-152). This view of computational explanation had great appeal in the early days of computational psychology, since it promised to bring precision to a discipline that had long suffered from the vagueness and expressive poverty of its concepts. For many, a computer program was a theory whose predictions were its outputs. Such was the view of Feigenbaum and Feldman in their introduction to the influential anthology Computers and Thought (1963). In one of the papers in that volume Newell and Simon pursued the input-output matching view of computational theorizing, comparing the operation of the General Problem Solver (GPS) program step by step against the reports of an experimental subject. (I will return to GPS repeatedly in subsequent chapters.) This was an extraordinary advance in psychological method: computer programming had made it imaginable to explain singular events, not just statistical averages over large populations. Newell and Simon went to great lengths to fashion an experimental situation for which a comparison could be made (Dreyfus 1972: 112-114). They chose a task with definite formal rules and a well-defined goal, limited the subject's choice of actions, entrusted the execution of the actions to the experimenter, ensured that the subject had no experience with the task before the experiment began, and generally arranged for the subject to conform to all of the premises of the theory. Subsequent research in their school has mounted impressive campaigns of this kind, systematically constraining aspects of theories through precisely contrived experimental setups (Anderson 1983; Newell 1990).

This approach has an honorable history, but it also assumes a definite conception of research. Bound to the artificial and easily manipulated conditions of the laboratory, it gains precision at the risk of detaching itself from the ordinary reality of human life; models that are wildly incongruent with that reality may seem reasonable in the laboratory. I would like to pursue a different approach, one that preserves a connection between technical practice and ordinary experience. In doing so, I must somehow replace the whole system of research values that begins with the testing of input-output predictions. I doubt if it is possible yet to draw meaningful comparisons between computers and people across the vast majority of human activities. Looser comparisons will suffice, however, provided that we bring some analytical rigor to the question of why human beings are the way they are. Forty years of computational studies have shown that it is easy to build systems that resemble human activities in one way or another. What is harder is to specify the *computational principles* that might make such a resemblance significant.

A computational principle is a verity of design, some kind of conclusion about the practicalities of the physical realization of complicated things. One example of a computational principle is commonly known as least commitment (Marr 1982: 106; Stefik 1981). This is the idea that a system should, other things being equal, only derive the logically necessary conclusions of each new item of information, thus keeping its options open to the greatest possible extent. Another is the slogan that "a good representation exposes domain constraint," meaning that a good representation scheme expresses things in a way that makes it easy to formulate patterns of deduction that have utility within a given activity. Another is the concept of nearly decomposable systems (Simon 1969: 100), which suggests that well-designed artifacts will consist of loosely coupled components whose interactions are simple enough to be understood. Computational principles are, of course, always open to debate and reformulation. Indeed, each of these principles contain assumptions that would be worth reconsidering. (For example, the first two are often interpreted as requiring that computation be understood in terms of mathematical logic.) Nonetheless, principles like these help articulate a design community's accumulated experience in a way that can offer provisional guidance to theorizing. A theory cast in these terms will seek to portray hypothesized mechanisms not simply as real but as necessary, at least in particular aspects, due to the constraints of physical realization. The complexity and variety of psychological phenomena make such a mode of explanation especially valuable.

The most influential conception of computational principles comes from David Marr (1982), who prescribed a definite format for computational theories. For Marr, each theory had three tiers: a computational theory of some problem, a specification of the algorithm by means of which the brain solves this problem, and the implementation of this algorithm in neural machinery. This scheme is attractive because it offers a way to decompose the research task, one problem at a time, and because it demands clarity about what the machinery is supposed to be explaining. Computational principles mediate between the analysis of a problem and the design of its solution, and an analysis of the problem can indeed clarify many issues without all the complications that mechanisms introduce. Unfortunately, Marr's conception of computation presupposes particular forms of machinery. Marr interprets the word "problem" in roughly the same way as the conventional theory of computation, as a mathematical function from an input representation to an output representation. In Marr's case these representations are derived from retinal images, but they could also be derived from auditory inputs or databases or a variety of other things. In any event, the technical notion of a problem found in both Marr and Turing presupposes a restrictive view of an agent's relationships to its surroundings, with single isolated inputs mapped to single isolated outputs. This is probably a good way to think about the earliest stages of visual processing, but it is not (I will argue) a good way to think about human activity in general. Computational inquiry into human activity thus requires a broader conception of computation itself.

Chapter 3 sketches a way of thinking about computation that provides an alternative to the conventional notion of a problem and is better suited to the study of activity. Whereas Marr proposed focusing research on distinct mappings from inputs to outputs, I propose focusing research on aspects of agents' interactions with their environments. Computational principles, on my view, relate the analysis of a form of interaction to the design of machinery. These principles are irreducibly intuitive, taking precise form only in the context of particular technical proposals. In computational psychology, they attempt to explain something about human nature starting from basic facts: that we have bodies, that our environment is extensively adapted to our culture's ways of doing things, and that everyday life contains a great deal that is cyclical, repetitive, and routine.

5. Critical orientation

Several previous authors have cast a critical eye on AI research. Weizenbaum (1976), for example, draws on the critique of technology in the Frankfurt School. Focusing on the culture of computer programmers and their use of machine metaphors for human thought, he argues that AI promotes an instrumental view of human beings as components in a rationalized society. In doing so, he largely accepts as practicable the construction of rationality found in AI and other engineering fields, even though he rejects it on ethical grounds.

Other authors have argued that AI as traditionally conceived is not just wrong but impossible, on the grounds that its technical methods presuppose mistaken philosophies. The first and most prominent of these critics was Dreyfus (1972), who pointed out that symbolic methods in AI are all based on the construction of rules that gloss English words and sentences as formally defined algorithms or data structures. Although these rules seem perfectly plausible when presented to audiences or displayed on computer screens, Dreyfus argued that this plausibility was misleading. Since philosophers such as Heidegger and Wittgenstein had shown that the use of linguistic rules always presupposes an embodied agent with a tacit background of understanding, attempts to program a computer with formal versions of the rules would necessarily fail. Unable to draw on tacit understandings to determine whether and how a given rule applied to a given situation, the computer would be forced into a regressive cycle of rules-about-how-to-apply-rules (Collins 1990). Later, Winograd and Flores (1986) extended this argument by describing the numerous ways in which language use is embedded in a larger way of life, including an individual's ceaseless construction of self and relationship, that cannot itself be framed in linguistic terms except on pain of a similar regress.

The AI community has, by and large, found these arguments incomprehensible. One difficulty has been AI practitioners' habit, instilled as part of a technical training, of attempting to parse all descriptions of human experience as technical proposals -- that is, as specifications of computing machinery. Given the currently available schemata of computational design, this method of interpretation will inevitably make the theories of Dreyfus and of Winograd and Flores sound naive or impossible, or as deliberate obscurantism, or even, in many cases, as mystical rejections of the realizability of human thought in the physical, causal world.

Another difficulty has been the hazardous procedure, shared by practitioners and critics alike, of "reading" computer programs and their accompanying technical descriptions as if they encoded a framework of philosophical stances. Of course, technical ideas and discourses do encode philosophical stances, and these stances generally are reflected in the programs that result. But as I have already observed, the programs themselves -- particular programs written on particular occasions -- inevitably also encode an enormous range of simplifications, stopgap measures, and practical expedients to which nobody is necessarily committed. As a result, many members of the AI community do not believe that they have actually embraced the philosophical stances that their critics have found wanting. In fact, AI's engineering mindset tends to encourage a pragmatic attitude toward philosophical stances: they are true if they are useful, they are useful if they help to build things that work, and they are never ends in themselves. If a particular stance toward rules, for example, really does not work, it can be abandoned. Instead, the fundamental (if often tacit) commitment of the field is to an inquiry into physical realization through reflective conversations with the materials of computer work. This is not always clear from the rhetoric of the field's members, but it is the only way I know to make sense of them.

Dreyfus as well as Winograd and Flores have conducted their critiques of AI from a standpoint outside of the field. Dreyfus is a philosopher by background, though in recent work with Stuart Dreyfus he has increased his constructive engagement with the field by promoting connectionism as a philosophically less objectionable alternative to the symbolic rule-making of classical AI (Dreyfus and Dreyfus 1988). Winograd began his career as a prominent contributor to AI research on natural language understanding and knowledge representation (1972), but his critical writing with Flores marked his departure from AI in favor of research on computer systems that support cooperative work among people (Winograd 1995). Dreyfus and Winograd both define themselves against AI as such, or the whole realm of symbolic AI, and they advocate a wholesale move to a different theory. Each of them effectively posits the field as a static entity, doomed to futility by the consequences of an impracticable philosophy.

Another approach, which I adopt in this book, takes its point of departure from the tacit pragmatism of engineering. I regard AI as a potentially valuable enterprise, but I am equally aware that right now it is a misguided enterprise as well. Its difficulties run deep: we could sink a probe through the practices of technology, past the imagery of Cartesianism, and into the origins of Western culture without hitting anything like a suitable foundation. And yet it is impossible simply to start over. The troubles run deeper than anyone can currently articulate, and until these troubles are diagnosed, any new initiative will inevitably reproduce them in new and obscure forms. This is why we need a critical technical practice.

The word "critical" here does not call for pessimism and destruction but rather for an expanded understanding of the conditions and goals of technical work. A critical technical practice would not model itself on what Kuhn (1962) called "normal science", much less on conventional engineering. Instead of seeking foundations it would embrace the impossibility of foundations, guiding itself by a continually unfolding awareness of its own workings as a historically specific practice. It would make further inquiry into the practice of AI an integral part of the practice itself. It would accept that this reflexive inquiry places all of its concepts and methods at risk. And it would regard this risk positively, not as a threat to rationality but as the promise of a better way of doing things.

One result of this work will be a renewed appreciation of the extent to which computational ideas are part of the history of ideas. The historicity of computational ideas is often obscured, unfortunately, by the notion that technical work stands or falls in practice and not in principle. Many times I have heard technical people reject the applicability of philosophical analysis to their activities, arguing that practical demonstration forms a necessary and sufficient criterion of success for their work. Technical ideas are held to be perfectly autonomous, defined in self-sufficient formal terms and bearing no constitutive relationship to any intellectual context or tradition. The Cartesian lineage of AI ideas, for example, is held to be interesting but incidental, and critiques of Cartesianism are held to have no purchase on the technical ideas that descend from it. This view, in my opinion, is mistaken and, moreover, forms part of the phenomenon needing explanation. Technical practitioners certainly put their ideas to the test, but their understandings of the testing process have placed important limits on their ability to comprehend or learn from it. Advances in the critical self-awareness of technical practice are intellectual contributions in their own right, and they are also necessary conditions for the progress of technical work itself.

Limitations in a certain historical form of technical practice do not, however, result from any failings in the people themselves. Such is the prestige of technical work in our culture that the AI community has attracted a great many intelligent people. But they, like you and I, are the products of places and times. The main units of analysis in my account of technical practice are discourses and practices, not the qualities of individual engineers and scientists. A given individual can see only so far in a fog. Periodic moments of clarity are but the consolidation of changes that have been gathering in the works; their full-blown emergence in the composition of great books is a convenient outcome to an orderly process. Equipped with some understanding of the mechanics of this process, critical inquiry can excavate the ground beneath contemporary methods of research, hoping thereby to awaken from the sleep of history.

In short, the negative project of diagnosis and criticism ought to be part and parcel of the positive project: developing an alternative conception of computation and an alternative practice of technology. A critical technical practice rethinks its own premises, revalues its own methods, and reconsiders its own concepts as a routine part of its daily work. It is concerned not with destruction but with reinvention. Its critical tools must be refined and focused: not hammers but scalpels, not a rubbishing of someone else but a hermeneutics and a dialectics of ourselves.

6. Outline

Chapter 2 states the book's theses: the reflexive thesis (which concerns the role of metaphors in computer modeling), the substantive thesis (which proposes replacing one set of metaphors with another), and the technical thesis (which describes the basis in technical experience for proposing such a shift). It then discusses the reflexive thesis at length, developing a vocabulary for analyzing the metaphors in technical research. The point is not simply to discover the right set of metaphors but to encourage a critical awareness of the role of metaphors in research. The chapter concludes with a sketch of reflexive issues that must await further work.

Chapter 3 concerns the substantive thesis. It describes the metaphor system of mentalism, which portrays the mind as an abstract territory set apart from the "outside world". Put into practice in day-to-day technical work, mentalism participates in characteristic patterns of success and failure, progress and frustration. An alternative is to ground AI in interactionist metaphors of involvement, participation, and reciprocal influence. I introduce some vocabulary and methodological ideas for doing interactionist AI research.

Chapter 4 analyzes the mentalist foundations of computing, starting with the tension between abstraction and implementation in conventional computer science. The technical notion of a variable provides an extended case study in this tension that will turn up in subsequent chapters. The tension between abstraction and implementation is also evident in the history of cognitive science, and its outlines provide some motivation for interactionist alternatives.

Chapter 5 continues the analysis of conventional computer science with a critical introduction to the workings of digital logic. Computers these days are made of digital logic, and throwing out digital logic altogether would leave little to build models with. Instead, this chapter prepares the way for a critical engagement with digital logic by describing the peculiar ideas about time that have accompanied it.

Chapter 6 shifts into a more technical voice, developing a set of fairly conventional ideas about the relationship between digital logic and human reasoning. It is a costly and difficult matter to think anything new, and so "dependencies" provide a means of recording, storing, and automatically recapitulating common lines of reasoning. In addition to presenting dependencies as a technical proposal, this chapter also briefly recounts the tradition of ideas about habit and learning from which they arise.

Chapter 7 introduces a simple rule-based programming language called Life that aids in the construction of artificial agents whose reasoning can be accelerated through dependency maintenance. Execution of Life programs depends on some formal properties of conventional AI rule languages that I define just well enough to permit an expert to reconstruct the details. Some cartoon programming examples demonstrate the Life language in use.

Chapter 8 presents a detailed analysis of the early planning literature, from Lashley's (1951) "serial order" paper to Newell and Simon's (1963) GPS program to Miller, Galanter, and Pribram's (1960) theory to the STRIPS program of Fikes, Hart, and Nilsson (1972). Through this history, a complicated pattern of difficulties develops concerning the relationship between the construction and execution of plans. Viewed in retrospect, this pattern has pushed AI research toward a different proposal, according to which activity arises through improvisation rather than the execution of plans.

Chapter 9 develops this proposal in more detail by introducing the notion of a running argument, through which an agent improvises by continually redeciding what to do. This scheme will not work in a chaotic world in which novel decisions must be made constantly, but it might work in a world in which more routine patterns of activity are possible. A set of Life rules is described through which an agent might conduct arguments about what to do. The chapter concludes by describing an architecture for such an agent, called RA.

Chapter 10 demonstrates RA in action on a series of simple tasks drawn from AI's conventional "blocks world". The chapter detects a series of difficulties with the program and, in the spirit of reductio ad absurdum, traces these back to common assumptions and practices. In particular, difficulties arise because of the way that the system's ideas about the blocks are connected to the blocks themselves.

Chapter 11 takes heed of this conclusion by reexamining the mentalist understanding of representation as a model of the world. The shortcomings of this view emerge through an analysis of indexical terms like "here" and "now", but they also emerge through the technical difficulty of maintaining and reasoning with such a model. The path to interactionist alternatives begins with the more fundamental phenomenon of intentionality: the "aboutness" of thoughts and actions. Some phenomenologists have proposed understanding intentionality in terms of customary practices for getting along in the world.

Chapter 12 attempts to convert this idea into a technical proposal. The basic idea is that an agent relates to things through time-extended patterns of causal relationship with it -- that is, through the roles that things play in its activities. The concept of deictic representation makes this proposal concrete.

Chapter 13 describes a computer program called Pengi that illustrates some of these ideas. Pengi plays a video game calling for flexible actions that must continually be rethought. As with RA, reflection on the strengths and weaknesses of this program yields lessons that may be valuable for future theorizing and model-building. Some of these lessons concern the tenacity of mentalism in the face of attempts to replace it; others concern the role of attention in the organization of improvised action.

Chapter 14 summarizes a variety of other research projects whose approaches converge with my own. It also offers some reflections on the reflexive thesis concerning the role of metaphor in technical modeling.

(c) Copyright 1997 by Philip E. Agre. All rights reserved.