Cognitive Ergonomics -- confidential draft - Rainer von Königslöw - - Page Cognitive Ergonomics Author: Rainer von Königslöw Date: July 18. 1995 Version 0.2 Date: Dec 17. 1995 Version 0.3 Introduction: A fairytale view of reality, including a model of perception and behaviour There is a common notion that information comes to us from the outside through the sense organs and into the brain, where it is assembled and interpreted and then leads to a response. This type of notion in its simplest form is known as stimulus-response. As part of this approach it is commonly assumed that the external parts, the stimulus and the behavioural response are most real, and best understood. I will propose an alternative approach, where the internal representation is "reality". In this view perception is a motivated activity, i.e., we use our senses to interpret the world to fit the internal reality. Similarly we behave in accordance with the internal reality. Let us furthermore posit that the "atomic" units of internal reality are stories and plays. Let us assume that we carry a collection of scripts for stories and plays. Let us also assume that we casts ourselves into these scripts, that we guide our perceptions from these scripts, and that we behave in accordance to the stories and plays, somewhat like acting in a theatre (story telling). Another way of expressing this concept is that we are forever acting "as if ...". We would claim that the stories or representations that underly the "as if" constitutes the core reality from which we act (c.f. Philosophy of As If). When we address the question: who am I, we often do it with analogies - I am like this person in this play or story. Both stories and plays are historically important means of communication, often formalized and ritualized. Initially in books, and now in movies, stories and plays are everywhere. Like morality plays they carry values on preferred ways of behaving. We have heroes whose behaviour we should imitate. Stories and plays are a compact representation of organized and systematic behaviour. It is a possible answer to the question: what do we learn when we learn to play a role, to be a mother, a doctor, a bus driver, etc. Using this approach, the concept of behaviour is fairly straightforward: we are acting a part in a story or play. The more difficult questions are how we select the story to use from our repertoire, and how perception fits into this model. Let us start with the problem of perception. According to this model, perception is an organized activity with a purpose. We use and focus some of our senses to collect information. These will normally be an anticipation of what kind of information we expect to perceive. Other information that is not relevant at that moment may be ignored. Use of senses is selective, i.e., the input from some senses may be ignored. For instance, let us assume that we are looking for a friend in a crowd. We will use our eyes to systematically scan the environment. We may even step on something to raise us up to see better. We may ignore sounds and touch, i.e., ignore conversations going on around us, and the occasional touch of other people against us. Following up further on this model, we can use a computer analogy, where input is filtered through a format to make it meaningful in the context. Thus the same information can be recognized as a number or alternatively as a string of characters with different format filters. In this analogy the incoming information is matched against a set of expectations which determine alternative interpretations of the stimulus information. This analogy is essentially equivalent to the Kantian notion of a filter. A restricted notion of SR can be accomodated within the model, as we can have unexpected perception events cause errors or interrupts which interrupt the normal sequence of purposeful perception. Examples of such errors or interrupts might include: (in our crowd scene) 1. A sudden, unexpected touch 2. The sound of one's name 3. Recognition of a known face not expected in this context Even with such interrupts, we would still claim that the stimulus does not carry meaning, but that the interrupt causes context switching, where new filters or formats are applied to interpret the information There are several extensions we need to make to the model. In most traditional views of perception, there is a concept of assembling complex meaning from basic "atomic" pieces of direct sensory data. Let us try an illustration, of recognizing a person running toward us: 1. We perceive a collection of colours and lights 2. We assemble the colours into shapes 3. We assemble the shapes into a faces, arms, torso, legs, and background objects 4. We note temporal changes is the positions of these body components 5. We infer that the person is running, and that the direction is toward us 6. We infer that the aim of the motion is to us (not just in our direction) 7. We infer that the intent of the person is to meet us (greet us, attack us, ...) With purposeful perception we might want to have top-down perception: 1. We have alternative hypotheses of a person running toward us or running away from us 2. The data best fits a model of a person running toward us (e.g., with Bayesian hypothesis testing) Furthermore, we might very high level scripts, like miniature movie parts against which we test our perception: 1. We have alternative models of attackers coming to hurt us and ... 2. We are alone, it is late, the person is running toward us, therefore.... One might extend the model, therefore, to see us as carrying around scripts or plot lines, like from Harlequin romances or from movies. Rather than "perceiving reality", accoring to this model, we would be applying all the different novel or movie scripts and selecting which of the scripts best fits the incoming perceptual data. We might borrow Herb Simon's notion of "satisficing" to postulate that we do not test againsts all the alternative scripts that we carry around. Rather we might test the alternative scripts in order until we find one that sufficiently fits the perception to satisfactorily explain what is going on. We "satisfice" rather than "optimize" fit. Some people may put in more effort than others to carry more alternative scripts, to note more discrepancies in matching, and to go through more alternative scripts before they are satisfied that they understand. Further integrating notions of primacy and recency found in memory research, we might explain the order in which we test and apply the scripts. Primacy would give emphasis to scripts we obtained early in our lives, which recency would emphasis recent new additions to the alternative script library we carry. Just as anecdotal support for the recency notion, I have certainly noticed that I am influenced by novels I read, and that I might see my wife as similar to the heroine of the last novel I read, and suddenly "perceive" new intentions and motivations, as laid out by the author of the novel. Introduction: models in science Above we have focussed on "people" stories, i.e., accounts that help to guide our own behaviour. However, the concept can be extended to other endeavors. We might see science as a systematic approach to story telling. Theories and formal models give us a systematic and "public" way of representing the story. Experimentation gives us a systematic way of evaluating stories and of comparing competing stories that purport to be about the same subject matter. Introduction: stories in business We can apply the notion of stories in business at a variety of levels. For a company as a whole, we have the fiction of a corporate entity with a mission. Within our departments, and for our jobs we have the concepts of job descriptions and roles which give a verbal description on how we should act in various contexts. Introduction: cognitive ergonomics How does this model apply to the design of information products such as electronic books and software applications? We can take the same approach and say that data has to be recognized to become information, and that information has to be recognized in context to become knowledge. We shall posit that we carry equivalent patterns to the movie scripts which allow us to recognize data and attribute meaning to it. However, the data often is ambiguous and will support more than one interpretation. We can be dreadfully wrong in interpreting data. One famous example arose with the nuclear plant at Three Mile Island. Signal gauges gave off warnings, alerting the operators. The operators saw the warnings and inferred that there was a problem with a pump and acted to counteract the problem. Unfortunately, the problem was not with the pump but with another signal. The operator actions, since it was addressed to "cure" the wrong problem in turn caused a serious malfunction. There are many such examples, all of them pointing to the fact that the correct action is not a response to the stimulus data but a response to our understanding of the data. There is also much anecdotal evidence for recency effects, where we assume that a current set of signals is cause by the same problem that cause the last similar display of signals. The above example dealt with process control. Similar examples can be constructed for legal reasoning, forensic accounting, project management, piloting aircraft, and in many other areas. The above examples looked at this process of understanding from a defensive orientation, i.e., how can we prevent errors of interpretation. One can certainly move along this path by helping the operator see alternative interpretations quickly, even when he/she is under pressure to respond immediately. An alternative approach is to look at this process of understanding as an opportunity to build systems that are easier to understand, by fitting them to mentals models and scripts that users carry in their heads. Structure and function: components of the model and core functionality The story as structure At the core of this model we have positioned the concept of a story. What we mean by that is: a structure that motivates both perception and action it should have predictive and explanatory power: it should allow us to predict what the person will do in some context it should be recognizable by the person it should have some match with a self-description a structure that provides continuity, i.e., it can have a beginning and an end, but it should have a transition to the next story introducing matching story partners should lead to predictable consequences. In other words, if a person is acting on a story, and the story involves other actors (players ...), and if other persons are present who share the story (or close variants), and who identify with the other roles in the story, then the sequence of behaviour should be roughly predictable by the story line. introducing incongruent elements into the context should lead either to changes in the story or to a switch to another story. In other words, if perceptual elements in the context do not match the story, then the person should either adapt the story or select another story to fit the context the story structure should be decomposable and composable it should be possible to build more complex stories out of simpler components, e.g., combining a quest and a romance it should be possible to decompose the story into simple story fragments it should be possible to edit and restructure the story by adding actors, story fragments, etc. There should be a relatively small set of skeletal stories (or story fragments) with well-understood structures (goal driven or event driven -- single actor or multi-actor -- ...) The skeletal stories should match against "known" and shared stories Literature Movies Jokes myths & fairy tales (e.g., Bruno Bettelheim) stereotypes & caricatures are usually set in minimal stories The skeletal stories should match against self-description and the assignment of roles to others Part of the model is the claim that a story is the "atomic" element of meaning. This contrasts with linguistic theories which claim that words or sentences are the smallest element. (Paragraphs - K. Pike). The concept developed here is that words, whether in reference to entities or actions are implicitly part of a story which assigns them their place. (There is some support for this notion in the anthropological literature, e.g., empty gas cans as entities or events). The story must be a representation of "programming", i.e., it must be usable as instructions on how to behave (both active perception and motor activity) The representation may be somewhat like a declarative form, in that we can verbally represent at least parts of this internal encoding - it functions as an instruction manual Limits of the representation Some behaviours - like tying shoelaces, swimming, or balancing on a bicycle - may not map readily into language or story representations. It is likely that these are coded in different ways. Representation and instantiation: The story as kept & as used We do not know how stories are "stored", or how they are represented when they are in use. It also seems likely that there are large individual differences. A reasonable goal at present is to look for functional analogies that have the appropriate properties. Books, movies, audio-tapes, video games, computer simulations, and electronic multi-media systems offer vehicles to construct functional analogies. Even though the story is the "atomic" building block for meaning, we normally work with small story fragments. I would claim that even in chivalrous gestures, such as a gentleman opening the door for a lady, the gesture is a fragment that casts the two people momentarily into the roles of lady and gentleman, and it is through that symbolic link to a set of "gentleman & lady" stories that the gesture becomes meaningful. I believe we normally participate in multiple stories more or less at the same time, as we act in more than one role. (Explore also the link to Goffman - phenomenology). When two people interact they have to match roles -- there is some subtle negotiation about the nature of the story they are acting out. If there is no agreement then there can be major disappointments & frustration. I find in generally in personal interactions, there is a lot of pressure toward a single story -- i.e., one becomes defensive if one is playing the role of a bad person in the other person's story. -- It is very difficult to maintain one's own story if the other person has a very strong and convincing view of who one is. Some people seem to project stronger and be more persuasive than others. Similar concepts apply in business and in science. One can be very influential in meetings just by having a very clear vision, and understandable story to tell. Once can exert leadership through the simplicity and persuasiveness of the story, even without any authority. The same is true in politics. There are typical stories such as: we are all victims and therefore we are justified in aggression. Predictive/interpretive power of story skeletons If we see ourself as participating in a story we have to match our past behaviour, and that of other participants to the story line, up to the "current" scene. By following the story to its conclusion we can predict how the current situation will play itself out (assuming we and other actor/participants will continue to adhere to the story line) The story assigns us and other actors a role in the story, and thus helps us understand our relationship to others. Requirements analysis We can look at cognitive ergonomics from a requirements perspective - what do we need to deal with a complex situation, a complex system. 1. We need a mental model that tells us how to act a) perception as action, what sensory input should we focus on, and what should we ignore i) visual ii) auditory iii) tactile iv) smell v) heat - cold vi) taste vii) internal feelings and emotions - internal states? pain, tiredness, heart rate, shortness of breath viii) internal feelings of limits & capacity - could push harder, run faster b) action as scripted or programmed activity Like programming a robot, there is a macroscopic control sequence and a microscopic "implementation" of macroscopic instructions There is a multi-layered representation: stories within stories Example 1: layered communications protocols Example 2: High level (5th generation) application programming script macro language command interpreter functions hardware - system calls (hardware & device independence) Example 3: interpersonal romantic evening - script embrace - macro -- active perception for perceptual cues to coordinate timing, engagement & release -- gross motor activity: hand motion, placement, pressure, timing, synchronization of engagement & release micro gestures to express our feelings - facial, hand, body fine (automated?) motor control to do the gestures According to this model, to put together an ideal 2 actor story we have to match story lines at all levels. Story acquisition: learning new roles & building new models Children practice role playing. Children observe role models & imitate. Kids play doctor, nurse, mother -- and act these out as play dramas - work on the core activities and interactions. We take formal courses and instruction to learn how to act systematically. We use apprenticeship. Story selection & story editing Matching our story to reality How do we detect when a story is inappropriate & needs change? There is always imperfect correspondence between our story and reality. We also often have the choice between changing our story and changing reality. It may well be that one cause of marital unhappiness is theis lack of match. In some cases it may be easier to change marriage partners than to change the story - especially if the story is central to our view of ourselves. In science we have explicity models of experimentation, establishing correspondences between the model and the world, using systematic measurements and established (statistical & other) evaluation procedures. Even in science the points of correspondence are sparse, i.e., there may be a lot of content in the story or model that does not have a direct correspondence with observable or predictable things. Our internal stories for our personal life are almost at the other extreme, in that there are no established methods for evaluating correctness. In science there are many examples where the scientist found it easier to adjust the data than to adjust the theory (e.g., inheritance of characteristics -- beans?) Multi-layered stories In the multi-layer model, correspondence is required at all the levels, but for complex stories is is unlikely to be exact. Furthermore, with our model of active perception, we would predict and active process of trying to match the elements. We can see the active perception and the active matching when the correspondence breaks down & is verbalized: If he loved me then he would ... , but instead he ... - indicating the expectation of what the other person in the story is expected to do, and what he was perceived to have done. Both the expectation and the "perception" in fact are story components and rarely describe actual behaviour in objective, detached detail. Rather they tend to include intent, feelings, etc. Errors in matching to reality A common type of error in matching to reality is illustrated by the multi-layered example of the romantic evening: His story of an embrace might include some cues that he expects from her that she is ready for an embrace. So he might be ready and willing for an embrace, and he might think that he has given the gestural signals that he is ready for an embrace, and he is now waiting for the gestural cues that tell him that she is ready for an embrace, but they do not come. Her story of an embrace might be that he should be tempestuous and sweep her off her feet. She might feel that she should pretend to be reluctant, as part of her story line -- rather than being forward and "initiating" When the embrace does not happen, both might be disappointed and feel that the other did not share the story of wanting to share a romantic evening He might feel that she was not ready and willing to be intimate at that level She might feel that he does not find her sufficiently attractive to be willing to be romantic In other words, both might misinterpret the mismatch to an element in the higher story line, the interest and willingness to have a romantic evening together. They might not realize or understand that they have a mismatch in a lower story: how to proceed to engage in an embrace Notes - Language is marginalized: 1. in SW projects, documentation is secondary, even help -- even though it is very important in the user interface & user perception -- RAADD - rapid application and documentation development 2. In the world of books, authors are paid very little, only on consignment, like pre-industrialization weavers -- among lawyers, lawyers working for publishers are paid very little, $25 an hour and up 3. Educators, who primarily work through language -- are in low esteem and paid little 4. Canada Council has less funding than NSERC or NRC 5. In science, even though models and language-based concepts are central, the focus and perception is about the world, not the language-based understanding 6. In psychiatry & clinical psychology, there is more comfort with gene and drug based accounts than with mental model based accounts 7. In politics 8. In movies, the author gets relatively little credit, compared to actors and directors, even though the author presumably had most of the creative concept (not always) Design of instructional materials There are two main approaches to the design of instructional materials. The two approaches complement each other, but they are quite different in assumptions underlying the learning. 1. Information flows from the outside into the "mind" through the eyes. The primary context providing examples are movies and novels. The individual is a passive receiver with eyes oriented at the screen or the page. The information is projected on the screen or the page, and flows from there through the eyes into the mind. Parts of it will be remembered like an internal recording system: like a VCR for visual information. What makes a movie or book interesting is in the information that is flowing through. 2. Information is collected and acquired through an active process from inside out and then back inside. In other words, the "mind" directs the eyes to look and focus to bring information back into the mind. The individual is an active acquirer of information. The "mind" builds an interpretation of the information, the "story". It is the interpretation that is recorded, not the information itself. What makes a movie or book interesting is the correspondence between the mental model and the information received from the outside. As we collect information through the eyes, we also anticipate and prdict -- interpolate. In other words we predict what we will see. A story is interesting if there is a mismatch between the prediction and the perceived information, causing us to restructure and reinterpret earlier information. a) absurd - if the information does not match and cannot be brought into alignment. b) predictable and boring - if the information matches and does not add to the story. c) interesting and surprising - if the information causes us to reinterpret and restructure earlier information (the twist to the story). There is a paradigm shift, more and/or new possibilities are added, the predictions for future detail change. d) interesting but not surprising - if the information fills in details but does not add to the structure. Not new possibilities or predictions are generated.